Cross-compiling Code Saturne for IBM BlueGene P

[AlexanderB]

Hello Code Saturne developers,
I am cross-compiling code_saturne-2.3.1 for IBM BlueGene/P. The configure for the CNK module run without a problem with this configuration options:

<working_direcotory>/Install/code_saturne-2.3.1/CNK>\
CC="mpicc" CFLAGS="-dynamic" \
CXX="mpicxx" CXXFLAGS="-dynamic" \
LDFLAGS="-Bdynamic" \
../configure --prefix=/<working_direcotory>/Install/code-saturne-2.3.1-cnk \
--host=bluegenep --build=ppc64 \
--enable-debug --enable-shared --disable-frontend --disable-gui \
--with-blas-type=ESSL --with-scotch=/<working_direcotory>/scotch_5.1.12 \
--with-libxml2=/<working_direcotory>/Install/libxml2-2.8.0-cnk \
--with-hdf5=/<working_direcotory>/Install/hdf5-1.8.9/bluegenep \
--with-cgns=/<working_direcotory>/Install/cgnslib_3.1.3-cnk \
--with-med=/<working_direcotory>/Install/med-3.0.5-cnk \
--without-zlib --without-salome-gui --without-salome-med --without-salome

After the successful configure, I run make. And then in the file cs_system_info.c, it says that it cannot find the files spi/kernel_interface.h and the other two files needed for the BlueGene P installation. I add this row and correct the LDFLAGS:

CPPFLAGS="-I/bgsys/drivers/ppcfloor/arch/include"
LDFLAGS="-Bdynamic -L/bgsys/drivers/ppcfloor/runtime/SPI"
LIBS="-lSPI.cna"

and it works fine, it finds the files, include them, but it gives errors on defined variable "ams" used by these files, which is defined somewhere else. I want to ask which compiler should I use the "mpicc" one or the "mpixlc(_r)" to build Code Saturne right? And when I use a XL compiler it can't configure right.

[Yvan Fournier]

Hello,

On our machine, we use "mpixlc(_r)", so it is surprising it does not work (unless your system setup is slightly different).

I released a 2.3.2 bugfix version yesterday, and installed it on Blue Gene/Q, but did not install it on our Blue Gene/P yet (our BG/P is scheduled to go offline near the end of November if I am not mistaken), so I may not have noticed a problem.

I'll install it today or tomorrow and keep you informed if I reproduce this issue.

Otherwise, when the configure or build fails, could you post the error message and config.log you obtain ?

Best regards,

Yvan

[AlexanderB]

Hello,
Thank you for the response. The configure pass without a mistake, without any errors. But when it goes up to the "make" command it gives error that it can't locate the header files needed for the file cs_system_info.c, I put the include path in CPPFLAGS as I do in the 1st message and then it detect them but for some reason don't include a header in which are all defined some of the types of BlueGene/P and I can't find it... As for the XL compilers I figured out how to build it, but the point is that I should make the build with GNU-mpi wrapper compilers (mpicc, mpicxx, mpif90). Also the database of "locate" is over 400 days old, could this be the problem for not finding some libraries or headers?

P.S. I attached the config.log and the errors from the make command in the two files.

[Yvan Fournier]

Hello,

It is possible that the functions used by cs_file_system_info.c depend on the IBM compiler runtime, which would explain why the missing "ams" is present.

Did you try removing "-L/bgsys/drivers/ppcfloor/runtime/SPI" from LDFLAGS and not setting LIBS ?

Also, to avoid multiple definitions of lex/yacc (or bison/flex) between Code_Saturne's MEI sublibrary and PT-SCOTCH's library, you may need to add "-Wl,--allow-multiple-definition" or -zmulfefs to LDFLAGS.

Finally, absolutely avoid using the MPI wrappers for the Fortran compiler, and use the compiler directly (whether gfortran or bgxlf*). Our build scripts use an autoconf macro to determine what libraries are required by Fortran, and those libraries are added to the link command automatically. This enables linking with C or C++ depending on the optional libraries used (C++ being required by MED for example). But when you use a Fortran compiler wrapper, more libraries are added to this list, and libtool wreaks havoc with those. Code_Saturne never uses MPI directly from Fortran, but always from C code, so setting FC directly to the Fortran compiler will probably avoid many issues.

Best regards,

Yvan

[AlexanderB]

Hello,

Actually I run configure script 3-5 times. First time without setting the FC, CPPFLAGS, LIBS and LDFLAGS set to "-Bdynamic" only - with this configure it can't find the headers. The Second run was only with CPPFLAGS add, this time it finds the headers, but gives error for the "asm" variable. The third time I add the FC compiler (no change to the result) and the fourth I add the path to LDFLAGS and LIBS (no change with the "asm" variable), today I find two directories with the names "asm-ppc" and "asm-ppc64" located in the /usr/include, I added the path to them in the CPPFLAGS, but once again no result.

The "make" and "make install" run perfectly with the IBM XL compilers. I will look around the include and lib paths for the XL compilers and try to set them for the GNU-wrapper compilers. For the XL configure the only differences with the GNU-wrapper configure are this:

CC="mpixlc_r" CCFLAGS="-qmkshrobj -qarch=450d -qtune=450" \
CXX="mpixlcxx_r" CXXFLAGS="-qmkshrobj -qarch=450d -qtune=450" \ (...)

of course I didn't set the FC compiler. But I am afraid that I should do this once again with mpicc and mpicxx compilers, because this is the main task for now.

I will write again to say does it work my idea and if yes, to give you a working configure options for the FEN and CNK modules of the IBM BlueGene/P, if no... I will continue with the thinking.

Best regards.

[Yvan Fournier]

Hello,

Could you post your the link command and matching error message that appear near the end of the compilation log ?

To check if the issues are due to the BGP personality info routines, you could add

#undef __bgp__

line 55 of cs_system_info.c (before the first test on this macro), to see if things improve.

Im am not sure whether the issue is due to this or due to the fact that you are trying to build Code_Saturne with dynamic libraries on Blue Gene/P (we build dynamic libraries on most machines, but default to static only when using the XL compilers (such as on Blue Genes), or on Mac/OS (due to different issues with their linker), and have never really needed to run using dynamic libraries so far on Blue Gene type machines, so I have not explored this.

So a useful test would be to use the GNU compilers, but add the --disable-shared option to configure, to see if that works (and only then move to a shared build).

I won't have access to our BG/P before Monday (nor to my computer for most of the coming week-end), but-I'll try to take a look sometime next week.

Is there a specific reason you are trying to build the code with the GNU compilers and shared librarises on this machine ?

Nest regards,

Yvan

[AlexanderB]

Hello,

Yes, there is a reason. It is no different if I use --disable-shared or --enable-shared for Code Saturne or some of the third-party-libraries as an option in configure (med, hdf5, cgns). I use -share, because I compiled libxml2 with the option --enable-shared, due to it didn't link to some of the system libraries, the other reason is the limited space allocated on the HDD and the huge number of users working on it, who "eat" the space like a dragon.

Best, regards.

[AlexanderB]

Hello,

I finally made it. I added

Code: Select all

#undefine __bgp__

in the beginning of all files which had

Code: Select all

#ifdef __bgp__

in them. And how I will post a installation guide here for the BlueGene/P with mpicc and mpicxx compilers.

1.) You need to download the third-party-libraries: libxml2, CGNS, HDF5, MED, SCOTCH;
2.) You need to cross-compile all of this libraries, except SCOTCH, which has PT-SCOTCH for parallel processing. You may have difficulties install HDF5 for the CNK module, you need to tune a little bit the configure script and/or some header files. If you install MED library version 2.3.6, there will be also difficulties with the cross-compilation, some of the header files when they are compiled with mpicxx should wrap parts of them so it should be compiled with C, no C++, or the opposite that is done by adding or commenting the closed between #ifdef and #endif directives:

#ifdef __cplusplus
extern "C"{
#endif
.....
<some_code>
.....
#ifdef __cplusplus
}
#endif

This is done for the header files: med.h, med_misc.h, med_hdf21.h, med_hdf231.h, MAJ_21_22.h, MAJ_231_232.h, med_outils.h. There is no problems with MED version 3.0.5 and above.
3.) After installing the third-party-libraries is time to install Code Saturne, if you use the IBM XL compilers you should look directly on the configure options, otherwise (using GNU-wrapper compilers) you should do the following: go to the untared directory and execute this command grep -rl '__bgp__' *, this should output all Code Saturne files that has "__bgp__" in them, open each one of them and in the beginning of the file add

#undef __bgp__

.
4.) NOTE: The following example configure options work for version 2.3.2.
4.1.) Configuring and compiling for the Frontend:
- go to the directory in which you untar Code Saturne and execute this commands:

cd <code_saturne_directory>
mkdir frontend
cd frontend

After this configure with this options:

../configure --prefix=<path_to_install_dir> \
--enable-debug --disable-gui --enable-shared \
--with-libxml2=<path_to_installed_dir>/libxml2-2.8.0-fen \
--with-blas-type=ESSL --without-metis \
--with-scotch=<path_to_installed_dir>/scotch_5.1.12 \
--with-hdf5=<path_to_installed_dir>/hdf5-1.8.9/frontend \
--with-cgns=<path_to_installed_dir>/cgnslib_3.1.3-fen \
--with-med=<path_to_installed_dir>/med-3.0.5-fen \
--without-salome --without-salome-gui --without-salome-med

NOTE: For IBM XL compilers, you should set CC and CXX compilers. You may not use the option --enable-shared
After successful configure execute this commands:

make
make install

4.2.) Configuring and compiling for the C:
- go to the directory in which you untar Code Saturne and execute this commands:

cd <code_saturne_directory>
mkdir bluegenep
cd bluegenep

After this configure with this options:

CC="mpicc" CXX="mpicxx" \
../configure --prefix=<path_to_install_dir> \
--host=bluegenep --build=ppc64 --enable-long-gnum \
--disable-frontend --disable-sockets --disable-nls \
--enable-debug --disable-gui --enable-shared \
--with-libxml2=<path_to_installed_dir>/libxml2-2.8.0-fen \
--with-blas-type=ESSL --without-metis \
--with-scotch=<path_to_installed_dir>/scotch_5.1.12 \
--with-hdf5=<path_to_installed_dir>/hdf5-1.8.9/frontend \
--with-cgns=<path_to_installed_dir>/cgnslib_3.1.3-fen \
--with-med=<path_to_installed_dir>/med-3.0.5-fen \
--without-salome --without-salome-gui --without-salome-med

NOTE: For IBM XL compilers, you should set CC and CXX compilers. You may not use the option --enable-shared
After successful configure execute this commands:

make
make install

[AlexanderB]

Thank you very much for the provided help.

Have a nice day.

Best regards.

[Yvan Fournier]

Hello,

Ok, thanks for the feedback.

I believe gcc does not handle the SIMD aspects of the Blue Gene/P Power 450 processor, so in optimized mode, at least for some operations, performance is probably less than that obtained with the IBM compilers (a comparison would be interesting).
If you need a shared library version, compiling the code with the IBM compilers in shared mode may be interesting. This may require removing the following section from the configure script (around line 6682, depending on the version of autoconf or the code revision):

Code: Select all

if test "x$cs_ibm_bg_type" = "xL" -o "x$cs_ibm_bg_type" = "xP" \
     -o "$host_os" = cygwin -o "$host_os" = mingw32 ; then
   enable_shared=no
fi

Also note that the static executable is about 50 Mb, so relinking this for every user case with user subroutines does take space, but probably much less that the average checkpoint/restart files for a case large enough that running on BG/P is worthwhile. To save space, encouraging the users to remove data such as checkpoint files which are no longer needed will probably benefit you more than just using a shared library version of the code.

Best regards,

Yvan