How install CS3.0 with parmetis

[dener]

Hello,
After searching in readme, setup, install_saturne.py and in this forum, I am not able to build and install CS 3 with parmetis.
I also use configure options --with-metis with the directory where parmetis-4.0.3 is, but every time I obtain :
ParMETIS (...) support: no

may you please explain how build, install CS 3 with parmetis ?
OS : ubuntu 12.04

Is CS 3 able to parallelize as CS2 without metis or scotch ?

thank you
Denis

[Yvan Fournier]

Hello,

There is a small bug in the 3.0 detection for ParaMetis. When trying to link with a statically compiled version of ParMetis, the check tries to link with libparmetis, but not with libmetis. This will be fixed in version 3.0.1, but in the meantime, you may add:

LIBS=-lmetis LDFLAGS=-L<metis_prefix>/lib

to the configure line.

Note that the is also a bug in the Parmetis install: it does not install metis.h, which is needed by parmetis.h. To work around this, when installing ParMetis, go to the metis subdirectory, and type the same "make config", "make", and "make install" commands.

Finally, due to constraints of modern linkers, it is usually not possible to link static libraries into dynamic libraries. So if you are building Code_Saturne with dynamic libraries (which is the case unless you are building on Mac, on IBM Blue Gene or using the IBM XL compilers, or if you added the "--disable-shared" option to configure), you also need to build ParMetis with shared libraries (make config shared=1 ...). Otherwise, the final link will probably fail, though detection may work.

Otherwise, the code can use a built-in space-filling curve based partitioning. V2 used the Morton curve in a domain bounding box, while V3 gives you the choice of Morton or Hibert curve, and Bounding box or bounding cuve (the default matches the one from V2).

ParMetis or PT-Scotch usually lead to 20-30% better performance, but for a few test cases, the partitioning from the space-filling curve leads to slightly faster runs. If you build the code with ParMetis and/or PT-SCOTCH, it is always a good idea for long running cases to compare performance on a few time steps, on a case by case basis.

Regards,

Yvan

[dener]

Hello,
merci M. Fournier !
metis and parmetis-4 make config shared=1 ok, make, make install
configure cs with option you give me: ok
detect metis and parmetis.
make
make install

declaring directory where metis and parmetis in LIB_LIBRARY_PATH
running CS with a simple pipe meshed with salome in MED format
importing: ok
testing mesh: ok
1st calcultation just mono process no option for partitionning: ok
2nd calcultation 2 processes, partitionning Metis/parMetis: ok
3rd calcultation 4 proc, partitionning default: error
copy of end of listing file: (same as error file)

----------------------------------------------------------

Partition de 230000 cellules vers 4 domaines sur 4 rangs
(ParMETIS_V3_PartKway).
Signal SIGTERM (terminaison) reçu.
--> calcul interrompu par l'environnement.

Pile d'appels :
1: 0x7f1d8552c09d <ompi_request_default_wait_all+0xad> (libmpi.so.1)
2: 0x7f1d85550a86 <PMPI_Waitall+0xb6> (libmpi.so.1)
3: 0x7f1d85b45670 <libparmetis__Match_Global+0xb50> (libparmetis.so)
4: 0x7f1d85b3c0fb <libparmetis__Global_Partition+0xab> (libparmetis.so)
5: 0x7f1d85b3cbb0 <ParMETIS_V3_PartKway+0x5b0> (libparmetis.so)
6: 0x7f1d8711cbcf <cs_partition+0x444f> (libsaturne.so.0)
7: 0x7f1d86e9fc9c <cs_preprocessor_data_read_mesh+0x26c> (libsaturne.so.0)
8: 0x7f1d86e11943 <cs_run+0x253> (libsaturne.so.0)
9: 0x7f1d86e1160a <main+0x14a> (libsaturne.so.0)
10: 0x7f1d8659d76d <__libc_start_main+0xed> (libc.so.6)
11: 0x4006d9 <> (cs_solver)
Fin de la pile

sorry but I dont know what to do ? what i do wrong ?
Rgds
Denis
PS : not able to add a file to my message, when I click add message extension n'est pas autorisée ?

[Yvan Fournier]

Hello,

The interesting info may be in one of the "error*" files (not the first "error" file if the crash occured on a non-zero rank), though if the error is in ParMetis itself, we may not have much info. If it is outside ParMetis, having configured Code_Saturne with --enable-debug will help troubleshoot things (though you also want a production build, as debug builds are usually 3x slower).

Regarding posting files, you may need to "cheat" with the extensions, such as adding ".txt" to a file...

Regards,

Yvan

[dener]

Hello,
thank you again for your help.
I test partitionning with Morton curve and 4 processes it's ok.
With metis AND sctoch: error
it's strange no ?
Is there a sample (easy and small) to test with the different options of partitionning and multi processing ?
I join my error and listing, and xml data file for options may be it helps.
regards
Denis

[Yvan Fournier]

Hello,

Do you have any othe "error_*" file ?

Is there a possibility that you have several MPI libraries installed, and Code_Saturne and PT-Scotch or ParMetis use different versions ? If you mix Open MPI and MPICH, you probably would have a link error, but if for example you have 2 different version of Open MPI (one installed by the Code_Saturne installer, the other through Linux packages), you could have this type of issue.

Regards,

Yvan