Problems with mpi on Ubuntu 18.04

All questions about installation
Forum rules
Please read the forum usage recommendations before posting.
Post Reply
AndrewH
Posts: 42
Joined: Thu Oct 02, 2014 11:03 am

Problems with mpi on Ubuntu 18.04

Post by AndrewH » Fri Jan 17, 2020 8:36 am

Hello,

I have been trying to install Code_Saturne v4.0.1/v4.08 on Ubuntu 18.04 and I have run into some mpi issues. I compiled it with the gcc and openmpi ubuntu packages along with libxml2. Running cases in serial mode, my Code_Saturne installation runs fine. However, in parallel mode my simulation crashes very randomly. Running with 100 iterations, sometimes the computation will finish without any issues but typically the computation will crash on random iterations with the stack originating from different files (e.g. cs_dot, cs_sles, multigrid, partitioning, etc...). I'm using the same computation that I have run multiple times on other computers and clusters, and it shouldn't crash randomly or at all. I also compiled Code_Saturne with mpich and get a consistent error of /usr/lib/x86_64-linux-gnu/libopen-pal.so.20 and /usr/lib/x86_64-linux-gnu/libpthread.so.0 before the computation stage of the simulation starts. Adding more mysterious to the problem, I compiled Code_Saturne with the same settings and packages on virtualbox VM on my laptop and I don't have any problems with parallels simulations; I compared the config files and they are exactly the same. I also modified the code_saturne.cfg after I compiled Code_Saturne. Are there any tests I can make to check my mpi libraries are properly built? I'm hoping this isn't a stupid mistake on my part but I'm also hoping this isn't a hardware problem.

Thank you,
Andrew

Yvan Fournier
Posts: 2869
Joined: Mon Feb 20, 2012 3:25 pm

Re: Problems with mpi on Ubuntu 18.04

Post by Yvan Fournier » Fri Jan 17, 2020 9:05 am

Hello,

I am not too sure about which tools may help chech your Open MPI installation, but in any case, if you have an error in libopenpal using MPICH, it means you are mixing Open MPI and MPICH (either compiling with one and running with the other, ore something more complex).

You can use ldd on the cs_solver executable to check if there are multiple linked versions of one MPI library. This sometimes happens when libraries used by code_saturne are compiled with another MPI version (especially when you have a default library in the parh and are trying to use another).

Do you use --with-mpi or CC=mpicc in your build/configure options ? The latter is often more robust.

Also, make sure the code_saturne.cfg points to the correct mpiexec command in case the automatic detection fails.

Since version 4.0 is quite old, I do not remember which improvements were made in the build to try to avoid issues with multiple MPI libraries.

Regards,

Yvan

AndrewH
Posts: 42
Joined: Thu Oct 02, 2014 11:03 am

Re: Problems with mpi on Ubuntu 18.04

Post by AndrewH » Fri Jan 17, 2020 9:51 am

Dear Yvan,

I will check the code_saturne.cfg file. I tried compiling Code_Saturne with both --with-mpi and CC=mpicc but the problem occured with both. When I installed mpich, I completely purge the openmpi package and recompiled my Code_Saturne installation, but I will double check that openmpi was completely removed.

Thank you,
Andrew

Post Reply