MVAPICH2 performance

[thomasburel]

Dear all users,

I am running some calculation on the cluster Briaree at Calcul Québec.
I installed Code Saturne 3.0.3.

I used OpenMPI with intel compiler and now, I use MVAPICH2 2.0b.
I obtained a big gain.

To do 2550 itérations, I took:

With OpenMPI 1.6.3 (include with Code Saturne), I spend 2846.5s (User CPU time: 2676.037s and System CPU time: 170.551 s).

With OpenMPI 1.6.5 (intel compiler), I spend 2898.7s (User CPU time: 2719.740s and System CPU time: 178.967s).

With MVAPICH2 2.0b (intel compiler), I spend 2071.1s (User CPU time: 2069.278s and System CPU time: 1.830s).

Roughly 30% of gain.

I remarked the time spend for each time step is more stable. With MVAPICH2, I spend a little less than 1s since 3 days. With OpenMPI, it is between 6 and 1s during my différents runs.

With NEK5000 (spectral element) and a mesh with 512 000 cells and 25 millions of collocation points, I obtained a gain of 19%.

I use 4 nodes and each node has 12 cores.
The mesh has 1.3 millions of cell.
The simulation is : incompressible and LES.
Metis is used.

[Yvan Fournier]

Hello,

Was the version of OpenMPI 1.6.5 (intel) you used pre-installed or compiled by you ?

If it was simply compiled and installed with default options, it might be not be configured to use the fast network or batch system correctly, explaining the performance difference. If this is not the case, it means that tuning parameters may have a lot of importance.

Thanks for the feedback !

Regards,

Yvan

[thomasburel]

Dear Yoan,

I used the module configured by Calcul Québec. I don't know exactly what are the option used to compile them.

I installed only Metis for all configurations and for openMPI 1.6.3, it was installed with the saturne's installer.

Best regards,

Thomas