code_saturne User's Forum

Hello,

I have been trying to use ParMETIS or Pt-Scotch partitioning library to increase speed of my simulation and I ran into some questions/problems. I previously used ParMETIS for a large mesh (~35 million cells) and observed a significant decrease in computing time per time step. However, I tried using ParMETIS to partition a mesh for a centrifugal compressor (~22 million cells) with a rotating mesh and it performed worst than using the built-in partitioning library. Is the efficiency of ParMETIS case specific? I also have tried using PT-Scotch/Scotch in the past and it works for a small case, but it crashes for large meshes with the following error in the output file:

Rank 8 [Mon Nov 21 08:01:14 2016] [c0-0c0s4n2] Fatal error in PMPI_Alltoallv: Other MPI error, error stack:
PMPI_Alltoallv(557)............: MPI_Alltoallv(sbuf=0xa00250, scnts=0xe6eb88, sdispls=0xa003d0, MPI_BYTE, rbuf=0xfcbf50, rcnts=0xe6e888, rdispls=0xa009d0, MPI_BYTE, MPI_COMM_WORLD) failed
MPIR_Alltoallv_impl(380).......:
MPIDI_CRAY_ugni_alltoallv(1373):

The listing file just ends at the partitioning step and it also states that the scotch library is being used.

Regards,
Andrew

Hello,

Yes, unfortunately there is no systematic winner. In my experience, graph-based partitioners such as PT-SCOTCH or ParMetis usually lead to better performance, but this is not always the case (I'd give a rough estimate of 4 out of 5 times, but it is case specific).

And space-filling curve based partitioning has the advantages of being deterministic and more robust...

Best regards,

Yvan