Mesh Partitioning

[AndrewH]

Hello,

I have been trying to use ParMETIS or Pt-Scotch partitioning library to increase speed of my simulation and I ran into some questions/problems. I previously used ParMETIS for a large mesh (~35 million cells) and observed a significant decrease in computing time per time step. However, I tried using ParMETIS to partition a mesh for a centrifugal compressor (~22 million cells) with a rotating mesh and it performed worst than using the built-in partitioning library. Is the efficiency of ParMETIS case specific? I also have tried using PT-Scotch/Scotch in the past and it works for a small case, but it crashes for large meshes with the following error in the output file:

Rank 8 [Mon Nov 21 08:01:14 2016] [c0-0c0s4n2] Fatal error in PMPI_Alltoallv: Other MPI error, error stack:
PMPI_Alltoallv(557)............: MPI_Alltoallv(sbuf=0xa00250, scnts=0xe6eb88, sdispls=0xa003d0, MPI_BYTE, rbuf=0xfcbf50, rcnts=0xe6e888, rdispls=0xa009d0, MPI_BYTE, MPI_COMM_WORLD) failed
MPIR_Alltoallv_impl(380).......:
MPIDI_CRAY_ugni_alltoallv(1373):

The listing file just ends at the partitioning step and it also states that the scotch library is being used.

Regards,
Andrew

[Yvan Fournier]

Hello,

Yes, unfortunately there is no systematic winner. In my experience, graph-based partitioners such as PT-SCOTCH or ParMetis usually lead to better performance, but this is not always the case (I'd give a rough estimate of 4 out of 5 times, but it is case specific).

And space-filling curve based partitioning has the advantages of being deterministic and more robust...

Best regards,

Yvan