Hello,
Is it possible users to define partitions manually with cs_partition or cs_preprocess?
If I understand correctly, MPI traffic for two adjoining partitions will be dependent on the number of mesh faces or elements in the halo. In some structures I am studying, there are parts of the geometry that look ideal for placement of a partition boundary. By controlling the partitioning, I'd expect to minimise MPI traffic and therefore the finish simulation sooner.
Regards,
Brennan
user defined partitions
Forum rules
Please read the forum usage recommendations before posting.
Please read the forum usage recommendations before posting.
-
Yvan Fournier
Re: user-defined partitions
Hello,
It is not normally possible to define partitions manually. If the Partitioner is built using METIS or Scotch, the partition quality should already be close to optimal, so you probably won't gain much by optimizing it further.
MPI traffic depends on the number of elements in the halo, which is pretty-well optimized by the partitioner, but latency may also depend on the number of neighboring domains (not explicitely optimized), and much of the time spent in MPI is due to reductions, which are independent of the partitioning and latency-bound.
If you do want to try a manual partitioning, the easiest solution would be to add src/base/cs_preprocess.c from the Code_Saturne kernel to your user subroutines, and adapt it to your needs; this simply requires replacing the definition of cell_rank[] in the "_cell_rank_by_sfc" function with your own values, and not providing a "domain_number_*" file to the code (i.e. not using the partitioner beforehand, by searching for and setting EXEC_PARTITION to "no" in the runcase script).
With no previous partitioner run, the code will automatically switch to "Space Filling Curve (Morton)" partitioning, which requires pre-loading the mesh with a non-optimized (i.e. block) partitioning, precomputing cell centers, determining a partition using those (the part which you may override), and finally redistribute the mesh based on that partitionning.
If you do try this, remember that load-balancing is just as important as minimizing the communication volume.
Regards,
Yvan
It is not normally possible to define partitions manually. If the Partitioner is built using METIS or Scotch, the partition quality should already be close to optimal, so you probably won't gain much by optimizing it further.
MPI traffic depends on the number of elements in the halo, which is pretty-well optimized by the partitioner, but latency may also depend on the number of neighboring domains (not explicitely optimized), and much of the time spent in MPI is due to reductions, which are independent of the partitioning and latency-bound.
If you do want to try a manual partitioning, the easiest solution would be to add src/base/cs_preprocess.c from the Code_Saturne kernel to your user subroutines, and adapt it to your needs; this simply requires replacing the definition of cell_rank[] in the "_cell_rank_by_sfc" function with your own values, and not providing a "domain_number_*" file to the code (i.e. not using the partitioner beforehand, by searching for and setting EXEC_PARTITION to "no" in the runcase script).
With no previous partitioner run, the code will automatically switch to "Space Filling Curve (Morton)" partitioning, which requires pre-loading the mesh with a non-optimized (i.e. block) partitioning, precomputing cell centers, determining a partition using those (the part which you may override), and finally redistribute the mesh based on that partitionning.
If you do try this, remember that load-balancing is just as important as minimizing the communication volume.
Regards,
Yvan
-
Brennan Sharp
Re: user defined partitions
Thanks Yvan,
In my case, the structure is very nearly periodic, so I believe the load balancing would be also be good if the partitions were formed where I'm thinking of. When I feel more brave, I'll try as you suggested.
Salut,
Brennan
In my case, the structure is very nearly periodic, so I believe the load balancing would be also be good if the partitions were formed where I'm thinking of. When I feel more brave, I'll try as you suggested.
Salut,
Brennan