Jacobi: error (divergence) solving for Velocity

[attene]

Hello everybody,

I am testing (with a small number of iterations for the moment ) my test case, which consists of two cell zones. One zone is a cylindrical domain including a turbine. The flow in the cylinder is solved in the relative frame of motion (frozen rotor). The second cell zone is a tank (rectangular cross section) which includes the cylinder. My mesh is quite big: around 46 millions of elements.

I managed to make a steady parallel simulation run for 10 iteration with 32 cores. wallclock time of 1h and 30 min.

I tried then to simulate the same test case with both 64 and 48 cores (in order to improve the calculation time) but both these jobs blown up giving the same error:

Jacobi: error (divergence) solving for Velocity

I am thinking that maybe the reason for that is related to either my physical setup or to the mesh partitioning!?

Any suggestions?

Please find attached a list of files which may be helpful.

Regards,
FA

[Yvan Fournier]

Hello,

Do you also have user subroutines ? I won't have acces to my computer before tomorrow night so can't easily check the setup, but looking at your listing, you are already starting to diverge at iteration 2 with huge velocity values. Do you have the same behavior on the case you ran to 10 iterations ?

Regards,

Yvan

[attene]

Hello,

Thank you for your answer.

I did not consider any user subroutines in these jobs.
What I can also tell you now, is that a restart with 15 more iterations ended without any errors. This happened for the job (exactly same setup of the file I forwarded to you) I mentioned in the post that run in parallel with 32 cores for 10 iterations.

Regards,

FA

[Yvan Fournier]

Hello,

This is strange, and might be a missing initialization somewhere in the code (which can cause intermittent "heisenbugs").

I'll run a frozen rotor test case next week under Valgrind to check.

Regards,

Yvan

[attene]

Hi Yvan,

Thank you for your answer!
Looking forward to hearing from you about this!

Meanwhile, do you think I can check anything to find more information regarding this behavior ?

Regards,

FA

[Yvan Fournier]

Hello,

I ran a standard test case (frozen rotor with joining, same as your case) under Valgrind on 2 MPI processes for a few iterations and did not observe any issue.

Could you run your case (same mesh, same partitioning) under a debug build ? If this does not reveal anything, a debug build using address sanitizer might do the trick. If you are familiar with debug builds, just go ahead. Otherwise, check the installation manual to see how to add options for debug (basically, adding "--enable-debug" to the configure options at installation). I can also point you to options for a "address sanitizer" build if necessary (for large meshes, Valgrind often leads to too many false positives on clusters with fast network drivers, so gcc or clang + debug with address sanitizer option can help get the relevant error messages).

Best regards,

Yvan

[attene]

Hi Yvan,

Thank you for your help!
I will try to do that and let you know.

just one thing and one question: The last setup.xml file I sent to you refers to a frozen-rotor without joining: In the mesh there are two different fluid cell zone and a conformal interface between them.

Can I improve the calculation speed by increasing the number of MPI rank steps under Partitioning options? any other things I can setup up in order to achieve this?

Regards,

FA

[Yvan Fournier]

Hello,

The partition rank step might slightly affect the partition quality, but no more, so it should not have a large impact on performance.

The factors which impact performance will be the number of sweeps, the required precision of the linear solvers, and so some degree the time step (which is constrained by CFL issues). The better the mesh quality, the better/faster the convergence in most cases, so given a number of cells, a higher quality mesh may lead to better performance through better convergence. If using least-squares gradients, with a good mesh quality, you do not need an extended neighborhood, which can improve performance. Limiting file I/O also helps.

The timer_stats.csv file helps you see which steps/operators take the most time (per time step). Which can help check everything is OK and where performance improvements or more agressive options may help.

Regards,

Yvan

[attene]

Hi everybody,

I would like your opinion regarding the last simulations I have done on my model. you can find attached the two setup.xml I tested as well as the two listing files:
1) steady simulation using the SIMPLEC algorithm.
2)////////////////////////// SIMPLE /////////.

Resuming: The computational domain consists of two parts, one rotating cylindrical fluid zone ( frozen-rotor approach) + one stationary tank domain with rectangular cross-section. The rotating domain is included in the tank domain. No faces joining was done.
All the simulations I carried out so far blew up!

Best regards,

FA

[attene]

Hi everybody,

I would like your opinion regarding the last simulations I have done on my model. you can find attached the two setup.xml I tested as well as the two listing files:
1) steady simulation using the SIMPLEC algorithm.
2)////////////////////////// SIMPLE /////////.

Resuming: The computational domain consists of two parts, one rotating cylindrical fluid zone ( frozen-rotor approach) + one stationary tank domain with rectangular cross-section. The rotating domain is included in the tank domain. No faces joining was done.
All the simulations I carried out so far blew up!

Best regards,

FA

...more files