Code Saturne Stability and Performance tips

[Antech]

Hello.

I'm continuing to test and learn Code Saturne on my PC. The basic level target is to be able to perform "pure aerodynamic" calculations on arbitrary, not very simple geometry (not a cube or cylinder) with adequate speed.

For test cases I chose an existing geometry of "tangential" boiler furnace (see attachment). Naturally, in tests it's just an isothermal cold flow with constant properties (defaults), but this flow is not very simple (not a flow in a tube e.t.c.).
Inlets are arranged in 3 tiers, each tier has 4 groups of inlets on two opposite walls. Inlet velocity is 20 m/s. At the top of the "furnace" is a pressure outlet PC (ambient). Other boundaries are smooth walls.

Code Saturne settings are almost defaults. Unsteady flow. Time step is 0.01 s.

Tetrahedral Mesh is built with Salome NetGen plugin that is easy to set up (important for engineering).

The first test case is based on mesh with ~1 million elements. There is mesh issue reported by Saturne's solver concerning Least Square gradient calculation method, but I don't use this method. The calculation ran for 500+ iterations on 4 Intel 2600K cores without problems (default partitioner, no Metis or Scotch yet installed). Increasing time step to 0.05 s for tens of iterations not rises any problems.
OK, I switch to 5.7-millions mesh (reduced min/max sizes and growth rate). It's also tetra with moderate growth rate (less than 0.2 in NetGen parameters). Calculation with time step 0.03 s diverges around 70th iteration (unreal velocity maximum). Calculation with time step 0.01 was almost OK up to 125 iterations (then needed to turn off my PC), no any unreal velocity peaks (maximum velocity magnitude ~24 m/s, monitoring with grep), solution in ParaView seems to be normal, but incomplete (that is expected).
But there is a problem. Average iteration duration in this case is 300...500 s (4 Intel 2600K cores, without HT). Not very fast, but it's unsteady and only 4 cores. The problem is some iterations that took about an order of magnitude more time to complete (up to 5500 s). I looked at listing and found that it's due to enormous linear solver iterations for velocity, and, on some iterations, for k. First part of calculation was with default linear solvers, maximum number of "fast" iterations in sequence was 7. Then I switched to Jacobi for Velocity, k and epsilon. After restart Saturne ran until iteration 122 with maximum time 1066 s per iteration. At that moment I thought that this issue is solved but... Iteration 123 was the long one, 4000+ s, and it was again too many "linear sub-iterations" for velocity.

Therefore, I encountered two problems. The first is "long iterations" (slow linear convergence) and second is a time per "normal" iteration (due to "unsteady" option).

I searched the forum and found several recomendations about stability. They are in various topics so I decided to combine them in a single list. IMHO, it's good for newbies like me.

1. Use Upwind 1st order spatial discretisation scheme (default is centered) for first estimation (then it's important for accuracy to switch to a 2nd order scheme, Centered or SOLU in case of Saturne).
2. Disable Flow reconstruction, at least for k and epsilon, for first estimation. I think it should then be enabled again for better precision.
3. Set gradient reconstruction method to "Iterative handling of non-orthogonalities" (default) or "Least squares method over partial extended cell neighborhood" in case of poor mesh quality. There are several Least Squares methods for gradient in Saturne, first in GUI list is definitely less stable in my test case than default (non-orthogonalities) on large timesteps (Courant number 60...100). I found Least squares ... partial ... neighborhood much more stable than default (non-orthogonalities) on mesh with prism layers.
4. Check mesh quality. Important criteria are Face and 3D Aspect ratio (was less then 5 in my "relatively good" cases), non-orthogonality and growth ratio (usually limited to max 1.2 for CFD). Particulary, NetGen may produde meshes with 3D Aspect ratio of hundreds that "crashes" solver (leads to velocity divergence on first iterations).
5. Check Courant and Fourier numbers. Courant number must be less than 20 (desired 1...5) for Saturne due to limitation of SIMPLEC velocity-pressure coupling algorythm. IMHO, this is the reason why commercial solvers like CFX are more robust with Coupled algorythms on big time steps.

Is this list complete?
I think, my case is Courant number (about 70), need to decrease time step (max 0.003 s). I also going to check effects of Upwind (1st order) discretisation for velocity and disabling flow reconstruction. It's in the same GUI table, am I right?

Also, many useful tips for stability described in official Best Practice Guides:
http://code-saturne.org/cms/documentation/BPG

Now about the second problem, performance. The only thing I know now from the forum is a linear solver precision than can be reduced from 10^-8 to 10^-5, apparently, without lack of solution precition.
I tried to turn to steady calculation, but surprisingly the time per iteration remained almost the same! What am I doing wrong? I understand that for unsteady flow performance obtained in my test is expected, but why does it not increase with "steady" option? Compared to CFX (steady) it's very slow, I can't get why...
Another performance issue is dependence of time per iteration on time step. I decreased time step form 0.01 s to 0.001 s in my "5.7-million tetra" test case and didn't see any sufficient speed-up. Is it normal?

Sorry for the long post, needed to describe details.
Thanks for your answers.

I attached the Saturne's listing for 5.7-million case, the case directory and a picture of streamlines from inlets (ParaView). Picture is for ~1-million mesh.

Additional info
OS: Ubuntu 12.04 with updates.
Code_Saturne 4.0.0 (compiled from official sources with SALOME option).
Salome Platform 7 (official distribution for Ubuntu).
OpenMPI

[Yvan Fournier]

Hello,

Most of the interesting performance info is not only in the "listing", but also in the "performance.log" file.

In any case, Code_Saturne's steady computation model is not a coupled solver like that of CFX, so it is less optimized for steady cases. Best results are often not obtained with the SIMPLEC option (idtvar = 2 in the listing), which simply uses a local time step to accelerate convergence, but you may also try the SIMPLE option, which is closer to a true steady algorithm.

Also, using large time steps, the computation may become unstable (too high CFL numbers), and for scalars and velocity, the linear systems solved will be less diagonal dominant, so Jacobi is nt a good fit anymore. You may want to try using a BiGCStab or BiCGStab2 for velocity and turbulent variables for those cases. In addition, you may try reducing the tolerance to 1.e-5.

Using upwind schemes usually degrades solution precision somewhat (and may even prevent obtaining some complex flow features, as it reduced the solution scheme's spatial order), but it does usually lead to faster computations.

Finally, with a million or so cells per core, your case probably has a lot of cache misses (we usually get best results using 30000-80000 cells per core, which means using clusters). On a workstation, there is no miracle solution.

I hope you know that when you need to turn off your machine, you can stop a computation cleanly using a control_file (and later restart it).

Regards,

Yvan

[Antech]

Yvan Fournier
Hello, thanks for your reply.

I hope I will not forget to upload a performance.log (to complete the description of this case). Code_Saturne is on my home PC. I don't get the administrative permission to install any software on our "battle" systems now, so at the moment experiments are on home PC and I haven't full case now on my workplace.

when you need to turn off your machine, you can stop a computation cleanly
Yes, surely, I did so. I continued calculation yesterday.
By the way, the "kill" button in GUI don't work on my system. If needed (incorrect setup, divergence e.t.c.), I stop Saturne's processes in terminal (via "killing" mpiexec).
Unfortunately, AFAIK, there is no simple way to interpolate results on other mesh when restarting in Code Saturne. It's a bit surprising because it has a mesh smoothing feature that is much more complex that interpolation (and it worked OK in my test case).

On a workstation, there is no miracle solution
I understand... We have a small blade cluster, but there is no permission to update any software now even on workstations (we have a strict rules for this). I very hope that we'll get a permission to install Saturne and Salome platform on our "battle" systems soon. These programs are 100% free for commercial use, am I right?
The target of experiments is not to calculate fast on usual PC (4 cores), but to archive performance that is enough to work on a powerful workstation (12 "real" Xeon cores or 24 virtual cores with HT) or a cluster. Let's assume 20 millions of cells for real case. If we will use a rough approximation of cores with equal performance, then we'll obtain almost the same calculation time (speed) on 12-real-core machine (relatively slow and not suitable for practical tasks). Nevertheless, it's still the other ways like reducing mesh or setting up on a cluster with hundreds of cores (but the cluster may be in use for other needs).

Using upwind schemes usually degrades solution precision somewhat
We observed considerable uncertainity (pressure drop) on a "long pipe + bends" case with 1st order upwind (CFX, rough walls, ~20 m/s, roughness 80 um), in that case both distributed and local pressure drops was of comparable magnitude. So we decided not to use 1st order spatial schemes for resulting solutions, only for intermediate calculation stages.

you may also try the SIMPLE option
It's grayed in the GUI (v4.0.0), is it OK?

Thanks for your suggestions. I really appreciate this free help for the free program.

I performed the test yesterday to try stability and performance tips found on this forum (case with 5.7-millions mesh). I reduced time step from 0.01 to 0.005 s, switched to Upwind for velocity, reduced linear precision (10^-5) and disabled Field Reconstruction. But I retained Jacobi algorythm for all variabls except pressure (default multigrid).
Resulting CFL was 32...34, no divergence or "long linear solving" problems, 45 iterations was made with ~130 s per iteration.

Next experiment is based on your suggestions and planned with Steady option, SIMPLE for velocity-pressure coupling, BiGCStab or BiGCStab2 for linear solver with various timesteps.

[Yvan Fournier]

Hello,

To answer of few of your questions/remarks, yes Code_Saturne and SALOME are free for commercial use. As they are under the GNU GPL V2 licence (with one part of Code_Saturne and most of SALOME under LGPL), the only restriction is related to their redistribution : you can only redistribute modified versions of the code under a compatible free licence. There is no restriction to usage.

Regarding the restart/interpolation feature, I have plans of implementing this feature later this year, but it has not been done yet. A workaround would be to run 2 coupled coarse/fine computations for one time step, with a restart for a coarse case, and exchange coarse to fine variables. Doable, but not so practical, and as it would go through rarely used code paths, so I wouln't recommend it to a non-expert user.

Regards,

Yvan

[Antech]

Yvan Fournier
Hello.
I performed new test calculations with Steady mode yesterday. Velocity-pressure coupling was set to SIMPLE (it's enabled with Steady), linear solver - to BiGCStab. Under these settings the only general numerical parameters was relaxation and number of "global" iterations. Default relaxation was 0.7 (I suppose for all variables), I changed it to 0.5 for stability.
Results are normal, but you were absolutely right: there is no miracle for PC Average time per iteration was ~125 s (almost the same as with Unsteady, SIMPLEC, dt=0.005 s). Calculation was stable, no problems like "hang iterations" (slow linear) or velocity divergence (|w|max=21...23 m/s, at inlets |w|=20 m/s). Then I compared obtained results with previous "unsteady" test (start was from the same state @126 iterations) and there was no obvious difference in velocity streamlines (calculation was incomplete, it's normal with just 170...180 "SIMPLE-like" iterations).
Therefore, stability and performance of these two setups (steady+SIMPLE and unsteady+SIMPLEC) was nearly the same. The next step I suppose to do is to increase relaxation coefficient in steady setup to 0.7...1.0 and look at stability and flow evolution.

Additionaly I attach complete archives of log files for first more or less successful run (unsteady + default settings) and recent steady test (SIMPLE + "stability" settings).

Thanks for your help and for info about licenses.

===Small update===
I tested my Steady "cold furnace" case (SIMPLE) with relaxation coefficient 0.8 instead of 0.5. Time per iteration increased from ~125 s to ~160 s (mainly due to linear iterations for pressue), that is expected and normal. There was no problems with hang iterations or divergence within tens of iterations. Meanwhile, flow structure (stream line shape) changed significantly (central vortex grew in diameter) on these iterations (that is correct but I still need more computational time).