code_saturne User's Forum

When using the command line interface provided by the code_saturne executable, multiple CS solver processes can be started with the `--nprocs` parameter, e.g., The command used by code_saturne to launch multiple processes can be changed in /etc/code_saturne.cfg.

Is it possible to call mpirun/srun/oarsub manually while still using the code_saturne executable (instead of calling cs_solver directly)?

For example, on a Slurm cluster it would be convenient if one could call
This does not work because, e.g., the preprocessor is fully sequential. The code_saturne executable supports the flags --initialize, --execute, --finalize which might solve my problem but it is not clear how to use them.

Hello,

The different run stages are described here (https://www.code-saturne.org/cms/sites/ ... ation.html), see especially "Execution steps".

If you run using "--initialize", the code execution will be prepared, but not started. In the execution directory, an "run_solver" script is created; containing environment setttings, and the mpiexec (or srun, or equivalent) command.

So if you need finer control, accessing that script might be interesting. For event finer control, using the Python API (cs_run/cs_case/cs_case_domain.py) allows buiding the mpiexec command.
Using a separate preprocessing run, and using the "mesh_input.csm" (mesh_input for older versions) allows avoidinf the serial preprocessor, but in any case, since the "code_saturne" script andles high-level operations such as directory creation, file copies, ... it is not intended to be used under mpiexec. Given a prepared case, a variant of "code_saturne run --id --execute" could be extended to return the required parameters for an "external" mpexec launch. We could easily add a dedicated option for that.

Hoping tat helps.

Best regards,

Yvan

The different run stages are described here (https://www.code-saturne.org/cms/sites/ ... ation.html), see especially "Execution steps".

This helped a lot. I did not see this before because I was only looking in the Code_Saturne 6.0 User Guide and Install Guide.

Is the following instruction sequence code a valid approach to launch multiple solver processes?

Hello,

You can directly run

code_saturne run --id=melissa --param=case1.xml --initialize

Without the --stage part, as t includes it.

For mpirun, you need to avoid the code_saturne script, so rather do something like:

cd RESU/melissa
mpirun -n 16 -- cs_solver

Then cleanup (optionally) the case going back to the original directory and use

code_saturne run --id=melissa --param=case1.xml --finalize

Best regards,

Yvan

Hi Yvan,

thank you.

One can assume that code_saturne will never pass parameters to the solver?

Hi Christoph,

Actually, it does pass a few options in some cases, but they are not directly the ones used by the script.

If you look at bin/cs_case_domain.py, you can see how options are handled in "solver_command", and in the case of MPMD executions, this is assembled in bin/cs_case.py (check call sites for solver_command).

Having a high-level "code_saturne" command to return these options and the (working directory) would be quite easy to do if needed from an "external" caller.

Best regards,

Yvan

Hi Yvan, I will look at the Python files you mentioned.

Yvan Fournier wrote: ↑Wed Feb 10, 2021 3:00 pm If you look at bin/cs_case_domain.py, you can see how options are handled in "solver_command", and in the case of MPMD executions, this is assembled in bin/cs_case.py (check call sites for solver_command).

Having a high-level "code_saturne" command to return these options and the (working directory) would be quite easy to do if needed from an "external" caller.

Hi Yvan,

thank you for the hint. I am not sure how to instantiate the class `case` but I do not want to pursue calling the Code_Saturne Python directly for another reason: I would have to locate the Python code.

Code_Saturne installs its Python code in prefix/lib/python_version/site-packages. This is predictable but depending on the Linux distribution not a directory that is searched by the Python interpreter. If it is not, then I have to determine the prefix and guess if the lib directory is really called "lib" or "lib64" like on RedHat-based distributions (CentOS 8 still does this). If I am on a Debian-based Linux distributions, the last directory will be called "dist-packages" instead of "site-packages" (see the code-saturne-data package). When there are loadable modules involved, it becomes even more complicated.

Please implement a high-level call returning the cs_solver arguments. Ideally the call does not return any MPI-related options because there is most likely no mpiexec on clusters with the Slurm batch scheduler, cf. Spack issue #10340 Do not remove mpirun when OpenMPI is built with Slurm.



By the way there is no `domain` child class in `bin/cs_case_domain.py` using the `app_id` argument passed to it in `bin/case_case.py`.

Hello Christoph,

Actually, the "mpiexec" command can be parametrized in the code_saturne post-installation step, by editing <install_prefix>/etc/code_saturne.cfg, so adaptation to SLURM is not an issue.

Other than that, most of the optional command-line arguments to cs_solver are probably not useful in an "ensemble" run such as when using Melissa..

These options include "--proprocess" and "--quality" for a mesh check mode only, but these are used to check mesh quality beforehand, so may be useful before running with Melissa, but probably not with Melissa.

The only options which might be useful is " --sig-defaults" if you need to replace the code_saturne signal handler with the default ones (for example to generate a core file instead of a backtrace), and "--wdir <path>" to force the working directory (in case the mpiexec/srun/ or equivalent command does not handle it). "The "--mpi" option can also be used to force MPI initialization if no known MPI environment variable indicating that we are undeer MPI is detected by code_saturne.

All information relative to the user physical and numerical setup is passed throuhg setup.xml, which has a fixed nam so does not need to be specified (at least not since code_saturne 6.0), or through user-defined functions.

In all cases, these are related more to the run environment than to the case setup, so for an "external" launch, it is probably simpler not to bother with Python at all, even at a high level, and just build your "mpiexec" or equivalent command around the "./cs_solver" executable.

If "cs_solver" is not present in the working directory, it means there are no user-defined sources at all and the default executable (<install_prefix>/libexec/code_saturne/cs_solver on mist systems, with libexec replaced by "bin" or "lib" on some systems such as OpenSUSE).

So since this is basically the only precaution needed, I can add a high-level python wrapper to check that, but simply testing the presence of cs_solver in the working directory is sufficient (and if you do not want to bother, if there is no file in a case's SRC, you may still have generated code, and you can even force things by adding an empty or default (reference) cs_user_parameters.c file in SRC.

For the working directory, you have two possibilities:

• You did not specify an directory when running "code_saturne run --initialize".
  in that case, you are stuck, as you have no idea od the directory used (actuallly, the default is YYYY-MM-DD, but if you launch several cases in quick succession, _1, _2, ... etc is added to avoid reusing an existing directory).

• You specified a run id using "code_saturne run --initialize --id <run_id>".
  In this case, the execution directory is under CASE/RESU/<run_id>, or when running coupled cases, under RESU_COUPLING/<run_id>/CASE.
  You can use a high level "code_saturne run --sugest-id" to obtain a run id suggestion (based on YYYY-MM-DD) before running the case if you have no idea of which run id to provide, but in parametric cases, generating a run id based on the parameters combination seems like a cleaner option.

If some parts of this are not clear, do not hesitate to ask me to clarify.

Thanks also for the remark on the unused "app_id". I'll clean that up soon.

Best regards,

Yvan