Free memory constantly decreasing

[daniele]

Hello,

I am using v7 and facing some memory issues: the free memory of the nodes I run the job on is constantly decreasing, and the job at the end crashes.
I have tried to run the job removing all user routines but the problem remains.
Running the code with Valgrind indicates 0 bytes definitely or indirectly lost, but non-zero possibly lost. I don't exactly know how to intrepret this result...
I have the same issue when compiling CS with openmpi/2.1.1-gcc820 or with openmpi/4.0.6-gcc850.

Do you have any suggestion about what to check to understand the source of the problem?

Thank you very much in advance.
Kind regards,
Daniele

[Yvan Fournier]

Hello,

This can be due to a memory leak either in code_saturne or in the libraries. If it is in the libraries it will be hard to solve, but we can first check whether it is in code_saturne.

Can you add
export CS_MEM_LOG=mem_log
in you run environment ?
- if you are runing from a terminal, directly in the terminal before running the code
- If you are submitting the job on a cluster, add it just after the batch directives in the run.cfg file

For each MPI rank, you will have a file tracing memory allocations and frees.
You only need to run for a few (10-20) time steps, and once you have the output, you can post a sample of 2 such files here (or send them to me).

If Valgrind detects nothing, I'm not sure this will be better, but it can certainly help detect other types of issues (buffers freed at the end but of larger and larger size during the computation), as these files allow generating plots of (ther instrumented part of) memory usage.

Best regards,

Yvan

[daniele]

Hello Yvan,

Thank you for your help.
I have found out that I had a beta function (for turbulence modeling) of CS v7 activated that seems to be the source of the memory leak. Switching to a "standard" turbulence model, memory does not seem to decrease anymore.
Anyway, I have done what you suggested, for both the simulations (standard and beta function), I attach two of the files that are generated: the ones named "..._leak" are generated by the set up that generates the leak, the ones "...no_leak" from the setup not generating the leak.

Hope this helps to analyse the problem.
Thank you very much.
Best regards,
Daniele

[daniele]

with the last attachement that couldn't be uploaded in the previous post...

[Yvan Fournier]

Hello,

There is no "instrumented" memory which seems to leek, and the instrumented memory usage does not seem to leak (importing your file under a spreadsheet and plotting the memory use over allocations_ I get this (which is good, as you seem memory being allocated and freed in equal amounts at each time step) :

So it seems this is in a non-instrumented part of the code (or possibly but with less probability only on some nodes, if this is for example due to a boundary array). Is the beta function for turbulence modeling a function in the code (in case which one ?) or your own modification (in which case can you provide it so I can check it ?).

Best regards,

Yvan

[daniele]

Hello,

Thank you for checking all this.
The beta function is already available in the CS sources v7. You can find it at le line 195 of the cs_turbulence_kw.c (DDES).
The only two additional allocated arrays that are created following the activation of this model are correctly freed at the end of the routine...
That looks weird. I can do a test once again, but my test clearly showed that not activating this model allowed to keep a constant free memory on nodes.

Thanks a lot for your time and help.
Bets regards,
Daniele

[daniele]

Hello Yvan,

I have realized some further testing on my memory leak issue. I recall that I originally posted a problem of continuously decreasing free RAM memory, that eventually leads the simulation crash because of zero available RAM.
I use v7. I added a "export CS_MEM_LOG="mem.log" ", and run only 3 time-steps.
The simulation consists of an ALE with imposed wall boundary displacements, prescribed through the "cs_user_boundary_conditions_ale.f90".

The results of my tests are the following:
- Running the case without the "cs_user_boundary_conditions_ale.f90" routine, the mem.log indicates zero non freed pointers (see attached file "mem.log.00_no_routine");
- Adding the "cs_user_boundary_conditions_ale.f90" routine, WITHOUT ADDING ANYTHING INSIDE IT BESIDE THE DEFAULT VARIABLE DECLARATION (I attached the routine), makes the mem.log indicate that a certain number of pointers are not freed (see attached file "mem.log.00_with_routine").

The preliminary conclusion would be that the simple addition of the "cs_user_boundary_conditions_ale.f90" routine (even if "empty") causes some non freed pointers. The non freed array seems to be "cs_meg_boundary_function.c".

Do you have any idea of the possible reasons of the observed behavior?

Thank you very much for your help.
Kind regards,
Daniele

[Yvan Fournier]

Hello Daniele,

I do not reproduce this solely by adding a user-defined function, but I do reproduce this by defining a fixed mesh displacement with the GUI.

I have pushed a fix in the master, v8.0, and v7.0 branches for this. So the fix will appear in v8.1, v8.0.3, and v7.0.7 when they are released (and is already available on the Git branches).

Best regards,

Yvan

[daniele]

Hello Yvan,

Thank you for testing and correcting the issue.
I am not experienced with Github, I saw the commit where you fix the bug on branch v7.0: does it mean that if I clone v7.0 and compile it, I will have the code with the bug fixed?

Thank you very much for your help.
Best regards,
Daniele

[Yvan Fournier]

Hello Daniele,

Yes, cloning the v7.0 branch allows you to have a version with this bug fix without waiting for the next v7.0.7 release.

After cloning, as a precaution, "cd" into the cloned repository, and run :

git checkout v7.0
./sbin/bootstrap && ./sbin/clean

You can then use that just the same as a .tar.gz version.

Best regards,

Yvan