System error: Too many open files in system

[Mohammad]

Hello,

I'm simulating a duct flow on a cluster using code_saturne 5.0.9.
Everything was OK, untill the simulation stopped after 4400 iterations with the following error:

System error: Too many open files in system
../../../../src/base/cs_base_fortran.c:544: Fatal error.

It is impossible to re-open the default output file:
listing

Call stack:
1: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
2: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
3: 0x2adea87764f2 <bft_error+0x92> (libsaturne.so.5)
4: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
5: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
6: 0x2adea87764f2 <bft_error+0x92> (libsaturne.so.5)
7: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
8: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
...

The error and listing files are attached.

listing_error.zip

(450.41 KiB) Downloaded 306 times

I set the listing file writing frequency to 10.

Where is the problem?

Regards,

Mohammad

[Yvan Fournier]

Hello,

Do you have user subroutines in tour case ?

Regards,

Yvan

[Mohammad]

Hello,

Yes! I defined 9 fields and I use cs_user_extra_operations.c file to compute the value of that fields.
I also compute about 14 time moments is cs_user_parameters-time-moments.c file.

Please note that I don't use bft_printf() function in my codes ,but I use write() function of fortran to write 6 lines at each iteration.

Code: Select all

write(nfecra, *) "Bulk velocity at time step N:", ubulk
write(nfecra, *) "Bulk velocity at time step N-1:", ubulka
write(nfecra, *) "Reference bulk velocity:", uref
write(nfecra, *) "Relaxation factor:", xrel
write(nfecra, *) "Total correction term:",ckp

Regards,

Mohammad

[Yvan Fournier]

Hello,

Could you post your user subroutines ? I suspect you are opening files indirectly through some operation that might be in the time loop and should be outside the time loop, but I can check with the files.

Otherwise an error leading to memory corruption (i..e. uninitialized value or array bounds overflow) could possibly cause this, as well as a bug.

Regards,

Yvan

[Mohammad]

Hello,

Thank you for your reply!

The files are attached.


Regards,

Mohammad

[Yvan Fournier]

Hello,

I do not see any obvious error (but as your file is quite long, errors may be hidden).

Did you check in single-processor mode on a small mesh using a tool such as Valgrind ?
Did you run a check with a build compiled in debug mode ? This can also be very useful.

Finally, though most fixes are not directly related to LES, I suggest upgrading from 5.0.9 to 5.0.11.
Upgrading to 6.0 is even better, but that may require minor changes in your setup (though the files you use are very similar between 5.0 and 6.0, so upgrading should be very easy).

Regards,

Yvan

[Mohammad]

Thank you very much Yvan,

I restarted the simulation using checkpoints and now it's on iteration #30,000 without any problems.
I'm on a new node.
I think the previous node on the cluster had some issues.

Regards,

Mohammad

[Antech]

Hello. Regarding open files. Check ulimit -a if you have problems related to the number of maximum open files. It may also be important to check (with the same command) maximum stack size and maximum amount of locked (reserved) memory (in some cases you will need unlimited for this). Persistent values for system limits are set in /etc/security/limits.conf (CentOS-7), reboot to apply changes in this file (or set the same with ulimit for current session).