System error: Too many open files in system

Questions and remarks about code_saturne usage
Forum rules
Please read the forum usage recommendations before posting.
Post Reply
Mohammad
Posts: 114
Joined: Thu Oct 25, 2018 12:18 pm

System error: Too many open files in system

Post by Mohammad »

Hello,

I'm simulating a duct flow on a cluster using code_saturne 5.0.9.
Everything was OK, untill the simulation stopped after 4400 iterations with the following error:
System error: Too many open files in system
../../../../src/base/cs_base_fortran.c:544: Fatal error.

It is impossible to re-open the default output file:
listing

Call stack:
1: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
2: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
3: 0x2adea87764f2 <bft_error+0x92> (libsaturne.so.5)
4: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
5: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
6: 0x2adea87764f2 <bft_error+0x92> (libsaturne.so.5)
7: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
8: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
...

The error and listing files are attached.
listing_error.zip
(450.41 KiB) Downloaded 146 times
I set the listing file writing frequency to 10.

Where is the problem?

Regards,

Mohammad
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: System error: Too many open files in system

Post by Yvan Fournier »

Hello,

Do you have user subroutines in tour case ?

Regards,

Yvan
Mohammad
Posts: 114
Joined: Thu Oct 25, 2018 12:18 pm

Re: System error: Too many open files in system

Post by Mohammad »

Hello,

Yes! I defined 9 fields and I use cs_user_extra_operations.c file to compute the value of that fields.
I also compute about 14 time moments is cs_user_parameters-time-moments.c file.

Please note that I don't use bft_printf() function in my codes ,but I use write() function of fortran to write 6 lines at each iteration.

Code: Select all

write(nfecra, *) "Bulk velocity at time step N:", ubulk
write(nfecra, *) "Bulk velocity at time step N-1:", ubulka
write(nfecra, *) "Reference bulk velocity:", uref
write(nfecra, *) "Relaxation factor:", xrel
write(nfecra, *) "Total correction term:",ckp
Regards,

Mohammad
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: System error: Too many open files in system

Post by Yvan Fournier »

Hello,

Could you post your user subroutines ? I suspect you are opening files indirectly through some operation that might be in the time loop and should be outside the time loop, but I can check with the files.

Otherwise an error leading to memory corruption (i..e. uninitialized value or array bounds overflow) could possibly cause this, as well as a bug.

Regards,

Yvan
Mohammad
Posts: 114
Joined: Thu Oct 25, 2018 12:18 pm

Re: System error: Too many open files in system

Post by Mohammad »

Hello,

Thank you for your reply!

The files are attached.


Regards,

Mohammad
Last edited by Mohammad on Mon Dec 16, 2019 7:12 am, edited 1 time in total.
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: System error: Too many open files in system

Post by Yvan Fournier »

Hello,

I do not see any obvious error (but as your file is quite long, errors may be hidden).

Did you check in single-processor mode on a small mesh using a tool such as Valgrind ?
Did you run a check with a build compiled in debug mode ? This can also be very useful.

Finally, though most fixes are not directly related to LES, I suggest upgrading from 5.0.9 to 5.0.11.
Upgrading to 6.0 is even better, but that may require minor changes in your setup (though the files you use are very similar between 5.0 and 6.0, so upgrading should be very easy).

Regards,

Yvan
Mohammad
Posts: 114
Joined: Thu Oct 25, 2018 12:18 pm

Re: System error: Too many open files in system

Post by Mohammad »

Thank you very much Yvan,

I restarted the simulation using checkpoints and now it's on iteration #30,000 without any problems.
I'm on a new node.
I think the previous node on the cluster had some issues.

Regards,

Mohammad
Antech
Posts: 197
Joined: Wed Jun 10, 2015 10:02 am

Re: System error: Too many open files in system

Post by Antech »

Hello. Regarding open files. Check ulimit -a if you have problems related to the number of maximum open files. It may also be important to check (with the same command) maximum stack size and maximum amount of locked (reserved) memory (in some cases you will need unlimited for this). Persistent values for system limits are set in /etc/security/limits.conf (CentOS-7), reboot to apply changes in this file (or set the same with ulimit for current session).
Post Reply