Page 1 of 1
System error: Too many open files in system
Posted: Sun Dec 15, 2019 8:45 am
by Mohammad
Hello,
I'm simulating a duct flow on a cluster using code_saturne 5.0.9.
Everything was OK, untill the simulation stopped after 4400 iterations with the following error:
System error: Too many open files in system
../../../../src/base/cs_base_fortran.c:544: Fatal error.
It is impossible to re-open the default output file:
listing
Call stack:
1: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
2: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
3: 0x2adea87764f2 <bft_error+0x92> (libsaturne.so.5)
4: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
5: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
6: 0x2adea87764f2 <bft_error+0x92> (libsaturne.so.5)
7: 0x2adea7fb7593 <cs_base_fortran_bft_printf_to_c+0x93> (libsaturne.so.5)
8: 0x2adea819566b <+0x32b66b> (libsaturne.so.5)
...
The error and listing files are attached.
I set the listing file writing frequency to 10.
Where is the problem?
Regards,
Mohammad
Re: System error: Too many open files in system
Posted: Sun Dec 15, 2019 11:21 am
by Yvan Fournier
Hello,
Do you have user subroutines in tour case ?
Regards,
Yvan
Re: System error: Too many open files in system
Posted: Sun Dec 15, 2019 12:15 pm
by Mohammad
Hello,
Yes! I defined 9 fields and I use
cs_user_extra_operations.c file to compute the value of that fields.
I also compute about 14 time moments is
cs_user_parameters-time-moments.c file.
Please note that I don't use bft_printf() function in my codes ,but I use write() function of fortran to write 6 lines at each iteration.
Code: Select all
write(nfecra, *) "Bulk velocity at time step N:", ubulk
write(nfecra, *) "Bulk velocity at time step N-1:", ubulka
write(nfecra, *) "Reference bulk velocity:", uref
write(nfecra, *) "Relaxation factor:", xrel
write(nfecra, *) "Total correction term:",ckp
Regards,
Mohammad
Re: System error: Too many open files in system
Posted: Sun Dec 15, 2019 8:26 pm
by Yvan Fournier
Hello,
Could you post your user subroutines ? I suspect you are opening files indirectly through some operation that might be in the time loop and should be outside the time loop, but I can check with the files.
Otherwise an error leading to memory corruption (i..e. uninitialized value or array bounds overflow) could possibly cause this, as well as a bug.
Regards,
Yvan
Re: System error: Too many open files in system
Posted: Sun Dec 15, 2019 11:34 pm
by Mohammad
Hello,
Thank you for your reply!
The files are attached.
Regards,
Mohammad
Re: System error: Too many open files in system
Posted: Mon Dec 16, 2019 12:05 am
by Yvan Fournier
Hello,
I do not see any obvious error (but as your file is quite long, errors may be hidden).
Did you check in single-processor mode on a small mesh using a tool such as Valgrind ?
Did you run a check with a build compiled in debug mode ? This can also be very useful.
Finally, though most fixes are not directly related to LES, I suggest upgrading from 5.0.9 to 5.0.11.
Upgrading to 6.0 is even better, but that may require minor changes in your setup (though the files you use are very similar between 5.0 and 6.0, so upgrading should be very easy).
Regards,
Yvan
Re: System error: Too many open files in system
Posted: Mon Dec 16, 2019 11:35 am
by Mohammad
Thank you very much Yvan,
I restarted the simulation using checkpoints and now it's on iteration #30,000 without any problems.
I'm on a new node.
I think the previous node on the cluster had some issues.
Regards,
Mohammad
Re: System error: Too many open files in system
Posted: Fri Dec 27, 2019 8:42 am
by Antech
Hello. Regarding open files. Check ulimit -a if you have problems related to the number of maximum open files. It may also be important to check (with the same command) maximum stack size and maximum amount of locked (reserved) memory (in some cases you will need unlimited for this). Persistent values for system limits are set in /etc/security/limits.conf (CentOS-7), reboot to apply changes in this file (or set the same with ulimit for current session).