Page 1 of 1

Lot of files "error_r****" and simulation stops every N iterations

Posted: Sat Jul 18, 2020 1:08 pm
by ptessier2
Hi everyone !

When I run a simulation of a cross flow over a cylinder, it only runs till the 50 000th iteration (out of 500 000 in total) and then stops leaving plenty of files of the type "error_rXXXX". XXXX vary from '0000' to a high number like '1400', it depends on the number of processors. And every of those files are empty

I already succesfully run such simulations but I got this type of error for the last simulations. I have enough storage left. The job.err.log file is ok, and no error message are in the result directory can help me

I can provide any information needed.

Re: Lot of files "error_r****" and simulation stops every N iterations

Posted: Sat Jul 18, 2020 2:01 pm
by Yvan Fournier
Hello,

If the error* files are empty (they should contain backtraces) you might have sole info on errors on the batch output or error log file. If a single error_r* file us non-empty that would be interesting.

Do you have user functions ? A specific output at 50000 iterations ? Doea the end of the run_solver.log/listing file contain useful info ?

Regards,

Yvan

Re: Lot of files "error_r****" and simulation stops every N iterations

Posted: Mon Jul 20, 2020 1:42 pm
by ptessier2
Hello

Tank you for your quick reply !

I have indeed a user function which is the cause of my problem : the simulation runs correctly without it.
I join the user function file and the batch error file which mainly says "segmentation fault".
The user function should print in the listing file the mean velocity of control aeras around a cylindrical geometry.

Re: Lot of files "error_r****" and simulation stops every N iterations

Posted: Mon Jul 20, 2020 2:07 pm
by Yvan Fournier
Hello,

Did you test this user subroutine on a smaller case on a workstation ? Ideally using a debug build ? This is the best practice, and usually make debugging much easier than on a large case with a batch system (even with good parallel debuggers such as DDT or TotalView, limiting "cluster debugging" to otherwise hard to reproduce cases is often a better use of resources).

Best regards,

Yvan