Hi all,
I'm having some problems using the restart function lately. I
already used it for several calculations without any problems, but now
when I start a calculation based on previous results (with all the same
parameters), the calculation either stops after one or two iterations
with the divergence of one variable, or I have to stop it myself
because it keeps running but prints stange messages : MAXIMUM NUMBER OF ITERATIONS REACHED FOR THE COMPUTATION OF UET: NITM= 100, DESIRED PRECISION EPS= 0.100E-02 CAUSTA SUBROUTINE CALLED FOR FACE IFAC= 75854
and everything
diverges (I got NaN everywhere)...
I don't understand why it doesn't work properly anymore, as I didn't
change anything in the installation or calculation procedure... maybe
the meshes I'm using now are a bit bigger, that all I can see that is
different from my previous restarted calculations...
I attach here one of the listing file with strange messages, from a
calculation I had to kill myself (I deleted most of the lines, as the massage appears for each face of the mesh... the file was too big to attach here !). The restart files are also too big too attach. I only have a suiavx and suiava in the restart directory (in .../RESU/RESTART.DATE). I'm using Code Saturne version 1.4.
Hope someone can help me...
Alicia
Problem using restart function
Forum rules
Please read the forum usage recommendations before posting.
Please read the forum usage recommendations before posting.
-
Alicia Consigny
Problem using restart function
- Attachments
-
- batch_100708_1106_reduc.txt
- (57.46 KiB) Downloaded 276 times
-
Yvan Fournier
Re: Problem using restart function
Hello,
Judging by your attachment, the code does indeed start diverging immediately, as we see velocities in the 10e4 at the end of the time step.
To see if the restart files have been corrupted somehow, you may run a zero-time step calculation with the restart files, so as to obtain a postprocessing output of the fields as they are written in the restart files.
If that seems OK, you may want to try :
- setting ileaux = 0 in usini1, so that mass fluxes are reconstructed, or
- running a time-dependent calculation instead of a steady calculation whne doing the restart. I am not sure in which conditions you used the restart before, but as the steady algorithm was quite recent when 1.4 came out, there might be issues when combining that with a restart (I'll try to check on that)
If all of this fails, does the calculation work if you run 55 iterations from the start for example (i.e. is there a possibility that it starts diverging around iteration 51 anyways, regardless of the restart aspect).
Best regards,
Yvan
Judging by your attachment, the code does indeed start diverging immediately, as we see velocities in the 10e4 at the end of the time step.
To see if the restart files have been corrupted somehow, you may run a zero-time step calculation with the restart files, so as to obtain a postprocessing output of the fields as they are written in the restart files.
If that seems OK, you may want to try :
- setting ileaux = 0 in usini1, so that mass fluxes are reconstructed, or
- running a time-dependent calculation instead of a steady calculation whne doing the restart. I am not sure in which conditions you used the restart before, but as the steady algorithm was quite recent when 1.4 came out, there might be issues when combining that with a restart (I'll try to check on that)
If all of this fails, does the calculation work if you run 55 iterations from the start for example (i.e. is there a possibility that it starts diverging around iteration 51 anyways, regardless of the restart aspect).
Best regards,
Yvan
-
Alicia Consigny
Re: Problem using restart function
ok so I'm quite lost now...
I was thinking that my restart files were having a problem, because as you said the divergence starts at the fisrt iteration of the restarted calculation. So I tried running a zero time step calculation, with the same restart files, and the results I have are the same as the ones I have when I look at the results of the restarted calculation (i.e. the results after the first 50 iterations are the same as the ones after the zero time step calculation with the restart files) -> the restart files are ok !
Then I tried setting ILEAUX = 0 in usini1.F. When running the calculation with this option and the restart files, the auxiliary restart file is not read, which means that the modification is taken into account... but still the calculation diverges after a few iterations. I attach the listing here.
I tried running more than 50 iterations from the start, and all is fine until at least 100 iterations, so the problem doesn't come from the calculation itself).
I already used the restart with some steady calculations without any problems, and as I didn't change anything since then, I don't know why it wouldn't work anymore suddenly !
So if you have other ideas or leads... Anyway thanks a lot for the previous ones, it's always interesting !
Alicia
I was thinking that my restart files were having a problem, because as you said the divergence starts at the fisrt iteration of the restarted calculation. So I tried running a zero time step calculation, with the same restart files, and the results I have are the same as the ones I have when I look at the results of the restarted calculation (i.e. the results after the first 50 iterations are the same as the ones after the zero time step calculation with the restart files) -> the restart files are ok !
Then I tried setting ILEAUX = 0 in usini1.F. When running the calculation with this option and the restart files, the auxiliary restart file is not read, which means that the modification is taken into account... but still the calculation diverges after a few iterations. I attach the listing here.
I tried running more than 50 iterations from the start, and all is fine until at least 100 iterations, so the problem doesn't come from the calculation itself).
I already used the restart with some steady calculations without any problems, and as I didn't change anything since then, I don't know why it wouldn't work anymore suddenly !
So if you have other ideas or leads... Anyway thanks a lot for the previous ones, it's always interesting !
Alicia
- Attachments
-
- batch_100708_1545.txt
- (39.79 KiB) Downloaded 286 times
-
Yvan Fournier
Re: Problem using restart function
Hello,
That seems hard to debug... As you seemed to be ready to move to Code_Saturne 2.0-rc2, testing if the issue still appears with that version may be best. If it does still appear, and you are able to reproduce it on small meshes, you may send us the case.
If you are only able to reproduce the error on the large case, what you could post would be:
- the "listing" log files for the first 50 iterations (before restart) and for the case wher you run 100 iterations directly.
- the xml and user Fortran subroutines for both the 50 initial iterations and the restarted run,
so that we could see if any discrepancy appears between the input data and the way the restart handles it.
Also, a final test you could try with the current version would be to run a 1 to 5 restarted iterations in single-processor mode, in case data read is corrupted by parallel I/O (improbable since yor 0 time-step restart seems OK, but constant values other than fields read from the restart could be garbled, and it's an easy check to run, so you may want to try it first.
Best regards,
Yvan
That seems hard to debug... As you seemed to be ready to move to Code_Saturne 2.0-rc2, testing if the issue still appears with that version may be best. If it does still appear, and you are able to reproduce it on small meshes, you may send us the case.
If you are only able to reproduce the error on the large case, what you could post would be:
- the "listing" log files for the first 50 iterations (before restart) and for the case wher you run 100 iterations directly.
- the xml and user Fortran subroutines for both the 50 initial iterations and the restarted run,
so that we could see if any discrepancy appears between the input data and the way the restart handles it.
Also, a final test you could try with the current version would be to run a 1 to 5 restarted iterations in single-processor mode, in case data read is corrupted by parallel I/O (improbable since yor 0 time-step restart seems OK, but constant values other than fields read from the restart could be garbled, and it's an easy check to run, so you may want to try it first.
Best regards,
Yvan
-
Alicia Consigny
Re: Problem using restart function
Hi,
I wasn't able to figure out what was the problem with the 1.4 version.
Anyway I moved to CS 2.0-rc2 and don't have the problem anymore... and don't have time to debug the old version, as I'm only going to use it for small cases, it doesn't really matter anymore... Thank you anyway for all your advices !
Alicia
I wasn't able to figure out what was the problem with the 1.4 version.
Anyway I moved to CS 2.0-rc2 and don't have the problem anymore... and don't have time to debug the old version, as I'm only going to use it for small cases, it doesn't really matter anymore... Thank you anyway for all your advices !
Alicia