Restart error-Invalid (not a number) values detected for a field

Questions and remarks about code_saturne usage
Forum rules
Please read the forum usage recommendations before posting.
Ruonan
Posts: 136
Joined: Mon Dec 14, 2020 11:38 am

Restart error-Invalid (not a number) values detected for a field

Post by Ruonan »

Dear developers,

Please could you help me with this restarting problem?

I am running version 7.2 alpha, compiled from GitHub source code, on a cluster. When I restart, I always found an error, "Invalid (not-a-number) values detected for a field." The listing, xml and subroutine I used are attached.

I use the ideal gas law rho=P/(R*T). I can see the error came from infinite density, which is because the minimum temperature went to 0. But the minimum temperature should be 293.15K, as I set it as boundary condition.

I also found one warning when reading the auxiliary restart file, so I tried not to read the auxiliary restart file, but the error was still there.

Many thanks,
Ruonan

Code: Select all

         READING THE AUXILIARY RESTART FILE             

 Start reading                                          
   Finished reading options                             
  restart/auxiliary.csc: section "ref_presstot01" not present.
@                                                            
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@                                                            
@ @@ WARNING: WHEN READING THE AUXILIARY RESTART FILE        
@    =======                                                 
@      Lecture des proprietes physiques                      
@                                                            
@    It was not possible to read some values from the        
@      auxiliary restart file.                               
@    They will be initialized by the default values.         
@                                                            
@     This situation can occur when the restart file         
@      originates from a run realised with different         
@      options or when the file is damaged.                  
@                                                            
@    The run will continue...                                
@                                                            
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@                                                            

   Fin de la lecture des proprietes physiques           
   Fin de la lecture du pas de temps                    
   Fin de la lecture des flux de masse                  
   Fin de la lecture des conditions aux limites         
  restart/auxiliary.csc: section "time_moments:sizes" not present.
 End reading                                            

 ----------------------------------------------------------- 


 ** VARIABLES INITIALIZATION
    ------------------------

 -----------------------------------------
 Property           Min. value  Max. value
 -----------------------------------------
  density           0.1021E+01    Infinity
  molecular_viscos  0.1810E-04  0.1810E-04
  turbulent_viscos  0.1793E-09    Infinity
 -----------------------------------------

 --- Diffusivity:
 -----------------------------------------------
 Scalar           Number  Min. value  Max. value
 -----------------------------------------------
 TempK                 1  0.2590E-01  0.2590E-01
 -----------------------------------------------



   ** INFORMATION ON BOUNDARY FACES TYPE
      ----------------------------------

-------------------------------------------------------------------------
Boundary type          Code    Nb faces
-------------------------------------------------------------------------
Inlet                     2           0
Smooth wall               5        2384
Rough wall                6           0
Symmetry                  4      357600
Free outlet               3           0
Free inlet               14           0
Convective inlet         16           0
Free surface             15           0
Undefined                 1           0
-------------------------------------------------------------------------


  ** Clippings for computed fields
     -----------------------------

   field     initial min     initial max  clips to min  clips to max
-  -----  --------------  --------------  ------------  ------------
a  TempK


  ** Field values on cells
     ---------------------

   field                    minimum         maximum        set mean    spatial mean
-  ----------------  --------------  --------------  --------------  --------------
v  Velocity[X]          -1.3814e-11      1.7239e-11     -5.1037e-16     -3.9398e-16
v  Velocity[Y]             -0.34354         0.31963      0.00012551     -0.00085584
v  Velocity[Z]             -0.33995         0.27782       -0.010218       -0.011866
v  ǁVelocityǁ                     0         0.35436        0.091759        0.095616
v  Pressure                -0.16049         0.17511       0.0076409       0.0011287
v  TempK                          0          345.84          163.23          164.36
v  Density                   1.0208             inf             inf             inf
v  TurbVisc              1.7931e-10             inf            -nan            -nan
v  CourantNb                      0               0               0               0
v  FourierNb                      0               0               0               0
v  total_pressure        1.0132e+05      1.0133e+05      1.0133e+05      1.0132e+05
v  Wall distance                  0        0.057692        0.013109        0.014458
v  Wall Y+                        0          207.28          23.596          25.743
v  volume_forces[X]               0               0               0               0
v  volume_forces[Y]         -7.2807          7.7347      -0.0014006       0.0054735
v  volume_forces[Z]         -7.8166          7.2972        0.005871      -0.0017102
v  ǁvolume_forcesǁ       0.00034577          8.0707          3.0387          3.1546

  ** Field values on boundary_faces
     ------------------------------

   field                        minimum         maximum        set mean    spatial mean
-  --------------------  --------------  --------------  --------------  --------------
v  Boundary temperature               0          345.84          163.21          164.36
v  Boundary Density              1.0208             inf             inf             inf
v  Yplus                              0               0               0               0



/users/rw00793/code_saturne_72alpha/code_saturne-master/src/base/cs_log_iteration.c:842: Fatal error.

Invalid (not-a-number) values detected for a field.


Call stack:
   1: 0x2af234c3006a <+0x16006a>                      (libsaturne-7.2.so)
   2: 0x2af234c32228 <cs_log_iteration+0x18>          (libsaturne-7.2.so)
   3: 0x2af234b8ec5d <caltri_+0x188d>                 (libsaturne-7.2.so)
   4: 0x2af2348d3fe9 <main+0x6d9>                     (libcs_solver-7.2.so)
   5: 0x2af237cc7c05 <__libc_start_main+0xf5>         (libc.so.6)
   6: 0x401209     <>                               (cs_solver)
End of stack
Best regards,
Ruonan
Attachments
setup.xml
(9.89 KiB) Downloaded 78 times
listing.txt
(25.7 KiB) Downloaded 90 times
cs_user_physical_properties.f90
(5.67 KiB) Downloaded 86 times
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: Restart error-Invalid (not a number) values detected for a field

Post by Yvan Fournier »

Hello,

From the log, it seems the field causing an issue is the boundary density.

Do you have a small mesh I could use for debugging ?

Regards,

Yvan
Ruonan
Posts: 136
Joined: Mon Dec 14, 2020 11:38 am

Re: Restart error-Invalid (not a number) values detected for a field

Post by Ruonan »

Hello Yvan,

Thank you very much for debugging!

I attached the mesh, xml and subroutines. I use two subroutines, one for changing the density, the other one for adding centrifugal source term explicitly to the momentum equation.

One strange thing is, when I run it for a shorter time and then restart, sometimes I have no error. But in most cases, especially when I run for a long time and then restart from that, I would have this error.

Best regards,
Ruonan
Attachments
cs_user_physical_properties.f90
(5.67 KiB) Downloaded 75 times
setup.xml
(11.83 KiB) Downloaded 73 times
Bohn_1995_configB_2D_Ra1E7.cgns
(9.49 MiB) Downloaded 79 times
Ruonan
Posts: 136
Joined: Mon Dec 14, 2020 11:38 am

Re: Restart error-Invalid (not a number) values detected for a field

Post by Ruonan »

Hello Yvan,

Attached is the second subroutine. Thank you!

Best regards,
Ruonan
Attachments
cs_user_source_terms-momentum.c
(5.09 KiB) Downloaded 79 times
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: Restart error-Invalid (not a number) values detected for a field

Post by Yvan Fournier »

Hello,

Running on a few iterations before restarting, I cannot reproduce the issue.

In cases where this fails for you, how many iterations do you run prior to restart ? Also, in cases where the restart fails, could you check the min/max values for boundary density in the run_solver.log file at the end of the previous computation ? So as to determine whether the error is only upon restarting, or upstream.

If the issue comes from the temperature, it would mean the boundary density is recomputed rather than using the value read (which is quite possible), but that the temperature used is not quite updated. I'll check again.

Also, how many processors are you running on ? In case this helps reproducing the issue.

Best regards,

Yvan
Ruonan
Posts: 136
Joined: Mon Dec 14, 2020 11:38 am

Re: Restart error-Invalid (not a number) values detected for a field

Post by Ruonan »

Hello Yvan,

Thank you very much. Yes, if I run it for only a few iterations and then restart, I won't have this issue either.

Regarding this case, I ran 366390 iterations before I restart. I used "n_procs: 56, n_threads: 1".

The run_solver.log for previous computation is too large to upload, so I copy the last two steps below. We can see the min/max boundary density values at the last step are 1.0208 and 1.2038, which are good.

As you said, maybe it's because when restart some values are not quite updated?

Today I did another case, with different geometry and boundary conditions, but ended up with this error again. It's very annoying, if it can't be restarted, it means all the previous computation is wasted. I suspect this is a bug?

Could you please help me to check?

Many thanks,
Ruonan

Code: Select all

 INSTANT    0.352832607E+03   TIME STEP NUMBER          366389
 =============================================================




   ** BOUNDARY MASS FLOW INFORMATION
      ------------------------------

---------------------------------------------------------------
Boundary type          Code    Nb faces           Mass flow
---------------------------------------------------------------
Inlet                     2           0         0.000000000E+00
Smooth wall               5        1584         0.000000000E+00
Rough wall                6           0         0.000000000E+00
Symmetry                  4      158400         0.000000000E+00
Free outlet               3           0         0.000000000E+00
Free inlet               14           0         0.000000000E+00
Convective inlet         16           0         0.000000000E+00
Free surface             15           0         0.000000000E+00
Undefined                 1           0         0.000000000E+00
---------------------------------------------------------------


   ** BOUNDARY CONDITIONS FOR SMOOTH WALLS
   ---------------------------------------
------------------------------------------------------------
                                         Minimum     Maximum
------------------------------------------------------------
   Rel velocity at the wall uiptn : -0.00000E+00-0.00000E+00
   Friction velocity        uet   :  0.42417E-03 0.65765E-01
   Friction velocity        uk    :  0.42417E-03 0.65765E-01
   Dimensionless distance   yplus :  0.28208E-02 0.37193E+00
   Friction thermal sca.    tstar : -0.30690E+02 0.79850E+02
   Dim-less thermal sca.    tplus :  0.28208E-02 0.37193E+00
   ------------------------------------------------------
   Nb of reversal of the velocity at the wall   :          0
   Nb of faces within the viscous sub-layer     :       1584
   Total number of wall faces                   :       1584
------------------------------------------------------------



   ** INFORMATION ON CONVERGENCE
      --------------------------
-----------------------------------------------------------------------------
   Variable      Rhs norm      N_iter  Norm. residual   Drift   Time residual
-----------------------------------------------------------------------------
c  Velocity      0.46731E-04       2   0.65215E-05   0.11257E-03 0.21425E+01
c  Velocity[X]                                       0.30210E-26
c  Velocity[Y]                                       0.57185E-04
c  Velocity[Z]                                       0.55381E-04
c  Pressure      0.37028E-09       6   0.83419E-04   0.40883E-02 0.86475E+00
c  TempK         0.15051E+01      20   0.35584E-03   0.78096E+00 0.86901E-01
c  Wall distance 0.25964E-06      22   0.39292E-03   0.00000E+00 0.00000E+00
c  Wall Y+       0.12786E+00       5   0.69080E-08   0.21155E+02 0.22790E+01
-----------------------------------------------------------------------------



  ** Clippings for computed fields
     -----------------------------

   field     initial min     initial max  clips to min  clips to max
-  -----  --------------  --------------  ------------  ------------
a  TempK


  ** Field values on cells
     ---------------------

   field                    minimum         maximum        set mean    spatial mean
-  ----------------  --------------  --------------  --------------  --------------
v  Velocity[X]           -8.997e-12      2.2525e-11     -3.4597e-16     -2.1204e-17
v  Velocity[Y]             -0.30008         0.29181      0.00024243     -2.9789e-05
v  Velocity[Z]             -0.33031         0.33517      0.00012093      0.00010256
v  ǁVelocityǁ            5.5188e-06         0.33608          0.1304         0.14316
v  Pressure                -0.17544         0.17743        0.014403     -1.4057e-17
v  TempK                     293.29          345.85          325.85          327.63
v  Density                   1.0208          1.2038          1.0848          1.0781
v  TurbVisc              1.9253e-17      3.6283e-07      4.4725e-08      5.7775e-08
v  CourantNb             1.5504e-05         0.33753         0.10771         0.11434
v  FourierNb               0.023066         0.71539         0.10715        0.051893
v  total_pressure        1.0132e+05      1.0133e+05      1.0133e+05      1.0132e+05
v  Wall distance         9.9922e-05         0.05712        0.022384        0.028817
v  Wall Y+                0.0028309          212.12          38.998          50.065
v  volume_forces[X]               0               0               0               0
v  volume_forces[Y]         -7.6959          7.4718      -0.0069542      -0.0072158
v  volume_forces[Z]         -6.9508          7.0087         0.01095       0.0023309
v  ǁvolume_forcesǁ       0.00027923           7.697          2.9323          3.1473

  ** Field values on boundary_faces
     ------------------------------

   field                        minimum         maximum        set mean    spatial mean
-  --------------------  --------------  --------------  --------------  --------------
v  Boundary temperature          293.15          345.92          325.79          327.63
v  Boundary Density              1.0208          1.2038           1.085          1.0781
v  Yplus                              0         0.37193       0.0017267      0.00074886


  ** Computed values on cells
     ------------------------

   criterion               minimum         maximum        set mean    spatial mean
   ---------------  --------------  --------------  --------------  --------------
   Courant/Fourier        0.028429         0.74628         0.21486         0.16623


 Options set or changed by "control_file":
 -------------------------

  max_time_step                          366390 (current: 366390)

 Finished reading "control_file".


 INSTANT    0.352833570E+03   TIME STEP NUMBER          366390
 =============================================================



   ** BOUNDARY MASS FLOW INFORMATION
      ------------------------------

---------------------------------------------------------------
Boundary type          Code    Nb faces           Mass flow
---------------------------------------------------------------
Inlet                     2           0         0.000000000E+00
Smooth wall               5        1584         0.000000000E+00
Rough wall                6           0         0.000000000E+00
Symmetry                  4      158400         0.000000000E+00
Free outlet               3           0         0.000000000E+00
Free inlet               14           0         0.000000000E+00
Convective inlet         16           0         0.000000000E+00
Free surface             15           0         0.000000000E+00
Undefined                 1           0         0.000000000E+00
---------------------------------------------------------------


   ** BOUNDARY CONDITIONS FOR SMOOTH WALLS
   ---------------------------------------
------------------------------------------------------------
                                         Minimum     Maximum
------------------------------------------------------------
   Rel velocity at the wall uiptn : -0.00000E+00-0.00000E+00
   Friction velocity        uet   :  0.44715E-03 0.65742E-01
   Friction velocity        uk    :  0.44715E-03 0.65742E-01
   Dimensionless distance   yplus :  0.29737E-02 0.37180E+00
   Friction thermal sca.    tstar : -0.28956E+02 0.75651E+02
   Dim-less thermal sca.    tplus :  0.29737E-02 0.37180E+00
   ------------------------------------------------------
   Nb of reversal of the velocity at the wall   :          0
   Nb of faces within the viscous sub-layer     :       1584
   Total number of wall faces                   :       1584
------------------------------------------------------------




 Write final restart files
   checkpoint at iteration     366390, Physical time    0.35283E+03


   ** Writing the main restart file
      -----------------------------

 Start writing
  Wrote field names and types to checkpoint at iteration 366390: checkpoint/main.csc
   End writing the options
  Wrote main variables to checkpoint: checkpoint/main.csc
 End writing

   ** Writing the auxiliary restart file
      ----------------------------------

 Start writing
   End writing the dimensions and options
   End writing the physical properties
   End writing the time step
  Wrote boundary condition coefficients to checkpoint: checkpoint/auxiliary.csc
   End writing the source terms
 End writing

   ** INFORMATION ON CONVERGENCE
      --------------------------
-----------------------------------------------------------------------------
   Variable      Rhs norm      N_iter  Norm. residual   Drift   Time residual
-----------------------------------------------------------------------------
c  Velocity      0.46732E-04       2   0.65223E-05   0.11261E-03 0.21429E+01
c  Velocity[X]                                       0.30286E-26
c  Velocity[Y]                                       0.57199E-04
c  Velocity[Z]                                       0.55414E-04
c  Pressure      0.37042E-09       6   0.83467E-04   0.40897E-02 0.86457E+00
c  TempK         0.15051E+01      20   0.36571E-03   0.78138E+00 0.86925E-01
c  Wall distance 0.25964E-06      22   0.39292E-03   0.00000E+00 0.00000E+00
c  Wall Y+       0.12788E+00       5   0.69152E-08   0.21198E+02 0.22809E+01
-----------------------------------------------------------------------------



  ** Clippings for computed fields
     -----------------------------

   field     initial min     initial max  clips to min  clips to max
-  -----  --------------  --------------  ------------  ------------
a  TempK


  ** Field values on cells
     ---------------------

   field                    minimum         maximum        set mean    spatial mean
-  ----------------  --------------  --------------  --------------  --------------
v  Velocity[X]          -9.0049e-12      2.2522e-11     -3.4647e-16     -2.1282e-17
v  Velocity[Y]             -0.30007         0.29148      0.00024215     -3.0071e-05
v  Velocity[Z]             -0.33035         0.33488      0.00012356      0.00010278
v  ǁVelocityǁ            5.4884e-06          0.3361         0.13041         0.14316
v  Pressure                -0.17547         0.17738        0.014401     -1.1647e-17
v  TempK                     293.29          345.85          325.85          327.63
v  Density                   1.0208          1.2038          1.0848          1.0781
v  TurbVisc              2.1207e-17      3.6415e-07      4.4733e-08      5.7784e-08
v  CourantNb             1.5194e-05         0.33763         0.10772         0.11434
v  FourierNb               0.023067         0.71539         0.10715        0.051893
v  total_pressure        1.0132e+05      1.0133e+05      1.0133e+05      1.0132e+05
v  Wall distance         9.9922e-05         0.05712        0.022384        0.028817
v  Wall Y+                0.0029841          211.99          39.006          50.074
v  volume_forces[X]               0               0               0               0
v  volume_forces[Y]         -7.6922          7.4727       -0.006989      -0.0072134
v  volume_forces[Z]          -6.949           7.002        0.010952       0.0023209
v  ǁvolume_forcesǁ       0.00033651          7.6933          2.9324          3.1473

  ** Field values on boundary_faces
     ------------------------------

   field                        minimum         maximum        set mean    spatial mean
-  --------------------  --------------  --------------  --------------  --------------
v  Boundary temperature          293.15          345.92          325.79          327.63
v  Boundary Density              1.0208          1.2038           1.085          1.0781
v  Yplus                              0          0.3718       0.0017271      0.00074898


  ** Computed values on cells
     ------------------------

   criterion               minimum         maximum        set mean    spatial mean
   ---------------  --------------  --------------  --------------  --------------
   Courant/Fourier        0.028153         0.74621         0.21487         0.16623



===============================================================



                 FINAL STAGE OF THE CALCULATION
                 ==============================


 ===========================================================





 ===========================================================




                      END OF CALCULATION
                      ==================


===============================================================

 Destroying structures and ending computation
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: Restart error-Invalid (not a number) values detected for a field

Post by Yvan Fournier »

Hello,

Checking you log snippets again, it seems in the example you posted, the issue is not just the boundary density: both the volume density and turbulent viscosity at cells seem yo have "infinite" values. Do you have this case even when using the auxiliary restart file ?

If this is the case, there may be an issue with that file, though I do not understand why.
Just in case, could you also check if you still have the issue when you deactivate parallel IO for your case, in case there is an I/O issue there (in the GUI, under the last "performance settings" section, "Input/Output", set read and write methods to "Standard IO/serial").

If your checkpoint/auxiliairy.csc file is not too large, you can also send it to for for verification.

Best regards,

Yvan
Ruonan
Posts: 136
Joined: Mon Dec 14, 2020 11:38 am

Re: Restart error-Invalid (not a number) values detected for a field

Post by Ruonan »

Hello Yvan,

Thank you very much, your help is much appreciated.

Unfortunately, I tried both "read auxiliary restart file" and "standard I/O serial", but the error was still there. I still can't restart.

But interestingly, when I remove the cs_user_physical_properties.f90, the case can restart with no error. So I guess there is something wrong with this density variation subroutine? I noticed this because in my log snippets, I can see "restart/auxiliary.csc: section "ref_presstot01" not present ...... It was not possible to read some values from the auxiliary restart file.". So I guess it's the variables used to calculate the density, i.e. pressure or temperature, that were not loaded properly?

Could you please kindly help me to check this subroutine (attached), or something other things related to density? In my case, the density variation is important, so I have to use this subroutine.

Best regards,
Ruonan

(The checkpoint folder and the failed restart folder are also attached)
Attachments
failed restart files from checkpoint.rar
(1.81 MiB) Downloaded 82 times
checkpoint of the previous calculation.rar
(9.39 MiB) Downloaded 74 times
cs_user_physical_properties.f90
(5.67 KiB) Downloaded 81 times
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: Restart error-Invalid (not a number) values detected for a field

Post by Yvan Fournier »

Hello,

This probably means the total pressure (prestot) is not yet computed at the first time step after restart.
Since the density from the previous time step is saved in the auxiliary restart file, you could add something like

Code: Select all

if (ntmabs .eq. ntpabs+1) return
at the beginning of cs_user_physical_properties.f90, so as to fall back to this density value when restarting.

Another solution(allowing computation at the current time step) is to force handling of the total pressure in restart files adding :

Code: Select all

const int restart_file_key_id = cs_field_key_id("restart_file");
 cs_field_set_key_int(cs_field_by_name("total_pressure"), restart_file_key_id_id, CS_RESTART_AUXILIARY);
You can also use CS_RESTART_MAIN instead of CS_RESTART_AUXILIARY if you prefer.

Best regards,

Yvan
Ruonan
Posts: 136
Joined: Mon Dec 14, 2020 11:38 am

Re: Restart error-Invalid (not a number) values detected for a field

Post by Ruonan »

Hello Yvan,

Thanks a lot for your suggestions. But sorry that I still can't get it work.

I tried the first method you suggested. The compilation was ok, but there seems no change. The case still can't restart, with the same error (infinite density) as before.

I then tried your second method, adding the two lines into cs_user_physical_properties.f90. I think this is a good idea, but when I compiled the subroutine, an error "Unclassifiable statement" occurred. I have no idea how to solve it.

Could you help me to check? My cannot-compiled subroutine is attached. Thank you!

Best regards,
Ruonan

(Hello Yvan, I checked the restart log snippets in my first post again, in the first timestep after restarting, I found the pressure values were ok, but the minimum temperature values (both domain and boundary values) went to 0, which may result in the infinite density values. So should we force the temperature value instead of the total pressure value?)
Attachments
cs_user_physical_properties.f90
(5.84 KiB) Downloaded 78 times
Post Reply