Values of source terms depend on the number of MPI processes!

[Mohammad]

Hello,

I just added a source term by using cs_user_source_terms.f90 file.

By adding the source term, the simulation diverged, so I decided to create three test fields in order to check the values of the source terms. These fields are added as property on mesh cells and named: CVAR1, CVAR2, CVAR3.
The source term code is as below (as mentioned, the CVAR*(iel) are added to check and visualize the values of source terms):

Code: Select all

double precision, dimension(:), pointer :: FSGS_1, FSGS_2, FSGS_3
double precision, dimension(:), pointer :: CVAR1, CVAR2, CVAR3
   call field_get_val_s_by_name('FSGS_1',FSGS_1)
   call field_get_val_s_by_name('FSGS_2',FSGS_2)
   call field_get_val_s_by_name('FSGS_3',FSGS_3)
   call field_get_val_s_by_name('CVAR1',CVAR1)
   call field_get_val_s_by_name('CVAR2',CVAR2)
   call field_get_val_s_by_name('CVAR3',CVAR3)

do iel = 1, ncel
      crvexp(1,iel) =  -volume(iel)*1.0*FSGS_1(iel)
      crvexp(2,iel) =  -volume(iel)*1.0*FSGS_2(iel)
      crvexp(3,iel) =  -volume(iel)*1.0*FSGS_3(iel)
      
      CVAR1(iel) = crvexp(1,iel)
      CVAR2(iel) = crvexp(2,iel)
      CVAR3(iel) = crvexp(3,iel)
enddo

FSGS_* are another fields which their values are calculated in another file. They are exactly defined in the same way as CVAR* s are defined.

After running the code on different number of MPI processes, I noticed the value of all fields remain unchanged, except the values of CVAR* fields which are exactly equal to the source terms that will be added to the N-S equations.
The maximum and minimum values increase by decreasing the number of MPI processes and vice versa! Also the magnitude of set means are about 100 times different.
For example with 8 MPI processes I get these values in the same time step:

field minimum maximum set mean spatial mean
- -------------- -------------- -------------- -------------- --------------
v FSGS_1 -1.6935e+05 1.4164e+05 -74.069
v FSGS_2 -4.2097e+05 4.0849e+05 415.27
v FSGS_3 -51602 54065 0.066343

v CVAR1 -0.031304 0.0053675 -2.6873e-08
v CVAR2 -0.031246 0.0047309 -1.945e-08
v CVAR3 -0.0031412 0.0029376 2.914e-12

And with 4 MPI processes I get:

field minimum maximum set mean spatial mean
- -------------- -------------- -------------- -------------- --------------
v FSGS_1 -1.6935e+05 1.4155e+05 -73.509
v FSGS_2 -4.2097e+05 4.0852e+05 415.83
v FSGS_3 -51602 54065 0.066343

v CVAR1 -0.066153 0.030618 -1.1741e-06
v CVAR2 -0.090241 0.034037 -1.1141e-06
v CVAR3 -0.0031412 0.0029376 2.914e-12

You can obviously see the huge difference between the values of CVARs except CVAR3 which is the z component of source term. The other fields like, FSGS_* don't have any difference!
It's really confusing and I think it's the cause of divergence.


Regards,
Mohammad

[Yvan Fournier]

Hello,

Could you post the file in which the FSGS_* values are computed ?

Regards,

Yvan

[Mohammad]

Hello,

Ofcourse, I edited viswal.f90 and it is calculated there at the last lines of the file:

I also attach source term files and field files:

cs_user_source_terms.f90

(7.4 KiB) Downloaded 185 times

cs_user_parameters-fields.c

(5.63 KiB) Downloaded 187 times

Regards,
Mohammad

[Yvan Fournier]

Hello,

Did you test this under a debug build (which includes bounds checks) ?

In your modified viswal.f90, lines 335/336, you have grads(iel,2) and +grads(iel,1) instead of grads(2,iel) and +grads(1,iel)

That probably explains the issue, but running under a debug build may catch other similar issues I may have missed.

Also, when modifying a non-user subroutine, you need to be extra-careful, as updates in code releases (including bug-fix releases in a same series) can introduce incompatibilities. So this may be useful for research and testing, but is not recommended for production studies.

Regards,

Yvan

[Mohammad]

Hello,

Tahnk you very very much dear Yvan,
I really appreciate it.

Yes! The problem caused by that line!

No I didn't run the code in debug mode. How can I do that?


Kind Regards,
Mohammad

[Yvan Fournier]

Hello,

What I call "running in debug mode" actually refers to running a debug build (preferably e second install, configured using --enable-debug). The install guide explains this in more detail. Such builds do not cat all errors, but definitively help.

Best regards,

Yvan

[Mohammad]

Hello,

Thank you Yvan.
I'll consider that.

Regards,
Mohammad