Convergence issues when running in parallel

[Nicolas Merigoux]

Dear all,
 
I would like to use Code_Saturne in parallel mode and so I performed a few tests without being successful yet.

You will find attached the same case which I ran on a single core and then on 12 cores (on the same workstation). This case is just a simple atmospheric dispersion case (with a block structured mesh), and I just tried to correctly set the wind flow for the moment. When ran on a single processor, I get some good results (with a max velocity of 6.9 m/s). But when ran on 12 cores, I got some strange results (with a max velocity of about 24 m/s instead of 6.9 m/s) and some really weird pressure, velocity, temperature.... distributions across the domain. I think that I have some convergence issues in parallel mode as it's not running faster on 12 cores than on a single core.

You will see in the attached files that I strictly use the same subroutines for both single and parallel cases. I think I might be doing something wrong as I saw some specifications about the use of user subroutines in parallel mode in the user guide. But I can't figure out what as I'm not very familiar with parallel calculations.
 
I'm using Code_Saturne V2.0.1 with the default partitioner (I did not compiled metis nor scotch). Perhaps this could help ? I also visualized the domain partitions in Paraview and saw some strange decompositions (I think the domain is automatically split in equal cell numbers). I'm not sure that such a decomposition will help to solve my convergence issues (it's just a suggestion) ? But I can't figure out how to manually split the domain (just to check if that can be a solution to my problem).
 
Anyway, I'm waiting for some suggestions. Thank you in advance for your help.

Regards.
 
Nicolas MERIGOUX

[Mickael Hassanaly]

Hello Nicolas,
could you send me your meshes, please?
Mickaël

[Nicolas Merigoux]

Hello Mickael,
My mesh is a very big one actually so I can't post it here... sorry.
But I don't think my problem is mesh related. Indeed, it works fine on a single core and the mesh is fully structured.  So I don't think that this should be difficult to handle for the partitionner ?
 
Nicolas.

[Yvan Fournier]

Hello,
In the last loop of your usiniv.f90 file, you use zent, but zent should be recomputed for each cell.
In your case, you assign the last computed value of zent to the temperature for all cells of the local processor. This value will be different for each processor, and is mesh-numbering dependent, so in any case, this is probably not what you wanted.
I do not know if this explains your convergence issues, but it could.
Best regards,
  Yvan

[Nicolas Merigoux]

Hello,

Thank you Yvan for this remark. That was a mistake I didn't noticed yet. As a consequence, I put the line with the temperature definition in the first loop on cells (just after the wind velocity definition), deleted this last loop and ran the case again.

Unfortunately I still got the same weird velocity distribution.

You'll find attached a picture of what I'm trying to simulate. This is the wind profile after 20 s (for the calculation on a single core and on 12 cores). The profile is not totally initialized yet but it gives a good idea of what I should obtain. I also put a slice of the domain decomposition for the parallel run.
 

Anyway, thank you for this first advice. I hope my pictures will help finding a solution.
 
Regards,

Nicolas

[Matthieu Guillaud]

Hello,
 
it's hard to find the problem, as Mickael said can you send your mesh if it's not confidential. With it we'll be able to run your calculation and it'll be easier to find the problem,... I hope.
Can you upload it on a website like filefront?
 
And also, can you send us the files result of the 12 core calculation after you made the correction on usiniv.f90 please?
 
 
Regards.
Matthieu

[Nicolas Merigoux]

Hello,

You'll find attached the result files of the 12 cores calculation after the correction made in usiniv.f90.

I'am sending the mesh and my whole setup for this file to the support mail address (saturne-support@edf.fr).

Thanks in advance for your answers.
 
Regards,

Nicolas

[Matthieu Guillaud]

Hello Nicolas,
 
we found what was wrong.
In your usclim.f90 file you overwrite ro0(iphas) in CALL GETFBR('15 and X >= 0.5 and X <= 0.5 and Z >=1.0 and Z <=2.0'...)
but this value is different from the one in your GUI. So it's the problem because ro0 is used in the calculation of the gravity term.
On a parallel calculation you had two values of ro0 on different processor.
You can fix the value of ro0 on the GUI equal to the one in your usclim.f90 or if you want a different one in usclim.f90, you have to use another name for ro0.
 
 
Best regards.
 
Matthieu

[Nicolas Merigoux]

Hello,
Thanks a lot for the support.
What I actually want is to calculate a kinetic and dissipation energy at the gas inlet using keendb. So I think I'll just rename ro0 (and also viscl0) in my call to keendb.
Thanks again.
 
Best regards,
Nicolas.