Page 1 of 2
[SOLVED] Problème calcul with cs_user_extra_operation
Posted: Mon Sep 08, 2014 10:04 am
by pleblanc86
Hello,
I am trying to code a stop criteria in cs_user_extra_operation subroutine. When the calculation is performed on a single processor, there is not problem and does it works !
Code: Select all
if(ippmod(ielion).ge.1) then
!Calcul du résidu courant
if (ntcabs.eq.1) then
allocate(residu(ntmabs))
call getcel('cell3',nlelt,lstelt)
do ilelt = 1, nlelt
icell = lstelt(ilelt)
!Calcul du résidu au premier pas de temps
residu(ntcabs) = (propce(icell,ipproc(iqelec))) - 0d0)/propce(icell,ipproc(iqelec))
rhoa = propce(icell,ipproc(iqelec))
enddo
elseif(ntcabs.gt.1) then
call getcel('cell3',nlelt,lstelt)
do ilelt = 1, nlelt
icell = lstelt(ilelt)
!Calcul du résidu durant le pas de temps courant
residu(ntcabs) = (propce(icell,ipproc(iqelec))) - rhoa)/propce(icell,ipproc(iqelec))
rhoa = propce(icell,ipproc(iqelec))
enddo
endif
! Gestion de l'arrêt du calcul
! Premier cas -- Le critère ne converge pas, on arrive donc à ntcabs = ntmabs
if (ntcabs.eq.ntmabs) then
impout = impusr(1)
open(impout,file="residu.dat")
do i = 1, ntcabs
write(impout,*) residu(i)
enddo
close(impout)
deallocate(residu)
endif
if(residu(ntcabs).gt.critere.and.ntcabs.gt.10) then
impout = impusr(1)
open(impout,file="residu.dat")
do i = 1, ntcabs
write(impout,*) residu(i)
enddo
close(impout)
deallocate(residu)
!Sachant que le calcul n'est pas arrivée à la valeur ntmabs
!Alors on provoque l'arrêt du calcul car il a atteint le critère
!de convergence
call csexit(0)
endif
endif
when we used several processor (for example 4), the computation is wrong ! I think it comes from the parallelization but I try several things without success.
Example :
if (irangp.ge.0.or.iperio.eq.1)
with function
then call synsca(...)
What is the way to run the script on several processor ? That's possible ?
Thanks for your help
Best regards
Paul Leblanc
Re: Problème calcul with cs_user_extra_operation
Posted: Mon Sep 08, 2014 2:41 pm
by Alexandra MartinS
Hello Paul,
It should be possible to run your code in several processors. However, what do you mean by "the computation is wrong"? I am guessing that your computation using several processors doesn't stop at the same point than when you use only one, and that you don't get the same residuals in both cases.
If so, you probably need to share information between the different processors. When you do
each processor selects different cells, and computes a different
value. In order to take into account all the cells at once, you need to perform a parallel sum, before comparing the residual to your criteria. Something like:
Code: Select all
cs_parall_sum(1, CS_DOUBLE, &residu(ntcabs));
After this, I would also make sure that it is the first processor the one that checks the criteria and writes the output:
Code: Select all
if(irangp.le.0.and.residu(ntcabs).gt.critere.and.ntcabs.gt.10) then
Hopefully this will help.
Cheers,
Alexandra
Re: Problème calcul with cs_user_extra_operation
Posted: Tue Sep 09, 2014 10:44 am
by pleblanc86
Hello Alexandra,
Thank you for your reply and you have quite understood my problem.
After your reply, I used the following command and the problem "improves"
Code: Select all
if(irangp.ge.0) then
call parsom(residu(ntcabs))
endif
I get for the residu (with 4 procs)
Code: Select all
[color=#00BF00] RESIDU : 4.41657690329410148E-004 (first time step)
RESIDU : 9.01763194227959974E-004
RESIDU : 1.36672258349849964E-003
RESIDU : 1.83751071852680781E-003[/color]
RESIDU : 0.25899514801465229
RESIDU : 0.25947211804845538
RESIDU : 0.25995225001166633
RESIDU : 0.26043545867728801
RESIDU : 0.26092167934237520
RESIDU : 0.26141086048555617
RESIDU : 0.26190296132051560
...
without this line, I obtain this values :
Code: Select all
RESIDU : 6.93134954333973094E-310 (first time step)
RESIDU : 6.93134954333973094E-310
RESIDU : 0.0000000000000000
RESIDU : 0.0000000000000000
RESIDU : 0.25668390119384454
RESIDU : 0.25668390119384454
RESIDU : 0.25668390119384454
RESIDU : 0.25668390119384454
RESIDU : 0.25668390119384454
RESIDU : 0.25668390119384454
...
Now, with 1 procs, the good results !!
Code: Select all
[color=#00BF00] 4.41657690329410148E-004(first time step)
9.01763194227959974E-004
1.36672258349849964E-003
1.83751071852680781E-003[/color]
2.31124682080768298E-003
2.78821685461084524E-003
3.26834881782159898E-003
3.75155748344300436E-003
4.23777814853046417E-003
...
We can see that when we call the "parsom function" (with 4 procs), the first 4 results of residu is good and are equal with the 1 procs case. But the rest is false.
I continued to search the solution but if someone has an idea, I'm interested !
Thank you
Best regards
Paul
Re: Problème calcul with cs_user_extra_operation
Posted: Tue Sep 09, 2014 11:54 am
by Yvan Fournier
Hello,
You residual does not seem to diminish monotonously, even on 1 processor, so I am not sure in what sense it is a stopping criteria (it may need more adjustment, or work for some flows, not others).
I am not convinced that the result on 1 proc is more correct than the one on 4 (in the case with parsom, of course) : after a few time steps, and especially if you mesh is large and or the flow field strongly varying), the truncation error differences due to summing thins in a different order can explain the differences. So if the values are different but the behavior is the same, you may not have an error. The smaller the mesh, the smaller (and later) this "natural" difference should appear, so f you do have differences on a very small mesh, you still have a bug. But on a large mesh, this behavior may be perfectly normal (you could have the same issue on 1 proc with a different mesh numbering).
Regards,
Yvan
Re: Problème calcul with cs_user_extra_operation
Posted: Tue Sep 09, 2014 1:06 pm
by pleblanc86
Hello Yvan,
Sorry, but for debug, I do not use the residu criteria itself. In this case, it's just a scalar which is calculated in the conduction module. And I known this values for each time step.
Between two run with 1 procs and multi procs the result is not the same and the problem is here for me.
Regards,
Paul
Re: Problème calcul with cs_user_extra_operation
Posted: Tue Sep 09, 2014 2:05 pm
by pleblanc86
Hello,
More precisions,
We have developed a conduction module for simulating the effect of electrical double layer. The conduction module has been modified and does it works.
To stop the code, we had decided to use the cs_user_extra_operations script. The reside is calculated by the value of scalar give by module and so I know this value. In the script (in the topic), residu(ntcabs) = propce(icell,ipproc(iqelec)) simply. How do you do to obtain the same value regardless of numbers of processor.
Do you understand my needs ?
Regards,
Paul
Re: Problème calcul with cs_user_extra_operation
Posted: Tue Sep 09, 2014 2:57 pm
by Yvan Fournier
Hello,
Yes, I understand your need, but if your mesh is large, there may be no way of guaranteeing that the result is exactly the same in parallel not. To be valid, your criteria should be "larger" than the "noise" you may have due to parallelism, or if it is not possible smoothed somehow (for example over multiple time steps).
To test if you differences are due to a bug in parallelism or are related to the "natural" truncation error, the simples solution is to run the same case on 1 processor, using a different mesh numbering. Depending on the code version or the tool you used for meshing, this may be easy or not.
In any case, I noticed a probable big error in your code , which may be a simpler explanation :
you use rhoa in your expressions, but rhoa is defined as a scalar, not a vector. I assume you defined rhoa with a save attribute, but it should be a vector, not a scalar. Otherwise, each cell uses the rhoa value from the previous cell, which is absolutely numbering-dependent. If propce(icell,ipproc(iqelec)) is constant in space, you code is OK. buf it the property (electric charge ?) is variable in space, your code is wrong.
Regards,
Yvan
Re: Problème calcul with cs_user_extra_operation
Posted: Tue Sep 09, 2014 6:11 pm
by pleblanc86
Dear Yvan,
Thank you for your reply. I just calculated two test cases. First one with 1 procs and second one with 4 procs. I am focused on the value of iqelec (particular physical) which in our module is space charge density. With Paraview (opening RESULT.case), I plotted the "space charge density" ( propce(icdll,ipproc(iqelec)) ) at center of only one cell as function of time. In both cases, I obtained the same result and for me it's ok. Regardless of numbers of processor the result is the same.
Now, if I want set up a criteria as function of space charge density ( propce(icdll,ipproc(iqelec)) ), I am using a user script -> cs_user_extra_operations.f90 .
Before setting up my criteria, I did a simple test script :
Code: Select all
if (ntcabs.eq.1) then
allocate(residu(ntmabs))
endif
call getcel('cell3',nlelt,lstelt) !cell3 is color of only on cell
do ilelt = 1, nlelt
icell = lstelt(ilelt)
residu(ntcabs) = (propce(icell,ipproc(iqelec))) !- 0d0)/propce(icell,ipproc(iqelec))
enddo
if(irangp.ge.0) then
call parsom(residu(ntcabs))
endif
if (ntcabs.eq.ntmabs.and.irangp.le.0) then
impout = impusr(1)
open(impout,file="residu.dat")
do i = 1, ntcabs
write(impout,*) residu(i)
enddo
close(impout)
deallocate(residu)
endif
In case of 1 procs, the residue.dat is exactly equal to the "iqelec" in RESULT.case, so that's wonderful
In case of 4 prods, the result is not the same between residu.dat and "iqelec".
I recall that in the both cases the RESULT.case give the same result.
So, I don't understand why when I want to use the cs_user_extra_operation.f90 and extract simply the value of iqelec (propce(icell,ipproc(iqelec))) in residu(ntcabs) and I don't obtain the same result in function of procs whereas the RESULT.case give exactly the same result.
I don't understand ! We can see that the iqelec value in RESULT.case (scalar compute in module) is the same regardless of number of procs. Why when I used the user script it's not the case.
Thank for your help
Regard
Paula
Re: Problème calcul with cs_user_extra_operation
Posted: Tue Sep 09, 2014 8:47 pm
by Yvan Fournier
Hello,
If you selection is the color of only one cell, you will have no selection on one processor.
But you still add the residue for that processor. How about initializing it to zero first, so you do not have a random value for the processor which does not go into the loop on selected cells ?
Regards,
Yvan
Re: Problème calcul with cs_user_extra_operation
Posted: Wed Sep 10, 2014 8:29 am
by pleblanc86
Hello,
I have initialise the value of residue to zero before calling the color of cell during the first time step. So, I have it :
Code: Select all
if (ntcabs.eq.0) then
do i = 1, ntmabs
residu(i) = 0.d0
enddo
endif
When I used only one procs, there is no problem. It's ok. But when I used 4 procs, I only get zero in residue.dat file. You're right, it does not select the cell with the color when we use several processor.
Is there a solution ?
Regards,
Paul