[SOLVED] Problème calcul with cs_user_extra_operation

[pleblanc86]

Hello,

I am trying to code a stop criteria in cs_user_extra_operation subroutine. When the calculation is performed on a single processor, there is not problem and does it works !

Code: Select all

if(ippmod(ielion).ge.1) then

  !Calcul du résidu courant
  if (ntcabs.eq.1) then
      allocate(residu(ntmabs))
      call getcel('cell3',nlelt,lstelt)
      do ilelt = 1, nlelt
        icell = lstelt(ilelt)
        !Calcul du résidu au premier pas de temps
        residu(ntcabs) = (propce(icell,ipproc(iqelec))) - 0d0)/propce(icell,ipproc(iqelec))
        rhoa = propce(icell,ipproc(iqelec))
      enddo
  elseif(ntcabs.gt.1) then
      call getcel('cell3',nlelt,lstelt)
      do ilelt = 1, nlelt
        icell = lstelt(ilelt)
        !Calcul du résidu durant le pas de temps courant
        residu(ntcabs) = (propce(icell,ipproc(iqelec))) - rhoa)/propce(icell,ipproc(iqelec))
        rhoa = propce(icell,ipproc(iqelec))
      enddo
  endif


  ! Gestion de l'arrêt du calcul
  ! Premier cas -- Le critère ne converge pas, on arrive donc à ntcabs = ntmabs
  if (ntcabs.eq.ntmabs) then
    impout = impusr(1)
    open(impout,file="residu.dat")
    do i = 1, ntcabs
      write(impout,*) residu(i)
    enddo
    close(impout)
    deallocate(residu)
  endif

  if(residu(ntcabs).gt.critere.and.ntcabs.gt.10) then
    impout = impusr(1)
    open(impout,file="residu.dat")
    do i = 1, ntcabs
      write(impout,*) residu(i)
    enddo
    close(impout)
    deallocate(residu)
     !Sachant que le calcul n'est pas arrivée à la valeur ntmabs
     !Alors on provoque l'arrêt du calcul car il a atteint le critère
     !de convergence
     call csexit(0)
  endif
endif

when we used several processor (for example 4), the computation is wrong ! I think it comes from the parallelization but I try several things without success.

Example :

if (irangp.ge.0.or.iperio.eq.1)

with function

then call synsca(...)

What is the way to run the script on several processor ? That's possible ?

Thanks for your help
Best regards
Paul Leblanc

[Alexandra MartinS]

Hello Paul,

It should be possible to run your code in several processors. However, what do you mean by "the computation is wrong"? I am guessing that your computation using several processors doesn't stop at the same point than when you use only one, and that you don't get the same residuals in both cases.

If so, you probably need to share information between the different processors. When you do

Code: Select all

call getcel('cell3',nlelt,lstelt)

each processor selects different cells, and computes a different

Code: Select all

residu(ntcabs)

value. In order to take into account all the cells at once, you need to perform a parallel sum, before comparing the residual to your criteria. Something like:

Code: Select all

cs_parall_sum(1, CS_DOUBLE, &residu(ntcabs));

After this, I would also make sure that it is the first processor the one that checks the criteria and writes the output:

Code: Select all

if(irangp.le.0.and.residu(ntcabs).gt.critere.and.ntcabs.gt.10) then

Hopefully this will help.

Cheers,

Alexandra

[pleblanc86]

Hello Alexandra,

Thank you for your reply and you have quite understood my problem.

After your reply, I used the following command and the problem "improves"

Code: Select all

  if(irangp.ge.0) then
    call parsom(residu(ntcabs))
  endif

I get for the residu (with 4 procs)

Code: Select all

[color=#00BF00] RESIDU :   4.41657690329410148E-004 (first time step)
 RESIDU :   9.01763194227959974E-004
 RESIDU :   1.36672258349849964E-003
 RESIDU :   1.83751071852680781E-003[/color]
 RESIDU :   0.25899514801465229     
 RESIDU :   0.25947211804845538     
 RESIDU :   0.25995225001166633     
 RESIDU :   0.26043545867728801     
 RESIDU :   0.26092167934237520     
 RESIDU :   0.26141086048555617    
 RESIDU :   0.26190296132051560
...

without this line, I obtain this values :

Code: Select all

 RESIDU :   6.93134954333973094E-310 (first time step)
 RESIDU :   6.93134954333973094E-310
 RESIDU :    0.0000000000000000     
 RESIDU :    0.0000000000000000     
 RESIDU :   0.25668390119384454     
 RESIDU :   0.25668390119384454     
 RESIDU :   0.25668390119384454     
 RESIDU :   0.25668390119384454     
 RESIDU :   0.25668390119384454     
 RESIDU :   0.25668390119384454
...

Now, with 1 procs, the good results !!

Code: Select all

[color=#00BF00] 4.41657690329410148E-004(first time step)
  9.01763194227959974E-004
  1.36672258349849964E-003
  1.83751071852680781E-003[/color]
  2.31124682080768298E-003
  2.78821685461084524E-003
  3.26834881782159898E-003
  3.75155748344300436E-003
  4.23777814853046417E-003
...

We can see that when we call the "parsom function" (with 4 procs), the first 4 results of residu is good and are equal with the 1 procs case. But the rest is false.

I continued to search the solution but if someone has an idea, I'm interested !

Thank you
Best regards
Paul

[Yvan Fournier]

Hello,

You residual does not seem to diminish monotonously, even on 1 processor, so I am not sure in what sense it is a stopping criteria (it may need more adjustment, or work for some flows, not others).

I am not convinced that the result on 1 proc is more correct than the one on 4 (in the case with parsom, of course) : after a few time steps, and especially if you mesh is large and or the flow field strongly varying), the truncation error differences due to summing thins in a different order can explain the differences. So if the values are different but the behavior is the same, you may not have an error. The smaller the mesh, the smaller (and later) this "natural" difference should appear, so f you do have differences on a very small mesh, you still have a bug. But on a large mesh, this behavior may be perfectly normal (you could have the same issue on 1 proc with a different mesh numbering).

Regards,

Yvan

[pleblanc86]

Hello Yvan,

Sorry, but for debug, I do not use the residu criteria itself. In this case, it's just a scalar which is calculated in the conduction module. And I known this values for each time step.

Between two run with 1 procs and multi procs the result is not the same and the problem is here for me.

Regards,
Paul

[pleblanc86]

Hello,

More precisions,

We have developed a conduction module for simulating the effect of electrical double layer. The conduction module has been modified and does it works.

To stop the code, we had decided to use the cs_user_extra_operations script. The reside is calculated by the value of scalar give by module and so I know this value. In the script (in the topic), residu(ntcabs) = propce(icell,ipproc(iqelec)) simply. How do you do to obtain the same value regardless of numbers of processor.

Do you understand my needs ?

Regards,
Paul

[Yvan Fournier]

Hello,

Yes, I understand your need, but if your mesh is large, there may be no way of guaranteeing that the result is exactly the same in parallel not. To be valid, your criteria should be "larger" than the "noise" you may have due to parallelism, or if it is not possible smoothed somehow (for example over multiple time steps).

To test if you differences are due to a bug in parallelism or are related to the "natural" truncation error, the simples solution is to run the same case on 1 processor, using a different mesh numbering. Depending on the code version or the tool you used for meshing, this may be easy or not.

In any case, I noticed a probable big error in your code , which may be a simpler explanation :

you use rhoa in your expressions, but rhoa is defined as a scalar, not a vector. I assume you defined rhoa with a save attribute, but it should be a vector, not a scalar. Otherwise, each cell uses the rhoa value from the previous cell, which is absolutely numbering-dependent. If propce(icell,ipproc(iqelec)) is constant in space, you code is OK. buf it the property (electric charge ?) is variable in space, your code is wrong.

Regards,

Yvan

[pleblanc86]

Dear Yvan,

Thank you for your reply. I just calculated two test cases. First one with 1 procs and second one with 4 procs. I am focused on the value of iqelec (particular physical) which in our module is space charge density. With Paraview (opening RESULT.case), I plotted the "space charge density" ( propce(icdll,ipproc(iqelec)) ) at center of only one cell as function of time. In both cases, I obtained the same result and for me it's ok. Regardless of numbers of processor the result is the same.

Now, if I want set up a criteria as function of space charge density ( propce(icdll,ipproc(iqelec)) ), I am using a user script -> cs_user_extra_operations.f90 .

Before setting up my criteria, I did a simple test script :

Code: Select all

  if (ntcabs.eq.1) then
      allocate(residu(ntmabs))
  endif

call getcel('cell3',nlelt,lstelt) !cell3 is color of only on cell
do ilelt = 1, nlelt
  icell = lstelt(ilelt)
  residu(ntcabs) = (propce(icell,ipproc(iqelec))) !- 0d0)/propce(icell,ipproc(iqelec))
enddo

if(irangp.ge.0) then
  call parsom(residu(ntcabs))
endif

if (ntcabs.eq.ntmabs.and.irangp.le.0) then
  impout = impusr(1)
  open(impout,file="residu.dat")
  do i = 1, ntcabs
    write(impout,*) residu(i)
  enddo
  close(impout)
  deallocate(residu)
endif

In case of 1 procs, the residue.dat is exactly equal to the "iqelec" in RESULT.case, so that's wonderful

In case of 4 prods, the result is not the same between residu.dat and "iqelec".
I recall that in the both cases the RESULT.case give the same result.

So, I don't understand why when I want to use the cs_user_extra_operation.f90 and extract simply the value of iqelec (propce(icell,ipproc(iqelec))) in residu(ntcabs) and I don't obtain the same result in function of procs whereas the RESULT.case give exactly the same result.

I don't understand ! We can see that the iqelec value in RESULT.case (scalar compute in module) is the same regardless of number of procs. Why when I used the user script it's not the case.

Thank for your help
Regard

Paula

[Yvan Fournier]

Hello,

If you selection is the color of only one cell, you will have no selection on one processor.

But you still add the residue for that processor. How about initializing it to zero first, so you do not have a random value for the processor which does not go into the loop on selected cells ?

Regards,

Yvan

[pleblanc86]

Hello,

I have initialise the value of residue to zero before calling the color of cell during the first time step. So, I have it :

Code: Select all

if (ntcabs.eq.0) then
  do i = 1, ntmabs
    residu(i) = 0.d0
  enddo
endif

When I used only one procs, there is no problem. It's ok. But when I used 4 procs, I only get zero in residue.dat file. You're right, it does not select the cell with the color when we use several processor.

Is there a solution ?

Regards,
Paul