MPI-GATHER

[Nazir Al Sayed]

Hello,

  I calculated in a scalar ustsns.F, i want the processor 0 collects the local values of other processors. I need the subroutine  MPI-GATHER  that i can not find in  cs_parall.c Could you help me to  solve my problem ? Thank you in advance.

Nazir AL SAYED.

[Yvan Fournier]

Hello,
 
I am not sure I understand your question, but since you found and checked cs_parall.c, I assume you simply need an appropriate wrapper function so as to be able to use MPI_Gather from ustsns.F.
 
To do this you could copy cs_parall.c in with your user source files (next to ustsnf.F), and modify it so as to add an appropriate function (you could also copy cs_parall.h, but since Fortran does not use C headers, it would only avoid a compiler warning, not really change anything.
 
Another solution would be to use the PARAGV function from cs_parall.c, which wraps MPI_Allgatherv (so it is slightly more costly but also more general than MPI_Gather).
 
Also, I assume that the scalar you computed is a single value or vector on each processor. If you need to gather a cell-based or face-based variable, using PARAGV is definitely necessary, as the number of values per processor may vary.
 
Best regards,
 
  Yvan Fournier

[Nazir Al Sayed]

Hello
 P lease find attached file USTSNS.F. In this file i  measure the volumetric flow DEBVOL in a section in the canal, then i calculate the  source term depending on the volumetric flow  at two different  times n and  n -1 (DEBDOL_ANT) My problem is that the volumetric flow  DEBVOL  in the parallelized case is different than the rate in the non-parallelized case !!!.
Could you help me to  solve my problem ? Thank you in advance.

Best regards

Nazir AL SAYED

[Yvan Fournier]

Hello,
I just looked at your code, and did not notice anything suspicious. To better understand your problem, it would be interesting to print DEBVOL just after your call to PARSOM(DEBVOL), just to make sure it is not modified later (it soes not seem to be). Also, using an integer counter of faces participating in the call to debvol, summing it across processors using PARCPT, and printing that would help to determine if you do not detect the same faces or if they simply have different values of PROPFB when running in parallel. I suspect the second case, as your geometric test (the first case) seems too simple for a bug to hide in it.
Is DEBVOL significantly different in the parallel case from the serial case ? If it is only slightly different, it could be simply due to different rounding errors in parallel and serial cases (as (a+b)+c != a+(b+c) in floating point, and sums may be grouped in a slightly different order in parallel, this may be enough to explain small differences.
We have also observed the case where in parallel, vortex shedding behind a cylinder would be slightly shifted in time in a serial or parallel run (due to the same rounding issues), so results appeared different at first but were identical if averaged or observed over time. If your case is highly unsteady, this could be the explanation for your different results (averaging over several time steps would make the comparison less strict).
If your case is steady or only slightly unsteady, and DEBVOL very different, I have no good explanation, so your problem may be due to a bug, either (hopefully) in another of your user subroutines, or in the code itself. In that case, posting the rest of your user subroutines would help, and if your mesh is small enough and non-confidential, it would be useful also.
Best regards,
  Yvan Fournier

[David Monfort]

Hello Nazir,

A possible issue would be a "double" count of some faces. Indeed, when the calculation is parallel, we split the domain in several parts but the frontier faces remain interior faces with two neighbours (one "real" cell and one "ghost" cell). This means that you could count the same face twice if ever the face where you sum the mass flux is just at a frontier between two domains.

To by-pass this difficulty, just check (for example) that the first neighbour is a "real" cell

Code: Select all

iflmas = ipprof(ifluma(iu(1))
debvol = 0.d0 

do ifac = 1, nfac
  if (cdgfac(1,ifac).le.XXX .and. cdgfac(1,ifac).ge.YYY) then
    if (ifacel(1,fac).le.ncel) then
      debvol = debvol + propfa(ifac,iflmas)
    endif
  endif
enddo

call parsom(debvol)

It should be enough ;-)

Let us know if it works.

David