Passing values around in parallel / which subroutine to call

[James McNaughton]

I wanted to output the values at boundary face cells to check one of my subroutines and got some strange results which led me to notice I wasn't paying attention to parallel. i.e. I am only listing the values of the boundary face in one (which I assume to be rank 0) part of the parallel split.

I calculate the total force on a boundary face in usproj as in the 3rd example and can run this in parallel using the parsom subrountine. But to list the values at each point is less simple.

For instance just to list the boundary face coordinates I tried using parbcr to broadcast the x,y,z values (my code's below) to the other ranks but it crashes, the terminal reads:

*** An error occurred in MPI_Bcast *** on communicator MPI_COMM_WORLD *** MPI_ERR_TRUNCATE: message truncated *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)

And the listing file stops after "CPU time for restart files" but there is no other output.

If I change  call parbcr(irangp,ndim,xyz) to call parbcr(irangv,ndim,xyz) the code runs without error but with the same result as if I run under one processor.

I think I am slightly missunderstanding the way in which the par*** subroutines are intended to be used here, if there's any point in the right direction it'd be great.

Thanks,

James

Code: Select all

 testfile = impusr(1) irangv = 0
 if (ntcabs.eq.ntmabs) then
   if (irangp.le.0) open(file='test.dat',unit=testfile)
   call getfbr('inner', nlelt, lstelt)
   !==========
   do ilelt = 1, nlelt
     ifac = lstelt(ilelt)
     do ii = 1, ndim
       xyz(ii) = cdgfbo(ii,ifac)
     enddo
     if (irangp.ge.0) then
       call parbcr(irangp,ndim,xyz)
     endif
     if (irangp.le.0) then
       write(testfile,'(3(F10.6,1X))')xyz(1),xyz(2),xyz(3)
     endif
   enddo
   if (irangp.le.0) close(unit=testfile)
 endif

[Yvan Fournier]

Hello,

PARBCR is a collective call, so it must be called simultaneously by all procs (procs with rank != irangp send, rank irangp receives). As in your case, each processor loops on its own list of faces, the loop sizes will almost always be different from one processor to another, so your approach won't work.

What you would need would be to use the PARAGV call, which is basically a wrapper to MPI_Gatherv (I am not sure if there are examples in USPROJ, but basically, you would use something like:

Code: Select all

double precision, dimension(3,: ), allocatable :: loc_face_centers, glob_face_centers 

call getfbr('inner', nlelt, lstelt)  
!========== 

n_glob_faces = nlelt
call parcpt(n_glob_faces)

allocate(loc_face_centers(3, nlelt))
allocate(glob_face_centers(3, n_glob_faces))

do ielt = 1, nlelt
  ifac = lstelt(ielt)
  do ii = 1, 3
    loc_face_centers(ii, ielt) = cdgfbo(ii, ifac)
  enddo
enddo

call paragv(nlelt, n_glob_faces, loc_face_centers, glob_face_centers)

Best regards,

  Yvan

[James McNaughton]

Thanks for your reply and sorry for my delay in testing it properly, I had some difficulties and found running the final timestep on one processor an easy solution to my problem. I would still like to be able to solve my initial problem though..

I have looked at the two parallel calls you suggest and (correct me if I'm wrong)
parcpt - a global sum of the local counters.
paragv - makes a global array out of local arrays.

So from the example provided I assume after the paragv call there exists an array of length n_glob_faces with the cell centres of x,y,z in them (in each rank?) and then I could (in rank 0 only) loop through this and print out the values. The code I use is below, where I print out "loc_face_centers" the result has a few correct values but the rest are wrong. If instead I print out "glob_face_centers" the correct values still appear but the rest of the output is zeros.

I should have 50 data points which I do get at least. In cs_parall the global array variable is IN only which is also confusing me.
 
Am I printing out in the right way, and should I be looking to print out the loc or glob faces?
 
Thanks a lot,
 
James

Code: Select all

 if (irangp.le.0) then
  open(file='test.dat',unit=file_test)
! Write headings to the dat file
! CX, CY, CZ are the x,y,z coefficients
  write(file_test,'(2(A,1X))')  &
   '      x     ',              &
   '      y     '              
endif
! 
!
!!!!!!    
! work on the upper surface
  call getfbr('wall',nlelt,lstelt)
! ===========

  n_glob_faces = nlelt
if (irangp.ge.0) then
  call parcpt(n_glob_faces)
  allocate(loc_face_centers(3, nlelt))
  allocate(glob_face_centers(3, n_glob_faces))
endif

! loop over the boundary faces we called up      
  do ilelt = 1, nlelt
        
! tmp variable IFAC is face we are on
    ifac  = lstelt(ilelt)

    do ii=1,ndim
      loc_face_centers(ii,ilelt) = cdgfbo(ii,ifac)
    enddo
! out the face loop
  enddo

if (irangp.ge.0) then
  call paragv(nlelt, n_glob_faces, loc_face_centers, glob_face_centers)
endif
if (irangp.le.0) then
  do ilelt = 1,n_glob_faces
    write(file_test,'(2(E12.5,1X))') &
     loc_face_centers(1,ilelt),loc_face_centers(2,ilelt)
  enddo
  ! close the file
  close(unit=file_test)
endif

[Yvan Fournier]

Hello,

After using paragv, loc_face_centers should be unchanged, but is as before usually incomplete on a given processor. The results of paragv's assembly is glob_face_centers, so that is what you want to print

But it seems I forgot in my previous post that we are using a 2D array, while paragv has no explicit provision for this. As it just appends data together, there is no issue if the array is interleaved (such as here), but the sizes passed to it must be multiplied by 3 in this case:

Code: Select all

nl_faces_3 = 3*nlelt
ng_faces_3 = 3*n_glob_faces

if (irangp.ge.0) then
  call paragv(nlelt, n_glob_faces, loc_face_centers, glob_face_centers)
endif
if (irangp.le.0) then
  do ilelt = 1,n_glob_faces
    write(file_test,'(2(E12.5,1X))') &
     glob_face_centers(1,ilelt),glob_face_centers(2,ilelt)
  enddo
  ! close the file
  close(unit=file_test)
endif 

Sorry for the mistake in the previous post. This version should work better...

Best regards,

  Yvan

[James McNaughton]

Hi Yvan,

Thanks for the quick reply, it's working well now,

Cheers,

James