Installation of code-saturne V2RC1 in parallel

[Serra Sylvain]

Hi,

I installed code-saturne on a cluster.

The first test I did was to run a calcul in batch and it was OK.

The same calcul using qsub on one proc is also OK.


My problem is when I submit the calcul on many proc. (eg : nodes=1:ppn=8)

I obtain the following error (:

------------------------------------------------------------------------------

Total number of processors: 8


Parallel Code_Saturne with partitioning in 8 sub-domains
Total number of processors: 8

Code_Saturne is running
***********************

Working directory (to be periodically cleaned) :
/home/sylvain.serra/tmp_Saturne/study.CASE.07191111

Kernel version: /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin
Preprocessor: /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin

********************************************
Preparing calculation
********************************************


********************************************
Starting calculation
********************************************

[chapo41.local:03137] pls:tm: failed to poll for a spawned proc, return status = 17002
[chapo41.local:03137] [0,0,0] ORTE_ERROR_LOG: In errno in file rmgr_urm.c at line 462
[chapo41.local:03137] mpiexec: spawn failed with errno=-11
Error running the calculation.

Check Kernel log (listing) and error* files for details


********************************************
Error in calculation stage.
********************************************


---------------------------------------------------------------------------------------


In the lispre.xxx and listpart.xxx files, it seems OK


.-----------------------.
| |
| Preprocessor finish |
| |
`-----------------------'


.------------------------.
| Partitioner finish |
`------------------------'


and there are neither error.xxx file nor listing file...

Anybody could help me?

Best regards,

Sylvain

[Yvan Fournier]

Hello,

The error message seems to be in the MPI execution start itself (I guess that you do not even obtain "listing*" or "error*" files at this stage; if you do obtain some of those files, what do they say ?).

It would be interesting to test your Open MPI installation itself (unless other codes already work in parallel on this cluster using the same MPI library). Problems might also come from your account configuration, for example if the spawning process requires automatic SSH login and your .ssh/* files are not configured for this. In this case, submitting a script with a PBS batch job header similar to Code_Saturne's and simply containing

mpiexec -n 8 /usr/bin/env

may be a good test: if it fails, you have a configuration or MPI installation issue. If it works the issue is in the Code_Saturne installation.

Finally, if you have more than one MPI library installed on the cluster, make sure that Code_Saturne is using the correct one: if you build with one library and the script tries to run with another, things will fail (but probably at a slightly later stage, during MPI_Init).

Best regards,

  Yvan

[Serra Sylvain]

Hello,

the installation always failed in parallel.

The OpenMPI works indeed, we can run a simulation using "mpiexec -n 8 /usr/bin/env" but I do not find where the installation of CS failed.

Here you could find the steps I follow for the installation:


Upgrading of qt4
rpm -ivh http://software.freivald.com/centos/sof ... noarch.rpm

yum update fontconfig fontconfig-devel qt4 qt4-devel qt4-doc qt4-postgresql qt4-odbc qt4-sqlite qt-creator

Installation of Salome_5.1.3
and SOURCE the environment of Salome: envSalome-V5_1_3.sh and prerequis-V5_1_3.sh

Installation of cgnslib_2.5.4
Cd /share/apps/code_saturne/2.0/librairies/cgnslib_2.5
mkdir include
mkdir lib
./configure --prefix=/share/apps/code_saturne/2.0/cgnslib_2.5
Make && make install

Installation of bft-1.1.2
cd /share/apps/code_saturne/2.0/librairies/bft-1.1.2
./configure --prefix=/share/apps/code_saturne/2.0/librairies/bft-1.1.2
Make && make install

Installation of fvm-0.15.0
Cd /share/apps/code_saturne/2.0/librairies/fvm-0.15.0
./configure --prefix=/share/apps/code_saturne/2.0/librairies/fvm-0.15.0 --with-hdf5=/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Hdf5-1.6.9 --with-med=/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Med-2.3.6 --with-mpi-include=/opt/openmpi/include --with-mpi-lib=/opt/openmpi/lib/
Make && make install

Installation of mei-1.0.1
cd /share/apps/code_saturne/2.0/librairies/mei-1.0.1
./configure –prefix=/share/apps/code_saturne/2.0/librairies/mei-1.0.1

Make && make install
 
Installation of ecs-2.0.0-rc1
Cd /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1
./configure –prefix=/share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1 --with-hdf5=/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Hdf5-1.6.9 --with-med=/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Med-2.3.6 --with-metis-include=/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Metis-4.0/Lib --with-metis-lib=/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Metis-4.0 --with-bft-lib=/share/apps/code_saturne/2.0/librairies/bft-1.1.2/lib
Make && make install

Installation of syrthes
In the directory : /share/apps/code_saturne/2.0/syrthes/syrthes3.4.2
Modification in bin/syrthes.profile :
PATHSYR=/share/apps/code_saturne/2.0/syrthes

Installation of code saturne
cd /share/apps/code_saturne/2.0/ncs-2.0.0-rc1
Creation of links
ln -s /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/PyQt-4.5.4_python2.4/bin/pylupdate4 /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Python-2.4.4/bin/pylupdate4
ln -s /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/PyQt-4.5.4_python2.4/bin/pyrcc4 /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Python-2.4.4/bin/pyrcc4
ln -s /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/PyQt-4.5.4_python2.4/bin/pyuic4 /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/Prerequis/Python-2.4.4/bin/pyuic4

Source the SALOME environnent
Source /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/ prerequis-V5_1_3.sh
Source /share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/envSalome-V5_1_3.sh

Then lunch the installation of code_saturne
./configure --prefix=/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin --with-mpi=/opt/openmpi --with-prepro=/share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1 --with-syrthes=/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2 LIBS=-lm --with-fvm=/share/apps/code_saturne/2.0/librairies/fvm-0.15.0 --with-mei=/share/apps/code_saturne/2.0/librairies/mei-1.0.1  --with-bft-lib=/share/apps/code_saturne/2.0/librairies/bft-1.1.2/lib
Make
Make install


Do you know what is wrong?

Thank you in advance

Best regards,

Sylvain

[Serra Sylvain]

Sorry for the head of my last message, I don't understand why...

[Yvan Fournier]

Hello,

I edited your message so as to delete the HTML junk at the top.

Do you obtain any error message in the batch result files (typically .out or .err files in the directory from which you submitted the job) indicating a problem with share llibraries not being found ? This could happen in certain cases, especially if your build is done on a different machine than the run, which may happen with clusters ?

Othewise, did you run your single mpiexec test in the same conditions as the code (i.e. both interactively or both as batch jobs ?)

The next step may be to test if Code_Saturne uses the correct mpiexec (in case more than one are installed):

Take your runcase script, then replace $RUN/$EXE wherever you find it by $RUN/env (this will be found at about 3/4 down, in the generation of a wrapper called localexec).

The run this modified script. It won't run the code, but will use the same mpiexec command and environment, so you'll see if you have the same error mpi spawn error or not.

Depending on these results, we'll know where to look...

Best regards,

  Yvan

[David Monfort]

Hello Sylvain,

Before testing Yvan's idea, could you test to run again a Code_Saturne simulation without sourcing / having sourced the SALOME environment (if you did so)? Unfortunately, sourcing it may pollute a lot of things in an environment... though this is needed to install Code_Saturne against SALOME pre-requisites. If that still doesn't, then try Yvan's proposals

David

[Serra Sylvain]

Hello,

- first, the answer for David is no, it did not works without sourcing salome.

The error is the following, in the listpre.X file :

/share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin/cs_preprocess: error while loading shared libraries: libhdf5.so.0: cannot open shared object file: No such file or directory

May be I can give the good way to find this librairie but I do not know how...

- second: the answers for Yvan:

I try to modifie the script and the error is the same for both  scripts:

/opt/torque/mom_priv/jobs/47327.chapo.interne.emd.SC: line 85: 4: command not found
Total number of processors: 4
 
Parallel Code_Saturne with partitioning in 4 sub-domains
Total number of processors: 4

                      Code_Saturne is running
                      ***********************

 Working directory (to be periodically cleaned) :
     /home/sylvain.serra/tmp_Saturne/study.CASE.09061523
 Kernel version:           /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin
 Preprocessor:             /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin
 ********************************************
            Preparing calculation
********************************************

********************************************
             Starting calculation
  ********************************************
[chapo38.local:08671] pls:tm: failed to poll for a spawned proc, return status = 17002
[chapo38.local:08671] [0,0,0] ORTE_ERROR_LOG: In errno in file rmgr_urm.c at line 462
[chapo38.local:08671] mpiexec: spawn failed with errno=-11
Error running the calculation.

Check Kernel log (listing) and error* files for details

  ********************************************
         Error in calculation stage.
  ********************************************

Have you got others ideas.

Do you think that it could be better to try to install conde-saturne without salome?

Thanks for all

Sylvain
 

[Yvan Fournier]

Hello,

It seems some command is not found in your batch script. I assume that the file:

/opt/torque/mom_priv/jobs/47327.chapo.interne.emd.SC

is your runcase script, copied to the batch system's spool, but the line numbers may not match those of the runcase (do you have a command line 85 ?)

Could you add "set -x" early in the runcase script and see what output thisprovids ?

Best regards,

  Yvan

[Serra Sylvain]

Hello,

it was a test I did and I forgot to replace it.

In fac, the line :

/opt/torque/mom_priv/jobs/47327.chapo.interne.emd.SC: line 85: 4: command not found

is not a matter.

the error is after :  

[chapo38.local:08671] pls:tm: failed to poll for a spawned proc, return status = 17002
[chapo38.local:08671] [0,0,0] ORTE_ERROR_LOG: In errno in file rmgr_urm.c at line 462
[chapo38.local:08671] mpiexec: spawn failed with errno=-11
Error running the calculation.

sorry

Sylvain

[Serra Sylvain]

Here is the file when I put "set -X" in the runcase:



+ SOLCOM=0
+ export TERM=xterm
+ TERM=xterm
+ STUDY=study
+ CASE=CASE
+ PARAM=chapo_mpi_canal_ke_1ms.xml
+ MESH=canal.msh
+ COMMAND_REORIENT=
+ COMMAND_JOIN=
+ COMMAND_CWF=
+ COMMAND_PERIO=
+ THERMOCHEMISTRY_DATA=
+ METEO_DATA=
+ NUMBER_OF_PROCESSORS=
+ PROCESSOR_LIST=
+ PARTITION_LIST=
+ USER_INPUT_FILES=
+ USER_OUTPUT_FILES='vitx003.dat vitx006.dat turb003.dat turb006.dat'
+ CS_TMP_PREFIX=
+ CS_LIB_ADD=
+ VALGRIND=
+ ARG_CS_VERIF=
+ ARG_CS_OUTPUT=
+ ADAPTATION=
+ summary=summary
+ CASEDIR=/home/sylvain.serra/saturne/study/CASE
+ DATA=/home/sylvain.serra/saturne/study/CASE/DATA
+ RESU=/home/sylvain.serra/saturne/study/CASE/RESU
+ SRC=/home/sylvain.serra/saturne/study/CASE/SRC
+ SCRIPTS=/home/sylvain.serra/saturne/study/CASE/SCRIPTS
+ RESTART_IN=/home/sylvain.serra/saturne/study/CASE/DATA/RESTART
+ PREPROCESSOR_OUTPUT_IN=/home/sylvain.serra/saturne/study/CASE/DATA/preprocessor_output
+ PARTITION_OUTPUT_IN=/home/sylvain.serra/saturne/study/CASE/DATA/PARTITION_OUTPUT
+ MESHDIR=/home/sylvain.serra/saturne/study/CASE/../MESH
+ DATA_SYR=/home/sylvain.serra/saturne/study/CASE/DATA_SYR
+ SRC_SYR=/home/sylvain.serra/saturne/study/CASE/SRC_SYR
+ SYRTHES_ENV=syrthes.env
+ COUPLING_MODE=MPI
+ ECHO_SYR_COMM=
+ EXEC_PREPROCESS=yes
+ EXEC_PARTITION=yes
+ EXEC_KERNEL=yes
+ prefix=/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin
+ exec_prefix=/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin
+ bindir=/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin/bin
+ datarootdir=/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin/share
+ datadir=/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin/share
+ pkgdatadir=/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin/share/ncs
+ ecs_bindir=/share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin
+ homard_prefix=
+ '[' '!' -z '' ']'
+ N_SYRTHES_COUPLINGS=0
+ '[' -f /home/sylvain.serra/saturne/study/CASE/SRC/ussyrc.f90 ']'
+ '[' '!' -z chapo_mpi_canal_ke_1ms.xml ']'
+ var=/home/sylvain.serra/saturne/study/CASE/DATA/chapo_mpi_canal_ke_1ms.xml
++ grep -c syrthes_app_num /home/sylvain.serra/saturne/study/CASE/DATA/chapo_mpi_canal_ke_1ms.xml
+ N_SYRTHES_COUPLINGS_GUI=0
+ (( N_SYRTHES_COUPLINGS = N_SYRTHES_COUPLINGS + N_SYRTHES_COUPLINGS_GUI ))
+ '[' 0 = 0 ']'
+ SYRTHES_COUPLING=no
+ '[' no = yes ']'
+ COUPLING_MODE=
+ THISSCRIPT=/opt/torque/mom_priv/jobs/47378.chapo.interne.emd.SC
++ whoami
+ USER=sylvain.serra
++ date +%m%d%H%M
+ DATE=09070906
+ SUFFIX=09070906
+ EXE=cs_solver
+ EXE_SYR=syrthes
+ cp /opt/torque/mom_priv/jobs/47378.chapo.interne.emd.SC /home/sylvain.serra/saturne/study/CASE/RESU/runcase.09070906
+ '[' '!' -z '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' -a '' '!=' /tmp ']'
+ RUN=/home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ '[' /home/sylvain.serra/tmp_Saturne/study.CASE.09070906 '!=' '' ']'
+ '[' '!' -d /home/sylvain.serra/tmp_Saturne/study.CASE.09070906 ']'
+ mkdir -p /home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ '[' '' = '--log 0' ']'
+ echo /home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ cd /home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ CS_MPI_PATH=/opt/openmpi/bin
+ . /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin/share/ncs/runcase_mpi_env
++ '[' -f /machines ']'
++ '[' '!' -z /opt/torque/aux//47378.chapo.interne.emd ']'
++ MPIHOSTS=/opt/torque/aux//47378.chapo.interne.emd
++ '[' -z '' ']'
++ '[' '!' -z '' ']'
++ '[' '!' -z '' ']'
++ '[' '!' -z '' ']'
++ '[' '!' -z /opt/torque/aux//47378.chapo.interne.emd ']'
+++ cat /opt/torque/aux//47378.chapo.interne.emd
+++ wc -l
++ NUMBER_OF_PROCESSORS=4
++ NUMBER_OF_NODES=1
++ '[' '!' -z /opt/torque/aux//47378.chapo.interne.emd ']'
+++ sort /opt/torque/aux//47378.chapo.interne.emd
+++ cut -f 1 '-d '
+++ uniq
+++ wc -l
+++ sed -e 's/:/ /'
++ NUMBER_OF_NODES=1
++ '[' '!' -z /opt/torque/aux//47378.chapo.interne.emd ']'
++ echo 'Total number of processors: 4'
Total number of processors: 4
++ '[' 4 -gt 1 ']'
+++ uname -s
++ NOM_ARCH=Linux
++ '[' Linux = Linux ']'
+++ domainname
++ '[' '(none)' = cluster-chatou ']'
++ '[' -d /bgl/BlueLight/ppcfloor ']'
++ '[' -d /bgsys/drivers/ppcfloor ']'
+++ uname -m
++ MACHINE=x86_64
++ case "$MACHINE" in
++ NOM_ARCH=Linux_x86_64
++ MPIBOOT=
++ MPIHALT=
++ MPIRUN='mpirun -np 4'
++ '[' -d /opt/openmpi/bin ']'
++ export PATH=/opt/openmpi/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin/Linux_x86_64:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/XDATA_V5_1_3/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Tepal:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Ghs3d-4.1:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Metis-4.0:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Scotch-4.0/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Vtk-5.0.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Sphinx-0.6.3_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Docutils-0.6_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Expat-2.0.1/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Libxml2-2.6.32_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Doxygen-1.6.1/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Graphviz-2.24/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Med-2.3.6/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Hdf5-1.6.9/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Swig-1.3.40/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/OmniORB-4.1.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/PyQt-4.5.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Sip-4.8.2_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Qt-4.5.2_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Python-2.4.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/etchForSalome/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/ParaView3.3-cvs/platforms/linux64Gcc/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/cmake-2.4.6/platforms/linux64/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/openmpi-1.2.6/platforms/linux64GccDPOpt/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/gcc-4.3.1/platforms/linux64/bin:/home/sylvain.serra/OpenFOAM/sylvain.serra-1.5/applications/bin/linux64GccDPOpt:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/applications/bin/linux64GccDPOpt:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/wmake:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/bin:/usr/kerberos/bin:/bin:/usr/bin:/opt/ganglia/bin:/opt/ganglia/sbin:/opt/hurricane/bin:/opt/hurricane/sbin:/opt/openmpi/bin/:/opt/torque/bin:/opt/torque/sbin:/opt/rocks/bin:/opt/rocks/sbin:/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin:/home/sylvain.serra/bin:/share/apps/gmsh/gmsh-2.4.2-Linux/bin:/share/apps/openfoam/OpenFOAM.x/OpenFOAM-1.5.x/bin:/share/apps/paraview/paraview-3.6.1-Linux-x86_64/bin:/share/apps/fluent/Fluent.Inc/bin:/home/sylvain.serra/OpenFOAM/ThirdParty-1.6/qt-everywhere-opensource-src-4.6.1/bin
++ PATH=/opt/openmpi/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin/Linux_x86_64:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/XDATA_V5_1_3/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Tepal:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Ghs3d-4.1:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Metis-4.0:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Scotch-4.0/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Vtk-5.0.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Sphinx-0.6.3_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Docutils-0.6_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Expat-2.0.1/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Libxml2-2.6.32_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Doxygen-1.6.1/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Graphviz-2.24/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Med-2.3.6/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Hdf5-1.6.9/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Swig-1.3.40/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/OmniORB-4.1.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/PyQt-4.5.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Sip-4.8.2_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Qt-4.5.2_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Python-2.4.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/etchForSalome/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/ParaView3.3-cvs/platforms/linux64Gcc/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/cmake-2.4.6/platforms/linux64/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/openmpi-1.2.6/platforms/linux64GccDPOpt/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/gcc-4.3.1/platforms/linux64/bin:/home/sylvain.serra/OpenFOAM/sylvain.serra-1.5/applications/bin/linux64GccDPOpt:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/applications/bin/linux64GccDPOpt:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/wmake:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/bin:/usr/kerberos/bin:/bin:/usr/bin:/opt/ganglia/bin:/opt/ganglia/sbin:/opt/hurricane/bin:/opt/hurricane/sbin:/opt/openmpi/bin/:/opt/torque/bin:/opt/torque/sbin:/opt/rocks/bin:/opt/rocks/sbin:/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin:/home/sylvain.serra/bin:/share/apps/gmsh/gmsh-2.4.2-Linux/bin:/share/apps/openfoam/OpenFOAM.x/OpenFOAM-1.5.x/bin:/share/apps/paraview/paraview-3.6.1-Linux-x86_64/bin:/share/apps/fluent/Fluent.Inc/bin:/home/sylvain.serra/OpenFOAM/ThirdParty-1.6/qt-everywhere-opensource-src-4.6.1/bin
++ '[' Linux_x86_64 = OSF1 ']'
++ '[' Linux_x86_64 = Linux_CCRT ']'
++ '[' Linux_x86_64 = Linux_IA64 ']'
++ '[' Linux_x86_64 = Linux_Ch ']'
++ '[' Linux_x86_64 = AIX ']'
++ '[' -d /opt/openmpi/bin ']'
++ '[' -f /opt/openmpi/bin/mpdboot ']'
++ '[' -f /opt/openmpi/bin/ompi_info ']'
++ MPIRUN='/opt/openmpi/bin/mpiexec -n 4'
++ '[' '!' -z /opt/torque/aux//47378.chapo.interne.emd ']'
++ MPIRUN='/opt/openmpi/bin/mpiexec -n 4 -machinefile /opt/torque/aux//47378.chapo.interne.emd'
+ nproc_kernel=4
+ echo

+ echo

+ '[' no = yes ']'
+ '[' 4 -gt 1 ']'
+ echo 'Parallel Code_Saturne with partitioning in 4 sub-domains'
Parallel Code_Saturne with partitioning in 4 sub-domains
+ '[' 0 = 1 ']'
+ '[' '!' -z /opt/torque/aux//47378.chapo.interne.emd ']'
+ echo 'Total number of processors: 4'
Total number of processors: 4
+ '[' 4 -gt 1 ']'
+ ARG_CS_MPI=' --mpi'
+ '[' '!' -z '' ']'
+ ECHOCOMMSYR=
+ echo ' '

+ echo ' Code_Saturne is running '
Code_Saturne is running
+ echo ' *********************** '
***********************
+ echo ' '

+ echo ' Working directory (to be periodically cleaned) : '
Working directory (to be periodically cleaned) :
+ echo ' ' /home/sylvain.serra/tmp_Saturne/study.CASE.09070906
/home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ echo

+ echo ' Kernel version: ' /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin
Kernel version: /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin
+ echo ' Preprocessor: ' /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin
Preprocessor: /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin
+ '[' yes = yes ']'
++ pwd
+ cur_dir=/home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ source_cas=/home/sylvain.serra/saturne/study/CASE/SRC
+ '[' '!' -z chapo_mpi_canal_ke_1ms.xml ']'
+ var=/home/sylvain.serra/saturne/study/CASE/DATA/chapo_mpi_canal_ke_1ms.xml
+ '[' -f /home/sylvain.serra/saturne/study/CASE/DATA/chapo_mpi_canal_ke_1ms.xml ']'
+ COMMAND_PARAM='--param chapo_mpi_canal_ke_1ms.xml'
+ cp /home/sylvain.serra/saturne/study/CASE/DATA/chapo_mpi_canal_ke_1ms.xml .
++ ls '/home/sylvain.serra/saturne/study/CASE/SRC/*.[fF]90' '/home/sylvain.serra/saturne/study/CASE/SRC/*.[ch]'
+ src_files=
+ '[' '!' -z '' ']'
+ '[' '!' -z chapo_mpi_canal_ke_1ms.xml ']'
+ /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin/bin/code_saturne check_consistency --param=chapo_mpi_canal_ke_1ms.xml --nproc=4
+ '[' 0 = 1 ']'
+ ln -s /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin/bin/cs_solver .
+ '[' no = yes ']'
+ echo

+ echo ' ********************************************'
********************************************
+ echo ' Preparing calculation '
Preparing calculation
+ echo ' ********************************************'
********************************************
+ echo

+ ERROR=false
+ PREPROCESS_ERROR=false
+ PARTITION_ERROR=false
+ EXECUTION_ERROR=false
+ '[' 0 = 1 ']'
+ '[' yes = yes ']'
+ for var in '$MESH'
+ ln -s /home/sylvain.serra/saturne/study/CASE/../MESH/canal.msh canal.msh
++ basename canal.msh .case
+ var2=canal.msh
+ '[' canal.msh '!=' canal.msh ']'
+ '[' 4 -eq 1 -a yes = yes ']'
+ '[' yes = no -a /home/sylvain.serra/saturne/study/CASE/DATA/PARTITION_OUTPUT '!=' '' ']'
+ '[' yes = yes ']'
+ for var in '${RESTART_IN}/*'
+ '[' -f '/home/sylvain.serra/saturne/study/CASE/DATA/RESTART/*' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' no = yes ']'
+ '[' '!' -z '' ']'
+ '[' 47378.chapo.interne.emd '!=' '' ']'
++ qstat -r 47378.chapo.interne.emd
++ grep 47378.chapo.interne.emd
++ sed '-es/ \{1,\}/ /g'
++ cut -d ' ' -f 9
+ CS_MAXTIME=
+ export CS_MAXTIME
++ unset LANG
++ date
+ CURDATE='Tue Sep 7 09:06:53 CEST 2010'
+ echo ========================================================
+ echo ' Start time : ' Tue Sep 7 09:06:53 CEST 2010
+ echo ' ----------------------------------------------------'
+ echo ' Kernel : ' /share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin
+ echo ' Preprocessor : ' /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin
+ echo ' ------------------------------------------------ '
+ echo ' HOMARD : '
+ echo ' ------------------------------------------------ '
+ echo ' CS_MPI_PATH : ' /opt/openmpi/bin
+ echo ' PATH : ' /opt/openmpi/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin/Linux_x86_64:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/XDATA_V5_1_3/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Tepal:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Ghs3d-4.1:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Metis-4.0:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Scotch-4.0/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Vtk-5.0.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Sphinx-0.6.3_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Docutils-0.6_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Expat-2.0.1/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Libxml2-2.6.32_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Doxygen-1.6.1/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Graphviz-2.24/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Med-2.3.6/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Hdf5-1.6.9/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Swig-1.3.40/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/OmniORB-4.1.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/PyQt-4.5.4_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Sip-4.8.2_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Qt-4.5.2_python2.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/Python-2.4.4/bin:/share/apps/code_saturne/2.0/Salome_5.1.3/SALOME5/V5_1_3/../Prerequis/etchForSalome/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/ParaView3.3-cvs/platforms/linux64Gcc/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/cmake-2.4.6/platforms/linux64/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/openmpi-1.2.6/platforms/linux64GccDPOpt/bin:/home/sylvain.serra/OpenFOAM/ThirdParty/gcc-4.3.1/platforms/linux64/bin:/home/sylvain.serra/OpenFOAM/sylvain.serra-1.5/applications/bin/linux64GccDPOpt:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/applications/bin/linux64GccDPOpt:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/wmake:/home/sylvain.serra/OpenFOAM/OpenFOAM-1.5/bin:/usr/kerberos/bin:/bin:/usr/bin:/opt/ganglia/bin:/opt/ganglia/sbin:/opt/hurricane/bin:/opt/hurricane/sbin:/opt/openmpi/bin/:/opt/torque/bin:/opt/torque/sbin:/opt/rocks/bin:/opt/rocks/sbin:/share/apps/code_saturne/2.0/ncs-2.0.0-rc1/bin:/share/apps/code_saturne/2.0/syrthes/syrthes3.4.2/bin:/home/sylvain.serra/bin:/share/apps/gmsh/gmsh-2.4.2-Linux/bin:/share/apps/openfoam/OpenFOAM.x/OpenFOAM-1.5.x/bin:/share/apps/paraview/paraview-3.6.1-Linux-x86_64/bin:/share/apps/fluent/Fluent.Inc/bin:/home/sylvain.serra/OpenFOAM/ThirdParty-1.6/qt-everywhere-opensource-src-4.6.1/bin
+ echo ' ------------------------------------------------ '
+ echo ' User : ' sylvain.serra
+ echo ========================================================
+ echo ' Machine : '
+ uname -a
+ '[' -z 4 ']'
+ echo ' N Procs : ' 4
+ '[' -z '' ']'
+ echo ' Processors : ' default
+ echo ========================================================
+ echo ' ----------------------------------------------------'
+ echo ' Case : ' CASE
+ echo ' DATA : ' /home/sylvain.serra/saturne/study/CASE/DATA
+ echo ' SRC : ' /home/sylvain.serra/saturne/study/CASE/SRC
+ echo ' RESU : ' /home/sylvain.serra/saturne/study/CASE/RESU
+ echo ' ----------------------------------------------------'
+ echo ' Exec. dir. : ' /home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ echo ' ----------------------------------------------------'
+ '[' '' = yes ']'
+ '[' yes = yes ']'
+ echo ' Partitioner : ' /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin/cs_partition
+ '[' yes = yes ']'
+ echo ' Executable : ' cs_solver
+ echo ' ----------------------------------------------------'
+ echo

+ echo ' ********************************************'
********************************************
+ echo ' Starting calculation'
Starting calculation
+ echo ' ********************************************'
********************************************
+ echo

+ '[' yes = yes ']'
+ /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin/cs_preprocess --mesh canal.msh --case CASE
+ '[' 0 '!=' 0 ']'
+ '[' yes = no ']'
+ '[' '!' -f /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin/cs_partition -a 4 -gt 1 ']'
+ '[' yes = yes ']'
+ '[' yes = yes ']'
+ /share/apps/code_saturne/2.0/librairies/ecs-2.0.0-rc1/bin/cs_partition 4
+ '[' 0 '!=' 0 -a false = false ']'
+ '[' yes = no ']'
+ '[' false '!=' true -a yes = yes ']'
+ '[' no = yes -a '' = MPI ']'
+ '[' no = yes ']'
+ '[' 0 = 0 ']'
+ localexec=/home/sylvain.serra/tmp_Saturne/study.CASE.09070906/localexec
+ echo '#!/bin/sh'
+ echo cd /home/sylvain.serra/tmp_Saturne/study.CASE.09070906
+ echo ' /home/sylvain.serra/tmp_Saturne/study.CASE.09070906/cs_solver $@ --mpi ' ' ' ' --param chapo_mpi_canal_ke_1ms.xml '
+ echo 'retour=$?'
+ echo 'exit $retour'
+ chmod 700 /home/sylvain.serra/tmp_Saturne/study.CASE.09070906/localexec
+ /opt/openmpi/bin/mpiexec -n 4 -machinefile /opt/torque/aux//47378.chapo.interne.emd /home/sylvain.serra/tmp_Saturne/study.CASE.09070906/localexec
[chapo59.local:03185] pls:tm: failed to poll for a spawned proc, return status = 17002
[chapo59.local:03185] [0,0,0] ORTE_ERROR_LOG: In errno in file rmgr_urm.c at line 462
[chapo59.local:03185] mpiexec: spawn failed with errno=-11
+ EXECUTION_ERROR=true
+ rm -f /home/sylvain.serra/saturne/study/CASE/SCRIPTS/runningext.09070906
+ '[' true = true ']'
+ '[' no = yes -a '' = MPI ']'
+ echo 'Error running the calculation.'
Error running the calculation.
+ echo

+ echo 'Check Kernel log (listing) and error* files for details'
Check Kernel log (listing) and error* files for details
+ echo

+ ERROR=true
+ '[' yes = yes ']'
+ RESTART_OUT=/home/sylvain.serra/saturne/study/CASE/RESU/RESTART.09070906
+ iok=1
+ mkdir /home/sylvain.serra/saturne/study/CASE/RESU/RESTART.09070906
+ '[' 1 = 1 ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f suiava ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f suiavx ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f t1dava ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f ctwava ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f vorava ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f rayava ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f 'lagava*' ']'
+ for f in suiava suiavx t1dava ctwava vorava rayava 'lagava*' 'lasava*'
+ '[' -f 'lasava*' ']'
+ resuser=0
+ for f in '${USER_OUTPUT_FILES}'
+ '[' -f vitx003.dat ']'
+ for f in '${USER_OUTPUT_FILES}'
+ '[' -f vitx006.dat ']'
+ for f in '${USER_OUTPUT_FILES}'
+ '[' -f turb003.dat ']'
+ for f in '${USER_OUTPUT_FILES}'
+ '[' -f turb006.dat ']'
+ '[' 0 = 1 ']'
+ for f in '$PARAM' '$THERMOCHEMISTRY_DATA' '$METEO_DATA'
+ '[' -f chapo_mpi_canal_ke_1ms.xml ']'
+ cp chapo_mpi_canal_ke_1ms.xml /home/sylvain.serra/saturne/study/CASE/RESU/chapo_mpi_canal_ke_1ms.xml.09070906
+ for f in 'probes_*.dat'
+ '[' -f 'probes_*.dat' ']'
+ for f in 'ush*'
+ '[' -f 'ush*' ']'
+ for f in 'list*' 'error*' '*.med' '*.cgns'
+ '[' -f listpart ']'
+ cp listpart /home/sylvain.serra/saturne/study/CASE/RESU/listpart.09070906
+ for f in 'list*' 'error*' '*.med' '*.cgns'
+ '[' -f listpre ']'
+ cp listpre /home/sylvain.serra/saturne/study/CASE/RESU/listpre.09070906
+ for f in 'list*' 'error*' '*.med' '*.cgns'
+ '[' -f 'error*' ']'
+ for f in 'list*' 'error*' '*.med' '*.cgns'
+ '[' -f '*.med' ']'
+ for f in 'list*' 'error*' '*.med' '*.cgns'
+ '[' -f '*.cgns' ']'
++ echo CASE
++ tr '[:upper:]' '[:lower:]'
+ cas=case
+ for dir in '*.ensight' '*.med'
+ '[' -d '*.ensight' ']'
+ for dir in '*.ensight' '*.med'
+ '[' -d '*.med' ']'
+ '[' yes = yes ']'
++ ls 'bord*'
+ rayt_list=
+ '[' '!' -z '' ']'
++ ls 'debug*' 'deplacement*' 'trajectoire*' 'frontiere*'
+ lagr_list=
+ '[' '!' -z '' ']'
+ '[' -f /home/sylvain.serra/tmp_Saturne/study.CASE.09070906/resuMatisse ']'
+ for dir in src_saturne
+ '[' -d src_saturne ']'
+ '[' no = yes ']'
+ '[' false = true ']'
+ echo ' Preprocessing : ' yes
+ '[' false = true ']'
+ echo ' Partitioning : ' yes
+ '[' true = true ']'
+ EXEC_KERNEL=failed
+ echo ' Calculation : ' failed
++ unset LANG
++ date
+ CURDATE='Tue Sep 7 09:06:54 CEST 2010'
+ echo ' ----------------------------------------------------'
+ echo ' Finish time : ' Tue Sep 7 09:06:54 CEST 2010
+ echo ========================================================
+ cp summary /home/sylvain.serra/saturne/study/CASE/RESU/summary.09070906
+ echo

+ echo ' ********************************************'
********************************************
+ '[' true = true ']'
+ echo ' Error in calculation stage.'
Error in calculation stage.
+ echo ' ********************************************'
********************************************
+ '[' true = true ']'
+ exit 1