Mesh partition and computing geometric quantities

[Yvan Fournier]

Hello,

The speedup seems OK from 5 to 16 procs (a bit less than optimal, though), and rather small from 16 to 32 procs. This is not so surprising, as the code is mostly bandwidth limited, and on most Xeon cores, performance increase from half node to full node is much less than 2 (more like 20%) as the memory bandwith limit is reached before the compute power limit. In the 60000 cells per core to 20000 cells per core range, performance can increase significantly, due to better cache behavior, but even on 32 procs, you are much above this range (so basically, getting better performance requires a bigger machine).

This does not explain why computations get "stuck". To determine that (or help determine it), it would be useful to attach a debugger, or at least obtain a "backtrace" (which you might be able to get when you kill the stuck job).

It would also be interesting to run a "debug" build of the code (about 3x slower, due to more internal instrumentation), which might contain info helping find the issue.

Regards,

Yvan

[fahad]

Hi Yvan: Thanks for replying.

I understand. Can you please guide me from where can i obtain this debugger and backtrace files. I am sorry I have never known about these. I will search too but if you could quickly point out, it will be of great help.

Thanks

[fahad]

Hi Yvan:
Instead of unsteady turbulent, I ran the simulation (on the same mesh) for steady laminar for 5 time steps and both 32MPI and 8MPI completed 5 time steps. I have attached both performance logs. Kindly take a look and see if you could find out whats wrong with unsteady LES set up of mine which causes the simulation to get stuck.

Regards
Fahad


Steady Laminar 32MPI
--------------------

Time-Step Time for Time-Step

1 20 min
2 5 min
3 88 min
4 85 min
5 154 min

Approximate Physical Run Time: 6.30hrs
------------------------------------------------------------
Steady Laminar 8MPI
-------------------

Time-Step Time for Time-Step

1 41 min
2 10 min
3 131 min
4 129 min
5 22 min

Approximate Physical Run Time: 7hrs

[Yvan Fournier]

Hello,

You have almost a factor 2 performance difference between 8 and 32 procs. It would be interesting to test at 16, as you probably have an improvement from 8 to 16 and almost no improvement from 16 to 32 (assuming the bandwith is saturated), but it can be interesting to check.

I do not know which difference between laminar and LES causes the blockings in LES (LES adds filters, and uses an 2nd order time scheme). In any case, I would recommend seeing if the latest 3.0 version (assuming you cannot move to 4.0, which would be better) also has the same issue.

Best regards,

Yvan

[fahad]

Hello Yvan:

I have attached performance and listing using 16MPI for steady laminar, as you suggested. Please do comment.

And I intend to install latest code saturne version 4.0. I have downloaded compressed version of it. The current code saturne was pre-installed on CAE Linux therefore I did not have to go through the expert level installation steps given in the installation guide.

I wish to retain the current version of code saturne and install code saturne 4.0 as well. If that is not recommended I want to upgrade the existing to version 4.0. If you do not mind kindly share the specific installation steps necessary for me so that I can have complete functionality that came with CAE linux. I am not familiar with Linux installation. I will be thankful.

Regards

[Yvan Fournier]

Hello,

The performance degradation is a little "smoother" than I expected, but still probably related to memory bandwidth limitations (i.e. to go faster with this code, you would need a more powerful machine).

For installation, I would recommend reading at least the beginning of the detailed installation manual, knowing that in the case of CAELinux, the optional libraries used by the code are probably all installed already (which makes things easier). I am just not too sure about where some optional libraries are found.

To make things easier, with the pre-installed version 3.0, could you type:
"ldd /opt/saturne-3.0/libexec/code_saturne/cs_solver" ? and post the results ? This will provide me the paths I can then use to recommend some configure options to you (ie. where to find the MED library provided/used by SALOME, for example).

Also, there is no problem with installing different versions of the code in different paths (this is what we do on our installs, as studies started with one version are usually continued with that version).

Regards,

Yvan

[fahad]

Hi Yvan:

The results are:

linux-vdso.so.1 => (0x00007fff61980000)
libsaturne.so.0 => /opt/saturne-3.0/lib/libsaturne.so.0 (0x00007fd165f40000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd165cf8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd165938000)
libple.so.0 => /opt/saturne-3.0/lib/libple.so.0 (0x00007fd165720000)
libcgns.so.3.1 => /opt/saturne-3.0/cgnslib-3.1.3//lib/libcgns.so.3.1 (0x00007fd165468000)
libmedC.so.1 => /opt/saturne-3.0/med-3.0.6/lib/libmedC.so.1 (0x00007fd165140000)
libparmetis.so.3.1 => /usr/lib/libparmetis.so.3.1 (0x00007fd164f00000)
libptscotch-5.1.so => /usr/lib/libptscotch-5.1.so (0x00007fd164c50000)
libmpi.so.0 => /usr/lib/openmpi/lib/libmpi.so.0 (0x00007fd164998000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fd164638000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007fd164320000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd164020000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd163e18000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd1677e8000)
libhdf5.so.7 => /opt/saturne-3.0/hdf5-1.8.9/lib/libhdf5.so.7 (0x00007fd163950000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd163650000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd163438000)
libmetis.so.3.1 => /usr/lib/libmetis.so.3.1 (0x00007fd1631e8000)
libptscotcherr-5.1.so => /usr/lib/libptscotcherr-5.1.so (0x00007fd162fe0000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd162dc8000)
libopen-rte.so.0 => /usr/lib/libopen-rte.so.0 (0x00007fd162b78000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007fd162940000)
libopen-pal.so.0 => /usr/lib/libopen-pal.so.0 (0x00007fd1626e8000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fd1624e0000)

Yes I had read the installation manual before contacting you. I was lost a bit.

I have downloaded and unzipped the files in a directory inside /usr. Now i need to make another directory where I will make and make install. Then I need to configure and compile third party libraries, then the environment variables setting, and then post installation step. Could you please share a set of commands that I run one after another for a clean installation with maximum functionality. I would be grateful.

PS: I would like to have separate paths for code-saturne 4.0.

[fahad]

hi Yvan

I have pasted the results in the previous posts. I shall wait for the installation instructions. Best Regards

[Yvan Fournier]

Hello,

Libraries such as Scotch and Metis seem to be in the general path, so the Code_Saturne installer should find those without any specific options. The HDF5, MED, and CGNS libraries for Code_Saturne 3.0 are installed in that build's subdirectory, so you unfortunately cannot use them for Code_Saturne 4.0 without risking linking issues.

The rest should be OK, so I recommend using the automatic installer:
- create a build directory (for example as $HOME/Code_Saturne/4.0/build)
- run <path_to_code_saturne_4.0_sources/install_saturne.py
- edit the setup file generated by this, choosing where to install the code, and which libraries to download and install. You'll need to tell the code to install HDF5, MED, and CGNS (assuming you need to read MED or CGNS files). For the rest, you should be able to use default settings.
- run <path_to_code_saturne_4.0_sources/install_saturne.py a second time

In case of error, you'll probably simply need to fix incorrect definitions in your setup file and run again.

Best regards,

Yvan

[fahad]

Hi Yvan

As per your instructions, I have clean installed code saturne 4.0.5 (no errors reported) and have performed the post-installation step (which means i have removed the word 'template' from the cfg file. But I have not touched anything inside the file (kindly guide me on this too).

But I am unable to launch code saturne 4 even from within the installed bin directory by typing code_saturne. It launches code saturne version 3 which you know is already installed.

I am utterly at loss with defining paths, setting commands, aliases, bash stuff. I tried whole day but to no avail.

path to version 4 installed is: /opt/Saturne-4.0.5/code_saturne-4.0.5/arch/Linux_x86_64/bin
path to version 3 installed is: /opt/saturne-3.0/bin
Alias could be: alias saturne4="/opt/Saturne-4.0.5/code_saturne-4.0.5/arch/Linux_x86_64/bin/code_saturne"

Kindly help me in setting short commands (where and how) to launch code saturne version 4 differently from code saturne 3. It would also be helpful where could i locate launch commands for code saturne 3 so that i could make it different than version 4.

Waiting for your reply. I have attached few files, If it helps.
Regards