code_saturne User's Forum

I wrote two posts about the compilation of Code_Saturne 2.0.0 on Ubuntu to share my experiences:

"Installation of Code_Saturne 2.0.0 on Ubuntu 9.04"

"Compile Code_Saturne with SALOME binary libraries"

The second one is for the people who also use SALOME. With an installed SALOME, its binary libraries can be re-used by Code_Saturne.

Hope they help and any comments are greatly appreciated.

Wayne

http://code-saturne.blogspot.com

Hi Wayne,

You did a really good job!

We've been following your blog for a while and really appreciate the tutorial you wrote -- especially concerning the installation, as it seems to remain an issue for many beginners (though we did a lot of work to improve the whole thing).

A few  comments on oyour post about installing Code_Saturne 2.0 beta2:

• Some libraries might already be available in Ubuntu, like HDF5 and MED, and the next version of Ubuntu may also have CGNS library and a newer version of ParaView (3.6.2) as they are available in Debian testing IIRC. You should also advice users to install the ATLAS library optimized for their processor (say e.g. SSE2, MMX, ...).
  
  As said in on post (or comment), part of Code_Saturne 2.0 beta2 is already available as a Debian package (and older stuff in Ubuntu as well). We'll communicate more on that topic in a fews days, but I can disclose that Code_Saturne 2.0 rc1 will be functional in Debian soon. Speaking of version 2.0 rc1, we updated the autoconf macros so that you do not need to add "-lm" anymore and it should also find the preprocessor automatically provided it is in your PATH; the configure now emits a warning or an error if it does not find some tools or libraries (except for LaTeX stuff); the MEI compilation issue is now fixed.
  
  Lastly, I need to check why you had to manually add two LaTeX packages to compile the documentation... seems weird to me :-/, plus I do not think epstopdf package is mandatory (need to check as well...)

Another bunch of comments about SALOME compilation:

• The Metis library included in the SALOME package is targeted towards Code_Aster, and has been slightly modified for their purpose. I cannot guaranty that is is working properly since I haven't tested it yet.
  With Code_Saturne 2.0 rc1, you will be able to use the Python executable and the PyQt libraries of SALOME for a better integration (thanks to the --with-python-exec --with-pyqt-dev-tools options). Also stay tune for more information about SALOME in the next weeks

I don't think at other comments at the moment...

Anyway, good job again!

David

Hi David,

How are you?

Many thanks for all your comments and the interesting information you provided.

1. I knew an old version of ParaView is available from apt-get. I remembered there is a package called 'hdf5-tools' (or similar like this, I cannot remember clearly). Unfortunately I didn't get the way to install HDF5 and MED by apt-get. If it is possible, can you please give hints? I would like to try and then update the post.

2. I suggested 'apt-get install libatlas-headers libatlas-base-dev' regarding installing the ATLAS library. I didn't encounter any problems by using this (in fact I didn't test whether the calculation performance is really improved or not, either). If I need to specify more detailed packages, please advise.

3. I am really looking forward to trying and using 2.0 rc1. I believe this new version can fix lots of problems. For example, users might not be forced to install mpi any more, I guess. MEI is also an expected function, which could make life easier. I know Code_Saturne has a Debian package, but unfortunately, I haven't tried it yet. A standard deb package could be expected as a ultimate solution of the installation problem. On the other hand, I am thinking, possibly, we can also try to make a PPA source for Code_Saturne, and people can then easily use Ubuntu-Tweak to install it. Launchpad can even automatically compile the source code and make packages for 32 and 64 bit platforms after we upload the source onto it. When I get time, I want to try this.

4. Er, I found epstopdf command is necessary because loads of eps figures need to be converted to pdf format when compiling the latex documents. At least when I wrote the post on Ubuntu 9.04, this is true. Sometimes I regard the Ubuntu package system is actually not as stable as I expect. When the distribution version number increase, some important packages don't behave like before and some are even obsoleted.

5. Thanks a lot for reminding me about the modified Metis library in SALOME. I didn't know about this. However, I didn't encounter problems yet up to now.

6. I believe the integration with SALOME is a amazing feature. Definitely I would try this.

Once again, thank you very much for your comments. I wrote the blog because I want to help and make effort to let more people know about Code_Saturne and then try and use it. I also hope I can help more in the future if I can.

Any comments will be welcome

Wayne

Hi Wayne,

Here are some new input:

HDF5 and MED can be installed as follow (I did so in my Ubuntu system): apt-get install libmedc-dev. The underlying reason of the package naming is that two interfaces are available for MED (Fortran and C). Please note that libmedc-dev should require the installation of libhdf5-serial-dev or libhdf5-dev (and of the correspondent binary packages of course .)
Different ATLAS packages are available on Ubuntu: libatlas-base-dev, libatlas-3dnow-dev, libatlas-sse-dev, libatlas-sse2-dev. The user may choose the cpu-optimized library for its computer CPU (if this latter supports the 3DNOW, SSE or SSE2 instruction set). The one you suggest is the generic one and so should work on every CPU supported by Ubuntu. As a side note, you can test performance by setting the variable ARG_CS_VERIF to --benchmark in the runcase script (or through the graphical interface, in the Advanced setup part of the Batch runnning page). It will run some basics tests on your mesh (like dot product, matrix-vector product, ...) and compare the results between without and with BLAS support. I would strongly advise in this case not to have any parameter file (i.e. outside the interface) because it can interfere with the default settings of the solver, and fail abruptly :-/

MPI and MEI installation issues are fixed in the RC1 release! I don't know about Ubuntu PPA, but if anyone wants to tackle the problem and needs help, I can help. Code_Saturne is currently in Debian "unstable" (just apt-get install code-saturne! without MPI support though) thanks to a huge work of the packager.

I know we need fig2dev tool to convert xfig picture in pdf format, but we do not have any ps files left in the documentation... Perhaps in the preprocessor documentation, but it will be merged with the user/theory manuals soon.

You may encounter no problem with this "EDF-tuned" Metis version... I just don't what they changed in the code (maybe the integer size...). Soon... soon...

I hope you'll continue posting on your blog; and feel free to also post update here in Code_Saturne forums about your blog so that other users can follow or discover it.

David

In continuation of this thread, I was wondering if it is possible to link CS to AMD ACML - it does wonders for Code_Aster speedups (nearly 100% in some cases!), I however, know very little about programming and linking of libraries.

In case you aren't familiar with it, heres their sales pitch (even though it's free)    :

ACML - the AMD Core Math Library - is a tuned math library designed for high performance on AMD64 machines, including Opteron(TM) and Athlon(TM) 64, and includes both 32-bit and 64-bit library versions. Different versions are available for Linux, Windows and Solaris operating systems.

A full suite of Basic Linear Algebra (BLAS) routines is provided, and key BLAS routines have been tuned for high performance on AMD processors in both 32-bit and 64-bit modes.

A full suite of Linear Algebra (LAPACK) routines is provided and takes advantage of the highly tuned BLAS for good performance.

Regards,
/C

Indeed, ACML should be linkable with Code_Saturne (more particularly with the solver in the kernel package "ncs").

However, we haven't test yet this set of libraries... but I cannot foresee any trouble in their usage. The only thing is that BLAS automatic detection only works, at the moment, with the following set of libraries: standard BLAS (also called REFBLAS), ATLAS, MKL (from Intel compiler suite) and ESSL (from IBM compiler suite, also available on BlueGene systems). We still miss the Goto BLAS and ACML libraries. If you can test for us, and provide any feedback on the libraries we should link with, it would be appreciated! So we could have a proper detectionof ACML in the final release of Code_Saturne 2.0.

The autodetection is done in the cs_blas.m4 file, located in ncs-2.0.0-rc1/config; if you edit it, you should bootstrap the package by ./sbin/bootstrap. But I suggest you to try a different path before by configuring the package with: ./configure --prefix=/path/to/install ... CPPFLAGS="-DHAVE_FBLAS -I/path/to/acml/include" LDFLAGS="-L/path/acml/lib" LIBS="-lacml_lib"

The difficult part is to find the right libraries to be set in the LIBS variable. IIRC, ACML BLAS only provide a Fortran interface for the different routines; if not, you can test the C interface by setting CPPFLAGS="-DHAVE_CBLAS".

A for the performance, I don't think that you can gain a factor of 2... but I would be interested in knowing the gain ;) Note that Code_Saturne matrices are sparse (contrary to most of Code_Aster ones) and that we only use BLAS level 1 routines (and perhaps some of level 2 as well in some rare conditions).

David

I've tried compiling ncs with AMD ACML libs now, using the MP version for obvious reasons.

./configure --prefix=/usr/local --with-mpi=/usr/lib/openmpi CPPFLAGS="-DHAVE_FBLAS -I/opt/acml4.3.0/gfortran64_mp/include" LDFLAGS="-L/opt/acml4.3.0/gfortran64_mp/lib/" LIBS="-lacml_mp"

Ran a small test with and without AMD ACML:

With AMD ACML:

Calculation time summary:
  User CPU time:            162.010 s
  System CPU time:            0.530 s
  Elapsed time:             171.143 s
  CPU / elapsed time          0.950
Memory use summary:
  Total memory used:                             204.113 mb
  Theoretical instrumented dynamic memory:        81.855 mb
  Fortran work arrays memory use:
   196215       integers needed (maximum reached in CREGEO)
   4746706      reals    needed (maximum reached in MEMNAV)
   Local maximum work memory requested       68.212 mb
                                  used       37.850 mb

Without AMD ACML:

Calculation time summary:
  User CPU time:            166.840 s
  System CPU time:            0.440 s
  Elapsed time:             177.417 s
  CPU / elapsed time          0.943
Memory use summary:
  Total memory used:                             171.805 mb
  Theoretical instrumented dynamic memory:        81.855 mb
  Fortran work arrays memory use:
   196215       integers needed (maximum reached in CREGEO)
   4746706      reals    needed (maximum reached in MEMNAV)
   Local maximum work memory requested       68.212 mb
                                  used       37.850 mb

So theres a small speedup of a couple of seconds using AMD ACML at the cost of increased memory use. This is however only a test of 10 time steps, so speedup might be much more significant with larger studies.

I might try for a longer test of 1000 steps at some point, but not tonight

I've attached the config.log file so you can check if I indeed configured ncs correctly; I had no errors during configuring nor build, so I assume things went well.

Regards,

/C

Hmmm...

I find it strange to have so little improvement with BLAS support :-/

Could you just try to launch benchmark tests (with --benchmark option defined in the ARG_CS_VERIF option) ? You shouldn't use any XML parameter file in this case (just your mesh is enough). And I don't think threaded library will change much, at least as long as you do not set something like OMP_NUM_THREADS variable (actually Code_Saturne only support OpenMP detection as of now, not its use ).

If you launch benchmark test suite, please post the result listing file, so that we can have a look it. I would expect something like 20% of improvement in standard simulation and perhaps a factor of 5 to 20 (in terms of Gflops) for dot products.

Regards, David

ps: I'll be outside of my office next week, but will try ACML porting the week after... if you do not succeed before

Previously David Monfort wrote:

Hmmm...

I find it strange to have so little improvement with BLAS support :-/

Could you just try to launch benchmark tests (with --benchmark option defined in the ARG_CS_VERIF option) ? You shouldn't use any XML parameter file in this case (just your mesh is enough). And I don't think threaded library will change much, at least as long as you do not set something like OMP_NUM_THREADS variable (actually Code_Saturne only support OpenMP detection as of now, not its use ).

If you launch benchmark test suite, please post the result listing file, so that we can have a look it. I would expect something like 20% of improvement in standard simulation and perhaps a factor of 5 to 20 (in terms of Gflops) for dot products.

Regards,

David

ps: I'll be outside of my office next week, but will try ACML porting the week after... if you do not succeed before

I just tried the benchmark while having my morning coffee and it made me wonder if the mesh I use, is dense enough to 'defeat' the variance created by random system calls etc, mainly because the difference between with and without ACML is yet again minuscule.

  Mesh
     Number of cells:          148984
     Number of interior faces: 292969
     Number of boundary faces: 9998
     Number of vertices:       28031
 
I don't have a more dense mesh readily at hand, but let me know if it'll change the benchmark significantly and I'll se what I can do.

Attaching listing with and without ACML - created with 'cs_solver --param case.xml --benchmark
' - let me know what you make of it.  

Hi David,

Thanks for your new hints. I will try them and definitely update my blog with interesting posts as I can

It is good to read you are discussing about AMD issues. Unfortunately I don't have an AMD machine on hand to test what you said.

Wayne