Compiling CS with Intel:

[Claus Andersen]

Looking to squeeze more out of my system, I have (finally) been able to install and compile Code_Aster with the latest Intel F/C++ compilers.

To do the same with Code_Saturne, I configure it with 'CC=icc FC=ifort'  + other needed terms.

This compiles the code with icc and ifort, but it does not use the MKL libs. provided by Intel, instead it used ATLAS.

checking for smp ESSL BLAS... no
checking for ESSL BLAS... no
checking for ESSL BLAS... no
checking for threaded MKL BLAS... no
checking for MKL BLAS... no
checking for threaded ATLAS BLAS... no
checking for ATLAS BLAS... yes

For Code_Aster I have the following line generated by Intel link-advisor:

MATHLIB='-Wl,--start-group /opt/intel/Compiler/11.1/073/mkl/lib/em64t/libmkl_intel_lp64.a -L/usr/lib -lg2c /opt/intel/Compiler/11.1/073/mkl/lib/em64t/mkl_intel_thread.a /opt/intel/Compiler/11.1/073/mkl/lib/em64t/libmkl_core.a /opt/intel/Compiler/11.1/073/mkl/lib/em64t/
libmkl_blacs_openmpi_lp64.a -Wl,--end-group'

How do I continue now, my lack of programming skills holds me back.

Regards,

Claus
 
Ps. Im just trying to compile NCS here - should other packages be compiled with Intel aswell?

[Yvan Fournier]

Hello,

You may add "--with-blas=" to your configure line. For example, our configure command for one of our clusters adds the following options to the configure line to use the Intel compilers an MKL.

Code: Select all

--with-blas=/home/logiciels/intel/Compiler/11.1/046/mkl \
CC=/home/logiciels/intel/Compiler/11.1/046/bin/intel64/icc \
FC=/home/logiciels/intel/Compiler/11.1/046/bin/intel64/ifort

We generally do not bother to force Icc for the other packages, as most of the time is spent in the Kernel, so they are usually build with gcc, though Icc should do also (we have observed one case were it seemed bft compiled with Icc 11.046 led to a crash if not compiled in debug mode, though adding "--disable-malloc-hooks" to BFT's configure solves the issue; as malloc hooks are useful only for memory use instrumentation on Linux kernels too old to have the "VmPeak" field in /proc/<pid>/status (circa 2.6.12), they will be disabled by default in 2.0.1, but 2.0.0 is frozen and should be released in a few days.

Best regards,

  Yvan

[Claus Andersen]

I will only bother with NCS then.

Hmm I still have the problem despite adding --with-blas to the configure command -

claus@claus-desktop:~/Downloads/ncs-2.0.0-rc2$ source /opt/intel/Compiler/11.1/073/bin/intel64/iccvars_intel64.sh
claus@claus-desktop:~/Downloads/ncs-2.0.0-rc2$ source /opt/intel/Compiler/11.1/073/bin/intel64/ifortvars_intel64.sh
claus@claus-desktop:~/Downloads/ncs-2.0.0-rc2$ source /opt/intel/Compiler/11.1/073/mkl/tools/environment/mklvarsem64t.sh

claus@claus-desktop:~/Downloads/ncs-2.0.0-rc2$ ./configure IC=/opt/intel/Compiler/11.1/073/bin/intel64/icc FC=/opt/intel/Compiler/11.1/073/bin/intel64/ifort --with-blas=/opt/intel/Compiler/11.1/073/mkl/ --with-mpi=/usr/lib/openmpi/ --with-syrthes=/usr/local/syrthes3.4.2

It still finds/prefers GNU Atlas over mkl - see attached config.log.  BLAS flags are around lines 2325+

Im running a calculation now, just to see if it works with GNU Atlas,

claus@claus-desktop:~$ lsof |grep atlas
cs_solver 20627      claus  mem       REG               8,33  7749816   665859 /usr/lib/libatlas.so.3gf.0
cs_solver 20676      claus  mem       REG               8,33  7749816   665859 /usr/lib/libatlas.so.3gf.0
claus@claus-desktop:~$

Can you spot the error in the configure log ?

Regards,

Claus

[Yvan Fournier]

Hello,

In your config.log, the check for MKL complains about not finding libguide (-lguide)

It searches in "/opt/intel/Compiler/11.1/073/mkl//lib/em64t", which I suppose is OK, so what libraries do you have in that directory ? We have compiled and tested Code_Saturne with several releases of ICC 10 and ICC 11.0, but I am not sure we have run with ICC .1. If there were changes in MKL, our autoconf scripts may not be up to date.

If you want to check this, you may edit the Kernel's config/cs_blas.m4 file (search for -lguide), and rebuild the configure script (run ./sbin/bootstrap, which is a developer script, and requires having the GNU autotools, gettext, bison, and flex installed), then re-run configure.

Otherwise, you may modify CFLAGS, LDFLAGS, and CPPFLAGS  and/or use --with-blas-type=BLAS and --with-blas-libs="<libs>" to force finer control over BLAS detection (don't use --with-blas-type=MKL, as this detection has already failed, and will use -lguide, whic seems to be the cause of the failure.)

In any case, we are interested in your feedback, and at least in the contents of your "/opt/intel/Compiler/11.1/073/mkl/lib/em64t" directory.

Best regards,

  Yvan

[Claus Andersen]

I seem to have got it working now, not entirely sure what the problem was right now. However, the binary acts a little strange: when launched from the GUI, nothing happens.

If I launch ./cs_solver --mpi --param WB.xml  from the ~/tmp_saturne/case/  the calculation runs and uses both cores.

Time for Intel + MKL: 458 s
Time for GNU + ATLAS: 374 s

claus@claus-desktop:~/tmp_Saturne/WB.CASE3.09072014$ ldd /usr/local/bin/cs_solver
        linux-vdso.so.1 =>  (0x00007fff1cf29000)
        libsaturne.so.0 => /usr/local/lib/libsaturne.so.0 (0x00007fd279bab000)
        libfvm.so.0 => /usr/local/lib/libfvm.so.0 (0x00007fd279908000)
        libmedC.so.1 => /usr/lib/libmedC.so.1 (0x00007fd2796cf000)
        libhdf5.so.6 => /usr/lib/libhdf5.so.6 (0x00007fd279121000)
        libfvm_coupl.so.0 => /usr/local/lib/libfvm_coupl.so.0 (0x00007fd278f1d000)
        libmei.so.0 => /usr/local/lib/libmei.so.0 (0x00007fd278d0d000)
        libbft.so.1 => /usr/local/lib/libbft.so.1 (0x00007fd278b01000)
        libz.so.1 => /lib/libz.so.1 (0x00007fd2788ea000)
        libmpi.so.0 => /usr/lib/libmpi.so.0 (0x00007fd278639000)
        libxml2.so.2 => /usr/lib/libxml2.so.2 (0x00007fd2782e9000)
        libmkl_intel_lp64.so => /opt/intel/Compiler/11.1/073/mkl/lib/em64t/libmkl_intel_lp64.so (0x00007fd277ded000)
        libmkl_intel_thread.so => /opt/intel/Compiler/11.1/073/mkl/lib/em64t/libmkl_intel_thread.so (0x00007fd2769b0000)
        libmkl_core.so => /opt/intel/Compiler/11.1/073/mkl/lib/em64t/libmkl_core.so (0x00007fd2764f9000)
        libguide.so => /opt/intel/Compiler/11.1/073/lib/intel64/libguide.so (0x00007fd276328000)
        libifport.so.5 => /opt/intel/Compiler/11.1/073/lib/intel64/libifport.so.5 (0x00007fd2761ee000)
        libifcore.so.5 => /opt/intel/Compiler/11.1/073/lib/intel64/libifcore.so.5 (0x00007fd275f77000)
        libimf.so => /opt/intel/Compiler/11.1/073/lib/intel64/libimf.so (0x00007fd275be3000)
        libsvml.so => /opt/intel/Compiler/11.1/073/lib/intel64/libsvml.so (0x00007fd2759cc000)
        libm.so.6 => /lib/libm.so.6 (0x00007fd275749000)
        libirc.so => /opt/intel/Compiler/11.1/073/lib/intel64/libirc.so (0x00007fd27560b000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00007fd2753ed000)
        libdl.so.2 => /lib/libdl.so.2 (0x00007fd2751e9000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fd274fd2000)
        libc.so.6 => /lib/libc.so.6 (0x00007fd274c4e000)
        libintlc.so.5 => /opt/intel/Compiler/11.1/073/lib/intel64/libintlc.so.5 (0x00007fd274b10000)
        libopen-rte.so.0 => /usr/lib/libopen-rte.so.0 (0x00007fd2748c4000)
        libopen-pal.so.0 => /usr/lib/libopen-pal.so.0 (0x00007fd27464f000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x00007fd274435000)
        libutil.so.1 => /lib/libutil.so.1 (0x00007fd274232000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fd273f1d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd27b0be000)

It's a bit surprising that gcc + atlas is faster than icc + mkl  and it makes me think something might be wrong. Or icc/ifort isn't a magic bullet.

I've attached the content of "/opt/intel/Compiler/11.1/073/mkl//lib/em64t" for reference

Regards,

Claus

[David Monfort]

Hello Claus,

First of all, thanks for the ls_lib.txt. It seems that our autoconf machinery is up-to-date.

As far your results are concerned, this is always difficult to draw conclusion of performance results. Have you reproduced several times the calculations and is your mesh big enough? An interesting test to do would be to run your simulation with the following Code_Saturne installation:

- Intel compilers
- Intel compilers + Intel MKL
- Inter compilers + Atlas
- GCC
- GCC + Intel MKL (might need to adapt the libraries to link against here)
- GCC + Atlas

I know it is a lot of work, but this way you will be able to conclude on where your lack of performance come from.

Last point, to fine-tune your installation you shoud edit the config/cs_auto_flags.sh file (and re-run the configure after) to adapt the cflags/fcflags variable according to your need.

Perhaps we missed some optimization for the Intel compilers.

David

ps: in my opinion, you should have better results with icc/ifort/mkl than with gcc/atlas...