Hi all,
I have to run coupled simulations for fluid flow with heat transfer, so I'd like to use the coupling capabilities of Saturne and Syrthes. I'm using Saturne for a few months now, but don't know at all Syrthes.
I started to look at the documentation, and tried to run a basic test case given in the user's manual. The calculation is running fine, but when it comes to post process the results, I use the tool syrthes2ensight to convert the results files in Ensight format, but I can't read them in Paraview. I have to files .case, . geom and xxx.TEMP_SOLID.xxx, when I open them they look like the files I have when I run a calculation with Saturne. I can open the .case file in Paraview, but when I click on 'apply' to load the file, Paraview just crashes !! (either with Big Endian or Little Endian)
I noticed a difference between the case files from Saturne and Syrthes (the converted file with syrthes2ensight) : in the first one, the type is Ensight Gold, in the second one the type is only Ensight. I can't figure out what is wrong... I'm using Paraview 3.3.0, Syrthes 3.4.2.
Another question regarding the conversion of the results files from Syrthes to Ensight files : I noticed that the executable syrthes2ensight needed to be in the same folder as the files I want to convert (geom and resu from Syrthes). Is there something I can do to leave the syrthes2ensight executable in my /opt/syrthes3.4.2/bin/arch/ folder, and my results files in the results folder, and still be able to do the conversion ? (it seems to me like a stupid question, and I'm sorry about that, but I just wasn't able to do it that way and it is bothering me !)
Last question on the topic : where can I find the tool syrthes2med ? (I only have a syrthes2med223.c in my syrthes folder, and no executable, and maybe I won't have the same problems with the conversion to med format)
I also wanted to know if you have somewhere in your files some basic tutorial for the coupling between Saturne and Syrthes ? Just to be sure that I'm doing everything correctly before starting with complex cases.
Last question : in the problem I have to model, I need to take into account forced convection, conduction of the walls but also of a piece of metal inside my domain, and radiation. Can I do it using the Saturne / Syrthes coupling ? Should I use the radiation module of Saturne or of Syrthes (It seems that it is more likely Saturne's but I'm not sure...)?
Thank you very much in advance !!
Alicia
Syrthes : post processing format / tutorial for coupling with Saturne
Forum rules
Please read the forum usage recommendations before posting.
Please read the forum usage recommendations before posting.
-
Yvan Fournier
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Hello,
As it seems the ParaView version you are using is a bit old, you may try a newer version. I beleive syrthes2ensight produces EnSight 6 output in text mode, but I do not know if ParaView has had recent bug corrections reading this (quite old) variant of the EnSight format.
Compiling syrthes2med would probably solve your problems, and you probably just need to know the correct compiler command (I do not know if there is a Makefile) to do this. I would suggest sending an e-mail to syrthes-support@edf.fr (I'll inform the Syrthes developpers 3 offices from mine to check their mail, but I am not sure of the best answer myself).
Finally, your calculation should be feasible with Code_Saturne/Syrthes coupling, if the fluid you are modeling is transparent (in which case you should use Syrthes's radiative module, and are in charted territory). If your fluid is semi-transparent, Code_Saturne's radiative model would be a better fit, but coupling the with Syrthes may not work yet (we may have recent patches for that we could probably send you, but they are yet untested and will only be standard in Code_Saturne 2.1, coming this fall).
Best regards,
Yvan
As it seems the ParaView version you are using is a bit old, you may try a newer version. I beleive syrthes2ensight produces EnSight 6 output in text mode, but I do not know if ParaView has had recent bug corrections reading this (quite old) variant of the EnSight format.
Compiling syrthes2med would probably solve your problems, and you probably just need to know the correct compiler command (I do not know if there is a Makefile) to do this. I would suggest sending an e-mail to syrthes-support@edf.fr (I'll inform the Syrthes developpers 3 offices from mine to check their mail, but I am not sure of the best answer myself).
Finally, your calculation should be feasible with Code_Saturne/Syrthes coupling, if the fluid you are modeling is transparent (in which case you should use Syrthes's radiative module, and are in charted territory). If your fluid is semi-transparent, Code_Saturne's radiative model would be a better fit, but coupling the with Syrthes may not work yet (we may have recent patches for that we could probably send you, but they are yet untested and will only be standard in Code_Saturne 2.1, coming this fall).
Best regards,
Yvan
-
Alicia Consigny
Re: Syrthes : post processing format / tutorial for coupling with Saturne
I sent a mail to the syrthes support team, and I'm waiting for their answer. I'll let you know the outcome.
I just have another question I'm interested in : do you have somewhere the specification of the time needed for calculation depending on the number of processors used ? (my boss wants to know if there is a real gain in calculation time when using more processors (I'm actually running on a computer with 6GB RAM and 2 cores) before buying anything !)
Thanks again for your help !!
Alicia
I just have another question I'm interested in : do you have somewhere the specification of the time needed for calculation depending on the number of processors used ? (my boss wants to know if there is a real gain in calculation time when using more processors (I'm actually running on a computer with 6GB RAM and 2 cores) before buying anything !)
Thanks again for your help !!
Alicia
-
David Monfort
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Hi Alicia,
The gain in calculation time when running parallel simulations really depends on the number of cells you have, and on the cluster you have. Code_Saturne scales quite well (depending on the case) up to thousands of processors; if the scaling is linear, and it is up to a certain number of processor (case-dependant again...), you'll have a factor two (gain in time) when you double the number of processors.
The only relevant parameter to have a good scaling is the network. You need a very good network (like Infiniband, Myrinet, ...) between nodes because the code communicates a lot between processors (equations are solved in an implicit way).
You need also to keep the number of cells by processor high enough so that there are enough calculations done on the processors compared to the cost of the communication between processors. A guess value (processor-dependant...) could be around 10 to 20 thousands cells.
To sum up, I would strongly advise you to test parallel simulations in your own with one standard test-case. By the way, what is the "standard" number of cells you need to run?
Hope this helps!
David
The gain in calculation time when running parallel simulations really depends on the number of cells you have, and on the cluster you have. Code_Saturne scales quite well (depending on the case) up to thousands of processors; if the scaling is linear, and it is up to a certain number of processor (case-dependant again...), you'll have a factor two (gain in time) when you double the number of processors.
The only relevant parameter to have a good scaling is the network. You need a very good network (like Infiniband, Myrinet, ...) between nodes because the code communicates a lot between processors (equations are solved in an implicit way).
You need also to keep the number of cells by processor high enough so that there are enough calculations done on the processors compared to the cost of the communication between processors. A guess value (processor-dependant...) could be around 10 to 20 thousands cells.
To sum up, I would strongly advise you to test parallel simulations in your own with one standard test-case. By the way, what is the "standard" number of cells you need to run?
Hope this helps!
David
-
Alicia Consigny
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Hi David,
Thanks for the reply, it is always helpful !!
The standard number of cells I'm usually using (and that I'm more likely going to use soon) is a few millions, I think it can be up to 10/20 millions. I will definitely need more than 2 procs to have a reasonable calculation time.
I already tried a little test with a standart test case, the gain was approximatively 40% between the calculation with 1 proc and with 2 procs.
What do you mean by 'you need a very good network between nodes' ? How can I control that parameter ? Is it something related to the meshing software I'm using, to the way the parallel computation is handled ?
Regards,
Alicia
Thanks for the reply, it is always helpful !!
The standard number of cells I'm usually using (and that I'm more likely going to use soon) is a few millions, I think it can be up to 10/20 millions. I will definitely need more than 2 procs to have a reasonable calculation time.
I already tried a little test with a standart test case, the gain was approximatively 40% between the calculation with 1 proc and with 2 procs.
What do you mean by 'you need a very good network between nodes' ? How can I control that parameter ? Is it something related to the meshing software I'm using, to the way the parallel computation is handled ?
Regards,
Alicia
-
Yvan Fournier
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Hello,
Parallel performance will depend on several factors, mostly on hardware (assuming you have built Code_Saturne using METIS or SCOTCH, or at least are running 2.0-rc1 with the built-in space-filling-curve base partitioning so as to obtain a relatively well optimized partitioning) :
For message-passing between nodes of a cluster, both bandwith and latency are important. Most modern networks have good bandwith (important for ghost cell updates), but iterative linear solvers require one or several global dot-products at each iteration (with only one real value per process, but which must be summed on all processors), so they are latency-bound. Gigabit Ethernet is thus a bad choice for a cluster running Code_Saturne or similar CFD codes, while Infiniband, Myrinet, or other specialized networks such as those of IBM Blue Genes or Cray XT's are a much better choice (basically, on 30 or so processors, a calculation could be expected to run at only half speed using Gigabit Ethernet compared to a high-performance network, and it robably gets worse for a higher number of processors) Inside a node, local memory bandwith and contention play a similar role. If you have multiple processors, you can expect much better performance with Opteron or Nehalem-type processors than with older Intel processors, as each processor has its own link to memory and they do not content for a shared memory bus. When you use several cores of a same processor, memory contention issues migh also occur Finally, inside a given processor, depending on the size of the local data, cache behavior may be good or bad, and this has a major effect on performance.
All these factors interact, so the only "simple" recommendations would be:
With typical hardward, for10 million cells, you would probably have better than linear speedup running on 128 to 256 processors than on 32 for example, as more data fits into cache (that might be way above the size of the machine you are planning to buy, if you are coming from just 2 processors). Scaling on pre-Nehalem Intel processors will be limited by the fact that processors share the same memory bus, so an AMD Opteron, Intel core I7 or Xeon 5500 series or better are highly recommended (I only consider Intel or AMD here as they are the most mainstream, but If your boss offers to buy you a cluster of IBM Power6's for example, those things can run very fast too).
Also, for better speed (independently of parallelism), and in most cases with no apparent adverse effects on the solution, you may reduce the required precision of the linear solvers from 1O -8 to 10 -5 , and thus gain quite a few iterations.
Best regards,
Yvan
Parallel performance will depend on several factors, mostly on hardware (assuming you have built Code_Saturne using METIS or SCOTCH, or at least are running 2.0-rc1 with the built-in space-filling-curve base partitioning so as to obtain a relatively well optimized partitioning) :
For message-passing between nodes of a cluster, both bandwith and latency are important. Most modern networks have good bandwith (important for ghost cell updates), but iterative linear solvers require one or several global dot-products at each iteration (with only one real value per process, but which must be summed on all processors), so they are latency-bound. Gigabit Ethernet is thus a bad choice for a cluster running Code_Saturne or similar CFD codes, while Infiniband, Myrinet, or other specialized networks such as those of IBM Blue Genes or Cray XT's are a much better choice (basically, on 30 or so processors, a calculation could be expected to run at only half speed using Gigabit Ethernet compared to a high-performance network, and it robably gets worse for a higher number of processors) Inside a node, local memory bandwith and contention play a similar role. If you have multiple processors, you can expect much better performance with Opteron or Nehalem-type processors than with older Intel processors, as each processor has its own link to memory and they do not content for a shared memory bus. When you use several cores of a same processor, memory contention issues migh also occur Finally, inside a given processor, depending on the size of the local data, cache behavior may be good or bad, and this has a major effect on performance.
All these factors interact, so the only "simple" recommendations would be:
With typical hardward, for10 million cells, you would probably have better than linear speedup running on 128 to 256 processors than on 32 for example, as more data fits into cache (that might be way above the size of the machine you are planning to buy, if you are coming from just 2 processors). Scaling on pre-Nehalem Intel processors will be limited by the fact that processors share the same memory bus, so an AMD Opteron, Intel core I7 or Xeon 5500 series or better are highly recommended (I only consider Intel or AMD here as they are the most mainstream, but If your boss offers to buy you a cluster of IBM Power6's for example, those things can run very fast too).
Also, for better speed (independently of parallelism), and in most cases with no apparent adverse effects on the solution, you may reduce the required precision of the linear solvers from 1O -8 to 10 -5 , and thus gain quite a few iterations.
Best regards,
Yvan
-
Alicia Consigny
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Hi Yvan,
Thanks a lot for all these explanations, I will discuss all that with my boss and see what we can do. Anyway as you understood what I can get is at most a machine with 4 or 8 processors (I am working for a very small engineering office which doesn't have the same means and needs as you may have at EDF !!).
I'm still waiting for an answer from the syrthes team about my post processing problem with Syrthes.
Regards,
Alicia
Thanks a lot for all these explanations, I will discuss all that with my boss and see what we can do. Anyway as you understood what I can get is at most a machine with 4 or 8 processors (I am working for a very small engineering office which doesn't have the same means and needs as you may have at EDF !!).
I'm still waiting for an answer from the syrthes team about my post processing problem with Syrthes.
Regards,
Alicia
-
Alicia Consigny
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Sorry to bother you again... I just found something interesting in my listing files, which I can't figure out what to do with... !!
At the very beginning of all my calculations, I got the following messages :
******************************************** Calculation starting ******************************************** libibverbs: Fatal: couldn't read uverbs ABI version. -------------------------------------------------------------------------- [0,1,0]: OpenIB on host tsl-ordi6 was unable to find any HCAs. Another transport will be used instead, although this may result in lower performance. -------------------------------------------------------------------------- libibverbs: Fatal: couldn't read uverbs ABI version. -------------------------------------------------------------------------- [0,1,1]: OpenIB on host tsl-ordi6 was unable to find any HCAs. Another transport will be used instead, although this may result in lower performance. --------------------------------------------------------------------------
As I don't have any idea of what a HCA might be (and googling HCA just gives millions of results of whatever those initials might stand for), or even openIB, I don't know if this is important in terms of use of memory, and if it is, how to fix it... Did you already encounter that problem ? Do you know what I should do to fix this problem ?
I'm using the 2009 distribution of CAELinux, with Ubuntu 8.04 and Saturne 1.4.
Thanks again for your help, and sorry for all those questions which are not really related to the proper use of code saturne... !!
Alicia
At the very beginning of all my calculations, I got the following messages :
******************************************** Calculation starting ******************************************** libibverbs: Fatal: couldn't read uverbs ABI version. -------------------------------------------------------------------------- [0,1,0]: OpenIB on host tsl-ordi6 was unable to find any HCAs. Another transport will be used instead, although this may result in lower performance. -------------------------------------------------------------------------- libibverbs: Fatal: couldn't read uverbs ABI version. -------------------------------------------------------------------------- [0,1,1]: OpenIB on host tsl-ordi6 was unable to find any HCAs. Another transport will be used instead, although this may result in lower performance. --------------------------------------------------------------------------
As I don't have any idea of what a HCA might be (and googling HCA just gives millions of results of whatever those initials might stand for), or even openIB, I don't know if this is important in terms of use of memory, and if it is, how to fix it... Did you already encounter that problem ? Do you know what I should do to fix this problem ?
I'm using the 2009 distribution of CAELinux, with Ubuntu 8.04 and Saturne 1.4.
Thanks again for your help, and sorry for all those questions which are not really related to the proper use of code saturne... !!
Alicia
- Attachments
-
- batch.xml
- (86.52 KiB) Downloaded 175 times
-
Yvan Fournier
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Hello,
This is an installation issue with OpenMPI (or with CAELinux 2009, as OpenMPi is probably packaged with the rest).
Downloading OpenMPI (www.open-mpi.org) or MPICH2 (http://www-unix.mcs.anl.gov/mpi/mpich2/) and installing them (configure && make && make install), then reinstalling Code_Saturne using the new MPI install would probably solve your problem.
You may also try to force OpenMPI to use shared memory exchanges only (instead of looking for specific libraries/networks as is seems to be trying to do), by setting environment variables or adding the appropriate entries in /etc/openmpi-mca-params.conf. to find the appropriate documentation or help (FAQ, mailing list), see the OpenMPI website (www.open-mpi.org), or check for similar issues with other codes on the CAELinux forums, as this issue is quite upstream of Code_Saturne.
Best regards,
Yvan
This is an installation issue with OpenMPI (or with CAELinux 2009, as OpenMPi is probably packaged with the rest).
Downloading OpenMPI (www.open-mpi.org) or MPICH2 (http://www-unix.mcs.anl.gov/mpi/mpich2/) and installing them (configure && make && make install), then reinstalling Code_Saturne using the new MPI install would probably solve your problem.
You may also try to force OpenMPI to use shared memory exchanges only (instead of looking for specific libraries/networks as is seems to be trying to do), by setting environment variables or adding the appropriate entries in /etc/openmpi-mca-params.conf. to find the appropriate documentation or help (FAQ, mailing list), see the OpenMPI website (www.open-mpi.org), or check for similar issues with other codes on the CAELinux forums, as this issue is quite upstream of Code_Saturne.
Best regards,
Yvan
-
Alicia Consigny
Re: Syrthes : post processing format / tutorial for coupling with Saturne
Hi,
Thanks a lot for your help. I had a look at the mailing list on the OpenMPI website, and found an answer for my problem. I'm not really a pro in all these libraries/tools things, but basically what I understood to be the source of the warning is that open mpi is installed by default taking into account infiniband, and the warning appears when it can not find the infiniband (whether because it is not installed or because the paths are not correct or whatever). I only had to deactivate the infiniband in the file /etc/openmpi-mca-parmas.conf with the line
btl = ^openib
Now the warning at the beginning of the calculation is gone and it doesn't change the performance.
Back to the first topic of this post, I still didn't have any answer from the syrthes team... so I don't want to keep running calculations as I can't see the results, so I don't really know if they're correct ! I sent another e-mail yesterday, and am still waiting. Do you have any idea where I can find the tool syrthes2med ?
As for the versions of Paraview and Ensight, I'm using Paraview 3.3.0, but tried the ensight files I obtained with my syrthes2ensight with Paraview 3.4.2 and 3.8.0 and it doesn't work either. The format of ensight I can load in my Paraviews obtained with Saturne is Ensight Gold and not Ensight. Does it have any influence on the fact I can not load the ensight files in Paraview ? Which versions (Paraview, ensight) are you using when you use Syrthes (if so)?
Regards,
Alicia
Thanks a lot for your help. I had a look at the mailing list on the OpenMPI website, and found an answer for my problem. I'm not really a pro in all these libraries/tools things, but basically what I understood to be the source of the warning is that open mpi is installed by default taking into account infiniband, and the warning appears when it can not find the infiniband (whether because it is not installed or because the paths are not correct or whatever). I only had to deactivate the infiniband in the file /etc/openmpi-mca-parmas.conf with the line
btl = ^openib
Now the warning at the beginning of the calculation is gone and it doesn't change the performance.
Back to the first topic of this post, I still didn't have any answer from the syrthes team... so I don't want to keep running calculations as I can't see the results, so I don't really know if they're correct ! I sent another e-mail yesterday, and am still waiting. Do you have any idea where I can find the tool syrthes2med ?
As for the versions of Paraview and Ensight, I'm using Paraview 3.3.0, but tried the ensight files I obtained with my syrthes2ensight with Paraview 3.4.2 and 3.8.0 and it doesn't work either. The format of ensight I can load in my Paraviews obtained with Saturne is Ensight Gold and not Ensight. Does it have any influence on the fact I can not load the ensight files in Paraview ? Which versions (Paraview, ensight) are you using when you use Syrthes (if so)?
Regards,
Alicia