Particle tracking: newbie's questions

Questions and remarks about code_saturne usage
Forum rules
Please read the forum usage recommendations before posting.
Antech
Posts: 201
Joined: Wed Jun 10, 2015 10:02 am

Re: Particle tracking: newbie's questions

Post by Antech »

Hello.

I'm sorry, the trunk version really outputs particle statistics on inlets/outlets. Thanks for suggestion and for the program! Other, statistical, variable (Part_bndy_mass_flux) is still zero, but it doesn't matter because it's not for particular boundary. I just have to mention that the statistics for boundary particle parameters is very desirable because changes of mass flow rate from iteration to iteration is very sufficient due to mechanism used (model particles instead of model tracks). For those who work with coal firing an averaged particle composition on outlets (ash, char, volatiles, moisture) is very important to estimate losses with unburnt carbon (for this case with cyclone it is, naturally, not needed).

Regarding the slowdown. This weekend I calculate the same cyclone geometry on different (simpler) mesh to check the mesh dependence (the same is being done with Fluent now). I used Saturne-5 (trunk version) on my home PC with Ubuntu. First time I started with settings I used earlier and when I woke up and checked the solution "my jaw was almost on the floor" how they say in English :). It was incredibly slow and made up to 2000 "linear" iterations for pressure on each top level (fluid flow) iteration. For linear solver, for all equations "BicGStab2" was selected (as in my case for 4.0 version, I didn't change it) because I remember that this algorythm worked better than default in Version 4.0 in other cases (not a cyclone). When I switched to Automatic linear algotythm for all equations the solver's speed became great and it prodused 4000+ iterations on a 1.0M-cells mesh during 2 days. I understand that I understand nothing in these linear algorythms :) but I think that there is a difference in them between versions 4.0 and trunk that makes cases good for 4.0 (with BicGStab2) very slow on new version. If an Automatic settigng for linear algotythm will be good for all cases, we can just forget about this slowdown issue.

Update
I found some inconsistence in resulting cyclone efficiency calculated with various methods on the same variant.
Method 1: Solver output for particle flow through the outlet (Efficiency=100%*[InletFlow-OutletFlow]/InletFlow).
Efficiency is 87%.
Method 2: Outlet particle flow is a numerical integral of [mean_particle_velocity_Y*mean_particle_volume_fraction*2300] over the outlet section (area) in ParaView (2300 kg/m3 is a particle density).
Efficiency is 94%.
I tried to move the slice for outlet section in ParaView, to switch from particle to gas velocity (it's reasonable for small particles like 10 microns or so), to change the number of iterations for averaging solver's particle flow output (because it "jumps" from iteration to iteration significantly). Results (efficiencies) almost didn't change. At the same time, the integral (in method 2) over the inlet (with Velocity_X) gives very good but not excellent agreement with given boundary flow (1 kg/s is given, about 0.98 kg/s is calculated, this is not the real particle flow, but it makes no sense because the fluid flow is frozen). Solver's particle mass flow for the inlet is absolutely as it should be (1 kg/s).
But this difference in efficiencies matters. What method is more precise, what method should I use in practice? May it be that there is something wrong with average volume fractions and velocities of particles in cells near the outlet, or I use these variables incorrectly? Particles at the outlet are distributed in relatively narrow annular area, mesh cell size is big enough compared with radial size of this area.

Update
I found another issues with particles...
1. Boundary particle mass flow information in listing is only a summary for the type of boundary, while to estimate the cyclone efficiency one needs to know particle flow rates through particular BC. It leaves only one useful method with mean volume parts and velocities in cells that may be not so precise as "real" flow rate calculated by solver. I found it when I introduced another one BC at the bottom of the cyclone to consume (deposit) particles, because, in realyty, there is an outlet for particles (but not for gas).
2. Resulting mass flow through the cyclone outlet is strongly dependent on the particle timestep. I work on mesh dependency study, and for "coarse" mesh with the 0.01s timestep for particles I reached almost steady state where there was very few particles passing through the outlet. But when I switched to 0.003s timestep lots of particles came from the bottom part of the cyclone and went to the outlet, particle mass flow at the outlet was several times higher than at the inlet (because all these particles that didn't leave the domain with larger timestep directed up to the outlet). I understand that the CFD is not so easy, but the user has no any clue to decide is this time step for particles is correct or not. Fluent has special parameter that tells the solver to limit spatial particle step with the size of the mesh cell or part of this size (it uses trajectory approach in steady state mode). With different meshes in Fluent, I have different cyclone efficiencies (92...98%), but not so different as with Saturne...

Update
I was lucky to obtain the reasonable efficiency (86%) with Saturne on a coarse mesh with the time step for particles 0.003s (almost the same as on finer mesh that has 3 times more cells, as I said before, the efficiency was 87% if estimated with outlet flow in listing). In previous case, on finer mesh, gas flow pattern was different and particles tended to be on outer walls of the cyclone. On coarse mesh flow pattern changed in some extent and particles began to go up from the bottom of the cyclone to the outlet. But, in reality, there should be less particles in that bottom region, so I introduced the deposit BC there and obtained "good" efficiency (this BC is not so important for fine mesh case).

But the time-step dependence of particle flow through the outlet, that I described before, is an open question.
Antech
Posts: 201
Joined: Wed Jun 10, 2015 10:02 am

Re: Particle tracking: newbie's questions

Post by Antech »

Hello. Sorry for being annoying but I still unable to figure out what is the problem with particle tracking in my case...

First, I get various results with various time steps for particles (fluid flow is frozen). Cyclone efficiency (100% minus fraction of particles leaving the domain through the top outlet) calculated with solvers's outlet particle flow is 86.7% for the time step 0.003s and 80.0% for step 0.0003s. In both cases calculation reached almost steady state (mean particle flow rates). These results are for 3+million cells mesh. For ~1-milion cells mesh efficiency is 86.9% with timestep 0.003s, 82.2% for step 0.001s and 83.9% for step 0.003s. As you can see, the mesh dependence is quite low, but there is a dependency of the time step for particles.

Second, I get inconsistent efficiencies calculated by various ways. For example, for the 3+M cells mesh efficiency is 80.0% if calculated simply by solver's particle flow at the top outlet but if I use Integrate variables filter in ParaView and calculate the particle flow as integral of MeanParticleVelocity[Y]*MeanParticleVolumeFraction*2300*dF I get efficiency 91.0% (2300 kg/m3 is particle density as specified in Saturne case for all fractions, integral is over the slice near the outlet). Both methods give good results for inlet (fro example, 1 kg/s and 0.95 kg/s of particle flow) but for the outlet the results are very different. I tried to move the control slice in ParaView but it didn't change the efficiency significantly (maximum - around 1% of difference), although there was different mesh cell sets in slice.

For background information: Fluent gives 92...93% efficiency for this cyclone and for the same particles except the number of fractions (classes). Fluent provides summary tables for particle statistics on each boundary so there is no problem to get the particle flow from it, I didn't apply the integration method for Fluent results.

Can anybody help me to determine what results are correct? Thanks.
Yvan Fournier
Posts: 4208
Joined: Mon Feb 20, 2012 3:25 pm

Re: Particle tracking: newbie's questions

Post by Yvan Fournier »

Hello,

Did you try the second-order algorithm for particles ? It should be useful mainly for cases such as cyclones. It was severly broken in version 3.0, still broken in 4.0, but should now be working well on trunk (i.e. was "repaired" some 2 months ago). In case your sensitivity to time step is due to a "still too coarse" step for particles, it may help.

Regards,

Yvan
Antech
Posts: 201
Joined: Wed Jun 10, 2015 10:02 am

Re: Particle tracking: newbie's questions

Post by Antech »

Thanks for your response. It seems that you're the only developer keeping the forum alive.

I used only the trunk (5) version for all calculations of efficiencies presented in my previous post (the program itself didn't change). It works now too (tracks particles on another cyclone geometry). So Lagrangian module should be almost up-to-date.

The second-order integration of turbulent dispersion equation is always enabled (and, naturally, the turbulent dispersion is also enabled). I didn't find any other integration order option for particles. And it's difficult to reduce the timestep because with 0.0003...0.001s it already takes too much time (up to two days) to calculate even on small meshes (2.5...3 mln of cells, 4 CPU cores). Velocity magnitude inside the cyclone is up to ~50m/s.
Fluent tracks much more particles on single core and comparable mesh in tens of minutes and outputs consistent data (I checked mesh dependence and variation of maximum number of steps for particles; spatial step for particles in my Fluent cases is limited by mesh cell size so the step-dependence was also checked and was not found). Maybe there is a way to speed-up particle tracking in Saturne to work with smaller steps but in reasonable time? You may say that I should just use Fluent but I need to do some validation time-to-time, CFX (in my opinion) has nothing to deal with particles in serious cases and the only other option for me now is Saturne.

Also, I calculated the particle outlet flow and efficiency for the same cyclone on Fluent results with CFD Post (Fluent gives mass concentration, not volume fraction as Saturne, so formula is Integral(ParticleMassFraction*Velocity*dF), result is in kg/s as expected and as CFD-Post says). I calculated efficiency on Saturne results (in ParaView) using the gas and particle (mean) velocities, results are almost the same, so for Fluent I integrated with gas velocity (normal to boundary component) because it doesn't write particle velocity in results (dat) file. So what is the result? I obtained ~92%! (91+...92+ for various positions of control section) Therefore, for Fluent, this method works and it is still unknown to me what of Saturne results is correct (the solver's particle flow or integral).
Yvan Fournier
Posts: 4208
Joined: Mon Feb 20, 2012 3:25 pm

Re: Particle tracking: newbie's questions

Post by Yvan Fournier »

Hello,

The second order scheme is not for particle turbulent dispersion, but for integration of SDE's but as this is in the same area in the GUI, I guess you have already activated it.

Are you using a debug build or a "default" build ? The performance difference between the 2 can easily be a factor of 3.

Also, could you post a performance.log and timer_stats.csv file, so I can check which parts are slowing the computation down ?

I probably won't be able to do it before a couple of weeks, but I could try to allow using a local time step for particle tracking in case of steady flows (no promise here, I have to check with the experts there are no hidden difficulties or theory issues, but it could speed up things).

Regards,

Yvan
Antech
Posts: 201
Joined: Wed Jun 10, 2015 10:02 am

Re: Particle tracking: newbie's questions

Post by Antech »

Yes, the second order scheme is activated, sorry, I didn't understand terminology correctly (I associated stochastic equations with turbulent dispersion). Local time step for particles would be very nice because the code performs well with local time step for fluid and I usually don't use constant time step in Saturne (because resources don't allow to do so many iterations to provide a local Courant number less that 5...10). On big cluster you use to not to mind about the number of iterations but on PCs it is actual.

About slowdown e.t.c. I don't actually think that it's a bug or so. It's more likely that you only feel it on workstations, because "there's no miracle for workstation" (I still remember your phrase :)). I turned all linear solvers to Automatic, precision is 10^-5 (as suggested on the forum) and fluid iterations advance normal (about 90s for the ~3-mln mesh on 4 cores with RSM turbulence). I didn't activated any debug options for compilation.

I attached an archive with logs from solve directory of one of cases and the simple script used to configure Saturne.
Attachments
RSM-Centered-Particles-DT=0.003.zip
(1.13 MiB) Downloaded 333 times
Yvan Fournier
Posts: 4208
Joined: Mon Feb 20, 2012 3:25 pm

Re: Particle tracking: newbie's questions

Post by Yvan Fournier »

Hello,

Are my eyes tired or did you forget the attachment ?

In frozen-field mode, we could probably save some intermediate results (such as gradients), which would require more memory but save some time. The performance.log can help us see if that would be useful or not.

Also, another Lagrangian expert tells me that if you deactivate turbulent dispersion, you avoid a large part of the stochastic differential equations resolution, which may be faster (not sure about the physical interpretation in that case).

Best regards,

Yvan
Antech
Posts: 201
Joined: Wed Jun 10, 2015 10:02 am

Re: Particle tracking: newbie's questions

Post by Antech »

Hello. The attachment is in place, I checked that it's downloadable. All files are in one archive, including the script "config-catalyst.sh" used to configure Saturne.

I wouldn't like to disable turbulent dispersion because I usually consider it to be important for physics... But, if there will be a version without gradient reconstruction, I'll try to calculate without turbulent dispersion (about half a time or even more is consumed by gradient reconstruction, please see the timer_stats.csv in archive).

Is it possible to make a spatial step for particles to be a part of the mesh cell size? (I mean in future releases of Saturne)
It also may be useful to limit the number of steps or residence time for particles, as in Fluent.

Update
1. I noticed one thing that I cannot understand. If you'll check out the timer_stats.csv in attached archive you'll see that the time for Lagrangian module doesn't depend strongly of the number of particles inside domain (that is proportional to the iteration number). In the log, there is a full run from "no particles" to "steady particle state" (when the average number of incoming/outcoming/depositing particles per iteration is almost stabilized, automatic averaging of particle-related quantities is done with my awk-scripts). So in the beginning of the log steep increase in particle number inside domain takes place, but the time for Lagrangian module per iteration doesn't change accordingly. Why? Does it spend time for something not dependent of the number of model particles?
2. I think I see the reason why Saturne is orders-of-magnitude slower on particles than Fluent, although we always considered Fluent the slow code. If we imagine one model particle that makes 10 steps from inlet to outlet than with the trajectory approach (Fluent, steady state) we must calculate the particle's position 10 times. But if the particle approach is used (Saturne) we must do much more position calculations (Sum(i), i=1...N, i.e. 55 in example). The more particles, the more is relative difference in number of required position calculations. For 1000 particle steps I estimated the time for the "Saturne-like" tracking (100 start points, only from inlet to outlet) as several hours (taking into account gradient reconstruction) while for the same variant in Fluent it takes only several minutes (3000 start points). This is the case if only the time per iteration is dependent on number of particles, that I don't see in log (or I don't understand it correctly). But if this is the case, maybe it's practical to implement trajectory approach too (for steady calculations) and provide the user with choice?

Thanks for attention.
Yvan Fournier
Posts: 4208
Joined: Mon Feb 20, 2012 3:25 pm

Re: Particle tracking: newbie's questions

Post by Yvan Fournier »

Hello,

Just had the time for a short look at your logs, but it seems that (as I had mentioned I suspected), half the time spent is in gradient reconstruction (for gradients of fields which do not change).

Another significant part is in postprocessing (probably in a large part in temporal means).

Gradients could relatively easily be saved in frozen field mode, which would shave almost half of the time here. I would need to check how much of the postprocessing is Lagrangian statistics updates, and how those could be optimized.

A though a bit more about a local time step, but is is not so easy: as time steps are local, mean presence would need to be adapted (maybe not too hard, but needs more checking on theory aspects).

Best regards,

Yvan
Antech
Posts: 201
Joined: Wed Jun 10, 2015 10:02 am

Re: Particle tracking: newbie's questions

Post by Antech »

Hello.
Thanks for you response and attention to my topic.

Local time step for particles will be a very useful option for those who deal with Lagrangian phase. Hope you will implement it in some future release. Is it impossible (or too complex) to add a steady-state (trajectory-based) particle tracking algorithm in Saturne? As I mentioned above, it's much faster than dynamic approach, although it's only suitable for steady state calculations.

I will continue to use Version 5 (trunk) for this cyclone case. The efficiencies calculated by integrating over the outlet section and with outlet particle flow rate from listing are still significantly different in Saturne for various meshes and particle time steps (in Fluent, these methods give the same results, so integrating itself is OK). It's important because I can't figure out what method is correct. What may be the reason? If you want and have time, you can check it out by yourself. I gave the link to CGNS mesh file in this topic and XML case files in attach. You just need to set the inlet velocity to ~15m/s. Density, viscosity, particle classes and numerical settings are already in XML. I suppose you got resources so it will not take long to obtain a flow-field solution and then introduce particles (with time step 0.001s). If the solver's particle flow will differ from the integral of ParticleConcentration*ParticleDensity*Velocity*dF over the outlet section (OUT_UP in CGNS mesh) you will have a "live" case to validate results. Sorry for being annoying and to ask too much, but, possibly, it will help people that work with particles.
Mesh: http://code-saturne.org/forum/viewtopic ... 006#p10859
Archive with XML file: http://code-saturne.org/forum/viewtopic ... =10#p10918
Post Reply