Track convergence

Questions and remarks about code_saturne usage
Forum rules
Please read the forum usage recommendations before posting.
Post Reply
SimBu
Posts: 30
Joined: Tue Sep 13, 2022 4:13 pm

Track convergence

Post by SimBu »

Dear users,

I don't know if it happen only to my code_saturne but the track convergence is not refreshing very well, espcially with the refresh time module.

While using the code_saturne gui, and calling track convergence tool from there, only the refresh time area seems not to work.
But when I launch code saturne on an other machine, and use my computer to plot the residuals using code_saturne trackcvg, neither the refresh time or the refresh buttons are working. I have to close the window and launch the trackcvg again and only then it is working perfectly.

I know the residuals are suposed to be updated when the command flush is in the control_file, or just calling touch control_file however I have noticed that the residuals.csv is not updated while the timer_stats.csv is.


Concerning the gui utilisation and the refresh time,
I have some clue :
In the code_saturne/gui/trackcvg/MainView.py :

Code: Select all

self.pushButtonRefresh.clicked.connect(self.slotRefresh)
The Refresh button is linked to slotRefresh wich is calling control_file (see the functions below)

While the refresh time area is only linked to slotRefreshTime (launching a timer). The end of the timer (timeout) is calling updateView,

Code: Select all

self.lineEditTime.textChanged[str].connect(self.slotRefreshTime) #launching a timer (see the function below)
#somewhere else in the file
self.timer.timeout.connect(self.updateView)
And this updateView function never calls the Refresh function, so the refresh time area does not call the control_file.... Do I conceive the refresh time area in the wrong way? I believed the goal of it was to automate the refresh clicking action...

Code: Select all

    def slotRefresh(self):
        """
        """
        if not self.caseName:
            return

        name = os.path.join(self.caseName, 'control_file')
        ficIn= open(name, 'w')
        ficIn.write('flush\n')
        ficIn.close()

Code: Select all

    def slotRefreshTime(self, text):
        """
        Private slot.
        """
        if self.lineEditTime.validator().state == QValidator.Acceptable:
            time = from_qvariant(text, float)
            self.timeRefresh = time
            if self.timeRefresh != 0.:
                self.timer.start(self.timeRefresh * 1000)
            else:
                self.timer.start(1.e8 * 1000)

Code: Select all

    def updateView(self):
        """
        """
        self.dc.clear()
        for (name, fle, status, subplot_id, probes_number) in self.fileList:
            if status == "on" or status == "onoff":
                base, ext = os.path.splitext(fle)
                if name == 'residuals.csv':
                    data = self.ReadCsvFile(fle, probes_number)
                    if status == "on" or status == "onoff":
                        self.dc.update_figure_listing(self.listingVariable,
                                                      data,
                                                      probes_number,
                                                      self.listFileProbes[name])
                elif name == 'residuals.dat':
                    data = self.ReadDatFile(fle, probes_number)
                    if status == "on" or status == "onoff":
                        self.dc.update_figure_listing(self.listingVariable,
                                                      data, probes_number,
                                                      self.listFileProbes[name])
                elif ext == ".csv":
                    data = self.ReadCsvFile(fle, probes_number)
                    nm, ext = os.path.splitext(name)
                    nm = nm[7:]
                    self.dc.update_figure(nm, data, probes_number,
                                          self.listFileProbes[name])
                elif ext == ".dat":
                    data = self.ReadDatFile(fle, probes_number)
                    nm, ext = os.path.splitext(name)
                    nm = nm[7:]
                    self.dc.update_figure(nm, data, probes_number,
                                          self.listFileProbes[name])
        self.dc.drawFigure()


Concerning the manual flush in control_file,
I do not understand why all the files are flushing exept the residuals wich are not updated (it does sometime but not all the time)... I tried to read the residuals log code but without success, does someone has the same issue ?
The residuals.csv is however updated when launching trackcvg (but then it won't with the refresh button)...

Otherwise I will just plot the residuals from the listing file but that is a pity because that was the purpose of the residuals.csv...

Many thanks in advance,
Regards,
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: Track convergence

Post by Yvan Fournier »

Hello,

It is strange that all files are flushing except residuals.csv. Does timer_stats.csv also flush ? The amount of output added to each at each time step is quite similar, so we could expect a similar flush behavior (while other files such as run_solver.log might have more output per time step and naturally be flushed more often. We have also observed incorrect flushing is some rare cases, which might be related to the filesystem (i..e do you have this problem on a simple workstation or on a cluster with a parallel file system) ?

There is an option to change the flushing behavior of various files such as probes, using the "cs_time_plot_set_flush_default" function, but the behavior should then be the same for all .csv or .dat files (probes, profiles, residuals, timer stats).

I'll check for the rest. I is possible that some automation is missing.

Also, the performance of the trackcvg took is known not to be good. There are 2 possible reasons, which we need to test for:
- Matplotlib might not be as fast for such purposes as some alternative options, such as qtcharts (not easy to test without re-coding, though comparing external benchmarks might help))
- When we update the trackcvg view, we re-read the whole file, instead of incrementally adding the new lines. Doing this properly would require keeping track of the read size, to add just what is needed, but would probably help quite a bit. I would actually expect this to be the main performance issue part.

Best regards,

Yvan
SimBu
Posts: 30
Joined: Tue Sep 13, 2022 4:13 pm

Re: Track convergence

Post by SimBu »

Thank you for your fast answer,

Yes indeed, the timer_stat is updated but not the residuals. I run the simulation on a cluster, and the touch control_file from a VM on my computer. At first I believed it was a owner file issue, but the timer_stat is actualised... Maybe it is linked with the parallel run you are right.

Ok, I did not know about the trackcvg tool, do you have an alternative ? I am not very familiar with Qt so...

By the way, I wrote a little c function that print the residuals in the terminal thanks to a

Code: Select all

cs_solving_info_t sinfo;
//some code
printf("Residuals %s : %f \n", f->name, sinfo.l2residual);
Does it slow my simulation a lot or it is just about milliseconds?
Yvan Fournier
Posts: 4070
Joined: Mon Feb 20, 2012 3:25 pm

Re: Track convergence

Post by Yvan Fournier »

Hello,

In most cases, the print statement you added will be buffered, and you also need to flush it (though it might work).

The performance hit really depends on the cluster's filesystem and how it is parametrized. Some years ago, on an IBM Blue Gene machine, if I forced the flush at each iteration for a test case I experimented with, I had a 5% performance hit per time step if I remember correctly (which is why we do not flush by default).

You best bet is to run you simulation with and without that addition for a few (10-30) iterations, and compare the performance (assuming you run on the same set of nodes and with the same partitioning, as this might have as much or more impact on performance).

Best regards,

Yvan
SimBu
Posts: 30
Joined: Tue Sep 13, 2022 4:13 pm

Re: Track convergence

Post by SimBu »

Ok, thank you very much, I'll try but indeed it seems that I need to flush anyway !
Post Reply