All non-trivial software has bugs, at least at first, so debugging is a part of code development.
When encountering or suspecting a bug, choosing the best debugging technique for the situation can lead to one or more orders of magnitude in time savings.
error*
, run_solver.log
, messages in batch output logs (or in the console) should be checked.error
and error_r*
are present, the latter are the ones which contain the useful information.postprocessing/error*
outputs are also available for boundary conditions and linear solvers.--enable-debug
to configure builds for debug.assert
checks in C code.The GNU debugger https://www.gnu.org/software/gdb is a broadly available, interactive debugger for compiled languages including C, C++, and Fortran.
gdb <executable>
help
for built-in help, q
to quit.b
(set breakpoint),c
(continue to next statement),s
(step into function),p
(print).-g
, and in the case of code_saturne, is provided using the --enable-debug
configure option at installation.When used directly, GDB runs in a single terminal frame, as shown here. Only the current line of code is shown, though the list
command allows showing more.
When started with the -tui
option, GDB runs in a split terminal, with source on top, commands on bottom.
CTRL+x+o
key combination allows changing focus from one to the other.CTRL+l
key allows refreshing the display.GDB may also be run under Emacs, which provides syntax highlighting of source code.
Many graphical front-ends are available for gdb. When evaluating a front-end, we recommend to check for the following features:
The DDD (Data Display Debugger) front-end is obsolete and uses a dated graphical toolkit, but has the advantage of combining a command prompt with graphical tools, and is very easy to use, so it might remain an option.
The Nemiver debugger also has a GDB back-end. It offers a clean display, but lacks the possibility of typing commands; everything must be done using the mouse and menus, which is often tedious. The project seems abandoned. KDbg is similar, slightly more practical, but does not seem to have been very active since 2018.
The gdbgui debugger seems promising, and a good potential successor to DDD. It is based on a web-browser interface,
Full integrated development environments (including Qt Creator, Visual Studio Code, Eclipse, Kdevelop, Anjuta) are outside the scope of this documentation. Most members of the code_saturne development team mostly use lighter, less integrated tools, so will not be able to provide recommendations regarding their use.
The Eclipse CDT and Eclipse PTP (Parallel Tools Platform) environments integrate debuggers, including a parallel debugger, but may use a different syntax than "standalone" GDB, so they are not considered here (though feedback and recommendations around these tools are welcome).
The LLDB debugger is an interesting competitor to GDB, with a different (similar but more verbose) syntax. Is is not as widely available yet, and is not yet handled by the code_saturne debug scripts, though a user familiar with it could of course set it up.
The Valgrind tool suite allows the detection of many memory management (and other) bugs.
Valgrind is easy to run:
valgrind
memcheck
tool.valgrind –tool=/cachegrind/callgrind/drd/massif/...
gdbserver
mode.valgrind –vgdb-error=<number>
Recent versions of the LLVM clang and GCC compilers have additional instrumentation options, allowing memory debugging with a lower overhead than Valgrind.
For the most common errors, use AddressSanitizer, a fast memory error detector.
CFLAGS=-fsanitize=address
FCFLAGS=-fsanitize=address
LDFLAGS=-fsanitize=address
export LD_LIBRARY_FLAGS=<path_to_compiler_libraries
when the compiler is installed in a nonstandard path on older systems.ulimit -c
AddressSanitizer also includes a memory leak checker, which is useful but may also report errors due to system libraries, so to allow a "clean" exit, we may use:
``` export ASAN_OPTIONS=detect_leaks=0 ```
The UndefinedBehaviorSanitizer instrumentation is also useful to detect other types of bugs, such as division by zero, some memory errors, integer overflows, and more.
-lubsan
and even in some cases specify LD_LIBRARY_FLAGS
CFLAGS=-fsanitize=undefined
FCFLAGS=-fsanitize=undefined
LDFLAGS=-fsanitize=undefined
export LD_LIBRARY_FLAGS=<path_to_compiler_libraries
as per AddressSanitizer.Several ways of running code_saturne under a debugger are possible:
domain.debug
setting in cs_user_scripts.py
to automatically run the code under a debugger.RESU
for each run and test.code_saturne run [options] –initialize
then running the debugger manually from the run directory.To allow for debugging parallel runs and combining GDB and Valgrind, GDB is run under a new terminal.
--terminal
option.gnome-terminal
crashes GDB. Running under the default xterm
or konsole
works fine.By default, xterm
will be used. This usually leads to very small, hard to read fonts. This can be fixed by editing $HOME/.Xresources
such as in the following example:
Starting the debugger manually in an execution directory avoids creating many directories and waiting for pre-processing before each run.
cd
to the run directory under RESU/<run_id>
.cat run_solver
to view the execution commands.cs_debug_wrapper.py
script, in the bin
directory of the source tree (and in the lib/python<version>/site-packages/code_saturne
directory of an installed build).cs_debug_wrapper.py --help
for instructions.code_saturne gui <file>
(ignoring the directory warning).code_saturne compile -s src_saturne
to update the cs_solver
executable.The code_saturne debug wrapper does not yet launching GDB under Vim. Various examples of use of that module are found on the web, explaining how Termdebug for example can be used.
Debugging parallel code_saturne runs is not very different from debugging serial runs.
run_solver
script in the exection directory), and ignore the rest of this slide.mpiexec -n 2 <program> : - n 1 <debug_wrapper> <program> : -n 3 <program>
to debug rank 2 of 6cs_debug_wrapper.py
script really becomes useful.print cs_glob_rank_id
Debugging OpenMP data races is much more tricky.
private
attributes in OpenMP pragmas.valgrind –tool=drd –check-stack-var=yes –read-var-info=yes
–disable-linux-futex
configure option, so this requires a special build of GCC.