Debugging

TotalView is a graphical, source-level, multiprocess debugger. When using this debugger you need to turn on X-forwarding, which is done when you login via ssh. This is done by adding the -Y on newer ssh version, and -X on older. Following is an example of using a new version of ssh.

ssh -Y username@hexagon.bccs.uib.no

If you don't know if you have an old or new version of ssh, you should run "man ssh" and look for an explanation of "-X" and/or "-Y".

The program you want to debug has to be compiled with the debug option. Normally this is the "-g" option, but that depends on the compiler. The executable from this compilation will further be called "filename".

First we have to load the totalview module to get the correct environment variables set:

module load xt-totalview

To start debugging run:

totalview "filename"

Which will start a graphical user interface.

Once inside the debugger, if you cannot see any source code, and keep the source files in a separate directory, add the search path to this directory via the main menu item File->Search path.

Source lines where it is possible to insert a breakpoint are marked with a box in the left column. Click on a box to toggle a breakpoint.

Double clicking a function/subroutine name in a source file should open the sourcefile. You can go back to the previous view by clicking on the left arrow on the top of the window.

The button "Go" runs the program from the beginning until the first breakpoint. "Next" and "Step" takes you one line forward. "Out" will continue until the end of the current subroutine/function. "Run to" will continue until the next breakpoint.

The value of variables can be inspected by right clicking on the name, then choose "add to expression list". The variable will now be shown in a pop up window. Scalar variables will be shown with their value, arrays with their dimensions and type. To see all values in the array, right click on the variable in the pop up window and choose "dive". You can now scroll through the list of values. Another option (very useful) is to visualize the array: after choosing "dive", open the menu item "Tools->Visualize" of the pop up window. If you did this with a 2D array, use middle button and drag mouse to rotate the surface that popped up, shift+middle button to pan, Ctrl+middle button to zoom in/out.

click to see full screenshot

Running totalview in batch system (compute nodes)

qsub -I -l mppwidth=[#procs],walltime=[time] -A [account] -j oe -X
mkdir -p /work/$USER/test_dir
cp $HOME/test_dir/a.out /work/$USER/test_dir
cd /work/$USER/test_dir
module load xt-totalview
totalview aprun -a -n [#procs] ./a.out

Replace [#procs] with core-count for the job. Note that totalview is licensed for a limited amount of cores.

When totalview starts it will get 'aprun' up first. Click GO and YES.)

An users guide for totalview can be found here.