Software Development
Most scientists on the UDE scientific computing systems use off the shelve applications. When these packages are available under open source licenses the HPC team can install these programs for you. In the case of commercial packages, or when you develop your own research software you may have to build your own versions.
In particular when you develop your own codes you should consider how efficiently your code is using the compute resources. While the general approach to performance analysis is similar across different kinds of machines, HPC clusters bring in additional dimensions. Especially when you run calculations across multiple nodes your application will need to communicate data between nodes. This is true in general and therefore a good overview of the code performance requires gathering its characteristics across all nodes, including its communication characteristics.
In response to these issues special performance analysis tools have been developed for HPC applications. This page describes the tools that the HPC team provides on the UDE clusters.
Before we go into details it is important to realize that there are essentially two different kinds of performance reports that you can collect for your code:
Performance profiles
Performance traces
Performance Profiles
A performance profile is a high level overview of how your code behaves. When your code runs statistics are collected that relate to the performance of your code. These statistics might include the total amount of time spent in a particular routine, the amount of data communicated in a particular routine, the amount of time spent computing in a routine versus the amount of time spent waiting, etc. These characteristics will be collected over the entire run of the code across all nodes.
Analysing the profile will give you an idea of how the code performs, for example, which routines take up the majority of the time in your calculation. Nevertheless, the profile is not time resolved nor “spatially” resolved. That is, you might see that a particular routine spends half its time waiting but you cannot see when that happens in the calculation. Does it happen every time the routine invoked, or does it happen only on some invocations? Alternatively, you would not be able to distinguish between a particular routine spending half its time waiting on all nodes, versus a routine spending all of its time waiting on half of the nodes and not waiting at all on the other half of the nodes.
Hence a performance profile lacks a lot of detail but therefore profiles tend to be small in data volume. Whether you run a given code across two nodes for an hour or across a thousand nodes for a week, the volume of data collected will be essentially the same. Usually a profile will be small enough that you can easily copy it to your laptop and analyze it locally.
Performance Traces
Performance traces provide a much more detailed view of how your code behaves. Throughout the run of your calculation data is collected about such things as when the program on a given node called a particular function, and when that function finished, when and which node sent how much data to what other node and when that data was received. In other words performance traces collect all the temporal and spatial data that profiles neglect. Obviously, because of all the additional details being collected trace data grows with the duration of the collection time as well as the number of nodes the calculation was run on. Nevertheless, performance traces may provide you with all the details to diagnose a specific performance problem.
As performance traces collect a lot of detailed information the volume of data recorded tends to be large. Often it is not going to be possible to copy the data off an HPC cluster to anywhere else, let alone your laptop, to analyze it. Special tools can be used to access the trace data remotely from your local machine, which we will describe below.
Performance analysis software
The performance analysis of your code involves three phases.
Build a version of your code suitable for performance data collection.
Run one or more calculations to gather performance data.
Analyze the performance data.
Obviously if you are optimizing the performance of your code then there are additional phases where you diagnose a performance problem, formulate a solution, implement it, test it, and then repeat the performance analysis. At this point in time we don’t provide tools for these additional phases but we might in future. Here we focus on the tools that support the initial three phases.
Dependent on what data you want to collect you might need to prepare your code specifically for that. For example, to attribute execution time to a particular function in a human readible way the executable needs to have the symbol names of each function. To record when a function starts and finishes the executable needs to be instrumented with calls to timing functions that collect timing data. To collect information about data movements the code needs use a communication library that stores such information, etc. With Score-P[1] the code can automatically be prepared for such data collection. Score-P will use special compiler features to insert such instrumentation when the code is compiled. Likewise Score-P provides the infrastructure required to collect the performance data when the code is run. The results may be stored as performance profiles or traces dependent on runtime flags that can be set in environment variables.
Once the data has been collected you will need to analyze it. Profiles can be analysed with tools such as Cube[2]. If needed profiles can even be converted to text and even analyzed manually, trace data typically requires special graphical representations to be intelligible. Vampir[3] is a tool that has been specifically developed for this purpose. It uses a timeline representation for every process in a calculation, showing all or a selection of processes side by side.
Performance analysis setup
As stated above performance profiles are usually small and can be analyzed on almost any machine whether a cluster or a desktop machine. Performance trace files are usually large and cannot be migrated easily. In addition there are cyber security requirements that limit how the data on a cluster can be accessed. As a result the one approach that is currently supported is to start a graphical user interface on the cluster and have the window displayed on your local screen. This does require an X-windows server to be installed on your local machine. More information about what packages are suitable for the operating system on your machine can be found in the section about X-Windows.
Once you have an X-Windows server up and running you can login on a Linux
machine and try to open an X-Windows application. A key thing at this point is
the DISPLAY environment variable. This variable tells the X-Windows
application on which screen to open it’s display. The X-Windows server on your
local machine will define and set this variable. Commonly you would have ssh
forward this variable to the cluster so it knows how to find the screen of
your local machine. Details about setting up your SSH configuration to forward
the DISPLAY variable are in the SSH access section.
Compiling an instrumented version of your code with Score-P
To start collecting performance data about your code it needs to be instrumented
and built first. This can be done using Score-P and here we’ll demonstrate this
on a simple example. This example is part of a collection of “mini-apps”[4].
Here we’ll focus on a matrix-matrix multiplication using MPI as this is
a compact example with non-trivial communication patterns so there will be
something to look at. The matrix-matrix multiplication is implemented in the
mpi-mxm mini app.
The mini apps can be accessed running
git clone https://git.uni-due.de/hpc-support-public/mini-apps.git
cd mini-apps/mpi-mxm
In this directory you’ll find three different kinds of scripts as well as a
small Fortran code consisting of calc-mxm.F and mpi-mxm.F. The code implements
a simple parallel version of DGEMM. DGEMM is the name for double precision
matrix-matrix multiplication in the BLAS[5] library.
The scripts are:
compile-openmpi.shto compile the code with the OpenMPI compiler wrapper scriptscompile-openmpi-scorep.shto compile the code with the OpenMPI compiler wrapper script while instrumenting it with Score-P.submit.shto submit the code to the batch queuerun-openmpi.shto run the program on the compute node with Score-P loaded
To compile the code generating a regular binary, simply run
./compile-openmpi.sh
To compile the code to obtain an instrumented binary, run
./compile-openmpi-scorep.sh
Either compilation script produces a binary called mpi-mxm. This binary takes one
argument that specifies the size of the matrices. For example
./mpi-mxm 1000
will perform the matrix multiplication C = A * B where matrices A, B, and C are all 1000 x 1000 in size.
To run this code enter
submit.sh openmpi
which will submit the run-openmpi.sh to run on the compute nodes. The
run-openmpi.sh always loads the Score-P module as the uninstrumented program
will simply ignore the Score-P related settings. Only the Score-P instrumented
code will utilize the relevant settings while producing the profile and trace data.
The Score-P instrumented code will generate a sub-directory with a name like
scorep-20250804_1530_9532255177661085
The parts
scorepsays this directory was produced by the Score-P infrastructure20250804represents the date when the calculation started, in this case the 4th of August 20251530represents the time when the calculation started, in this case 15:30 hoursthe last set of digits is some (arbitrary) increasing number
the entire string is guaranteed to be unique.
In the directory there are two key files
profile.cubexthe file containing the profile of the runtraces.otf2the file that outlines the available trace files
Analysing the profile
The profile can be analyzed with Cube. To run this code program you first need to load the corresponding module and then run the code
module load cube
cube profile.cubex
this will result in a view like
Cube displays 3 columns:
the left most column lists the various metrics that have been recorded across the entire application (i.e. including all processors) in the profile, such as
Visits for the number of times a routine was called
Time for the amount of time in aggregate spent in routines
Bytes_sent and bytes_received for the amount of data sent and received
the middle column displays the call tree off the application so you can see where in the program what happened
the right most column display the “structure” of the machine
the top level represents the entire machine
the next level down represents the switches that were used, that is the components of the communication infrastructure that moves data around in the machine
then the compute node involved
and finally the MPI ranks on each node.
By selecting different metrics you can look particular behaviours of your code, such as:
Where does your code spend most of its time?
Where does your code generate most of the communications?
Does your code spend any time waiting?
Analyzing the traces
The performance traces can be analyzed with Vampir (allegedly the only German word that contains “mpi” as a sub-string). You can do this be running
module load vampir
vampir traces.otf2
This will result in an image like shown:
In this window on the left the program’s time line is shown. The red bits in the timelines are processors waiting for communication (in this particular case broadcast operations). The green parts are the processors doing computation. The diamonds indicate the start and finish of the broadcast operations. As this job was run on 64 processors the program used an 8x8 processors grid. This implies that there were 8 phases where each processor received data and performed some computation. You can recognize phases as the 8 green parts between the columns of diamonds.
On the right side of the window you can see other properties of the program. The
function summary, for example, shows where the program spent its time. In this
example the code spent most of its time doing the matrix-matrix multiplication. If
you run this example on more processors, or if you make the dimension of the matrices
smaller you will see the data communication in MPI_Bcast becomes more important.
Exercise: try the code with a number of processors that is not the square of an integer
In the example above you will see some interesting behaviour if you choose a number of processors that is not a square of an integer number.
In the profile what do you see happening with the
MPI_Barriertime?In the traces what do you see that is related to previous question?
What trick was played in the program to make the parallelization simple?