An introduction to IPM
IPM is a profiling tool that helps you analyse MPI programs. A thing that has been said about IPM in comparison to other profiling tools, is that it gives you "a relative small amount of information for a relative small amount of effort". IPM is in fact very easy to use, it requires no modifications of your code (unless you wan't more information), and you have the opportunity to create nice html output that include graphical representation of the data.
How to run a program with IPM profiling
There is two ways of using IPM. One of the ways, is to set an environment variable before you run your program:
$ export LD_PRELOAD=/path/to/ipm/lib/libipm.so
The other way, is to recompile your program with IPM enabled:
$ mpicc -L/path/to/ipm/lib -lipm your_program.c -o your-program
When you run a program with ipm, there will be created an xml file that you can parse to text or html using ''ipm_parse xmlfile'', ''ipm_parse -full xmlfile'' or ''ipm_parse -html xmlfile''. For more information, see http://ipm-hpc.sourceforge.net/userguide.html.
Using Hardware Performance Monitor Counters (HPM) with IPM
IPM has the ability to hook up to a system's hardware performance monitors. By doing so, we would be able to study for instance cache misses (and many other counters). Using HPM, also means that IPM gets less overhead, and our results are more accurate. To modify what HPM counters to include in the report, modify the HPM_IPM environment variable. For more information, see http://ipm-hpc.sourceforge.net/userguide.html.
At the time of writing, IPM on Njord uses HPM, while IPM on Kongull does not.
Defining regions (optional)
By using the command MPI_Pcontrol, it is possible to partition your program into regions. This regions can then be analysed separately, and they can be compared to each other. Here is an example:
int main(int argc, char **argv)
...
MPI_Pcontrol( 1, "region-a");
function_a();
MPI_Pcontrol(-1, "region-a");
MPI_Pcontrol( 1, "region-b");
function_b();
MPI_Pcontrol(-1, "region-b");
....
MPI_Finalaize();
How to understand the output
Most of the representations in the HTML file from ipm_parse -html should be rather straight forward to understand. Make sure that you run the same program (without modifications) more than once, so that you can separate changes in the output that derive from changes in the code from natural variations.
Try to keep in mind that the statistics you get can be influenced by other programs that is running on the cluster.
More information on understanding the ipm reports, is available in the Tutorial, or in this 1,5 hour video lecture by Nick Wright (free registration required): http://www.sdsc.edu/us/training/elearning/login/index.php
