An introduction to performance analyses
Performance analysis is a journey you take in order to discover the flaws of your program.
Profiling vs tracing
Profiling involves obtaining overall statistics for your entire application, investigating total runtime, time used on communication etc. Tracing involves obtaining data that can place events along the global timeline. Both strategies will give you the opportunity to analyse and and optimize your parallelization of the program.
Synchronizing load balance
A buzzy bee, is a good bee, a sleeping one, well he is a disgrace. In terms of profiling, you can analyse the load balance of a system by looking at system time vs user time, the amount of memory used on each node and time spent on functions like MPI_Wait etc. When using tracing, the load balance is more about getting a graphic overview of different parts of the program, and try to put all those parts together so that no single process is inactive for too long.
Scalability
When you increase the number of processes you run, the percentage of time spent on communication, will increase (for most programs that use communication that is). Sometimes it might be wise not to super-seed a certain amount of processes/nodes for a given program. Other times you can obtain better scalability for your program by modifying the message size and number of messages, or by running on a cluster with a faster network structure.
Communication analysis
Communication analysis helps you to see how the different processes of a program communicate with each other, and how much data is sent between each of the processes. You can also analyze how much time is used on communication, vs how much time that is spent on computation.
