Compilers, Libraries and Tools
by Henrik R. Nagel — last modified Mar 07, 2011 04:20 PM
Refer to the Notur software list to learn what revision of the below software is currently installed on njord.
The default object mode on njord is 64-bit, which means
you by default builds programs which executes in 64-bit address
$ export OBJECT_MODE=32
XL Fortran, XL C and XL C/C++ comprise the family of XL compilers installed on the system. Use one of the following commands when invoking the compiler
$ xlf90_r -O3 -qstrict -qarch=auto -qtune=auto -qcache=auto my_program.f90
Invoking any of the compilers starting with "mp" compile programs which use MPI, e.g.
$ mpxlf90_r my_mpi_program.f90
When compiling code using OpenMP ($OMP) directives specify the -qsmp=omp option, e.g.
$ xlf90_r -qsmp=omp my_omp_program.f90
If you use xlf_r (F77 compiler) to compile your OpenMP program, you should also use the -qnosave option. For xlf90_r and xlf95_r the -qnosave option is used by default.
For more information see Compiling XL Fortran programs.
To compile mixed MPI and OpenMP source codes:
$ mpxlf90_r -qsmp=omp my_mpi_omp_program.f90
The GCC Compiler Suite
Like IBM C/C++, the GCC compilers reads OBJECT_MODE from the environment. That is, if the object mode is set to 64, gcc/g++ will create 64-bit XCOFF executables, otherwise 32-bit executables are created. Furthermore, an explicit object mode might be requested by supplying -maix32 or -maix64 on the command line. Refer to the manual page on gcc for further descriptions compiler switches, including power architecture switches.
It is simple to compile and link parallel programs with GCC. Add -mpe to the command line to build an MPI application. To build an OpenMP application, compile and/or link with -fopenmp. Parallel execution is as for parallel applications developed with XL C/C++, obeying the very same environment variables to run interactively or in batch with the queueing system.
Mixing GCC and IBM XL C/C++Due to a common back-end, object files compiled with gcc and xlc might be freely mixed. If parts of an application contains code specific to gcc, this part might be compiled with gcc, and the remaining files of the application with xlc. The sample code on mainfunc.c, func1.c and func2.c might be compiled and linked partially with gcc and xlc:
$ xlc -c func2.c $ gcc mainfunc.c func1.c func2.o $ ./a.out func1() is compiled with gcc 4.4.3 func2() is compiled with xlc 10.1.0.0 $ gcc -c func2.c $ xlc mainfunc.c func1.c func2.o $ ./a.out func1() is compiled with xlc 10.1.0.0 func2() is compiled with gcc 4.4.3
MPI ProgrammingIf you want to learn about MPI programming on njord.hpc.ntnu.no, then have a look at IBM's redbook: Practical MPI Programming by Scott Vetter, Yukiya Aoyama, Jun Nakano, ISBN: 0738413658.
System Math Libraries
It is recommended to specify -lxlopt -lblas to link with the Basic Linear Algebra Subprogram (BLAS) library. Prefixing with -lxlopt provides optimized versions of the standard BLAS subroutines xGEMV() and xGEMM().
The Engineering and Scientific Subroutine Library (ESSL) is a collection of subroutines providing a wide range of mathematical functions for many different scientific and engineering applications. Its primary characteristics are performance, functional capability, and usability. To link with the ESSL library specify this at command line
$ xlf90 filename.f90 -lessl
The Mathematical Acceleration Subsystem (MASS) consist of a library of scalar routines and a set of vector libraries tuned for specific architectures. Compiling with -qhot, -O4 or -O5 will cause the compiler to try using a subset of the MASS library. Use
$ xlf ... -lmass -lmassvp5to explicitly link with the MASS library. For more information on available routines, see Using the Mathematical Acceleration Subsystem (MASS).
The IBM High Performance Computing Toolkit is a suite of performance-related tools and libraries to assist in application tuning. This toolkit is an integrated environment for performance analysis of sequential and parallel applications using the MPI and OpenMP paradigms.
Compiling a Program in Debug Mode
If a program crashes or behaves erroneously, it is usually easiest to find the error if one compiles the program in debug mode. This can be done both with the GCC compiler and with IBM's XL* compilers by adding the -g option to the compile command and avoiding optimizing with more than -O2. However, if you are able to compile your program with IBM's XL* compilers, then there are a few more options for debugging, which are very helpful:
$ xlc -g -qcheck=all -qfullpath ... $ xlc++ -g -qcheck=all -qfullpath ... $ xlf -g -qcheck -qfullpath ...
With these options set, programs with errors will in many cases crash as soon as an error occurs. To get the full list of debug options for XL* compilers, type "man xlc" or "man xlf" on a command line on njord and scroll down to the "Error Checking and Debugging Options" section.
The Totalview debugger is an X11 application, so you must first make sure that you can display X-windows on your local desktop.
If your are logging in to njord.hpc.ntnu.no from a Linux computer, then it is easy to get X11 forwarding. All you have to do is to add the -X option to your ssh command:
$ ssh -X njord.hpc.ntnu.no
Notice, that there is a difference between lower-case x and upper-case X. -x disables X11 forwarding and -X enables it.
Now that you have compiled your program in full debug mode and logged in to njord.hpc.ntnu.no with X11-forwarding enabled, it is time to start the Totalview debugger:
$ cd «directory_where_the_executable_is» $ module load totalview $ totalview ./«name_of_executable»
There are many good pages on how to use Totalview on the Internet and most of what is written there also applies to njord.hpc.ntnu.no. See for example:
Other Debugging Resources