Compilers, Libraries and Tools
- Object Mode
- System Compilers
- The GCC Compiler Suite
- MPI Programming
- System Math Libraries
Refer to the Notur software list to learn what revision of the below software is currently installed on njord.
The default object mode on njord is 64-bit, which means
you by default builds programs which executes in 64-bit address
To change bit mode specify 32 for the environment variable OBJECT_MODE, e.g. (bash/ksh shells):
$ export OBJECT_MODE=32
XL Fortran, XL C and XL C/C++ comprise the family of XL compilers installed on the system. Use one of the following commands when invoking the compiler
Language Serial MPI thread-safe OpenMP Fortran 77 xlf mpxlf_r xlf_r Fortran 90 xlf90 mpxlf90_r xlf90_r Fortran 95 xlf95 mpxlf95_r xlf95_r C xlc mpcc_r xlc_r C++ xlC mpCC_r xlC_r
As a set of standard compile line options for production codes we recommend the following, e.g. for a Fortran program
$ xlf90_r -O3 -qstrict -qarch=auto -qtune=auto -qcache=auto my_program.f90
Option Description -O3 Optimization level, including memory intensive and compile-time intensive optimizations -qstrict Ensures that optimizations done by -O3 and greater do not alter the semantics of a program. -qarch=auto Specifies the correct target processor that allows the compiler to select more efficient machine instructions and generate instruction sequences that perform best for the selected architecture -qtune=auto Tunes instruction selection, scheduling, and other implementation-dependent performance enhancements for the selected hardware architecture -qcache=auto Automatically detects the specific cache configuration of the compiling machine.
Invoking any of the compilers starting with "mp" compile programs which use MPI, e.g.
$ mpxlf90_r my_mpi_program.f90
When compiling code using OpenMP ($OMP) directives specify the -qsmp=omp option, e.g.
$ xlf90_r -qsmp=omp my_omp_program.f90
If you use xlf_r (F77 compiler) to compile your OpenMP program, you should also use the -qnosave option. For xlf90_r and xlf95_r the -qnosave option is used by default.
For more information see Compiling XL Fortran programs.
To compile mixed MPI and OpenMP source codes:
$ mpxlf90_r -qsmp=omp my_mpi_omp_program.f90
The GCC Compiler Suite
|On AIX, the IBM XL compilers are the preferred tools for programming i C/C++ and Fortran. The GCC compilers are provided to compile software developed with GCC, which does not easily port to IBM XL C/C++. If performance is important, the necessary time should be invested to port your software to IBM XL C/C++. As to be expected, applications optimized with XL C/C++ usually perform better sequentially and in parallel than their GCC counterparts.|
Like IBM C/C++, the GCC compilers reads OBJECT_MODE from the environment. That is, if the object mode is set to 64, gcc/g++ will create 64-bit XCOFF executables, otherwise 32-bit executables are created. Furthermore, an explicit object mode might be requested by supplying -maix32 or -maix64 on the command line. Refer to the manual page on gcc for further descriptions compiler switches, including power architecture switches.
It is simple to compile and link parallel programs with GCC. Add -mpe to the command line to build an MPI application. To build an OpenMP application, compile and/or link with -fopenmp. Parallel execution is as for parallel applications developed with XL C/C++, obeying the very same environment variables to run interactively or in batch with the queueing system.
Mixing GCC and IBM XL C/C++Due to a common back-end, object files compiled with gcc and xlc might be freely mixed. If parts of an application contains code specific to gcc, this part might be compiled with gcc, and the remaining files of the application with xlc. The sample code on mainfunc.c, func1.c and func2.c might be compiled and linked partially with gcc and xlc:
$ xlc -c func2.c $ gcc mainfunc.c func1.c func2.o $ ./a.out func1() is compiled with gcc 4.4.3 func2() is compiled with xlc 10.1.0.0 $ gcc -c func2.c $ xlc mainfunc.c func1.c func2.o $ ./a.out func1() is compiled with xlc 10.1.0.0 func2() is compiled with gcc 4.4.3
- Using the GNU C/C++ compiler on AIX
- Contains general advice on using gcc/g++ on AIX. Notice GCC on njord reads OBJECT_MODE and works with -mpe, which is not necessarily the case with the GCC RPM's as provided by IBM.
- GCC homepage
- The ultimate reference to programming with GCC.
MPI ProgrammingIf you want to learn about MPI programming on njord.hpc.ntnu.no, then have a look at IBM's redbook: Practical MPI Programming by Scott Vetter, Yukiya Aoyama, Jun Nakano, ISBN: 0738413658.
System Math Libraries
It is recommended to specify -lxlopt -lblas to link with the Basic Linear Algebra Subprogram (BLAS) library. Prefixing with -lxlopt provides optimized versions of the standard BLAS subroutines xGEMV() and xGEMM().
The Engineering and Scientific Subroutine Library (ESSL) is a collection of subroutines providing a wide range of mathematical functions for many different scientific and engineering applications. Its primary characteristics are performance, functional capability, and usability. To link with the ESSL library specify this at command line
$ xlf90 filename.f90 -lessl
The Mathematical Acceleration Subsystem (MASS) consist of a library of scalar routines and a set of vector libraries tuned for specific architectures. Compiling with -qhot, -O4 or -O5 will cause the compiler to try using a subset of the MASS library. Use
$ xlf ... -lmass -lmassvp5to explicitly link with the MASS library. For more information on available routines, see Using the Mathematical Acceleration Subsystem (MASS).
The IBM High Performance Computing Toolkit is a suite of performance-related tools and libraries to assist in application tuning. This toolkit is an integrated environment for performance analysis of sequential and parallel applications using the MPI and OpenMP paradigms.
- used to profile both serial and parallel applications. It uses procedure-profiling information to construct a graphical display of the functions within an application. Xprofiler provides quick access to the profiled data and helps users identify the functions that are the most CPU-intensive.
- Hardware performance monitor (HPM)
- provides reports of events that are critical to performance, e.g. the number of misses on all cache levels.
- MPI Tracer/Profiler
- consists of a set of libraries that collect profiling and tracing data for MPI programs. Performance metrics, such as, the time used by MPI function calls and message sizes, are reported.
- OpenMP Profiler
- dynamically instruments OpenMP applications, and reports performance related information such as timings and the overhead of OpenMP constructs.
Compiling a Program in Debug Mode
If a program crashes or behaves erroneously, it is usually easiest to find the error if one compiles the program in debug mode. This can be done both with the GCC compiler and with IBM's XL* compilers by adding the -g option to the compile command and avoiding optimizing with more than -O2. However, if you are able to compile your program with IBM's XL* compilers, then there are a few more options for debugging, which are very helpful:
$ xlc -g -qcheck=all -qfullpath ... $ xlc++ -g -qcheck=all -qfullpath ... $ xlf -g -qcheck -qfullpath ...
With these options set, programs with errors will in many cases crash as soon as an error occurs. To get the full list of debug options for XL* compilers, type "man xlc" or "man xlf" on a command line on njord and scroll down to the "Error Checking and Debugging Options" section.
The Totalview debugger is an X11 application, so you must first make sure that you can display X-windows on your local desktop.
If your are logging in to njord.hpc.ntnu.no from a Linux computer, then it is easy to get X11 forwarding. All you have to do is to add the -X option to your ssh command:
$ ssh -X njord.hpc.ntnu.no
Notice, that there is a difference between lower-case x and upper-case X. -x disables X11 forwarding and -X enables it.
Now that you have compiled your program in full debug mode and logged in to njord.hpc.ntnu.no with X11-forwarding enabled, it is time to start the Totalview debugger:
$ cd «directory_where_the_executable_is» $ module load totalview $ totalview ./«name_of_executable»
There are many good pages on how to use Totalview on the Internet and most of what is written there also applies to njord.hpc.ntnu.no. See for example:
- Nersc has this page on how to use Totalview on their Bassi supercomputer.
- LLNL has a thorough tutorial on using Totalview here.
- Totalview Technologies has placed the documentation about their Totalview debugger on the Internet here.
Other Debugging Resources
- Debugging with GDB (or the curses variant GDB TUI) is a bit limited on AIX as only 32-bit binaries are supported. With 32-bit applications, gdb works as expected from other systems. See for instance this page for a tutorial on how to use GDB.
- dbx Symbolic Debug Program Overview
- Using the pdbx debugger
- IBM's Parallel Environment for AIX 5L V4.1.1 Hitchhiker's Guide also contains a section about using the pdbx debugger. Select the topic "The program runs but..." on the contents list.