Compilers, Libraries and Tools

Contents

• Object Mode
• System Compilers
• MPI
• OpenMP
• Hybrid MPI/OpenMP
• The GCC Compiler Suite
• Compiling Programs
• Parallel Programming
• Mixing and IBM XL C/C++
• References
• MPI Programming
• System Math Libraries
• BLAS
• ESSL
• MASS
• Optimization
• Performance Tools
• Debugging
• Compiling a Program in Debug Mode
• Starting Totalview
• Using Totalview
• Other Debugging Resources

 

Refer to the Notur software list to learn what revision of the below software is currently installed on njord.

 

Object Mode

The default object mode on njord is 64-bit, which means you by default builds programs which executes in 64-bit address mode.
To change bit mode specify 32 for the environment variable OBJECT_MODE, e.g. (bash/ksh shells):

 $ export OBJECT_MODE=32

 

System Compilers

XL Fortran, XL C and XL C/C++ comprise the family of XL compilers installed on the system. Use one of the following commands when invoking the compiler

Language	Serial	MPI thread-safe	OpenMP
Fortran 77	xlf	mpxlf_r	xlf_r
Fortran 90	xlf90	mpxlf90_r	xlf90_r
Fortran 95	xlf95	mpxlf95_r	xlf95_r
C	xlc	mpcc_r	xlc_r
C++	xlC	mpCC_r	xlC_r

As a set of standard compile line options for production codes we recommend the following, e.g. for a Fortran program

 $ xlf90_r -O3 -qstrict -qarch=auto -qtune=auto -qcache=auto my_program.f90

Option	Description
-O3	Optimization level, including memory intensive and compile-time intensive optimizations
-qstrict	Ensures that optimizations done by -O3 and greater do not alter the semantics of a program.
-qarch=auto	Specifies the correct target processor that allows the compiler to select more efficient machine instructions and generate instruction sequences that perform best for the selected architecture
-qtune=auto	Tunes instruction selection, scheduling, and other implementation-dependent performance enhancements for the selected hardware architecture
-qcache=auto	Automatically detects the specific cache configuration of the compiling machine.

With auto, the compiler automatically detects the processor type of the compiling machine. For more information see the XL Fortran for AIX library and XL C/C++ for AIX library guides.

 

MPI

Invoking any of the compilers starting with "mp" compile programs which use MPI, e.g.

 $ mpxlf90_r my_mpi_program.f90

 

OpenMP

When compiling code using OpenMP ($OMP) directives specify the -qsmp=omp option, e.g.

 $ xlf90_r -qsmp=omp my_omp_program.f90

If you use xlf_r (F77 compiler) to compile your OpenMP program, you should also use the -qnosave option. For xlf90_r and xlf95_r the -qnosave option is used by default.

For more information see Compiling XL Fortran programs.

 

Hybrid MPI/OpenMP

To compile mixed MPI and OpenMP source codes:

 $ mpxlf90_r -qsmp=omp my_mpi_omp_program.f90

 

The GCC Compiler Suite

On AIX, the IBM XL compilers are the preferred tools for programming i C/C++ and Fortran. The GCC compilers are provided to compile software developed with GCC, which does not easily port to IBM XL C/C++. If performance is important, the necessary time should be invested to port your software to IBM XL C/C++. As to be expected, applications optimized with XL C/C++ usually perform better sequentially and in parallel than their GCC counterparts.

Compiling programs

Like IBM C/C++, the GCC compilers reads OBJECT_MODE from the environment. That is, if the object mode is set to 64, gcc/g++ will create 64-bit XCOFF executables, otherwise 32-bit executables are created. Furthermore, an explicit object mode might be requested by supplying -maix32 or -maix64 on the command line. Refer to the manual page on gcc for further descriptions compiler switches, including power architecture switches.

Parallel programming

It is simple to compile and link parallel programs with GCC. Add -mpe to the command line to build an MPI application. To build an OpenMP application, compile and/or link with -fopenmp. Parallel execution is as for parallel applications developed with XL C/C++, obeying the very same environment variables to run interactively or in batch with the queueing system.

Mixing GCC and IBM XL C/C++

Due to a common back-end, object files compiled with gcc and xlc might be freely mixed. If parts of an application contains code specific to gcc, this part might be compiled with gcc, and the remaining files of the application with xlc. The sample code on mainfunc.c, func1.c and func2.c might be compiled and linked partially with gcc and xlc:

 $ xlc -c func2.c
 $ gcc mainfunc.c func1.c func2.o
 $ ./a.out
         func1() is compiled with gcc 4.4.3
         func2() is compiled with xlc 10.1.0.0
 $ gcc -c func2.c
 $ xlc mainfunc.c func1.c func2.o
 $ ./a.out
          func1() is compiled with xlc 10.1.0.0
          func2() is compiled with gcc 4.4.3

References

Using the GNU C/C++ compiler on AIX

Contains general advice on using gcc/g++ on AIX. Notice GCC on njord reads OBJECT_MODE and works with -mpe, which is not necessarily the case with the GCC RPM's as provided by IBM.

GCC homepage

The ultimate reference to programming with GCC.

 

MPI Programming

If you want to learn about MPI programming on njord.hpc.ntnu.no, then have a look at IBM's redbook: Practical MPI Programming by Scott Vetter, Yukiya Aoyama, Jun Nakano, ISBN: 0738413658.

 

System Math Libraries

BLAS

It is recommended to specify -lxlopt -lblas to link with the Basic Linear Algebra Subprogram (BLAS) library. Prefixing with -lxlopt provides optimized versions of the standard BLAS subroutines xGEMV() and xGEMM().

ESSL

The Engineering and Scientific Subroutine Library (ESSL) is a collection of subroutines providing a wide range of mathematical functions for many different scientific and engineering applications. Its primary characteristics are performance, functional capability, and usability. To link with the ESSL library specify this at command line

 $ xlf90 filename.f90 -lessl

For more information see the ESSL 4.2 and Parallel ESSL 3.2 guides.

MASS

The Mathematical Acceleration Subsystem (MASS) consist of a library of scalar routines and a set of vector libraries tuned for specific architectures. Compiling with -qhot, -O4 or -O5 will cause the compiler to try using a subset of the MASS library. Use

$ xlf ... -lmass -lmassvp5

to explicitly link with the MASS library. For more information on available routines, see Using the Mathematical Acceleration Subsystem (MASS).

 

Optimization

See Optimizing C/C++ Programs and Optimizing Fortran Programs.

 

Performance tools

The IBM High Performance Computing Toolkit is a suite of performance-related tools and libraries to assist in application tuning. This toolkit is an integrated environment for performance analysis of sequential and parallel applications using the MPI and OpenMP paradigms.

• Xprofiler

  used to profile both serial and parallel applications. It uses procedure-profiling information to construct a graphical display of the functions within an application. Xprofiler provides quick access to the profiled data and helps users identify the functions that are the most CPU-intensive.

• Hardware performance monitor (HPM)

  provides reports of events that are critical to performance, e.g. the number of misses on all cache levels.

• MPI Tracer/Profiler

  consists of a set of libraries that collect profiling and tracing data for MPI programs. Performance metrics, such as, the time used by MPI function calls and message sizes, are reported.

• OpenMP Profiler

  dynamically instruments OpenMP applications, and reports performance related information such as timings and the overhead of OpenMP constructs.

 

Debugging

Compiling a Program in Debug Mode

If a program crashes or behaves erroneously, it is usually easiest to find the error if one compiles the program in debug mode. This can be done both with the GCC compiler and with IBM's XL* compilers by adding the -g option to the compile command and avoiding optimizing with more than -O2. However, if you are able to compile your program with IBM's XL* compilers, then there are a few more options for debugging, which are very helpful:

$ xlc -g -qcheck=all -qfullpath ...
$ xlc++ -g -qcheck=all -qfullpath ...
$ xlf -g -qcheck -qfullpath ...

With these options set, programs with errors will in many cases crash as soon as an error occurs. To get the full list of debug options for XL* compilers, type "man xlc" or "man xlf" on a command line on njord and scroll down to the "Error Checking and Debugging Options" section.

 

X11 Forwarding

The Totalview debugger is an X11 application, so you must first make sure that you can display X-windows on your local desktop.

Linux Users

If your are logging in to njord.hpc.ntnu.no from a Linux computer, then it is easy to get X11 forwarding. All you have to do is to add the -X option to your ssh command:

$ ssh -X njord.hpc.ntnu.no

Notice, that there is a difference between lower-case x and upper-case X. -x disables X11 forwarding and -X enables it.

Windows Users

If you used the Putty ssh client to log in to njord.hpc.ntnu.no from a Windows computer, then you must enable X11 forwarding on it, as described on this page.

 

Starting Totalview

Now that you have compiled your program in full debug mode and logged in to njord.hpc.ntnu.no with X11-forwarding enabled, it is time to start the Totalview debugger:

$ cd «directory_where_the_executable_is»
$ module load totalview
$ totalview ./«name_of_executable»

 

Using Totalview

There are many good pages on how to use Totalview on the Internet and most of what is written there also applies to njord.hpc.ntnu.no. See for example:

• Nersc has this page on how to use Totalview on their Bassi supercomputer.
• LLNL has a thorough tutorial on using Totalview here.
• Totalview Technologies has placed the documentation about their Totalview debugger on the Internet here.

 

Other Debugging Resources

• Debugging with GDB (or the curses variant GDB TUI) is a bit limited on AIX as only 32-bit binaries are supported. With 32-bit applications, gdb works as expected from other systems. See for instance this page for a tutorial on how to use GDB.
• dbx Symbolic Debug Program Overview
• Using the pdbx debugger
• IBM's Parallel Environment for AIX 5L V4.1.1 Hitchhiker's Guide also contains a section about using the pdbx debugger. Select the topic "The program runs but..." on the contents list.