Main differences between Snowstorm and Stallo.

Login environment.

When you log into stallo.uit.no, you will be logged into stallo-1 or stallo-2 which are the interactive nodes.

Compute nodes.

• Each node has 8 cpu-cores, so they look like an eight-cpu machine.
• 50 nodes have 32GB memory each, the rest have 16GB.
• there are 704 compute nodes where 384 have infiniband interconnect.

Login nodes vs. compute nodes.

On Snowstorm the front-end node, snowstorm.uit.no, had CPU's of a different type than the compute nodes. Therefore you had to log in on a different node, snykov.uit.no, to perform interactive work. So the preferred login node had a different name then the cluster it self, this was very confusing for new users.

But on Stallo we have "done this right” and the login nodes are associated with the name of the cluster. The login nodes on Stallo are also based on the same CPU architecture as the compute nodes, so all kind of interactive work (compiling, testing, etc) should be done on the login nodes. However, the login nodes are not suited for real benchmarking on Stallo, as they are not completely identical to the compute nodes and their performance may therefore be different from the comput enodes.

File system.

Both home and /global/work is on a shared global file system. /global/work should be the preferred file system for jobs, /local/work is also available but has much lower performance than /global/work and the local work area used on snowstorm.

Backup.

Backup are now (February 12th) taken of files stored on /home/<username> on Stallo.

SW.

All SW must be recompiled when moving from Snowstorm to Stallo, since the CPU's are different!

Compilers.

The compilers are the same as on snowstorm, icc and ifort.

MPI library.

The default mpi library is OpenMPI v1.2.4. mpicc/mpif90 use the intel compilers. The job launcher is called orterun (mpirun and mpiexec are just aliases for orterun).

If you for some reason cannot use the mpicc/mpif90 compiler wrappers you need to compile with -I/opt/openmpi/1.2.4/include and link with -L/opt/openmpi/1.2.4/lib -lmpi_f90 -lmpi (you only need mpi_f90 if you are compiling fortran code).

Submitting jobs.

There is no itanium queue, if you have -q itanium in your job script you will have to remove it, you do not need a queue specification at all.

To choose nodes with infiniband interconnect you use the ib flag as on snowstorm: -lnodes=20:ppn=8:ib

Scheduling.

The scheduling system is currently set up to run only jobs for one user per node. This has the advantage of users not stepping on each others toes and we can support grid jobs that tend mostly to be single cpu jobs.