The PBS queuing system and job submission

The queuing system

The queuing system used for batch jobs on most of the clusters available through Notur is the TORQUE Resource Manager, from Cluster Resources Inc, which is based on the Open PBS queuing system.

“On top of” Torque we are running the Maui Cluster Scheduler and the Gold Allocation Manager. However, most users should not pay much attention to this.

Basic commands.

Some basic commands that everybody should know before they start running jobs on these systems:

qsub                - to submit a job to the queuing system.

qdel <jobid>        - to delete a job that has been submitted to the queuing system.

showq and qstat     - to get information about the queue situation.

checkjob <jobid>    - detailed information about at job submitted to the queuing system, 
                      especially when a job doesn't start.
cost                - to check how much CPU time that is available on you account.

For a complete reference on these commands, check out their man pages ('man <command>' or '<command> --help').

Useful / advanced commands.

Some commands that are useful to get more detailed / advanced information:

qsig -sNULL <jobid>    - kills jobs that refuse to be killed with qdel.

diagnose -n            - for detailed information about ALL the compute nodes in the cluster

The command 'diagnose' is sometimes very useful to understand why Your job doesn’t start when it looks like there are available resources.

Running interactive jobs in the queuing system.

If you are running jobs outside the queue your jobs will be killed by the support personal. The simplest thing is to run an interactive job through the queuing system using the -I flag, for instance:

qsub -lnodes=1,walltime=1:0:0 -I

this will give you a shell where you can work as normal until you hit the walltime limit (or takes an Exit from it).

Sample job script(s).

When your program has been compiled and works, a batch script should be prepared. Example scripts with PBS job parameters is available here:

• General job script (for stallo.uit.no, but it can be used as an example)
  
• ADF job script (not available yet)
  
• Amber job script (not available yet)
• Gaussian 03 job script (not available yet)

Submit your batch job from the frontend with the command

qsub {your_jobscript}

You get the jobname and -id assigned, which can be used with various commands, for instance the commends listed above.

"Keywords" in the a typical job script.

Headers

• All headers significant to the queuing system start with #PBS.

• All command line options to 'qsub' can be placed in a #PBS comment in the script.
• Command line options take precedence over the options placed in the script.

-lnodes
This keyword is used you specify the number of nodes/CPUs you want to allocate for your job. Be as vague as possible.

An example on two ways to specify that you wants 4 CPU's:

-lnodes=4                  (nodes = servers / machines)

will give you 4 CPU's on the system, and the queuing system is free to decide how to pick them.
It is therefore a more flexible / "better" way to specify it than

-lnodes=2:ppn=2:ib            (ppn = processors per node)

which should be read as

• up to 2 nodes
  
• with at least 2 CPU's per node
  
• and the nodes must be hooked up on the Infiniband interconnect (":ib")
  

The last example (The "better" one) will force the queuing system to give you either 1 node with 4 CPU's, or 2 nodes with 2 CPU's.

The first example is actually equivalent to -lnodes=4:ppn=1

As you understand, the scheduler is very liberal in how it interprets your "processors per node" request. All you know for sure is that when specifying -lnoeds=M:ppn=N you will get MxN CPU's, and you will never get less than N CPU's per node.

Multiple interconnects

The ":ib" option in the example above tells the system that this job needs to use the Infiniband interconnect. If the option is omitted, the job may either run on the Infiniband interconnect or the Gigabit Ethernet (or other interconnects available), but you don't know in advance. If your application doesn't need Infiniband you should omit this option, to avoid being stuck in the queuing system waiting for Infiniband powered nodes becoming available.

-lwalltime
This keyword is used to specify the total time the job will require to finish.

Example:

-lwalltime=10:00:00   - a 10 hours job

Be as accurate as you dare, because: If you play it too safe (specifying a very large walltime) your job will get a very low priority in the queuing system, but if you specify the limit too tight the job might be killed before it is finished. If the limit has been set to tight you can try to contact the support personnel, since they can increase the limit. However, don't relay on that: we are not working 24 hours...

-lpvmem
This keyword is used to specify the amount of memory you need allocated per process for your job.

Please mark that this is a hard limit! If your program exceeds this limit it will be killed bye the queuing system!

Remember that this is the per process memory allocation, so if you have a parallel application requiring 4GB memory and want to run it on 8 CPU's you should specify this as:

-lpvmem=500MB

NB! The queuing system detects that the nodes have 4000 MB (well, some have 8000 and some have 16 000) of memory, which is slightly less than 4 GB. So if specifying 1000 MB the queuing system will be able to pack 4 processes on one node, while specifying 1 GB only will make it able to pack 3 processes on one node.

-lfile
To specify the amount of disk space that needs to be available per process on a compute node. Only necessary if you are using the local work area (i.e. local disk), /work, on the nodes. If using the global work area, /global/work, this should not be neccesary - unless you needs HUGE amounts of disk space.

-lfile=nGB     - to specify that a minimum of n GB must be available per process on a node.

-m

This keyword is used to specify that you wants the queuing system to send you email messages about your job.

-m a      - the queuing sends you an email on job Abort

-m b      - the queuing sends you an email when the job Begins

-m e      - the queuing sends you an email when the job Ends

-m abe    - all three of them

-A
To specify which account you want to use for this job. Only necessay if you have multiple accounts.

-A nn9999k   - my Notur quota for project X
-A nn6666k   - my Notur quota for project Y

pbsdsh - PBS Distributed Shell

pbsdsh [-c copies] [-o] [-s] [-u] [-v] program [args]
pbsdsh [-n node] [-o] [-s] [-u] [-v] program [args]
pbsdsh [-h nodename] [-o] [-v] program [args]

Executes (spawns) a normal Unix program on one or more nodes under control of the Portable Batch System, PBS. When run without the -c or the -n option, pbsdsh will spawn the program on all nodes allocated to the PBS job. The spawns take place concurrently - all execute at (about) the same time. See the manual pages, 'man pbsdsh' , for a complete description.