PBS at TPAC
The Portable Batch Systems (PBS) used at TPAC vary between compute systems. Vortex and Katabatic use Maui/Torque package while Eddy and Kunanyi use open-source PBS Pro. The following is intended to be generic across both PBS types.
Quick Syntax guide
|Standard queue status command supplied by PBS. See man qstat for details of options. (But see the local nqstat command below.)|
||Local version of qstat. The queue header of
|qdel jobid||Delete your unwanted jobs from the queues. The jobid is returned by
||Submit jobs to the queues. The simplest use of the
You submit this script for execution by PBS using the command:
You may need to enter data to the program and may be used to doing this interactively when prompted by the program.
There are two ways of doing this in batch jobs.
If, for example, the program requires the numbers 1000 then 50 to be entered when prompted. You can either create a file called, say, input containing these values
then run the program as
or the data can be included in the batch job script as follows:
Notice that the PBS directives are all at the start of the script, that there are no blank lines between them, and there are no other non-PBS commands until after all the PBS directives.
qsub options of note:
|qps jobid||show the processes of a running job|
|qls jobid||list the files in a job’s jobfs directory|
|qcat jobid||show a running job’s stdout, stderr or script|
|qcp jobid||copy a file from a running job’s jobfs directory|
The man pages for these commands on the system detail the various options you will probably need to use.
An SSH client is required to connect to all HPC systems. For windows users the PuTTY client is a good free client but there are many others. Mac OS X users can use the builtin Terminal app.
Your account will have been enabled only for the HPC system appropriate for your project(s).
How to get online help
To find a command or library routine that performs a required function, try searching by keyword e.g.
man -k keyword
Use “man command_name” to find details on how to use a unix command or library e.g.
If no man page is found, try “info command_name” e.g.
Manipulating files and directories
List contents of current directory
Remove file of directory
Make a new directory
Use the “man” command for more information on the above.
A few notes on Unix directory names.
A Unix file full path name is constructed of the directory and subdirectory names separated by slashes “/”. ie. /u/jsmith/work/file1. When you first login you will be in your “home” directory at TPAC this is usually /u/username. In most shells this can also be referenced as ~username.
For example if your username is asmith then “cd ~asmith/work” will take you to the “work” directory in your home directory.
All Unix full path names start with “/”, (There are no Drive/Volume names as in Windows). Hence any filename starting with “/” is a full pathname.
A filename containing one or more slashes “/” will refer to a subdirectory of the “current working directory”. The current working directory may also be referenced as dot “.” i.e. ./subdirectory/file.
The parent of the “current working directory” may be referenced as dot-dot “..”. For example if you have two directories in your home directory work1 and work2 and you cd to work1 you can then change to work2 by typing the command “cd ../work2”.
If the application you are running on a node produces output on STDOUT it is important to ensure that this output is captured to a file in your home directory. If it isn’t redirected it will be captured to a file within the node’s local storage which has limited space. If the local storage file system fills up it may cause the job that is running to terminate early or produce inconsistent data.
It is recommended that you use the -e and -o options when running qsub. To ensure these are not forgotten it would be best to create a shell script to start your job as follows:
qsub -e errors.txt -o output.txt pbs_job.sh
Alternatively the output can be redirected in the pbs_job.sh using the standard STDOUT and STDERR redirection operators. Refer to “What are the shells control and redirection operators” for more information.
Use “qstat -f” to provide information about the current status of the job.
In your job startup script you should redirect STDOUT to a file in your home directory. This may give you information about what the job is doing depending on your application.
The queuing system is now enforcing memory limits on jobs. If you do not specify a “mem” limit for your job you will receive the default mem limit of 600mb. This corresponds to:
#PBS -l mem=600mb
If your job uses more memory per thread than this and you do not explicitly ask for more, your job will be killed by the pbs scheduler. If your job uses less than this and you do not specify, you will simply be telling pbs that your job needs more memory than it actually uses which will make that memory unavailable to other jobs. It is best if your mem setting accurately reflects your jobs actual memory requirements.