PBS (Portable Batch System) can schedule jobs based on cpu and memory resources. We use Torque and Maui to queue large jobs on whiteout. When you subject a job, you can specify the resources needs (such as number of CPUs, length of time, memory etc...) and the batch system will queue up your job. Your job will be run when your specified resources are available.
Commands:
- qstat / showq – show status of batch jobs.
- qdel – delete a job from the queue. is returned by qsub when submitted.
- qsub – submit a job to the queue. (see below for more information)
- qalter – alter the resources needed for a job (handy if you underestimated the time needed for a job)
- diagnose -? - display diagnostic data about the system.
- Tracejob - show how a job passed through the system.
Other batch commands: qhold, qmsg, qrerun, qrls, qselect, qsig (see man pages)
Example usage:
What's in the queue?
qstat
qstat -a # more information
qstat -f # FULL information
showq # MAUI queue information
Specifying on the command line:
# 2 cpus, 5 gigs of memory, 2 hours
qsub myscript.sh -lncpus=2,mem=5gb,walltime=2:00:00
Altering (some) of the resources of the job:
# change to 5 hours
qalter -lwalltime=5:00:00
Note: this seems to be mostly useful if you have queued up the job but it hasn't started yet. The current scheduler does not let you change the walltime after the job has started.
Using qsub:
qsub can be run from the command line. Options to qsub can be specified on the command line or through comments in a qsub script. Qsub is designed to work with scripts, so if you wish to run a single executable, you will have to wrap it in a script.
The simplest job submission with default resource limits:
% qsub myscript.sh
Specifying resources in command line:
% qsub -l walltime=20:00:00 myscript.sh
(if you leave out the script filename, qsub will read from stdin)
Specifying resources in script:
% cat jobscript
#!/bin/csh
#PBS -l walltime=2:00:00
#PBS -l mem=6gb
#PBS -l ncpus=3
./a.out
You submit this script for execution by PBS using the command:
% qsub jobscript
Note that PBS commands are at the top of the script, there are no blank lines between resources and that there are no other commands until all resources are specified.
Interactive input
Some programs require interactive input. If running from a script, this may be difficult. You can create a file and pipe it to the running program.
Eg:
% cat input
1000
50
% ./a.out < input
or in a script:
% cat jobscript
#!/bin/csh
#PBS -P a99
#PBS -q normal
#PBS -l walltime=20:00:00,vmem=300MB
#PBS -wd
./a.out << EOF
1000
50
EOF
qsub options:
- -p
- specify project
- -l walltime=20:00:00 – specify wall clock time limit for the job.
- -l vmem=??MB – total virtual memory limit for the job. Specify with MB or GB.
- -l ncpus – specify the number of cpus necessary
- -l software – specify what software you are using
- -r y – job is restartable. It is assumed that jobs are not to be restartable as programs have to have checkpointing installed to take advantage of this.
- -I – interactive shell.
- -v - pass on environment variable into job (handy for passing DISPLAY variable)
Interactive Job
It is possible to run an interactive shell from qsub. Once run, it will wait until resources are available for you to enter the shell. This can be useful if you wish to run an interactive process that requires a lot of memory resources and will not run normally in the interactive cpu set.
% qsub -I -l
If you are using X11 display, include “-v DISPLAY” to pass the DISPLAY variable. (you don't need to modify this if you are using SSH to tunnel your X traffic)
eg
% qsub -I -l -v DISPLAY
Scheduling & Fair Share
When specifying resource limits, keep in mind that the batch job processor will only run the job if there is enough of that resource available. Therefore silly memory limits may never be processed.
Priority of jobs in the queue depends on several factors, including how long the job was in the queue and the usage of that particular usage.
It is important that the user minimises the resource requests as much as possible. Otherwise they may be queued for longer than necessary. Experimentation is the key here.
The PBS scheduler has recently been replaced with Maui. One of the handy features of this new scheduler is that it allows the system to automatically kill jobs that go over their memory/processes limit. What this means for the user is that they really need to be careful when specifying the memory limits of your job.
