If the application you are running on a node produces output on STDOUT it is important to ensure that this output is captured to a file in your home directory. If it isn’t redirected it will be captured to a file within the node’s local storage which has limited space. If the local storage file system fills up it may cause the job that is running to terminate early or produce inconsistent data.
It is recommended that you use the -e and -o options when running qsub. To ensure these are not forgotten it would be best to create a shell script to start your job as follows:
qsub_job.sh:
#!/bin/bash
#PBS -e errors.txt
#PBS -o output.txt
cmds
Alternatively the two output streams can be joined together. The following sends the error stream to the output stream:
qsub_job.sh:
#!/bin/bash
#PBS -j oe
#PBS -o output.txt
cmds
It is also possible on kunanyi to see the output and error streams while the job is still running using jobtail.sh and jobcat.sh.
