Storage in HPC systems and transferring data

/u Home directory

Note: Rosalind users should refer to documentation provided separately regarding storage.

For kunanyi the home directory file system is provided by a Hierarchical Storage System (HSM).  As disk space gets short the HSM will automatically move files which have been least accessed to tape.  When the file is again accessed it will automatically be retrieved to disk.  This can result in a brief delay before the file can be accessed while the tape is found and read.

The tape system does not perform well with large numbers of files.  Please use “tar” or “zip” utilities to group together large numbers of files into a single archive.  Both utilities will conveniently work on a directory tree.

Your jumpbox home directory is different to your cluster home directory but you can still
access your cluster home directory on jumpbox via /cluster-home/{username}

/scratch

Each cluster has a separate /scratch filesystem on high performance infrastructure suitable for jobs to read and write data from at speed.  /scratch should be considered to be ephemeral i.e. files may be deleted from this file system  with short notice.  There is also very little data resiliency provided by these systems.  It is your responsibility to ensure you have a copy of critical data in your home directory.  Please consider the use of “tar” for this action.

Transferring data into your HPC home directory

For small amounts of data < 100GB you can use ‘scp’ like tools e.g. WinSCP.   Also SmarTTY and MobaXterm both have inbuilt tools to transfer files as well as to make ssh connections.  Transfer the data to jumpbox.tpac.org.au:/cluster-home/{username}

For large amounts of data > 100GB please contact helpdesk@tpac.org.au.