Jump to content

Compute Storage: Difference between revisions

From HPCwiki
No edit summary
No edit summary
 
Line 33: Line 33:


* <code>/tmp</code> — local disk for temporary caching. Clean up after yourself so your files don't get in the way of other users. You can request a node with enough free space in your sbatch script:
* <code>/tmp</code> — local disk for temporary caching. Clean up after yourself so your files don't get in the way of other users. You can request a node with enough free space in your sbatch script:
<syntaxhighlight lang="bash">
#SBATCH --tmp=<required space>
</syntaxhighlight>


* <code>/dev/shm</code> — a virtual filesystem that lives directly in memory, for extremely fast access. Anything you put here counts against your job's memory, but it is the fastest storage available. After the job ends, the files you put there stay there, so clean up after you are done, otherwise you might have a surprise for follow up jobs failing with strange OUT_OF_MEMORY failures. '''sacct/seff''' do not show this usage.
* <code>/dev/shm</code> — a virtual filesystem that lives directly in memory, for extremely fast access. Anything you put here counts against your job's memory, but it is the fastest storage available. After the job ends, the files you put there stay there, so clean up after you are done, otherwise you might have a surprise for follow up jobs failing with strange OUT_OF_MEMORY failures. '''sacct/seff''' do not show this usage.

Latest revision as of 14:15, 6 November 2026

For active compute work, use the Lustre parallel filesystem rather than your home directory. Lustre is large and fast: it serves files from multiple redundant servers over the fast Omnipath network, which is what you want when many jobs or many nodes read and write at once.

Lustre layout

Lustre is mounted at /lustre and organised by group:

/lustre/[tier]/[partner]/[unit]

for example:

/lustre/backup/WUR/ABGC/

It is divided into several tiers, each with a different backup guarantee and cost:

Path Purpose Backup
/lustre/nobackup The default working area. Cheaper, because the data is stored only once. None — data cannot be recovered if it is lost.
/lustre/backup Working data that needs a safety net. Stored a second time on a separate machine. Disaster-recovery copy only. It may help revert a mistake if you report it very quickly, but there is no guarantee.
/lustre/scratch Temporary working data. None. Files may be removed once the filesystem gets too full (typically after 30 days) — tidy up yourself when work is done.
/lustre/shared Truly shared data that isn't tied to a single person. Same as /lustre/backup, but publicly readable. See Shared Storage.

See Backup Policy for the full picture and Quotas for limits.

Node-local storage

Each worker node also offers fast storage that is local to that node and exists only for the lifetime of your job:

  • /tmp — local disk for temporary caching. Clean up after yourself so your files don't get in the way of other users. You can request a node with enough free space in your sbatch script:
  • /dev/shm — a virtual filesystem that lives directly in memory, for extremely fast access. Anything you put here counts against your job's memory, but it is the fastest storage available. After the job ends, the files you put there stay there, so clean up after you are done, otherwise you might have a surprise for follow up jobs failing with strange OUT_OF_MEMORY failures. sacct/seff do not show this usage.

See also