Compute Storage
Jump to navigation
Jump to search
For active compute work, use the Lustre parallel filesystem rather than your home directory. Lustre is large and fast: it serves files from multiple redundant servers over the fast Omnipath network, which is what you want when many jobs or many nodes read and write at once.
Lustre layout
Lustre is mounted at /lustre and organised by group:
/lustre/[tier]/[partner]/[unit]
for example:
/lustre/backup/WUR/ABGC/
It is divided into several tiers, each with a different backup guarantee and cost:
| Path | Purpose | Backup |
|---|---|---|
/lustre/nobackup |
The default working area. Cheaper, because the data is stored only once. | None — data cannot be recovered if it is lost. |
/lustre/backup |
Working data that needs a safety net. Stored a second time on a separate machine. | Disaster-recovery copy only. It may help revert a mistake if you report it very quickly, but there is no guarantee. |
/lustre/scratch |
Temporary working data. | None. Files may be removed once the filesystem gets too full (typically after 30 days) — tidy up yourself when work is done. |
/lustre/shared |
Truly shared data that isn't tied to a single person. | Same as /lustre/backup, but publicly readable. See Shared Storage.
|
See Backup Policy for the full picture and Quotas for limits.
Node-local storage
Each worker node also offers fast storage that is local to that node and exists only for the lifetime of your job:
/tmp— local disk for temporary caching. Clean up after yourself so your files don't get in the way of other users. You can request a node with enough free space in your sbatch script:
#SBATCH --tmp=<required space>
/dev/shm— a virtual filesystem that lives directly in memory, for extremely fast access. Anything you put here counts against your job's memory, but it is the fastest storage available.