Storage Systems Overview: Difference between revisions

From HPCwiki
Jump to navigation Jump to search
← Older edit
Haars0011 (talk | contribs)
Bureaucrats, Administrators
261 edits
VisualWikitext
No edit summary
IA migration §6: rewrite Filesystems as the storage overview / landing page; detail content split into Home Directory, Compute Storage, Shared Storage, Archival Storage, Backup Policy, Quotas (via update-page on MediaWiki MCP Server)
 
(25 intermediate revisions by 6 users not shown)
Line 1: Line 1:
The AgroGenomics HPC currently has multiple filesystem mounts that are available cluster-wide:
Anunna provides several storage areas, each meant for a different purpose. They differ in size, speed, cost, and — importantly — whether they are backed up. This page is a map of the storage landscape; follow the links for the details of each area.


* /home - This mount uses NFS to mount the home directories directly from nfs01. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.
== Storage areas at a glance ==


* /cm/shared - This mount provides a consistent set of binaries for the entire cluster.
{| class="wikitable"
|-
! Area !! Use it for !! Backed up? !! Details
|-
| <code>/home</code> || Personal files: programs, scripts, configuration, small results. Reliable but slower. || Yes || [[Home Directory]]
|-
| <code>/lustre/nobackup</code> || The default area for active compute work. Large and fast. || No || [[Compute Storage]]
|-
| <code>/lustre/backup</code> || Compute data that also needs a safety copy. || Yes (disaster-recovery copy) || [[Compute Storage]]
|-
| <code>/lustre/scratch</code> || Short-lived working data. || No (auto-purged) || [[Compute Storage]]
|-
| <code>/lustre/shared</code> || Data shared between people or groups. || Yes || [[Shared Storage]]
|-
| <code>/archive</code> || Keeping data you are no longer computing on (login nodes, WUR users). Cheaper than Lustre. || Yes || [[Archival Storage]]
|-
| <code>/tmp</code>, <code>/dev/shm</code> || Fast, node-local scratch for the lifetime of a job. || No || [[Compute Storage#Node-local storage|Compute Storage]]
|}


* /lustre - This large mount uses the Lustre filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
There is also a read-only <code>/shared</code> mount that provides a consistent set of binaries and configuration for the whole cluster; it is maintained centrally and is not for your own data.
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of three major parts (and some minor):
* /lustre/backup - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/nobackup - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recivered.
* /lustre/scratch - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.
* /lustre/shared - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.


== Choosing where to put your data ==


Specific to certain machines are some other filesystems that are available to you:
* '''Running jobs?''' Read and write on [[Compute Storage|Lustre]] (<code>/lustre/nobackup</code> by default), not on your home directory.
* /archive - an archive mount only accessible from nfs01. Files here are sent to the Isilon for deeper storage. The cost of storing data here is much less than on the Lustre, but it cannot be used for compute work. This is only available to WUR users, as the Isilon is unable to resolve local groups (without additional work).
* '''Keeping something safe?''' Your [[Home Directory|home directory]] and the <code>/lustre/backup</code> tier are backed up; <code>/lustre/nobackup</code> and <code>/lustre/scratch</code> are not. See [[Backup Policy]].
 
* '''Done computing, but want to keep the data?''' Move it to [[Archival Storage|/archive]], where long-term data is archived to tape.
* /tmp - On each worker node there is a (large) /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become accessible to other users.
* '''Sharing with a team?''' See [[Shared Storage]].
 
* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.


Storage is not free; see [[Tariffs]] for costs, and [[Quotas]] for the limits that apply.


== See also ==
== See also ==
* [[B4F_cluster | B4F cluster]]
* [[Home Directory]]
 
* [[Compute Storage]]
== External links ==
* [[Shared Storage]]
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]
* [[Archival Storage]]
* [[Backup Policy]]
* [[Quotas]]
* [[Tariffs]]

Latest revision as of 11:54, 18 June 2026

Anunna provides several storage areas, each meant for a different purpose. They differ in size, speed, cost, and — importantly — whether they are backed up. This page is a map of the storage landscape; follow the links for the details of each area.

Storage areas at a glance

Area Use it for Backed up? Details
/home Personal files: programs, scripts, configuration, small results. Reliable but slower. Yes Home Directory
/lustre/nobackup The default area for active compute work. Large and fast. No Compute Storage
/lustre/backup Compute data that also needs a safety copy. Yes (disaster-recovery copy) Compute Storage
/lustre/scratch Short-lived working data. No (auto-purged) Compute Storage
/lustre/shared Data shared between people or groups. Yes Shared Storage
/archive Keeping data you are no longer computing on (login nodes, WUR users). Cheaper than Lustre. Yes Archival Storage
/tmp, /dev/shm Fast, node-local scratch for the lifetime of a job. No Compute Storage

There is also a read-only /shared mount that provides a consistent set of binaries and configuration for the whole cluster; it is maintained centrally and is not for your own data.

Choosing where to put your data

  • Running jobs? Read and write on Lustre (/lustre/nobackup by default), not on your home directory.
  • Keeping something safe? Your home directory and the /lustre/backup tier are backed up; /lustre/nobackup and /lustre/scratch are not. See Backup Policy.
  • Done computing, but want to keep the data? Move it to /archive, where long-term data is archived to tape.
  • Sharing with a team? See Shared Storage.

Storage is not free; see Tariffs for costs, and Quotas for the limits that apply.

See also

Retrieved from "https://wiki.anunna.wur.nl/index.php?title=Storage_Systems_Overview&oldid=2819"