Storage Systems Overview: Difference between revisions

From HPCwiki
Jump to navigation Jump to search
← Older edit
Haars0011 (talk | contribs)
Bureaucrats, Administrators
261 edits
VisualWikitext
IA migration §6: rewrite Filesystems as the storage overview / landing page; detail content split into Home Directory, Compute Storage, Shared Storage, Archival Storage, Backup Policy, Quotas (via update-page on MediaWiki MCP Server)
 
(36 intermediate revisions by 7 users not shown)
Line 1: Line 1:
== Fast Scratch ==
Anunna provides several storage areas, each meant for a different purpose. They differ in size, speed, cost, and — importantly — whether they are backed up. This page is a map of the storage landscape; follow the links for the details of each area.


On the Lustre PFS scratch space is organised per partner. Users can only create directory and files in the folders of the organisation they belong to. The Fast Scratch is meant for temporary files and folders. Files and folders should be removed once the jobs are finished. Files and folder older than one month will automatically be removed. Since the Fast Scratch is in an integrated part of the compute infrastructure, no additional cost is incurred based on use in either throughput or volume stored.
== Storage areas at a glance ==


  /lustre/scratch/[partner]/[unit]
{| class="wikitable"
e.g.
|-
  /lustre/scratch/WUR/ABGC/
! Area !! Use it for !! Backed up? !! Details
|-
| <code>/home</code> || Personal files: programs, scripts, configuration, small results. Reliable but slower. || Yes || [[Home Directory]]
|-
| <code>/lustre/nobackup</code> || The default area for active compute work. Large and fast. || No || [[Compute Storage]]
|-
| <code>/lustre/backup</code> || Compute data that also needs a safety copy. || Yes (disaster-recovery copy) || [[Compute Storage]]
|-
| <code>/lustre/scratch</code> || Short-lived working data. || No (auto-purged) || [[Compute Storage]]
|-
| <code>/lustre/shared</code> || Data shared between people or groups. || Yes || [[Shared Storage]]
|-
| <code>/archive</code> || Keeping data you are no longer computing on (login nodes, WUR users). Cheaper than Lustre. || Yes || [[Archival Storage]]
|-
| <code>/tmp</code>, <code>/dev/shm</code> || Fast, node-local scratch for the lifetime of a job. || No || [[Compute Storage#Node-local storage|Compute Storage]]
|}


== Fast Protected ==
There is also a read-only <code>/shared</code> mount that provides a consistent set of binaries and configuration for the whole cluster; it is maintained centrally and is not for your own data.
Data that needs to remain on the Lustre PFS *and* needs to be backed up as well (i.e. requires redundancy in case the PFS experiences a fatal failure) can be placed in the Fast Protected area.


  /lustre/backup/[name partner]
== Choosing where to put your data ==


e.g.  
* '''Running jobs?''' Read and write on [[Compute Storage|Lustre]] (<code>/lustre/nobackup</code> by default), not on your home directory.
* '''Keeping something safe?''' Your [[Home Directory|home directory]] and the <code>/lustre/backup</code> tier are backed up; <code>/lustre/nobackup</code> and <code>/lustre/scratch</code> are not. See [[Backup Policy]].
* '''Done computing, but want to keep the data?''' Move it to [[Archival Storage|/archive]], where long-term data is archived to tape.
* '''Sharing with a team?''' See [[Shared Storage]].


  /lustre/backup/WUR�/ABGC/
Storage is not free; see [[Tariffs]] for costs, and [[Quotas]] for the limits that apply.


Note that this map will not be backed up yet. It is planned that daily syncing will commence 1-1-2014.
== See also ==
 
* [[Home Directory]]
== fast unprotected ==
* [[Compute Storage]]
== fast shared ==
* [[Shared Storage]]
== compute nodes scratch ==
* [[Archival Storage]]
* [[Backup Policy]]
* [[Quotas]]
* [[Tariffs]]

Latest revision as of 11:54, 18 June 2026

Anunna provides several storage areas, each meant for a different purpose. They differ in size, speed, cost, and — importantly — whether they are backed up. This page is a map of the storage landscape; follow the links for the details of each area.

Storage areas at a glance

Area Use it for Backed up? Details
/home Personal files: programs, scripts, configuration, small results. Reliable but slower. Yes Home Directory
/lustre/nobackup The default area for active compute work. Large and fast. No Compute Storage
/lustre/backup Compute data that also needs a safety copy. Yes (disaster-recovery copy) Compute Storage
/lustre/scratch Short-lived working data. No (auto-purged) Compute Storage
/lustre/shared Data shared between people or groups. Yes Shared Storage
/archive Keeping data you are no longer computing on (login nodes, WUR users). Cheaper than Lustre. Yes Archival Storage
/tmp, /dev/shm Fast, node-local scratch for the lifetime of a job. No Compute Storage

There is also a read-only /shared mount that provides a consistent set of binaries and configuration for the whole cluster; it is maintained centrally and is not for your own data.

Choosing where to put your data

  • Running jobs? Read and write on Lustre (/lustre/nobackup by default), not on your home directory.
  • Keeping something safe? Your home directory and the /lustre/backup tier are backed up; /lustre/nobackup and /lustre/scratch are not. See Backup Policy.
  • Done computing, but want to keep the data? Move it to /archive, where long-term data is archived to tape.
  • Sharing with a team? See Shared Storage.

Storage is not free; see Tariffs for costs, and Quotas for the limits that apply.

See also

Retrieved from "https://wiki.anunna.wur.nl/index.php?title=Storage_Systems_Overview&oldid=2819"