Storage Systems Overview - Revision history

Haars0011: IA migration §6: rewrite Filesystems as the storage overview / landing page; detail content split into Home Directory, Compute Storage, Shared Storage, Archival Storage, Backup Policy, Quotas (via update-page on MediaWiki MCP Server)

2026-06-18T11:54:01Z

IA migration §6: rewrite Filesystems as the storage overview / landing page; detail content split into Home Directory, Compute Storage, Shared Storage, Archival Storage, Backup Policy, Quotas (via update-page on MediaWiki MCP Server)

@@ Line 1: / Line 1: @@
-Anunna currently has multiple filesystem mounts that are available cluster-wide:
+Anunna provides several storage areas, each meant for a different purpose. They differ in size, speed, cost, and — importantly — whether they are backed up. This page is a map of the storage landscape; follow the links for the details of each area.
-== Global ==
+== Storage areas at a glance ==
-* /shared - This mount provides a consistent set of binaries and configuration files for the entire cluster.
+{| class="wikitable"
-* /lustre - This large and fast mount uses the Lustre parallel filesystem to provide files from multiple redundant servers over the fast Omnipath network. Access is provided per group, thus:
+There is also a read-only <code>/shared</code> mount that provides a consistent set of binaries and configuration for the whole cluster; it is maintained centrally and is not for your own data.
-And additionally:
+== Choosing where to put your data ==
-=== Private shared directories ===
+* '''Running jobs?''' Read and write on [[Compute Storage|Lustre]] (<code>/lustre/nobackup</code> by default), not on your home directory.
-If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.
+* '''Keeping something safe?''' Your [[Home Directory|home directory]] and the <code>/lustre/backup</code> tier are backed up; <code>/lustre/nobackup</code> and <code>/lustre/scratch</code> are not. See [[Backup Policy]].
-== Local ==
+Storage is not free; see [[Tariffs]] for costs, and [[Quotas]] for the limits that apply.
 == See also ==
-* [[Tariffs | Costs associated with resource usage]]
+* [[Home Directory]]
+* [[Compute Storage]]
-== External links ==
+* [[Shared Storage]]
-* [http://wiki.lustre.org/index.php/Main_Page Lustre website]
+* [[Archival Storage]]

Haars0011: Haars0011 moved page Filesystems to Storage Systems Overview: IA migration §6/§1: rename Filesystems → Storage Systems Overview (leaving redirect) (via move-page on MediaWiki MCP Server)

2026-06-18T11:53:46Z

Haars0011 moved page Filesystems to Storage Systems Overview: IA migration §6/§1: rename Filesystems → Storage Systems Overview (leaving redirect) (via move-page on MediaWiki MCP Server)

← Older revision	Revision as of 11:53, 18 June 2026
(No difference)

Haars0011 at 15:25, 19 December 2025

2025-12-19T15:25:46Z

← Older revision		Revision as of 15:25, 19 December 2025
Line 50:		Line 50:
	* We have a function to ease uploads (use -h for help) : <code>itape</code>		* We have a function to ease uploads (use -h for help) : <code>itape</code>
	* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.		* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
	* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To ~~fix that, either use <code>irm -f</code> or the icommand to~~ empty it, see <code>irmtrash -h</code>.		* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To empty it, see <code>irmtrash -h</code>.

			Because of hardware limitations on the backend tape storage, the size limit per file for our tape archive is 5T.
	Because of hardware limitations on the backend tape storage, the ~~filesize~~ limit for our tape archive is 5T.

	== See also ==		== See also ==

Prins0891: add 5T filesize limit for tape

2025-11-28T15:15:36Z

add 5T filesize limit for tape

← Older revision		Revision as of 15:15, 28 November 2025
Line 51:		Line 51:
	* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.		* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
	* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.		* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.


			Because of hardware limitations on the backend tape storage, the filesize limit for our tape archive is 5T.

	== See also ==		== See also ==

Haars0011: Clearer descriptions

2025-11-21T14:14:17Z

Clearer descriptions

← Older revision		Revision as of 14:14, 21 November 2025
Line 2:		Line 2:

	== Global ==		== Global ==
	* /home - This mount uses NFS to mount the home directories ~~directly~~ from the active master. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.		* /home - This mount uses NFS to mount the home directories over the slower internal network from the active master. Each user has a 200G quota for this filesystem,it is regularly backed up to tape, and can reliably be restored from up to a week's history. Use this for programs and configuration files.

	* /shared - This mount provides a consistent set of binaries for the entire cluster.		* /shared - This mount provides a consistent set of binaries and configuration files for the entire cluster.

	* /lustre - This large mount uses the Lustre parallel filesystem to provide files from multiple redundant servers. Access is provided per group, thus:		* /lustre - This large and fast mount uses the Lustre parallel filesystem to provide files from multiple redundant servers over the fast Omnipath network. Access is provided per group, thus:
	/lustre/[level]/[partner]/[unit]		/lustre/[level]/[partner]/[unit]
	e.g.		e.g.

Pelle016: typo

2025-11-17T12:36:30Z

typo

← Older revision		Revision as of 12:36, 17 November 2025
Line 46:		Line 46:

	* The zone is HPC		* The zone is HPC
	* With <code>~~init~~</code> you can init the irods env. Use your account password.		* With <code>iinit</code> you can init the irods env. Use your account password.
	* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>		* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>
	* We have a function to ease uploads (use -h for help) : <code>itape</code>		* We have a function to ease uploads (use -h for help) : <code>itape</code>

Pelle016: irods commands added

2025-11-12T16:10:53Z

irods commands added

← Older revision		Revision as of 16:10, 12 November 2025
Line 46:		Line 46:

	* The zone is HPC		* The zone is HPC
	* ~~We have a function to ease uploads :~~ <code>~~itape~~</code>		* With <code>init</code> you can init the irods env. Use your account password.
	* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>		* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>
			* We have a function to ease uploads (use -h for help) : <code>itape</code>
	* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.		* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
	* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.		* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.

Pelle016: add command to help use itape command

2025-11-12T14:10:45Z

add command to help use itape command

← Older revision		Revision as of 14:10, 12 November 2025
Line 47:		Line 47:
	* The zone is HPC		* The zone is HPC
	* We have a function to ease uploads : <code>itape</code>		* We have a function to ease uploads : <code>itape</code>
	* We have aliases to ease checking of the status : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.		* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>
			* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
	* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.		* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.

Haars0011: Added info on how to empty the iRODS trashbin

2025-07-18T09:14:58Z

Added info on how to empty the iRODS trashbin

← Older revision		Revision as of 09:14, 18 July 2025
Line 48:		Line 48:
	* We have a function to ease uploads : <code>itape</code>		* We have a function to ease uploads : <code>itape</code>
	* We have aliases to ease checking of the status : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.		* We have aliases to ease checking of the status : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
			* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.

	== See also ==		== See also ==

Prins0891 at 07:08, 15 July 2025

2025-07-15T07:08:30Z

← Older revision		Revision as of 07:08, 15 July 2025
Line 37:		Line 37:
	With that you can push data to the WUR tape storage for archiving at very low cost.		With that you can push data to the WUR tape storage for archiving at very low cost.

	More info on how to use it, please see https://irods.~~wurnet~~.nl/.		More info on how to use it, please see https://irods.wur.nl/.

	The best course of action is to loosely follow the course, using your own data, and use your personal space for data upload and transfer to tape.		The best course of action is to loosely follow the course, using your own data, and use your personal space for data upload and transfer to tape.