Archival Storage - Revision history

Honfi001: /* Group pages */

2026-08-25T12:52:03Z

Group pages

← Older revision		Revision as of 12:52, 25 August 2026
Line 85:		Line 85:
	=== Group pages ===		=== Group pages ===

	Group admins have an additional set of pages for their group scope:		Group admins have a separate additional set of pages for their group scope:
	* '''Group overview'''		* '''Group overview'''
	* '''Group datasets'''		* '''Group datasets'''

Honfi001: /* Group pages */

2026-08-25T12:51:14Z

Group pages

← Older revision		Revision as of 12:51, 25 August 2026
Line 85:		Line 85:
	=== Group pages ===		=== Group pages ===

	Group admins have ~~a separate~~ set of pages for their group scope:		Group admins have an additional set of pages for their group scope:
	* '''Group overview'''		* '''Group overview'''
	* '''Group datasets'''		* '''Group datasets'''

Honfi001 at 12:48, 25 August 2026

2026-08-25T12:48:22Z

← Older revision		Revision as of 12:48, 25 August 2026
Line 53:		Line 53:
	This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.		This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.
	[[File:Screenshot from 2026-03-17 15-20-23.png\|none\|thumb\|1400x1400px]]		[[File:Screenshot from 2026-03-17 15-20-23.png\|none\|thumb\|1400x1400px]]
	Important:		'''Important:'''
	* If a dataset already has an active move candidate, scheduling controls are disabled.		* If a dataset already has an active move candidate, scheduling controls are disabled.
	The dataset list is informational; move decisions are handled through the Schedule page.		The dataset list is informational; move decisions are handled through the Schedule page.

Honfi001 at 12:43, 25 August 2026

2026-08-25T12:43:47Z

← Older revision		Revision as of 12:43, 25 August 2026
Line 29:		Line 29:
	# You will be notified by email that Tapeworm plans to move data you own.		# You will be notified by email that Tapeworm plans to move data you own.
	# You can review and change what should happen, or block the move(s) entirely.		# You can review and change what should happen, or block the move(s) entirely.
	# If no action is taken, after a wait period of 4 weeks, planned moves ~~can~~ become scheduled and then executed.		# If no action is taken, after a wait period of 4 weeks, planned moves will become scheduled and then executed.
	# Data is moved to tape, and removed from <code>/archive</code>.		# Data is moved to tape, and removed from <code>/archive</code>.

Honfi001 at 12:17, 25 August 2026

2026-08-25T12:17:33Z

← Older revision		Revision as of 12:17, 25 August 2026
Line 25:		Line 25:
	# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.		# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.
	# At the dataset discovery step, it groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.		# At the dataset discovery step, it groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.
	# A policy engine checks which discovered datasets look ~~stale~~ (for example: 60+ days old and larger than 1GB).		# A policy engine checks which discovered datasets look inactive (for example: 60+ days old and larger than 1GB).
	# Matching datasets are marked as '''planned''' and shown in your overview.		# Matching datasets are marked as '''planned''' and shown in your overview.
	# You will be notified by email that Tapeworm plans to move data you own.		# You will be notified by email that Tapeworm plans to move data you own.

Honfi001 at 12:16, 25 August 2026

2026-08-25T12:16:29Z

← Older revision		Revision as of 12:16, 25 August 2026
Line 24:		Line 24:

	# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.		# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.
	# A dataset discovery step groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.		# At the dataset discovery step, it groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.
	# A policy engine checks which discovered datasets look stale (for example: 60+ days old and larger than 1GB).		# A policy engine checks which discovered datasets look stale (for example: 60+ days old and larger than 1GB).
	# Matching datasets are marked as '''planned''' and shown in your overview.		# Matching datasets are marked as '''planned''' and shown in your overview.

Honfi001 at 12:13, 25 August 2026

2026-08-25T12:13:17Z

← Older revision		Revision as of 12:13, 25 August 2026
Line 5:		Line 5:
	<code>/archive</code> is a mount that is only accessible from the login nodes, it cannot be used for compute work, and it is only available to WUR users.		<code>/archive</code> is a mount that is only accessible from the login nodes, it cannot be used for compute work, and it is only available to WUR users.

	<code>/archive</code> is backed up ~~to tape~~ on the same schedule as your [[Home Directory\|home directory]] — restorable from roughly a week of history (see [[Backup Policy]]). That backup is separate from the long-term tape archival described below: a backup is a short-term safety copy, while archival moves data off <code>/archive</code> for long-term keeping.		<code>/archive</code> is backed up periodically on the same schedule as your [[Home Directory\|home directory]] — restorable from roughly a week of history (see [[Backup Policy]]). That backup is separate from the long-term tape archival described below: a backup is a short-term safety copy, while archival moves data off <code>/archive</code> for long-term keeping.

	== Tapeworm ==		== Tapeworm ==

Haars001 at 12:12, 14 July 2026

2026-07-14T12:12:27Z

← Older revision		Revision as of 12:12, 14 July 2026
Line 3:		Line 3:
	== The /archive filesystem ==		== The /archive filesystem ==

	<code>/archive</code> is a mount that is only accessible from the login nodes~~. It is cheaper than [[Compute Storage\|Lustre]]~~, ~~but~~ it cannot be used for compute work, and it is only available to WUR users.		<code>/archive</code> is a mount that is only accessible from the login nodes, it cannot be used for compute work, and it is only available to WUR users.

	<code>/archive</code> is backed up to tape on the same schedule as your [[Home Directory\|home directory]] — restorable from roughly a week of history (see [[Backup Policy]]). That backup is separate from the long-term tape archival described below: a backup is a short-term safety copy, while archival moves data off <code>/archive</code> for long-term keeping.		<code>/archive</code> is backed up to tape on the same schedule as your [[Home Directory\|home directory]] — restorable from roughly a week of history (see [[Backup Policy]]). That backup is separate from the long-term tape archival described below: a backup is a short-term safety copy, while archival moves data off <code>/archive</code> for long-term keeping.
Line 141:		Line 141:
	If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.		If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.

	== Manual tape access with iRODS ==		== Manual tape access with iRODS besides Tapeworm ==

	<!-- TODO: clarify how this manual iRODS/itape workflow relates to Tapeworm now that Tapeworm manages /archive → tape automatically. Is itape still the recommended way to push data to tape, or is iRODS now used mainly for retrieving archived datasets? Confirm with FB-IT. -->		Anunna hosts its own iRODS instance, with which you can push data manually to the WUR tape storage for archiving. For general usage information, see https://irods.wur.nl/. The best course of action is to loosely follow the course using your own data, and use your personal space for data upload and transfer to tape.

	Anunna hosts its own iRODS instance, with which you can push data to the WUR tape storage for archiving ~~at very low cost~~. For general usage, see https://irods.wur.nl/. The best course of action is to loosely follow the course using your own data, and use your personal space for data upload and transfer to tape.

	'''Be sure to check whether the data is correctly stored on tape before you remove your data.'''		'''Be sure to check whether the data is correctly stored on tape before you remove your data.'''
Line 158:		Line 156:
	* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trash bin. The advantage is that you can retrieve it if the removal was in error; the disadvantage is that the data will keep costing money. To empty it, see <code>irmtrash -h</code>.		* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trash bin. The advantage is that you can retrieve it if the removal was in error; the disadvantage is that the data will keep costing money. To empty it, see <code>irmtrash -h</code>.

	Because of hardware limitations on the backend tape storage, the size limit per file for our tape archive is 5~~ ~~TB.		Because of hardware limitations on the backend tape storage, the size limit per file for our tape archive is 5 TB.

			== Retrieving data from tape ==
			As the tape storage is maintained by a different team, it is best to follow their documentation on retrieval.

			For that, see here: https://irods.wur.nl/userguide/archiving_research_data_using_icommands/

	== See also ==		== See also ==

Haars001: IA migration §6: expand into Archival Storage — add /archive + iRODS sections, convert →bold, fix backup schedule, frame Tapeworm as a section (via update-page on MediaWiki MCP Server)

2026-06-18T11:50:57Z

IA migration §6: expand into Archival Storage — add /archive + iRODS sections, convert <b>→bold, fix backup schedule, frame Tapeworm as a section (via update-page on MediaWiki MCP Server)
Show changes

Haars001: Haars0011 moved page Tapeworm to Archival Storage: IA migration §6: rename Tapeworm → Archival Storage (leaving redirect) (via move-page on MediaWiki MCP Server)

2026-06-18T11:50:19Z

Haars0011 moved page Tapeworm to Archival Storage: IA migration §6: rename Tapeworm → Archival Storage (leaving redirect) (via move-page on MediaWiki MCP Server)

← Older revision Revision as of 11:50, 18 June 2026
(No difference)

← Older revision	Revision as of 11:50, 18 June 2026
(No difference)