Archival Storage - Revision history

Haars0011: IA migration §6: expand into Archival Storage — add /archive + iRODS sections, convert →bold, fix backup schedule, frame Tapeworm as a section (via update-page on MediaWiki MCP Server)

2026-06-18T11:50:57Z

IA migration §6: expand into Archival Storage — add /archive + iRODS sections, convert →bold, fix backup schedule, frame Tapeworm as a section (via update-page on MediaWiki MCP Server)
Show changes

Haars0011: Haars0011 moved page Tapeworm to Archival Storage: IA migration §6: rename Tapeworm → Archival Storage (leaving redirect) (via move-page on MediaWiki MCP Server)

2026-06-18T11:50:19Z

Haars0011 moved page Tapeworm to Archival Storage: IA migration §6: rename Tapeworm → Archival Storage (leaving redirect) (via move-page on MediaWiki MCP Server)

← Older revision Revision as of 11:50, 18 June 2026
(No difference)

Prins0891 at 09:26, 19 March 2026

2026-03-19T09:26:45Z

← Older revision Revision as of 09:26, 19 March 2026
Line 1: Line 1:

= Tapeworm: Automated tape-archival of old datasets =
= Tapeworm: Automated tape-archival of old datasets =

~~<div style="border:2px solid #d32f2f; background:#ffebee; padding:12px; margin:12px 0;">~~

~~Under construction:~~

~~<ul style="margin:8px 0 0 18px;">~~

~~<li>This documentation page is under construction and may still contain some gaps.</li>~~

~~<li>The Tapeworm application will be "In Production" from March 19.</li>~~

~~</ul>~~

~~</div>~~

=== https://tapeworm.anunna.wur.nl/ ===
=== https://tapeworm.anunna.wur.nl/ ===

Prins0891: images updated to new GUI

2026-03-17T14:25:07Z

images updated to new GUI

← Older revision Revision as of 14:25, 17 March 2026
Line 45: Line 45:

=== 1) Overview ===
=== 1) Overview ===

This is your action page. It shows items that currently need your decision.
This is your action page. It shows items that currently need your decision.

[[File:~~Tapeworm screen1~~.png|~~frame~~|~~left~~]]
[[File:Screenshot from 2026-03-17 15-19-46.png|none|thumb|1400x1400px]]

For each candidate, you can:
For each candidate, you can:

* Approve: proceed with the tape move. It will schedule for the next day
* Approve: proceed with the tape move. It will schedule for the next day

Line 56: Line 56:

=== 2) Datasets ===
=== 2) Datasets ===

This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.
This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.

[[File:Screenshot from 2026-03-17 15-20-23.png|none|thumb|1400x1400px]]

[[File:~~User datasets~~.png|~~frame~~|~~none~~]]

Important:
Important:

* If a dataset already has an active move candidate, scheduling controls are disabled.
* If a dataset already has an active move candidate, scheduling controls are disabled.

Line 65: Line 63:

=== 3) Schedule ===
=== 3) Schedule ===

This page shows move candidates and their status over time.
This page shows move candidates and their status over time.

[[File:Screenshot from 2026-03-17 15-20-58.png|none|thumb|1400x1400px]]

[[File:~~User schedule~~.png|~~frame~~|~~none~~]]

Common statuses:
Common statuses:

* Planned: under review
* Planned: under review

Line 78: Line 74:

=== 4) Overrides ===
=== 4) Overrides ===

Overrides tell Tapeworm to ignore specific paths in future planning.
Overrides tell Tapeworm to ignore specific paths in future planning.

[[File:Screenshot from 2026-03-17 15-21-49.png|none|thumb|1400x1400px]]

[[File:~~User override~~.png|~~frame~~|~~none~~]]

Use overrides when:
Use overrides when:

* a project is still active and needs to remain on /archive
* a project is still active and needs to remain on /archive

Line 92: Line 86:

This page shows completed archival moves.
This page shows completed archival moves.

When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.
When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.

[[File:Screenshot from 2026-03-17 15-24-40.png|none|thumb|1400x1400px]]

[[File:~~User history~~.png|~~frame~~|~~none~~]]

== Group pages ==
== Group pages ==

Prins0891 at 14:05, 17 March 2026

2026-03-17T14:05:42Z

← Older revision Revision as of 14:05, 17 March 2026
Line 45: Line 45:

=== 1) Overview ===
=== 1) Overview ===

This is your action page. It shows items that currently need your decision.
This is your action page. It shows items that currently need your decision.

[[File:Tapeworm screen1.png|~~none~~|~~frame~~]]
[[File:Tapeworm screen1.png|frame|left]]

For each candidate, you can:
For each candidate, you can:

* Approve: proceed with the tape move. It will schedule for the next day
* Approve: proceed with the tape move. It will schedule for the next day

Prins0891 at 14:04, 17 March 2026

2026-03-17T14:04:56Z

← Older revision Revision as of 14:04, 17 March 2026
Line 45: Line 45:

=== 1) Overview ===
=== 1) Overview ===

This is your action page. It shows items that currently need your decision.
This is your action page. It shows items that currently need your decision.

[[File:Tapeworm screen1.png|none|frame]]

[[File:~~User overview 3~~.png|frame~~|none~~]]

For each candidate, you can:
For each candidate, you can:

* Approve: proceed with the tape move. It will schedule for the next day
* Approve: proceed with the tape move. It will schedule for the next day

Prins0891 at 13:58, 17 March 2026

2026-03-17T13:58:21Z

← Older revision Revision as of 13:58, 17 March 2026
Line 29: Line 29:

# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.
# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.

# A dataset discovery step groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.
# A dataset discovery step groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.

# A policy engine checks which discovered datasets look stale (for example: 30+ days old and larger than 1GB).
# A policy engine checks which discovered datasets look stale (for example: 60+ days old and larger than 1GB).

# Matching datasets are marked as planned and shown in your overview.
# Matching datasets are marked as planned and shown in your overview.

# You will be notified by email that Tapeworm plans to move data you own
# You will be notified by email that Tapeworm plans to move data you own

Prins0891 at 14:55, 13 March 2026

2026-03-13T14:55:50Z

← Older revision Revision as of 14:55, 13 March 2026
Line 22: Line 22:

If you do nothing, Tapeworm will continue with the planned move after the review period.
If you do nothing, Tapeworm will continue with the planned move after the review period.

That is why we recommend checking your pending actions regularly. You will also receive notification emails about pending actions.
That is why we recommend checking your pending actions regularly. You will also receive notification emails about pending actions.

From time to time, the user pages may temporarily be unavailable during maintenance. In that case, Tapeworm will show a short maintenance page instead of the normal interface.

== How Tapeworm works ==
== How Tapeworm works ==

# Tapeworm scans <code>/archive</code> and builds an index of datasets, size, owner, and last active use.
# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.

# A policy engine checks which datasets look stale (for example: 30+ days old and larger than 1GB).
# A dataset discovery step groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.

# A policy engine checks which discovered datasets look stale (for example: 30+ days old and larger than 1GB).

# Matching datasets are marked as planned and shown in your overview.
# Matching datasets are marked as planned and shown in your overview.

# You will be notified by email that Tapeworm plans to move data you own
# You will be notified by email that Tapeworm plans to move data you own

Line 68: Line 71:

Common statuses:
Common statuses:

* Planned ~~(or planned + notified)~~: under review
* Planned: under review

* Scheduled: move is planned for a specific date
* Scheduled: move is planned for a specific date

* Executing / Tape staged / On tape: move is in progress or completed
* Executing / Tape staged / On tape: move is in progress or completed

Line 148: Line 151:

Usually because the move has already progressed (executing/staged/on tape/error), so schedule edits are no longer valid.
Usually because the move has already progressed (executing/staged/on tape/error), so schedule edits are no longer valid.

=== Why do ~~I see “planned + notified”~~? ===
=== Why do some paths on /archive/ not appear as datasets in Tapeworm? ===

~~That means the dataset move is planned~~ and a ~~notification has already been sent~~.
Tapeworm only shows paths that are discovered as datasets and pass a minimum-size threshold. Very small paths, single text files, and other tiny items are intentionally filtered out.

== Need help? ==
== Need help? ==

If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.
If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.

Itter003 at 11:57, 12 March 2026

2026-03-12T11:57:43Z

← Older revision Revision as of 11:57, 12 March 2026
Line 2: Line 2:

<div style="border:2px solid #d32f2f; background:#ffebee; padding:12px; margin:12px 0;">
<div style="border:2px solid #d32f2f; background:#ffebee; padding:12px; margin:12px 0;">

~~Warning~~:
Under construction:

<ul style="margin:8px 0 0 18px;">
<ul style="margin:8px 0 0 18px;">

 <li>This documentation page is under construction and may contain ~~errors~~.</li>
 <li>This documentation page is under construction and may still contain some gaps.</li>

 <li>The Tapeworm application ~~is in beta and may contain errors~~.</li>
 <li>The Tapeworm application will be "In Production" from March 19.</li>

</ul>
</ul>

</div>
</div>

Haars0011: Don't flow text around images

2026-03-03T17:31:45Z

Don't flow text around images

← Older revision Revision as of 17:31, 3 March 2026
Line 43: Line 43:

This is your action page. It shows items that currently need your decision.
This is your action page. It shows items that currently need your decision.

[[File:User overview 3.png|~~left~~|~~frame~~]]
[[File:User overview 3.png|frame|none]]

For each candidate, you can:
For each candidate, you can:

Line 56: Line 56:

This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.
This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.

[[File:User datasets.png|~~left~~|~~frame~~]]
[[File:User datasets.png|frame|none]]

Important:
Important:

Line 65: Line 65:

This page shows move candidates and their status over time.
This page shows move candidates and their status over time.

[[File:User schedule.png|~~left~~|~~frame~~]]
[[File:User schedule.png|frame|none]]

Common statuses:
Common statuses:

Line 78: Line 78:

Overrides tell Tapeworm to ignore specific paths in future planning.
Overrides tell Tapeworm to ignore specific paths in future planning.

[[File:User override.png|~~left~~|~~frame~~]]
[[File:User override.png|frame|none]]

Use overrides when:
Use overrides when:

Line 92: Line 92:

When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.
When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.

[[File:User history.png|~~left~~|~~frame~~]]
[[File:User history.png|frame|none]]

== Group pages ==
== Group pages ==

← Older revision	Revision as of 11:50, 18 June 2026
(No difference)

← Older revision		Revision as of 09:26, 19 March 2026
Line 1:		Line 1:
	= Tapeworm: Automated tape-archival of old datasets =		= Tapeworm: Automated tape-archival of old datasets =

	~~<div style="border:2px solid #d32f2f; background:#ffebee; padding:12px; margin:12px 0;">~~
	~~<b>Under construction:</b>~~
	~~<ul style="margin:8px 0 0 18px;">~~
	~~<li>This documentation page is under construction and may still contain some gaps.</li>~~
	~~<li>The Tapeworm application will be "In Production" from March 19.</li>~~
	~~</ul>~~
	~~</div>~~

	=== https://tapeworm.anunna.wur.nl/ ===		=== https://tapeworm.anunna.wur.nl/ ===

← Older revision		Revision as of 14:25, 17 March 2026
Line 45:		Line 45:
	=== 1) Overview ===		=== 1) Overview ===
	This is your action page. It shows items that currently need your decision.		This is your action page. It shows items that currently need your decision.
	[[File:~~Tapeworm screen1~~.png\|~~frame~~\|~~left~~]]		[[File:Screenshot from 2026-03-17 15-19-46.png\|none\|thumb\|1400x1400px]]
	For each candidate, you can:		For each candidate, you can:
	* <b>Approve</b>: proceed with the tape move. It will schedule for the next day		* <b>Approve</b>: proceed with the tape move. It will schedule for the next day
Line 56:		Line 56:
	=== 2) Datasets ===		=== 2) Datasets ===
	This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.		This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.
			[[File:Screenshot from 2026-03-17 15-20-23.png\|none\|thumb\|1400x1400px]]
	[[File:~~User datasets~~.png\|~~frame~~\|~~none~~]]

	Important:		Important:
	* If a dataset already has an active move candidate, scheduling controls are disabled.		* If a dataset already has an active move candidate, scheduling controls are disabled.
Line 65:		Line 63:
	=== 3) Schedule ===		=== 3) Schedule ===
	This page shows move candidates and their status over time.		This page shows move candidates and their status over time.
			[[File:Screenshot from 2026-03-17 15-20-58.png\|none\|thumb\|1400x1400px]]
	[[File:~~User schedule~~.png\|~~frame~~\|~~none~~]]

	Common statuses:		Common statuses:
	* <b>Planned</b>: under review		* <b>Planned</b>: under review
Line 78:		Line 74:
	=== 4) Overrides ===		=== 4) Overrides ===
	Overrides tell Tapeworm to ignore specific paths in future planning.		Overrides tell Tapeworm to ignore specific paths in future planning.
			[[File:Screenshot from 2026-03-17 15-21-49.png\|none\|thumb\|1400x1400px]]
	[[File:~~User override~~.png\|~~frame~~\|~~none~~]]

	Use overrides when:		Use overrides when:
	* a project is still active and needs to remain on /archive		* a project is still active and needs to remain on /archive
Line 92:		Line 86:
	This page shows completed archival moves.		This page shows completed archival moves.
	When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.		When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.
			[[File:Screenshot from 2026-03-17 15-24-40.png\|none\|thumb\|1400x1400px]]
	[[File:~~User history~~.png\|~~frame~~\|~~none~~]]

	== Group pages ==		== Group pages ==

← Older revision		Revision as of 14:05, 17 March 2026
Line 45:		Line 45:
	=== 1) Overview ===		=== 1) Overview ===
	This is your action page. It shows items that currently need your decision.		This is your action page. It shows items that currently need your decision.
	[[File:Tapeworm screen1.png\|~~none~~\|~~frame~~]]		[[File:Tapeworm screen1.png\|frame\|left]]
	For each candidate, you can:		For each candidate, you can:
	* <b>Approve</b>: proceed with the tape move. It will schedule for the next day		* <b>Approve</b>: proceed with the tape move. It will schedule for the next day

← Older revision		Revision as of 14:04, 17 March 2026
Line 45:		Line 45:
	=== 1) Overview ===		=== 1) Overview ===
	This is your action page. It shows items that currently need your decision.		This is your action page. It shows items that currently need your decision.
			[[File:Tapeworm screen1.png\|none\|frame]]
	[[File:~~User overview 3~~.png\|frame~~\|none~~]]

	For each candidate, you can:		For each candidate, you can:
	* <b>Approve</b>: proceed with the tape move. It will schedule for the next day		* <b>Approve</b>: proceed with the tape move. It will schedule for the next day

← Older revision		Revision as of 13:58, 17 March 2026
Line 29:		Line 29:
	# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.		# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.
	# A dataset discovery step groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.		# A dataset discovery step groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.
	# A policy engine checks which discovered datasets look stale (for example: 30+ days old and larger than 1GB).		# A policy engine checks which discovered datasets look stale (for example: 60+ days old and larger than 1GB).
	# Matching datasets are marked as <b>planned</b> and shown in your overview.		# Matching datasets are marked as <b>planned</b> and shown in your overview.
	# You will be notified by email that Tapeworm plans to move data you own		# You will be notified by email that Tapeworm plans to move data you own

← Older revision		Revision as of 14:55, 13 March 2026
Line 22:		Line 22:
	If you do nothing, Tapeworm will continue with the planned move after the review period.		If you do nothing, Tapeworm will continue with the planned move after the review period.
	That is why we recommend checking your pending actions regularly. You will also receive notification emails about pending actions.		That is why we recommend checking your pending actions regularly. You will also receive notification emails about pending actions.

			From time to time, the user pages may temporarily be unavailable during maintenance. In that case, Tapeworm will show a short maintenance page instead of the normal interface.

	== How Tapeworm works ==		== How Tapeworm works ==

	# Tapeworm scans <code>/archive</code> and builds an index of datasets, size, owner, and last active use.		# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.
	# A policy engine checks which datasets look stale (for example: 30+ days old and larger than 1GB).		# A dataset discovery step groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.
			# A policy engine checks which discovered datasets look stale (for example: 30+ days old and larger than 1GB).
	# Matching datasets are marked as <b>planned</b> and shown in your overview.		# Matching datasets are marked as <b>planned</b> and shown in your overview.
	# You will be notified by email that Tapeworm plans to move data you own		# You will be notified by email that Tapeworm plans to move data you own
Line 68:		Line 71:

	Common statuses:		Common statuses:
	* <b>Planned</b> ~~(or <b>planned + notified</b>)~~: under review		* <b>Planned</b>: under review
	* <b>Scheduled</b>: move is planned for a specific date		* <b>Scheduled</b>: move is planned for a specific date
	* <b>Executing / Tape staged / On tape</b>: move is in progress or completed		* <b>Executing / Tape staged / On tape</b>: move is in progress or completed
Line 148:		Line 151:
	Usually because the move has already progressed (executing/staged/on tape/error), so schedule edits are no longer valid.		Usually because the move has already progressed (executing/staged/on tape/error), so schedule edits are no longer valid.

	=== Why do ~~I see “planned + notified”~~? ===		=== Why do some paths on /archive/ not appear as datasets in Tapeworm? ===
	~~That means the dataset move is planned~~ and a ~~notification has already been sent~~.		Tapeworm only shows paths that are discovered as datasets and pass a minimum-size threshold. Very small paths, single text files, and other tiny items are intentionally filtered out.

	== Need help? ==		== Need help? ==
	If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.		If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.

← Older revision		Revision as of 11:57, 12 March 2026
Line 2:		Line 2:

	<div style="border:2px solid #d32f2f; background:#ffebee; padding:12px; margin:12px 0;">		<div style="border:2px solid #d32f2f; background:#ffebee; padding:12px; margin:12px 0;">
	<b>~~Warning~~:</b>		<b>Under construction:</b>
	<ul style="margin:8px 0 0 18px;">		<ul style="margin:8px 0 0 18px;">
	<li>This documentation page is under construction and may contain ~~errors~~.</li>		<li>This documentation page is under construction and may still contain some gaps.</li>
	<li>The Tapeworm application ~~is in beta and may contain errors~~.</li>		<li>The Tapeworm application will be "In Production" from March 19.</li>
	</ul>		</ul>
	</div>		</div>

← Older revision		Revision as of 17:31, 3 March 2026
Line 43:		Line 43:
	This is your action page. It shows items that currently need your decision.		This is your action page. It shows items that currently need your decision.

	[[File:User overview 3.png\|~~left~~\|~~frame~~]]		[[File:User overview 3.png\|frame\|none]]

	For each candidate, you can:		For each candidate, you can:
Line 56:		Line 56:
	This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.		This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.

	[[File:User datasets.png\|~~left~~\|~~frame~~]]		[[File:User datasets.png\|frame\|none]]

	Important:		Important:
Line 65:		Line 65:
	This page shows move candidates and their status over time.		This page shows move candidates and their status over time.

	[[File:User schedule.png\|~~left~~\|~~frame~~]]		[[File:User schedule.png\|frame\|none]]

	Common statuses:		Common statuses:
Line 78:		Line 78:
	Overrides tell Tapeworm to ignore specific paths in future planning.		Overrides tell Tapeworm to ignore specific paths in future planning.

	[[File:User override.png\|~~left~~\|~~frame~~]]		[[File:User override.png\|frame\|none]]

	Use overrides when:		Use overrides when:
Line 92:		Line 92:
	When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.		When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.

	[[File:User history.png\|~~left~~\|~~frame~~]]		[[File:User history.png\|frame\|none]]

	== Group pages ==		== Group pages ==