Jump to content

Archival Storage: Difference between revisions

From HPCwiki
No edit summary
iRODS section: iput vs itape and rdm_archive_this, status values (no failure status), how long it takes, keep your copy until archive_completed, USER_FILE_TOO_LARGE (HPC-3254, answers of the iRODS team 2026-11-05) (via update-page on MediaWiki MCP Server)
 
(19 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Tapeworm: Automated tape-archival of old datasets =
<code>/archive</code> is Anunna's warm storage tier for data you want to keep but are not actively computing on. Storing data there costs less than on [[Compute Storage|Lustre]], and datasets that have not been used for a long time are moved on to long-term tape archive — automatically, through the [[#Tapeworm|Tapeworm]] service — to keep the warm tier free for active work.


<div style="border:2px solid #d32f2f; background:#ffebee; padding:12px; margin:12px 0;">
== The /archive filesystem ==
<b>Warning:</b>
<ul style="margin:8px 0 0 18px;">
  <li>This documentation page is under construction and may contain errors.</li>
  <li>The Tapeworm application is in beta and may contain errors.</li>
</ul>
</div>


=== https://tapeworm.anunna.wur.nl/ ===
<code>/archive</code> is a mount that is only accessible from the login nodes, and it cannot be used for compute work. It is available to all Anunna users, external users included.


Tapeworm helps you manage data on <code>/archive</code> by identifying datasets that are no longer actively used and preparing them for tape archival.
<code>/archive</code> is backed up periodically on the same schedule as your [[Home Directory|home directory]] — restorable from roughly a week of history (see [[Backup Policy]]). That backup is separate from the long-term tape archival described below: a backup is a short-term safety copy, while archival moves data off <code>/archive</code> for long-term keeping.
The goal is simple: keep our warm storage available for active work, while safely preserving older data on tape.
 
== Tapeworm ==
 
Tapeworm (https://tapeworm.anunna.wur.nl/) helps you manage data on <code>/archive</code> by identifying datasets that are no longer actively used and preparing them for tape archival. The goal is simple: keep our warm storage available for active work, while safely preserving older data on tape.


With Tapeworm, you can:
With Tapeworm, you can:
Line 20: Line 17:
* Add metadata to help describe archived datasets. The metadata is included on tape and can be used to view/retrieve from tape, should you need to do so in the future
* Add metadata to help describe archived datasets. The metadata is included on tape and can be used to view/retrieve from tape, should you need to do so in the future


If you do nothing, Tapeworm will continue with the planned move after the review period.
If you do nothing, Tapeworm will continue with the planned move after the review period. That is why we recommend checking your pending actions regularly. You will also receive notification emails about pending actions.
That is why we recommend checking your pending actions regularly. You will also receive notification emails about pending actions.


== How Tapeworm works ==
From time to time, the user pages may temporarily be unavailable during maintenance. In that case, Tapeworm will show a short maintenance page instead of the normal interface.


# Tapeworm scans <code>/archive</code> and builds an index of datasets, size, owner, and last active use.
=== How Tapeworm works ===
# A policy engine checks which datasets look stale (for example: 30+ days old and larger than 1GB).
# Matching datasets are marked as <b>planned</b> and shown in your overview.
# You will be notified by email that Tapeworm plans to move data you own
# You can review and change what should happen, or block the move(s) entirely
# If no action is taken, after a wait period of 4 weeks, planned moves can become scheduled and then executed.
# Data is moved to tape, and removed from /archive/


== Who sees what? ==
# Tapeworm scans <code>/archive</code> and builds an index of paths and filesystem metadata.
# At the dataset discovery step, it groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.
# A policy engine checks which discovered datasets look inactive (for example: 60+ days old and larger than 1GB).
# Matching datasets are marked as '''planned''' and shown in your overview.
# You will be notified by email that Tapeworm plans to move data you own.
# You can review and change what should happen, or block the move(s) entirely.
# If no action is taken, after a wait period of 4 weeks, planned moves will become scheduled and then executed.
# Data is moved to tape, and removed from <code>/archive</code>.


* <b>Regular users</b> see only their own datasets and actions.
=== Who sees what? ===
* <b>Group admins/contacts</b> see data for their configured group(s), in addition to their own data.


== User pages ==
* '''Regular users''' see only their own datasets and actions.
* '''Group admins/contacts''' see data for their configured group(s), in addition to their own data.


=== 1) Overview ===
=== User pages ===
 
==== 1) Overview ====
This is your action page. It shows items that currently need your decision.
This is your action page. It shows items that currently need your decision.
 
[[File:Screenshot from 2026-03-17 15-19-46.png|none|thumb|1400x1400px]]
[[File:User overview 3.png|left|frame]]
 
For each candidate, you can:
For each candidate, you can:
* <b>Approve</b>: proceed with the tape move. It will schedule for the next day
* '''Approve''': proceed with the tape move. It will schedule for the next day
* <b>Deny</b>: stop this move, configure an override for this path. Tapeworm will not try to move this dataset/path again, until you choose to remove the override
* '''Deny''': stop this move, configure an override for this path. Tapeworm will not try to move this dataset/path again, until you choose to remove the override
* <b>Snooze</b>: postpone the decision to a future date
* '''Snooze''': postpone the decision to a future date
* <b>Edit metadata</b>: add key/value notes for archived data. These values are included on tape and can be used to view/retrieve datasets on tape
* '''Edit metadata''': add key/value notes for archived data. These values are included on tape and can be used to view/retrieve datasets on tape


You can also select multiple rows and apply actions in bulk.
You can also select multiple rows and apply actions in bulk.


=== 2) Datasets ===
==== 2) Datasets ====
This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.
This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.
 
[[File:Screenshot from 2026-03-17 15-20-23.png|none|thumb|1400x1400px]]
[[File:User datasets.png|left|frame]]
'''Important:'''
 
Important:
* If a dataset already has an active move candidate, scheduling controls are disabled.
* If a dataset already has an active move candidate, scheduling controls are disabled.
  The dataset list is informational; move decisions are handled through the Schedule page.
  The dataset list is informational; move decisions are handled through the Schedule page.


=== 3) Schedule ===
==== 3) Schedule ====
This page shows move candidates and their status over time.
This page shows move candidates and their status over time.
 
[[File:Screenshot from 2026-03-17 15-20-58.png|none|thumb|1400x1400px]]
[[File:User schedule.png|left|frame]]
 
Common statuses:
Common statuses:
* <b>Planned</b> (or <b>planned + notified</b>): under review
* '''Planned''': under review
* <b>Scheduled</b>: move is planned for a specific date
* '''Scheduled''': move is planned for a specific date
* <b>Executing / Tape staged / On tape</b>: move is in progress or completed
* '''Executing / Tape staged / On tape''': move is in progress or completed
* <b>Error</b>: move needs admin attention. You may be contacted, maybe we resolve it ourselves :)
* '''Error''': move needs admin attention. You may be contacted, maybe we resolve it ourselves :)


Once a move is already executing or completed, schedule-changing actions are locked.
Once a move is already executing or completed, schedule-changing actions are locked.


=== 4) Overrides ===
==== 4) Overrides ====
Overrides tell Tapeworm to ignore specific paths in future planning.
Overrides tell Tapeworm to ignore specific paths in future planning.
 
[[File:Screenshot from 2026-03-17 15-21-49.png|none|thumb|1400x1400px]]
[[File:User override.png|left|frame]]
 
Use overrides when:
Use overrides when:
* a project is still active and needs to remain on /archive
* a project is still active and needs to remain on /archive
Line 88: Line 79:
Overrides apply to the selected path and everything below it.
Overrides apply to the selected path and everything below it.


=== 5) History ===
==== 5) History ====
This page shows completed archival moves.
This page shows completed archival moves. When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.
When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.
[[File:Screenshot from 2026-03-17 15-24-40.png|none|thumb|1400x1400px]]


[[File:User history.png|left|frame]]
=== Group pages ===


== Group pages ==
Group admins have a separate additional set of pages for their group scope:
 
* '''Group overview'''
Group admins have a separate set of pages for their group scope:
* '''Group datasets'''
* <b>Group overview</b>
* '''Group schedule'''
* <b>Group datasets</b>
* '''Group overrides'''
* <b>Group schedule</b>
* '''Group history'''
* <b>Group overrides</b>
* <b>Group history</b>


If you manage more than one group, you can switch group scope in the selector at the top of the group pages.
If you manage more than one group, you can switch group scope in the selector at the top of the group pages.


== Notifications (email) ==
=== Notifications (email) ===


Tapeworm sends email updates when actions are pending, dates are approaching, or move state changes happen.
Tapeworm sends email updates when actions are pending, dates are approaching, or move state changes happen.
Line 117: Line 106:


Notification types you may receive:
Notification types you may receive:
* <b>Action required</b>: please approve, snooze, or deny
* '''Action required''': please approve, snooze, or deny
* <b>Reminder</b>: review date is approaching
* '''Reminder''': review date is approaching
* <b>Informational</b>: move status changed (for example scheduled, staging, or completed)
* '''Informational''': move status changed (for example scheduled, staging, or completed)
* <b>Escalation</b>: sent to group contacts when no user response is received
* '''Escalation''': sent to group contacts when no user response is received


Please read these emails carefully — they are your chance to adjust decisions before execution.
Please read these emails carefully — they are your chance to adjust decisions before execution.


== Best practices for users ==
=== Best practices for users ===


* Check your <b>Overview</b> page regularly
* Check your '''Overview''' page regularly
* Use <b>Snooze</b> if you need time to validate impact
* Use '''Snooze''' if you need time to validate impact
* Add <b>metadata</b> when approving important datasets
* Add '''metadata''' when approving important datasets
* Use <b>Overrides</b> for known and persisting exceptions
* Use '''Overrides''' for known and persisting exceptions
* If unsure, contact HPC support before a scheduled move date
* If unsure, contact HPC support before a scheduled move date


== FAQ ==
=== Tapeworm FAQ ===


=== What happens if I do nothing? ===
==== What happens if I do nothing? ====
Planned items can move forward automatically after the review window (4 weeks)
Planned items can move forward automatically after the review window (4 weeks).


=== Can I undo after tape staging? ===
==== Can I undo after tape staging? ====
Not directly in Tapeworm. Retrieval is done via the tape/iRODS workflow.
Not directly in Tapeworm. Retrieval is done via the tape/iRODS workflow. See: https://irods.wur.nl/userguide/tape_retrieval/
See: https://irods.wur.nl/userguide/tape_retrieval/


=== What does “completed” mean? ===
==== What does "completed" mean? ====
Completed means Tapeworm saw the tape workflow finish and finalized the move.
Completed means Tapeworm saw the tape workflow finish and finalized the move. Before finalization, the system verifies the archive in iRODS and only then removes the staged source copy.
Before finalization, the system verifies the archive in iRODS and only then removes the staged source copy.


=== Why is an action button disabled? ===
==== Why is an action button disabled? ====
Usually because the move has already progressed (executing/staged/on tape/error), so schedule edits are no longer valid.
Usually because the move has already progressed (executing/staged/on tape/error), so schedule edits are no longer valid.


=== Why do I see “planned + notified”? ===
==== Why do some paths on /archive/ not appear as datasets in Tapeworm? ====
That means the dataset move is planned and a notification has already been sent.
Tapeworm only shows paths that are discovered as datasets and pass a minimum-size threshold. Very small paths, single text files, and other tiny items are intentionally filtered out.


== Need help? ==
=== Need help? ===
If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.
If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.
== Manual tape access with iRODS besides Tapeworm ==
Anunna hosts its own iRODS instance, with which you can push data to the WUR tape storage yourself. Use it instead of [[#Tapeworm|Tapeworm]] when you want to decide what goes into each archive file, where it lands, and when it goes to tape. For general usage information, see [https://irods.wur.nl/ iRODS in WUR]. The best course of action is to loosely follow the course there using your own data, and use your personal space for data upload and transfer to tape.
On Anunna there are some differences and additions to the linked site:
* The zone is <code>HPC</code>.
* With <code>iinit</code> you can initialise the iRODS environment. Use your account password.
* With <code>ils</code> you can see your available iRODS collections. You need that as a destination location for <code>itape</code>.
* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trash bin. The advantage is that you can retrieve it if the removal was in error; the disadvantage is that the data will keep costing money. To empty it, see <code>irmtrash -h</code>.
=== Getting data onto tape ===
Uploading alone does not put data on tape. A plain <code>iput</code> leaves it on disk in iRODS (resource <code>hot_1</code>) until the rule <code>rdm_archive_this</code> is run on it. The <code>itape</code> function does both: it uploads with a checksum (<code>iput -K</code>) and then runs the rule.
<syntaxhighlight lang="bash">
itape [-r] <local path> <iRODS collection>
</syntaxhighlight>
To request tape for data you uploaded with <code>iput</code> yourself, run the rule on the file or collection. Upload with <code>iput -K</code>: files without a checksum are skipped.
<syntaxhighlight lang="bash">
irule -r irods_rule_engine_plugin-irods_rule_language-instance rdm_archive_this "*file_or_collection=<file or collection>" ruleExecOut
</syntaxhighlight>
<code>itape</code>, <code>itapestat</code> and <code>itapestatnp</code> only exist in an interactive shell. In a script, a job, or a command run over <code>ssh</code>, they give <code>command not found</code>; use the <code>iput</code> and <code>irule</code> commands above, and <code>imeta ls -d <path></code> for the status.
Tape works best with a few large files. Pack your data into tar files of several GB or more each (for example one per sample or per run) before you upload it, instead of sending many small files.
=== Checking the status ===
<code>itapestat</code> (paginated, for reading) and <code>itapestatnp</code> (everything at once, for <code>grep</code>) show the <code>archive_status</code> of your files. It moves through these values:
{| class="wikitable"
! Status !! Meaning
|-
| <code>archive_requested</code> || Tape has been requested; the data is still only on disk.
|-
| <code>archive_performed</code> || The data has been written to tape and is being verified, which reads it back from tape.
|-
| <code>archive_completed</code> || The data is on tape and verified. From now on it can no longer be changed or removed.
|-
| <code>completed_and_hot_deleted</code> || The copy on disk in iRODS has been removed too.
|}
There is no failure status. Faults are recorded elsewhere and fixed by the iRODS team, so a status that does not change is waiting, not failed.
Plan for days, not hours. The steps run in batches: in a test in November 2026, a small tar reached <code>archive_performed</code> after 20 minutes and <code>archive_completed</code> after 12.5 hours. When the tape system is busy, for example while Tapeworm archives a lot, requests queue and it takes longer, for small files too.
'''Keep your own copy until the status is <code>archive_completed</code>.''' It is easiest to work in two rounds: upload and request tape for everything, then come back days later and remove the local copies of what has reached <code>archive_completed</code>.
=== Upload errors ===
Because of hardware limitations on the backend tape storage, the size limit per file for our tape archive is 5 TB.
An upload can also fail with <code>USER_FILE_TOO_LARGE</code> for a file far below that limit, while small files still go in. The disk in front of tape then has less free space than the minimum it keeps for new files. <code>hot_1</code> passes data on to a storage resource below it, and the minimum is set on that resource, whose name may change:
<syntaxhighlight lang="bash">
ilsresc hot_1              # shows the resource below hot_1, e.g. local_filestore_2
ilsresc -l <resource>      # compare its free space with minimum_free_space_for_create_in_bytes
</syntaxhighlight>
If the free space is below the minimum, try again later, or open an HPC support ticket.
== Retrieving data from tape ==
As the tape storage is maintained by a different team, it is best to follow their documentation on retrieval.
For that, see here: https://irods.wur.nl/userguide/archiving_research_data_using_icommands/
== See also ==
* [[Storage Systems Overview]]
* [[Backup Policy]]
* [[Compute Storage]]
* [[Tariffs|Costs associated with resource usage]]
== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Latest revision as of 15:54, 6 November 2026

/archive is Anunna's warm storage tier for data you want to keep but are not actively computing on. Storing data there costs less than on Lustre, and datasets that have not been used for a long time are moved on to long-term tape archive — automatically, through the Tapeworm service — to keep the warm tier free for active work.

The /archive filesystem

/archive is a mount that is only accessible from the login nodes, and it cannot be used for compute work. It is available to all Anunna users, external users included.

/archive is backed up periodically on the same schedule as your home directory — restorable from roughly a week of history (see Backup Policy). That backup is separate from the long-term tape archival described below: a backup is a short-term safety copy, while archival moves data off /archive for long-term keeping.

Tapeworm

Tapeworm (https://tapeworm.anunna.wur.nl/) helps you manage data on /archive by identifying datasets that are no longer actively used and preparing them for tape archival. The goal is simple: keep our warm storage available for active work, while safely preserving older data on tape.

With Tapeworm, you can:

  • See which of your datasets are being considered for tape archival
  • Review planned moves before they happen
  • Approve, snooze, or block moves when needed
  • Add metadata to help describe archived datasets. The metadata is included on tape and can be used to view/retrieve from tape, should you need to do so in the future

If you do nothing, Tapeworm will continue with the planned move after the review period. That is why we recommend checking your pending actions regularly. You will also receive notification emails about pending actions.

From time to time, the user pages may temporarily be unavailable during maintenance. In that case, Tapeworm will show a short maintenance page instead of the normal interface.

How Tapeworm works

  1. Tapeworm scans /archive and builds an index of paths and filesystem metadata.
  2. At the dataset discovery step, it groups those paths into datasets, determines size, owner, and last active use, and makes them available in the GUI. Very small datasets are filtered out and will not be shown.
  3. A policy engine checks which discovered datasets look inactive (for example: 60+ days old and larger than 1GB).
  4. Matching datasets are marked as planned and shown in your overview.
  5. You will be notified by email that Tapeworm plans to move data you own.
  6. You can review and change what should happen, or block the move(s) entirely.
  7. If no action is taken, after a wait period of 4 weeks, planned moves will become scheduled and then executed.
  8. Data is moved to tape, and removed from /archive.

Who sees what?

  • Regular users see only their own datasets and actions.
  • Group admins/contacts see data for their configured group(s), in addition to their own data.

User pages

1) Overview

This is your action page. It shows items that currently need your decision.

For each candidate, you can:

  • Approve: proceed with the tape move. It will schedule for the next day
  • Deny: stop this move, configure an override for this path. Tapeworm will not try to move this dataset/path again, until you choose to remove the override
  • Snooze: postpone the decision to a future date
  • Edit metadata: add key/value notes for archived data. These values are included on tape and can be used to view/retrieve datasets on tape

You can also select multiple rows and apply actions in bulk.

2) Datasets

This page shows your discovered datasets, their sizes, and last activity times. The application has no concept of what data belongs together and should be considered a 'dataset'. If the selections on this page are wrong, you can change how Tapeworm should handle these datasets instead.

Important:

  • If a dataset already has an active move candidate, scheduling controls are disabled.
The dataset list is informational; move decisions are handled through the Schedule page.

3) Schedule

This page shows move candidates and their status over time.

Common statuses:

  • Planned: under review
  • Scheduled: move is planned for a specific date
  • Executing / Tape staged / On tape: move is in progress or completed
  • Error: move needs admin attention. You may be contacted, maybe we resolve it ourselves :)

Once a move is already executing or completed, schedule-changing actions are locked.

4) Overrides

Overrides tell Tapeworm to ignore specific paths in future planning.

Use overrides when:

  • a project is still active and needs to remain on /archive
  • policy suggestions are not appropriate for that location

If you agree that the dataset can in principle be moved to tape, but you don't (yet) know when, you can choose to postpone/snooze the archival instead of overriding it.

Overrides apply to the selected path and everything below it.

5) History

This page shows completed archival moves. When a dataset has been successfully archived and finalized, it is removed from active scheduling pages and moved into history.

Group pages

Group admins have a separate additional set of pages for their group scope:

  • Group overview
  • Group datasets
  • Group schedule
  • Group overrides
  • Group history

If you manage more than one group, you can switch group scope in the selector at the top of the group pages.

Notifications (email)

Tapeworm sends email updates when actions are pending, dates are approaching, or move state changes happen.

Emails typically include:

  • Dataset path
  • Size
  • Last activity
  • Current status
  • Review/scheduled date

Notification types you may receive:

  • Action required: please approve, snooze, or deny
  • Reminder: review date is approaching
  • Informational: move status changed (for example scheduled, staging, or completed)
  • Escalation: sent to group contacts when no user response is received

Please read these emails carefully — they are your chance to adjust decisions before execution.

Best practices for users

  • Check your Overview page regularly
  • Use Snooze if you need time to validate impact
  • Add metadata when approving important datasets
  • Use Overrides for known and persisting exceptions
  • If unsure, contact HPC support before a scheduled move date

Tapeworm FAQ

What happens if I do nothing?

Planned items can move forward automatically after the review window (4 weeks).

Can I undo after tape staging?

Not directly in Tapeworm. Retrieval is done via the tape/iRODS workflow. See: https://irods.wur.nl/userguide/tape_retrieval/

What does "completed" mean?

Completed means Tapeworm saw the tape workflow finish and finalized the move. Before finalization, the system verifies the archive in iRODS and only then removes the staged source copy.

Why is an action button disabled?

Usually because the move has already progressed (executing/staged/on tape/error), so schedule edits are no longer valid.

Why do some paths on /archive/ not appear as datasets in Tapeworm?

Tapeworm only shows paths that are discovered as datasets and pass a minimum-size threshold. Very small paths, single text files, and other tiny items are intentionally filtered out.

Need help?

If anything is unclear, or you think a move is incorrect but you cannot alter it in the provided GUI, please open an HPC support ticket.

Manual tape access with iRODS besides Tapeworm

Anunna hosts its own iRODS instance, with which you can push data to the WUR tape storage yourself. Use it instead of Tapeworm when you want to decide what goes into each archive file, where it lands, and when it goes to tape. For general usage information, see iRODS in WUR. The best course of action is to loosely follow the course there using your own data, and use your personal space for data upload and transfer to tape.

On Anunna there are some differences and additions to the linked site:

  • The zone is HPC.
  • With iinit you can initialise the iRODS environment. Use your account password.
  • With ils you can see your available iRODS collections. You need that as a destination location for itape.
  • If you remove data with irm within iRODS, the data isn't actually removed but moved to a trash bin. The advantage is that you can retrieve it if the removal was in error; the disadvantage is that the data will keep costing money. To empty it, see irmtrash -h.

Getting data onto tape

Uploading alone does not put data on tape. A plain iput leaves it on disk in iRODS (resource hot_1) until the rule rdm_archive_this is run on it. The itape function does both: it uploads with a checksum (iput -K) and then runs the rule.

itape [-r] <local path> <iRODS collection>

To request tape for data you uploaded with iput yourself, run the rule on the file or collection. Upload with iput -K: files without a checksum are skipped.

irule -r irods_rule_engine_plugin-irods_rule_language-instance rdm_archive_this "*file_or_collection=<file or collection>" ruleExecOut

itape, itapestat and itapestatnp only exist in an interactive shell. In a script, a job, or a command run over ssh, they give command not found; use the iput and irule commands above, and imeta ls -d <path> for the status.

Tape works best with a few large files. Pack your data into tar files of several GB or more each (for example one per sample or per run) before you upload it, instead of sending many small files.

Checking the status

itapestat (paginated, for reading) and itapestatnp (everything at once, for grep) show the archive_status of your files. It moves through these values:

Status Meaning
archive_requested Tape has been requested; the data is still only on disk.
archive_performed The data has been written to tape and is being verified, which reads it back from tape.
archive_completed The data is on tape and verified. From now on it can no longer be changed or removed.
completed_and_hot_deleted The copy on disk in iRODS has been removed too.

There is no failure status. Faults are recorded elsewhere and fixed by the iRODS team, so a status that does not change is waiting, not failed.

Plan for days, not hours. The steps run in batches: in a test in November 2026, a small tar reached archive_performed after 20 minutes and archive_completed after 12.5 hours. When the tape system is busy, for example while Tapeworm archives a lot, requests queue and it takes longer, for small files too.

Keep your own copy until the status is archive_completed. It is easiest to work in two rounds: upload and request tape for everything, then come back days later and remove the local copies of what has reached archive_completed.

Upload errors

Because of hardware limitations on the backend tape storage, the size limit per file for our tape archive is 5 TB.

An upload can also fail with USER_FILE_TOO_LARGE for a file far below that limit, while small files still go in. The disk in front of tape then has less free space than the minimum it keeps for new files. hot_1 passes data on to a storage resource below it, and the minimum is set on that resource, whose name may change:

ilsresc hot_1               # shows the resource below hot_1, e.g. local_filestore_2
ilsresc -l <resource>       # compare its free space with minimum_free_space_for_create_in_bytes

If the free space is below the minimum, try again later, or open an HPC support ticket.

Retrieving data from tape

As the tape storage is maintained by a different team, it is best to follow their documentation on retrieval.

For that, see here: https://irods.wur.nl/userguide/archiving_research_data_using_icommands/

See also