HPCwiki - User contributions [en]

Courses

2025-11-27T14:48:12Z

Pelle016:

== HPC Courses Autumn 2025 ==

17 oct 2025: [https://www.wur.nl/en/activity/linux-basic-course.htm Linux Basics]
[[File:WUR Linux Basics Course Slides.pdf|thumb]]

12 nov 2025: [https://www.wur.nl/en/activity/hpc-basic-course.htm HPC Basics]

21 nov 2025: [https://www.wur.nl/en/activity/hpc-advanced-course.htm HPC Advanced]

== HPC Advanced Course - June 2024 ==

A course for experienced users of the Anunna HPC was organised in June 2024, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[https://public.anunna.wur.nl/hpc-courses/HPC-Advanced.pdf Slides]

== HPC Basic Course - June 2024==

A course for beginners shall be given in June 2024, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[https://public.anunna.wur.nl/hpc-courses/HPC-Basics.pdf Slides]

== Linux Basic Course - May 2024 ==

A beginners course to the linux (bash) command line. This course is meant to prepare students for the upcoming HPC and Containers courses.

[https://public.anunna.wur.nl/hpc-courses/Linux-Basics.pdf Slides]

== Linux Basic Course - June 2022 ==

A course for beginners shall be given in June 2022, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - November 2021 ==

A course for experienced users oth the Anunna HPC was organised in November 2021, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20211021.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - November 2021==

Two courses for beginners was given in November 2021, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20211012.pdf|HPC Basic Course]]

== Linux Basic Course - November 2021 ==

A course for beginners was given in November 2021, to help beginner Linux users gain some skills in using Linux.

== HPC Basic Course - 2019-11-14==

A course for beginners was given on the 14th of November, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20191111.pdf|HPC Basic Course]]

== HPC Basic Course - 2019-06-28==

A course for beginners was given on the 28th of June, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Linux Basic Course - 2019-06-27 ==

A course for beginners was given on the 13th of June, to help beginner Linux users gain some skills in using Linux.

== HPC CUDA/AI Course - 2019-06-21 ==

A course for interested users for deep learning and neural networks, combined with some deep level manipulation of graphics cards was given by Dell on the 21st of July.

[[File:WUR_CUDA_210619.pdf|WUR CUDA Course]]
[[File:WUR_AI_101_210619.pdf|WUR AI Course 101]]
[[File:WUR_AI_201_210619.pdf|WUR AI Course 201]]

[[File:WUR_CUDA_2_210619.pdf|WUR CUDA Course]]
[[File:WUR_Deep_Learning_Frameworks_210619.pdf|WUR Deep Learning Frameworks Primer]]
[[File:WUR_Deep_Learning_Lab_210619.pdf|WUR Deep Learning Lab]]

== HPC Advanced Course - 2019-05-28 ==

A course for experienced users was organised on the 28th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20190506.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - 2019-05-07 ==

A course for beginners will be organised on the 7th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20190506.pdf|Basic Course]]

== Linux Basic Course - 2019-04-16 ==

A course for beginners was organised on the 16th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2018-10-16 ==

A course for experienced users was organised on the 16th of October, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_Slides_20181016.pdf|Advanced Course (Gwen)]]

[[File:HPC_advanced_course_20181008.pdf|Advanced Course (Jeremie)]]

== HPC Basic Course - 2018-10-11 ==

A course for beginners was organised on the 11th of October, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC_basic_course_20181008.pdf|Basic Course]]

== Basic Linux Course - 2018-10-02 ==

A course basic Linux usage was organised on the 2nd of October, to help beginner Linux users gain some skills in using Linux.

[https://etherpad.lug.wur.nl/p/UpkF2KXDVh]

== HPC Advanced Course - 2018-05-18 ==

A course for experienced users was organised on the 18th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_20180518-GD.pdf|Advanced Course (Gwen)]]

== HPC Basic Course - 2018-05-17 ==

A course for beginners was organised on the 17th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Basic Linux Course - 2018-04-19 ==

A course basic Linux usage was organised on the 19th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2017-11-09 ==

A course for experienced users was organised on the 9th of November, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_course_2017-11-08-JV.pdf|Advanced Course (Jeremie)]]

[[File:HPC_Advanced_course_2017-11-08-GD.pdf|Advanced Course (Gwen)]]

[[File:Checkpointing_2017-11-08.pdf|Checkpointing]]

== HPC Basic Course - 2017-10-30 ==

A course for beginners was organised on the 30th of October, aiming to help absolute beginners to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:HPC_basic_course_20171025.pdf | Basic introduction to Linux]]

== HPC Teaching - 2017-06-07 ==

A course for was organised on the 7th of June, aiming to help absolute beginners (and moderately experienced users) to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:Connecting_with_Secure_Shell_to_the_HPC_20170606.pdf | Basic introduction to Linux]]

[[File:Submitting_and_monitoring_jobs_on_the_HPC_20170602.pdf | Submitting and Monitoring Jobs]]

== Old Courses ==
* [http://www.basgen.nl/sdac/ Sequence Data Analysis Course (Dec. 2012)]

Courses

2025-11-27T14:45:00Z

Pelle016:

== HPC Courses Autumn 2025 ==

17 oct 2025: [https://www.wur.nl/en/activity/linux-basic-course.htm Linux Basics]

[[Slides]]

12 nov 2025: [https://www.wur.nl/en/activity/hpc-basic-course.htm HPC Basics]

21 nov 2025: [https://www.wur.nl/en/activity/hpc-advanced-course.htm HPC Advanced]

== HPC Advanced Course - June 2024 ==

A course for experienced users of the Anunna HPC was organised in June 2024, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[https://public.anunna.wur.nl/hpc-courses/HPC-Advanced.pdf Slides]

== HPC Basic Course - June 2024==

A course for beginners shall be given in June 2024, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[https://public.anunna.wur.nl/hpc-courses/HPC-Basics.pdf Slides]

== Linux Basic Course - May 2024 ==

A beginners course to the linux (bash) command line. This course is meant to prepare students for the upcoming HPC and Containers courses.

[https://public.anunna.wur.nl/hpc-courses/Linux-Basics.pdf Slides]

== Linux Basic Course - June 2022 ==

A course for beginners shall be given in June 2022, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - November 2021 ==

A course for experienced users oth the Anunna HPC was organised in November 2021, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20211021.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - November 2021==

Two courses for beginners was given in November 2021, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20211012.pdf|HPC Basic Course]]

== Linux Basic Course - November 2021 ==

A course for beginners was given in November 2021, to help beginner Linux users gain some skills in using Linux.

== HPC Basic Course - 2019-11-14==

A course for beginners was given on the 14th of November, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20191111.pdf|HPC Basic Course]]

== HPC Basic Course - 2019-06-28==

A course for beginners was given on the 28th of June, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Linux Basic Course - 2019-06-27 ==

A course for beginners was given on the 13th of June, to help beginner Linux users gain some skills in using Linux.

== HPC CUDA/AI Course - 2019-06-21 ==

A course for interested users for deep learning and neural networks, combined with some deep level manipulation of graphics cards was given by Dell on the 21st of July.

[[File:WUR_CUDA_210619.pdf|WUR CUDA Course]]
[[File:WUR_AI_101_210619.pdf|WUR AI Course 101]]
[[File:WUR_AI_201_210619.pdf|WUR AI Course 201]]

[[File:WUR_CUDA_2_210619.pdf|WUR CUDA Course]]
[[File:WUR_Deep_Learning_Frameworks_210619.pdf|WUR Deep Learning Frameworks Primer]]
[[File:WUR_Deep_Learning_Lab_210619.pdf|WUR Deep Learning Lab]]

== HPC Advanced Course - 2019-05-28 ==

A course for experienced users was organised on the 28th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20190506.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - 2019-05-07 ==

A course for beginners will be organised on the 7th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20190506.pdf|Basic Course]]

== Linux Basic Course - 2019-04-16 ==

A course for beginners was organised on the 16th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2018-10-16 ==

A course for experienced users was organised on the 16th of October, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_Slides_20181016.pdf|Advanced Course (Gwen)]]

[[File:HPC_advanced_course_20181008.pdf|Advanced Course (Jeremie)]]

== HPC Basic Course - 2018-10-11 ==

A course for beginners was organised on the 11th of October, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC_basic_course_20181008.pdf|Basic Course]]

== Basic Linux Course - 2018-10-02 ==

A course basic Linux usage was organised on the 2nd of October, to help beginner Linux users gain some skills in using Linux.

[https://etherpad.lug.wur.nl/p/UpkF2KXDVh]

== HPC Advanced Course - 2018-05-18 ==

A course for experienced users was organised on the 18th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_20180518-GD.pdf|Advanced Course (Gwen)]]

== HPC Basic Course - 2018-05-17 ==

A course for beginners was organised on the 17th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Basic Linux Course - 2018-04-19 ==

A course basic Linux usage was organised on the 19th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2017-11-09 ==

A course for experienced users was organised on the 9th of November, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_course_2017-11-08-JV.pdf|Advanced Course (Jeremie)]]

[[File:HPC_Advanced_course_2017-11-08-GD.pdf|Advanced Course (Gwen)]]

[[File:Checkpointing_2017-11-08.pdf|Checkpointing]]

== HPC Basic Course - 2017-10-30 ==

A course for beginners was organised on the 30th of October, aiming to help absolute beginners to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:HPC_basic_course_20171025.pdf | Basic introduction to Linux]]

== HPC Teaching - 2017-06-07 ==

A course for was organised on the 7th of June, aiming to help absolute beginners (and moderately experienced users) to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:Connecting_with_Secure_Shell_to_the_HPC_20170606.pdf | Basic introduction to Linux]]

[[File:Submitting_and_monitoring_jobs_on_the_HPC_20170602.pdf | Submitting and Monitoring Jobs]]

== Old Courses ==
* [http://www.basgen.nl/sdac/ Sequence Data Analysis Course (Dec. 2012)]

Filesystems

2025-11-17T12:36:30Z

Pelle016: typo

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from the active master. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre parallel filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of two major parts (and some minor):
* /lustre/'''nobackup''' - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recovered.
* /lustre/'''backup''' - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/'''shared''' - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

And additionally:
* /lustre/'''scratch''' - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* '''/archive''' - an archive mount only accessible from the login nodes. The cost of storing data here is less than on Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<pre>
#SBATCH --tmp=<required space>
</pre>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== iRods ==
On Anunna we host our own iRods instance.

With that you can push data to the WUR tape storage for archiving at very low cost.

More info on how to use it, please see https://irods.wur.nl/.

The best course of action is to loosely follow the course, using your own data, and use your personal space for data upload and transfer to tape.

Be sure to check whether the data is correctly stored on tape before you remove your data!

On Anunna there are some differences and additions to the above site:

* The zone is HPC
* With <code>iinit</code> you can init the irods env. Use your account password.
* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>
* We have a function to ease uploads (use -h for help) : <code>itape</code>
* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Courses

2025-11-13T11:36:27Z

Pelle016: added Linux Basics Course Slides

== HPC Courses Autumn 2025 ==

17 oct 2025: [https://www.wur.nl/en/activity/linux-basic-course.htm Linux Basics]
[[File:Pdf-file.png|left|thumb|79x79px|Linux Basics Slides]]

12 nov 2025: [https://www.wur.nl/en/activity/hpc-basic-course.htm HPC Basics]

21 nov 2025: [https://www.wur.nl/en/activity/hpc-advanced-course.htm HPC Advanced]

== HPC Advanced Course - June 2024 ==

A course for experienced users of the Anunna HPC was organised in June 2024, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[https://public.anunna.wur.nl/hpc-courses/HPC-Advanced.pdf Slides]

== HPC Basic Course - June 2024==

A course for beginners shall be given in June 2024, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[https://public.anunna.wur.nl/hpc-courses/HPC-Basics.pdf Slides]

== Linux Basic Course - May 2024 ==

A beginners course to the linux (bash) command line. This course is meant to prepare students for the upcoming HPC and Containers courses.

[https://public.anunna.wur.nl/hpc-courses/Linux-Basics.pdf Slides]

== Linux Basic Course - June 2022 ==

A course for beginners shall be given in June 2022, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - November 2021 ==

A course for experienced users oth the Anunna HPC was organised in November 2021, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20211021.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - November 2021==

Two courses for beginners was given in November 2021, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20211012.pdf|HPC Basic Course]]

== Linux Basic Course - November 2021 ==

A course for beginners was given in November 2021, to help beginner Linux users gain some skills in using Linux.

== HPC Basic Course - 2019-11-14==

A course for beginners was given on the 14th of November, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20191111.pdf|HPC Basic Course]]

== HPC Basic Course - 2019-06-28==

A course for beginners was given on the 28th of June, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Linux Basic Course - 2019-06-27 ==

A course for beginners was given on the 13th of June, to help beginner Linux users gain some skills in using Linux.

== HPC CUDA/AI Course - 2019-06-21 ==

A course for interested users for deep learning and neural networks, combined with some deep level manipulation of graphics cards was given by Dell on the 21st of July.

[[File:WUR_CUDA_210619.pdf|WUR CUDA Course]]
[[File:WUR_AI_101_210619.pdf|WUR AI Course 101]]
[[File:WUR_AI_201_210619.pdf|WUR AI Course 201]]

[[File:WUR_CUDA_2_210619.pdf|WUR CUDA Course]]
[[File:WUR_Deep_Learning_Frameworks_210619.pdf|WUR Deep Learning Frameworks Primer]]
[[File:WUR_Deep_Learning_Lab_210619.pdf|WUR Deep Learning Lab]]

== HPC Advanced Course - 2019-05-28 ==

A course for experienced users was organised on the 28th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20190506.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - 2019-05-07 ==

A course for beginners will be organised on the 7th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20190506.pdf|Basic Course]]

== Linux Basic Course - 2019-04-16 ==

A course for beginners was organised on the 16th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2018-10-16 ==

A course for experienced users was organised on the 16th of October, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_Slides_20181016.pdf|Advanced Course (Gwen)]]

[[File:HPC_advanced_course_20181008.pdf|Advanced Course (Jeremie)]]

== HPC Basic Course - 2018-10-11 ==

A course for beginners was organised on the 11th of October, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC_basic_course_20181008.pdf|Basic Course]]

== Basic Linux Course - 2018-10-02 ==

A course basic Linux usage was organised on the 2nd of October, to help beginner Linux users gain some skills in using Linux.

[https://etherpad.lug.wur.nl/p/UpkF2KXDVh]

== HPC Advanced Course - 2018-05-18 ==

A course for experienced users was organised on the 18th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_20180518-GD.pdf|Advanced Course (Gwen)]]

== HPC Basic Course - 2018-05-17 ==

A course for beginners was organised on the 17th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Basic Linux Course - 2018-04-19 ==

A course basic Linux usage was organised on the 19th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2017-11-09 ==

A course for experienced users was organised on the 9th of November, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_course_2017-11-08-JV.pdf|Advanced Course (Jeremie)]]

[[File:HPC_Advanced_course_2017-11-08-GD.pdf|Advanced Course (Gwen)]]

[[File:Checkpointing_2017-11-08.pdf|Checkpointing]]

== HPC Basic Course - 2017-10-30 ==

A course for beginners was organised on the 30th of October, aiming to help absolute beginners to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:HPC_basic_course_20171025.pdf | Basic introduction to Linux]]

== HPC Teaching - 2017-06-07 ==

A course for was organised on the 7th of June, aiming to help absolute beginners (and moderately experienced users) to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:Connecting_with_Secure_Shell_to_the_HPC_20170606.pdf | Basic introduction to Linux]]

[[File:Submitting_and_monitoring_jobs_on_the_HPC_20170602.pdf | Submitting and Monitoring Jobs]]

== Old Courses ==
* [http://www.basgen.nl/sdac/ Sequence Data Analysis Course (Dec. 2012)]

File:Pdf-file.png

2025-11-13T11:32:37Z

Pelle016:

Pdf icon

File:WUR Linux Basics Course Slides.pdf

2025-11-13T11:31:21Z

Pelle016:

Slideset of course material of Linux basics course in autumn 2025

Filesystems

2025-11-12T16:10:53Z

Pelle016: irods commands added

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from the active master. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre parallel filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of two major parts (and some minor):
* /lustre/'''nobackup''' - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recovered.
* /lustre/'''backup''' - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/'''shared''' - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

And additionally:
* /lustre/'''scratch''' - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* '''/archive''' - an archive mount only accessible from the login nodes. The cost of storing data here is less than on Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<pre>
#SBATCH --tmp=<required space>
</pre>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== iRods ==
On Anunna we host our own iRods instance.

With that you can push data to the WUR tape storage for archiving at very low cost.

More info on how to use it, please see https://irods.wur.nl/.

The best course of action is to loosely follow the course, using your own data, and use your personal space for data upload and transfer to tape.

Be sure to check whether the data is correctly stored on tape before you remove your data!

On Anunna there are some differences and additions to the above site:

* The zone is HPC
* With <code>init</code> you can init the irods env. Use your account password.
* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>
* We have a function to ease uploads (use -h for help) : <code>itape</code>
* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Filesystems

2025-11-12T14:10:45Z

Pelle016: add command to help use itape command

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from the active master. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre parallel filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of two major parts (and some minor):
* /lustre/'''nobackup''' - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recovered.
* /lustre/'''backup''' - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/'''shared''' - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

And additionally:
* /lustre/'''scratch''' - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* '''/archive''' - an archive mount only accessible from the login nodes. The cost of storing data here is less than on Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<pre>
#SBATCH --tmp=<required space>
</pre>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== iRods ==
On Anunna we host our own iRods instance.

With that you can push data to the WUR tape storage for archiving at very low cost.

More info on how to use it, please see https://irods.wur.nl/.

The best course of action is to loosely follow the course, using your own data, and use your personal space for data upload and transfer to tape.

Be sure to check whether the data is correctly stored on tape before you remove your data!

On Anunna there are some differences and additions to the above site:

* The zone is HPC
* We have a function to ease uploads : <code>itape</code>
* With <code>ils</code> you can see your available irods collections. You need that as a destination location for <code>itape</code>
* We have aliases to ease checking of the status of your archive process. (it takes a while) : <code>itapestat</code> and <code>itapestatnp</code>, the first is for human use, is shows a paginated status of all your files. The latter dumps all the info, so you can e.g. use grep to filter.
* If you remove data with <code>irm</code> within iRODS, the data isn't actually removed but moved to a trashbin. The advantage is that you can retrieve it if the removal was in error, the disadvantage is that the data will keep costing money. To fix that, either use <code>irm -f</code> or the icommand to empty it, see <code>irmtrash -h</code>.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Node usage graph

2025-04-08T08:21:29Z

Pelle016: you first need to load legacy before loading anunna

There is a graphing tool that uses elements directly from sacct to display information about the current cluster usage, node_usage_graph (located at in the anunna module ).

Example:
<pre>
[user@login0 ~]# module load legacy
[user@login0 ~]# module load anunna
[user@login0 ~]# usage_graph
node: |0% 100%|
fat001: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
fat002: CCCCCCCCC
MMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node001:

node002:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmm
node003:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MM
node004:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node005:CCCCCCCCCC

node006:CCCCCCCCCC

node007:CCCCCCCCCC

node008:CCCCCCCCCCccccc
MMMMMMMMMMMMMMMMMMMMM
node009:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
node010:

node011:

node012:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node013:

node014:

node015:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node016:CCCCCCCCCCCCCCCCCCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node017:

node018:

node019:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node020:

node021:

node022:

node023:

node024:CCCCCCCCCCCCCCC

node025:CCCCCCCCCCCCCCCCCCCCC

node026:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node027:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node028:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMM
node029:

node030:

node031:

node032:

node033:

node034:

node035:

node036:

node037:

node038:

node039:

node040:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node041:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCcccccc
MMMMMMMMMMMMMMMMMMMMMM
node042:RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
node049:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node050:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node051:

node052:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMMMMMmmmmmmmmmmmmmmm
node053:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node054:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
</pre>

This gives an overview of the current per-node resource usage. There are four types of letter:
* M: Memory reserved and in use
* m: Memory reserved and not in use
* C: CPU reserved and in use
* c: CPU reserved and not in use
* D: Drained node (not available for job submission)
* R: Reserved node
* P: Node is powered off (for energy-saving)

It cannot however give you an indication of how much the queue is right now for any node. for that, squeue is a better resource.