HPCwiki - User contributions [en]

File:Pi-tutorial-speedup-vs-cores.png

2026-06-24T09:15:54Z

Honfi001: Speedup against the serial baseline. The dashed line is perfect (linear) speedup; real runs sit just below it.

== Summary ==
Speedup against the serial baseline. The dashed line is perfect (linear) speedup; real runs sit just below it.

File:Pi-tutorial-walltime-vs-cores.png

2026-06-24T09:14:55Z

Honfi001: Wall time falls as cores are added. Note both axes are logarithmic; a straight line here means near-ideal scaling.

== Summary ==
Wall time falls as cores are added. Note both axes are logarithmic; a straight line here means near-ideal scaling.

Tutorials

2026-06-24T09:01:06Z

Honfi001: /* Parallelism */

Hands-on tutorials for working on Anunna. For instructor-led courses see [[Workshops]], and for course slides and self-study material see [[Training Materials]].

== Apptainer (containers) ==

* [[Tutorials/Apptainer-Introduction|Introduction]]
* [[Tutorials/Apptainer-PullingImages|Pulling images]]
* [[Tutorials/Apptainer-FakerootAndOverlays|Fakeroot and overlays]]
* [[Tutorials/Apptainer-FakerootAndSandbox|Fakeroot and sandbox]]
* [[Tutorials/Apptainer-DefinitionFiles|Definition files]]
* [[Tutorials/Apptainer-GPUs|GPUs]]

== Linux ==

* [[Linux Basic/Linux Self Assessment]] — check whether your command-line skills are ready for the HPC and container courses.

== Parallelism ==

* [[Tutorials/Parallelism-Estimating-Pi|Estimating Pi]]

== See also ==
* [[Workshops]]
* [[Training Materials]]
* [[Apptainer]]

File:Pi-tutorial-best-per-approach.png

2026-06-24T08:59:50Z

Honfi001: Plot of comparing the performance of different parallelization strategies for the MonteCarlo Pi Tutorial

== Summary ==
Plot of comparing the performance of different parallelization strategies for the MonteCarlo Pi Tutorial

Tutorials/Parallelism-Estimating-Pi

2026-06-24T08:40:03Z

Honfi001: Created page with "This tutorial is a gentle, hands-on introduction to '''parallel computing on Anunna'''. Instead of a heavy scientific code, it uses one friendly problem — estimating the number π by throwing random darts — and runs the ''same'' calculation across every kind of parallelism the cluster offers: a single core, many threads, many processes, multiple nodes, and a GPU. Because the problem stays the same, you can see exactly what each step up the ladder buys you. You do no..."

This tutorial is a gentle, hands-on introduction to '''parallel computing on Anunna'''. Instead of a heavy scientific code, it uses one friendly problem — estimating the number π by throwing random darts — and runs the ''same'' calculation across every kind of parallelism the cluster offers: a single core, many threads, many processes, multiple nodes, and a GPU. Because the problem stays the same, you can see exactly what each step up the ladder buys you.

You do not need to write any C code. The whole exercise comes down to filling in '''two''' blanks in two job scripts, launching a batch of jobs, and reading three plots.

== What you will learn ==

* How a single program can be run serially, with threads (OpenMP), with processes (MPI), across several nodes, and on a GPU.
* How Slurm ''allocates'' resources and why your program still has to be ''told'' how many to use.
* The difference between '''strong scaling''' (same work, more workers, less time) and '''weak scaling''' (more workers, more work, same time).
* Why more workers make a calculation '''faster, not more accurate'''.
* How to read scaling and speedup plots — and where their limits are.

== Before you begin ==

You should already be comfortable logging into Anunna and submitting a basic Slurm job. If you are not sure, work through the [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] first.

You also need the '''course environment loaded''', because it sets the <code>$myScratch</code> variable that the tutorial relies on. If your instructor has not told you otherwise, it is loaded for you when you log in. You can check with:

<syntaxhighlight lang="bash">
echo "$myScratch"
</syntaxhighlight>

If that prints a path under <code>/lustre/scratch/...</code>, you are ready. If it prints nothing, load the course environment before continuing.

Everything you run lives in two places:

* '''The shared tutorial''' at <code>/lustre/shared/hpcCourses/pi-tutorial</code> — read-only, the same for everyone.
* '''Your personal copy''' at <code>$myScratch/hpcCourse/pi-tutorial</code> — created the first time you launch, and where you make your edits.

== The big idea: estimating π with random darts ==

Imagine a square dartboard with a quarter-circle drawn inside it. If you throw darts at random and they land anywhere in the square with equal chance, the fraction that land ''inside'' the quarter-circle is exactly the ratio of the two areas — which works out to π/4. So if you throw '''N''' darts and '''H''' of them land inside:

<syntaxhighlight lang="text">
π ≈ 4 × H / N
</syntaxhighlight>

Throw more darts, and the estimate gets better. This is a '''Monte Carlo''' method: a calculation built out of huge numbers of independent random trials. It has one property that makes it perfect for teaching parallelism — every dart is independent, so the work splits cleanly across as many workers as you like, with almost no coordination between them.

The tutorial implements this ''same'' dart-throwing kernel six ways:

{| class="wikitable"
! Rung !! Program !! What it uses
|-
| serial || <code>pi_serial</code> || one core (the reference)
|-
| OpenMP || <code>pi_omp</code> || many threads on one node
|-
| MPI || <code>pi_mpi</code> || many processes on one node
|-
| multi-node || <code>pi_mpi</code> || processes spread across several nodes (same program)
|-
| GPU || <code>pi_gpu</code> || thousands of device threads on one GPU
|-
| hybrid || <code>pi_hybrid</code> || MPI processes, each running OpenMP threads
|}

Because the random-number generator is identical everywhere and each worker draws its own independent stream, the ''answer'' for a given number of darts and seed is the same no matter how the work is divided. Only the ''time'' changes. That is the whole point.

== Step 1: Make the tutorial modules available ==

The tutorial ships as a set of environment modules. Point Lmod at them once per session:

<syntaxhighlight lang="bash">
module use /lustre/shared/hpcCourses/pi-tutorial/modules
module avail pi-tutorial
</syntaxhighlight>

You will see three modules. They are mutually exclusive — loading one unloads the others:

* <code>pi-tutorial</code> — the CPU rungs (serial, OpenMP, MPI, hybrid).
* <code>pi-tutorial-cuda</code> — the NVIDIA GPU rung.
* <code>pi-tutorial-rocm</code> — the AMD GPU rung.

You do not need to load these by hand for the exercise — each job script loads the right one for you. The <code>module use</code> line above is enough.

== Step 2: Get your personal copy of the scripts ==

Run the launcher once:

<syntaxhighlight lang="bash">
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

On this first run it does three things: it copies the job scripts into your personal space at <code>$myScratch/hpcCourse/pi-tutorial/slurm</code>, clears any old results, and submits the whole batch of jobs. Watch them with:

<syntaxhighlight lang="bash">
squeue --me
</syntaxhighlight>

Most of the jobs (serial, GPU, multi-node, hybrid) are complete and will run correctly. '''Two of them — the OpenMP and the MPI jobs — contain a deliberate blank''' and will not give a useful result yet. Fixing those blanks is your job, and it is the next step.

== Step 3: Fill in the two blanks ==

When you ask Slurm for resources, Slurm sets aside the cores or tasks you requested — but it does not automatically tell your ''program'' about them. You have to pass the number along. Slurm makes that easy by exporting the counts as environment variables. Your task is to connect the two.

Open the OpenMP script in your personal copy (use <code>nano</code> if you are not comfortable with <code>vi</code>):

<syntaxhighlight lang="bash">
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multithread.slurm
</syntaxhighlight>

You will find this line:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=FIXME
</syntaxhighlight>

The script asked Slurm for cores with <code>--cpus-per-task</code>, and Slurm exported that count as <code>SLURM_CPUS_PER_TASK</code>. Tell OpenMP to use exactly that many threads:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
</syntaxhighlight>

Now the MPI script:

<syntaxhighlight lang="bash">
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multiprocess.slurm
</syntaxhighlight>

Here the blank is the number of processes to launch:

<syntaxhighlight lang="bash">
mpirun -np FIXME pi_mpi --total 1e10 --seed 12345
</syntaxhighlight>

This script asked Slurm for tasks with <code>--ntasks</code>, and Slurm exported that as <code>SLURM_NTASKS</code>. Launch one process per task:

<syntaxhighlight lang="bash">
mpirun -np ${SLURM_NTASKS} pi_mpi --total 1e10 --seed 12345
</syntaxhighlight>

That is the entire editable surface — two lines. Save both files.

== Step 4: Launch the full sweep ==

Run the launcher again:

<syntaxhighlight lang="bash">
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

This time it '''keeps your edited scripts''' (it never overwrites your copy), clears the old results, and resubmits everything. The launcher sweeps a range of sizes for you automatically — OpenMP from 1 up to 128 threads, MPI from 1 up to 128 processes — plus the GPU, multi-node, and hybrid runs.

Keep an eye on the queue:

<syntaxhighlight lang="bash">
squeue --me
</syntaxhighlight>

Each finished job writes one result line into <code>$myScratch/hpcCourse/pi-tutorial/results</code>. Wait until the queue is empty before moving on.

== Step 5: Analyze your results ==

The analysis script reads every result file, prints a summary table, and writes three plots. It needs matplotlib, which lives in the SciPy bundle:

<syntaxhighlight lang="bash">
module load 2024 SciPy-bundle/2024a
python /lustre/shared/hpcCourses/pi-tutorial/analyze.py $myScratch/hpcCourse/pi-tutorial/results
</syntaxhighlight>

When it finishes you will have three PNG files in your results directory:

* <code>walltime_vs_cores.png</code> — how long each run took, against the number of cores.
* <code>speedup_vs_cores.png</code> — how much faster than the serial run, against the ideal straight line.
* <code>best_per_approach.png</code> — the single fastest run of each approach, side by side.

Copy them to your laptop (for example with <code>scp</code>) to view them, or open them through the OnDemand file browser.

== Results ==

[[File:Pi-tutorial-walltime-vs-cores.png|thumb|right|360px|Wall time falls as cores are added. Note both axes are logarithmic; a straight line here means near-ideal scaling.]]
[[File:Pi-tutorial-speedup-vs-cores.png|thumb|right|360px|Speedup against the serial baseline. The dashed line is perfect (linear) speedup; real runs sit just below it.]]
[[File:Pi-tutorial-best-per-approach.png|thumb|right|360px|The fastest run of each approach. The GPU finishes the same 10-billion-dart job in a fraction of the time the CPU rungs need.]]

A correct run produces a table like the one below (trimmed; your exact times will vary by a few percent). Every run threw the same '''10 billion''' darts.

<syntaxhighlight lang="text">
variant nodes cores device wall_s pi pct_err
serial 1 1 cpu 26.008 3.1416052 0.00040
omp 1 1 cpu 26.499 3.1416052 0.00040
omp 1 16 cpu 1.663 3.1415960 0.00011
omp 1 128 cpu 0.235 3.1416075 0.00047
mpi 1 1 cpu 69.676 3.1416052 0.00040
mpi 1 128 cpu 0.555 3.1416075 0.00047
mpi 4 64 cpu 1.090 3.1416039 0.00036
hybrid 4 128 cpu 0.217 3.1416075 0.00047
gpu 1 1 AMD_Instinct_MI210 0.080 3.1415897 0.00009
gpu 1 1 NVIDIA_A100_80GB_PCIe 0.045 3.1415897 0.00009
</syntaxhighlight>

Two things are worth noticing straight away:

* The serial run took about '''26 seconds'''; the 128-thread OpenMP run took about '''0.24 seconds'''; the A100 GPU finished in about '''0.045 seconds'''. The work was identical — only the time changed.
* Look at the <code>pi</code> column. For any given total work and seed, the estimate is '''identical''' across serial, OpenMP, MPI, multi-node, and hybrid — for example every 128-core CPU run lands on 3.1416075. Splitting the darts across more workers did not change the answer, because each worker draws its own independent random stream. (The GPU differs in the last few digits because it uses far more streams; it still lands well within the expected error.)

== Discussion ==

The plots tell a tidy story: add workers, get proportionally faster, almost along the ideal line. That is real — but it is important to understand ''why'' it looks so clean, and where the limits are.

'''This problem is unusually friendly.''' Throwing darts is "embarrassingly parallel": the workers never need to talk to each other until the very end, when their hit-counts are summed. Most real scientific codes are not like this. A fluid-dynamics solver, a molecular-dynamics simulation, or a genome assembler has data dependencies and communication between workers, and that communication eventually caps how much faster you can go (this ceiling is known as '''Amdahl's law'''). Do not expect your own code to scale as cleanly as π does — this tutorial shows you the ''best'' case, not the typical one.

'''More workers buy speed, not accuracy.''' The error of a Monte Carlo estimate shrinks with the number of darts as roughly 1/√N — so to halve the error you must throw '''four times''' as many darts. Adding cores lets you throw a fixed number of darts ''sooner'', or throw ''more'' darts in the same time, but it never makes a fixed number of darts more accurate. Speed and accuracy are separate knobs.

'''Compare equal core counts.''' In the table, the 4-node MPI run (1.09 s) looks ''slower'' than the 1-node MPI run (0.56 s). It is not a penalty for crossing nodes — the 4-node run used '''64''' cores while the 1-node run used '''128'''. When you compare like with like (64 cores on 1 node ≈ 1.11 s versus 64 cores across 4 nodes ≈ 1.09 s) the times are essentially equal: for this workload the network between nodes is not a bottleneck. Always check the core count before reading a comparison as good or bad news.

'''The measurements are deliberately rough.''' Each point is a single run with a single seed, and the fastest runs finish in well under a second — far too short to time precisely on a shared machine. That noise is why a step in the curve can occasionally look ''better'' than perfect ("super-linear"); it is timing jitter, not magic. A proper benchmark repeats each run several times and reports the median. This tutorial trades that rigour for clarity and a short queue.

'''A real-world wrinkle.''' You may notice the single-process MPI run (about 70 s) is markedly slower than the single-thread OpenMP run (about 26 s) for the same work — and that gap persists across the sweep. That is a property of how this particular MPI program starts up and is being investigated; it is not a general rule that MPI is slower than OpenMP. It is a good reminder that ''how'' you measure matters as much as ''what'' you measure.

== Conclusion ==

You ran one calculation on a single core and on hundreds, on CPUs and on GPUs, on one node and across four — and the answer never changed, only the time. That is the core lesson of parallel computing on a cluster: '''you choose a parallel model to match the structure of your problem''', and the reward is speed, not correctness.

A few takeaways to carry forward:

* If your work is made of independent pieces (parameter sweeps, many input files, Monte Carlo trials), it will parallelise well — threads, processes, or a GPU all help.
* If your work has steps that depend on each other, expect communication to limit your speedup, and measure before assuming more cores will help.
* More cores make things faster; only more samples (or a better method) make a Monte Carlo result more accurate.

To go further, try editing a script to use <code>--per-worker</code> instead of <code>--total</code> and watch the difference between weak and strong scaling, or move on to packaging your own software with the [[Tutorials/Apptainer-Introduction|Apptainer tutorials]].

== Troubleshooting ==

* '''<code>myScratch is not set</code>''' — the course environment is not loaded. Load it and try again.
* '''A GPU job sits in the queue (pending)''' — the NVIDIA cards may be busy; the job will start when one frees up. The AMD GPU run uses a reservation and is the reliable one for the course.
* '''You edited a script badly and want a clean copy''' — delete your script folder and re-run the launcher to get fresh copies:
<syntaxhighlight lang="bash">
rm -rf "$myScratch/hpcCourse/pi-tutorial/slurm"
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

== See also ==

* [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] — check your command-line readiness.
* [[Tutorials/Apptainer-Introduction|Apptainer: Introduction]] — packaging software for reproducible runs.
* [[Tutorials|All tutorials]] · [[Workshops]] · [[Training Materials]]

Tutorials

2026-06-24T08:38:43Z

Honfi001:

Hands-on tutorials for working on Anunna. For instructor-led courses see [[Workshops]], and for course slides and self-study material see [[Training Materials]].

== Apptainer (containers) ==

* [[Tutorials/Apptainer-Introduction|Introduction]]
* [[Tutorials/Apptainer-PullingImages|Pulling images]]
* [[Tutorials/Apptainer-FakerootAndOverlays|Fakeroot and overlays]]
* [[Tutorials/Apptainer-FakerootAndSandbox|Fakeroot and sandbox]]
* [[Tutorials/Apptainer-DefinitionFiles|Definition files]]
* [[Tutorials/Apptainer-GPUs|GPUs]]

== Linux ==

* [[Linux Basic/Linux Self Assessment]] — check whether your command-line skills are ready for the HPC and container courses.

== Parallelism ==

* Estimating Pi

== See also ==
* [[Workshops]]
* [[Training Materials]]
* [[Apptainer]]

HPC Advanced

2026-06-17T12:14:06Z

Honfi001:

In this course, more experienced Anunna users will explore advanced topics such as job optimization, efficient resource usage, and installing applications and modules themselves. You are encouraged to bring your own use cases, which we will discuss and address in a plenary session where possible.

Join this free 5-hour course covering advanced HPC topics.

The course is organized by the Anunna team and SRF, with contributions from WUR scientists experienced in working on Anunna.

See [[2026 Course dates]] for scheduling and registration details.

==== Preparation ====
The course assumes that the student has attended the HPC basics course, if that is not the case, the student is able to "catch up" by completing the exercises in this page: [[HPC Advanced/Preparation Exercise|HPC Advanced Preparation Exercise]]

HPC Advanced/Preparation Exercise

2026-06-17T06:24:53Z

Honfi001: /* Part 3 — The Launcher Script weather.sh */

= HPC Basics Refresher — Weather Data Analyser =

{{Note|'''Who is this for?''' This is a self-paced preparation exercise for people who are attending the '''HPC Advanced''' course but did '''not''' attend '''HPC Basics'''. Working through it end to end means you arrive at the advanced course already comfortable with environment variables, virtual environments, launcher scripts, interactive jobs, and SLURM batch submission on '''Anunna'''.}}

'''The application:''' a small Python script that loads weather data from a Dutch weather station and draws some figures.

Source: <code>/lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py</code>

By the end you will have:

* Set up handy Lustre path variables (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) and aliases
* Created a Python virtual environment on Lustre scratch
* Written an executable launcher script (<code>weather.sh</code>)
* Measured the job's memory requirement interactively
* Written and submitted a SLURM batch script (<code>weather.slurm</code>) and collected the figures

''Conventions in this document:'' lines starting with <code>$</code> are commands you type at the shell (do not type the <code>$</code>). <code><nowiki><placeholder></nowiki></code> means ''substitute your own value''.

== Part 1 — Environment Variables & Aliases ==

The Anunna <code>getLustreDir</code> helper writes your personal Lustre directories (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) into your environment so you never have to type long paths again.

=== 1.1 — Set up the variables ===

<syntaxhighlight lang="bash">
# Provided it does not exist
# Create an empty ~/.bash_aliases if you do not already have one
$ touch ~/.bash_aliases

# Load the anunna module
$ module load anunna

# Inspect what getLustreDir does
$ getLustreDir -h

# Append the export lines to your ~/.bash_aliases
$ getLustreDir -e >> ~/.bash_aliases

# Check the result (cat, less, more or nano all work)
$ cat ~/.bash_aliases

# Reload your environment so the new variables take effect
$ source ~/.bashrc
</syntaxhighlight>

=== 1.2 — Test the variables ===

'''Test 1 — does this print your scratch folder location?'''

<syntaxhighlight lang="bash">
$ echo $myScratch
</syntaxhighlight>

'''Test 2 — does this take you to your scratch folder?'''

<syntaxhighlight lang="bash">
$ cd $myScratch
</syntaxhighlight>

If both work, your variables are live.

=== 1.3 — Bonus: convenience aliases ===

Add these lines to your <code>~/.bash_aliases</code> and reload your environment again:

<syntaxhighlight lang="bash">
alias cds="cd $myScratch"
alias cdb="cd $myBkp"
alias cdn="cd $myNoBkp"
</syntaxhighlight>

{{Note|'''Important:''' the alias lines must come '''after''' the variable definitions in <code>~/.bash_aliases</code>, because they reference those variables.}}

If your aliases still do not work after <code>source ~/.bashrc</code>, your <code>~/.bashrc</code> may not be sourcing <code>~/.bash_aliases</code> at all. Add this block to the '''end''' of your <code>~/.bashrc</code>:

<syntaxhighlight lang="bash">
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
</syntaxhighlight>

Then <code>source ~/.bashrc</code> once more.

== Part 2 — Python Virtual Environment ==

We isolate the analyser's dependencies in a virtual environment living on Lustre scratch.

=== 2.1 — Load the software stack and Python ===

<syntaxhighlight lang="bash">
$ module purge
$ module load 2024
$ module load Python/3.12.3
</syntaxhighlight>

=== 2.2 — Create the virtual environment (in Lustre scratch) ===

<syntaxhighlight lang="bash">
$ mkdir $myScratch/hpcCourse; cd $_
$ python -m venv $myScratch/hpcCourse/venv
$ source $myScratch/hpcCourse/venv/bin/activate
</syntaxhighlight>

{{Note|<code>cd $_</code> reuses the last argument of the previous command — here, the directory you just created.}}

=== 2.3 — Check you are using the venv's Python ===

<syntaxhighlight lang="bash">
$ which python # should point inside .../hpcCourse/venv/bin
$ python -V # should report Python 3.12.3
</syntaxhighlight>

=== 2.4 — Install the required libraries ===

<syntaxhighlight lang="bash">
$ pip install matplotlib pandas datetime
$ pip freeze # for checking what is installed
</syntaxhighlight>

== Part 3 — The Launcher Script <code>weather.sh</code> ==

=== Tasks ===

# Copy the Python script into your course folder:
#: <syntaxhighlight lang="bash" inline>cp /lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py $myScratch/hpcCourse/</syntaxhighlight>
# Write a bash script called '''<code>weather.sh</code>''' that loads the modules and activates the virtual environment you created, and make it executable.
# The script should end with the run command below. '''Hint:''' use variables for paths.
#: <syntaxhighlight lang="bash" inline>python $myScratch/hpcCourse/weer_vanaf_2000.py 240</syntaxhighlight>
#: (<code>240</code> is the argument passed to the analyser — the number of months of data to plot, i.e. ~20 years.)

=== Example solution — <code>weather.sh</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
# weather.sh — load the environment and run the Weather Data Analyser

# Load the software stack and Python
module purge
module load 2024
module load Python/3.12.3

# Activate the virtual environment
source "$myScratch/hpcCourse/venv/bin/activate"

# Run the analyser (240 = ID of this specific weather station)
python "$myScratch/hpcCourse/weer_vanaf_2000.py" 240
</syntaxhighlight>

Make it executable:

<syntaxhighlight lang="bash">
$ chmod +x $myScratch/hpcCourse/weather.sh
</syntaxhighlight>

== Part 4 — Find the Job Requirements ==

Before submitting a batch job you need to know how much memory it needs. We start with a deliberately small allocation and increase it until the job stops crashing.

=== Task ===

Find the job requirements (e.g. memory) of the Python script using <code>sinteractive</code>. Increase the amount of RAM until the job no longer crashes.

<syntaxhighlight lang="bash">
$ sinteractive --mem 100M --time=0-0:10 -c 1
$ cd $myScratch/hpcCourse
$ ./weather.sh
</syntaxhighlight>

If the job is killed (out-of-memory), exit the interactive session, re-launch <code>sinteractive</code> with a larger <code>--mem</code> value (e.g. <code>200M</code>, <code>500M</code>, <code>1G</code>, …) and run again. Repeat until <code>weather.sh</code> completes cleanly. '''Note the smallest <code>--mem</code> value that works''' — you will use it in Step 3.

== Part 5 — Write and Submit a SLURM Script ==

=== Tasks ===

# '''Write''' a SLURM script for the job using the resource requirements you determined in Step 2. Label it '''<code>weather.slurm</code>'''.
# '''Submit''' the job with <code>sbatch</code>.
# '''Check the status''' of your job and cancel it if needed, though probably not since the jobs are very short.
# '''Check the figures''' generated by the Python script.

{{Note|'''Tip:''' a SLURM template is available at <code>/lustre/shared/hpcCourses/Basics/script_slurm.sh</code>}}

=== Example solution — <code>weather.slurm</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=weather
#SBATCH --mem=<RAM-from-Step-2> # e.g. 500M — the value you found in Step 2
#SBATCH --time=0-0:10
#SBATCH --cpus-per-task=1
#SBATCH --output=weather-%j.out
#SBATCH --error=weather-%j.err

cd "$myScratch/hpcCourse"
./weather.sh
</syntaxhighlight>

=== Submit, monitor, and collect results ===

<syntaxhighlight lang="bash">
# Submit the job
$ sbatch weather.slurm

# Check the status of your jobs
$ squeue --me

# Cancel a job if needed (replace <jobid> with the ID from squeue / sbatch)
$ scancel <jobid>

# Once finished, inspect the output and the generated figures
$ ls $myScratch/hpcCourse
$ cat weather-<jobid>.out
</syntaxhighlight>

The analyser writes its figures into your course folder — open them to confirm the run succeeded.

== Checklist ==

* <code>echo $myScratch</code> prints your scratch path and <code>cd $myScratch</code> works
* <code>cds</code> / <code>cdb</code> / <code>cdn</code> aliases jump to the right folders
* Virtual environment activates and <code>which python</code> points inside it
* <code>weather.sh</code> is executable and runs the analyser interactively
* You know the minimum <code>--mem</code> the job needs (from Step 2)
* <code>weather.slurm</code> submits with <code>sbatch</code> and the figures are generated

You are now ready for the '''HPC Advanced''' course.

HPC Advanced/Preparation Exercise

2026-06-17T06:20:26Z

Honfi001: Fix headers

= HPC Basics Refresher — Weather Data Analyser =

{{Note|'''Who is this for?''' This is a self-paced preparation exercise for people who are attending the '''HPC Advanced''' course but did '''not''' attend '''HPC Basics'''. Working through it end to end means you arrive at the advanced course already comfortable with environment variables, virtual environments, launcher scripts, interactive jobs, and SLURM batch submission on '''Anunna'''.}}

'''The application:''' a small Python script that loads weather data from a Dutch weather station and draws some figures.

Source: <code>/lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py</code>

By the end you will have:

* Set up handy Lustre path variables (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) and aliases
* Created a Python virtual environment on Lustre scratch
* Written an executable launcher script (<code>weather.sh</code>)
* Measured the job's memory requirement interactively
* Written and submitted a SLURM batch script (<code>weather.slurm</code>) and collected the figures

''Conventions in this document:'' lines starting with <code>$</code> are commands you type at the shell (do not type the <code>$</code>). <code><nowiki><placeholder></nowiki></code> means ''substitute your own value''.

== Part 1 — Environment Variables & Aliases ==

The Anunna <code>getLustreDir</code> helper writes your personal Lustre directories (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) into your environment so you never have to type long paths again.

=== 1.1 — Set up the variables ===

<syntaxhighlight lang="bash">
# Provided it does not exist
# Create an empty ~/.bash_aliases if you do not already have one
$ touch ~/.bash_aliases

# Load the anunna module
$ module load anunna

# Inspect what getLustreDir does
$ getLustreDir -h

# Append the export lines to your ~/.bash_aliases
$ getLustreDir -e >> ~/.bash_aliases

# Check the result (cat, less, more or nano all work)
$ cat ~/.bash_aliases

# Reload your environment so the new variables take effect
$ source ~/.bashrc
</syntaxhighlight>

=== 1.2 — Test the variables ===

'''Test 1 — does this print your scratch folder location?'''

<syntaxhighlight lang="bash">
$ echo $myScratch
</syntaxhighlight>

'''Test 2 — does this take you to your scratch folder?'''

<syntaxhighlight lang="bash">
$ cd $myScratch
</syntaxhighlight>

If both work, your variables are live.

=== 1.3 — Bonus: convenience aliases ===

Add these lines to your <code>~/.bash_aliases</code> and reload your environment again:

<syntaxhighlight lang="bash">
alias cds="cd $myScratch"
alias cdb="cd $myBkp"
alias cdn="cd $myNoBkp"
</syntaxhighlight>

{{Note|'''Important:''' the alias lines must come '''after''' the variable definitions in <code>~/.bash_aliases</code>, because they reference those variables.}}

If your aliases still do not work after <code>source ~/.bashrc</code>, your <code>~/.bashrc</code> may not be sourcing <code>~/.bash_aliases</code> at all. Add this block to the '''end''' of your <code>~/.bashrc</code>:

<syntaxhighlight lang="bash">
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
</syntaxhighlight>

Then <code>source ~/.bashrc</code> once more.

== Part 2 — Python Virtual Environment ==

We isolate the analyser's dependencies in a virtual environment living on Lustre scratch.

=== 2.1 — Load the software stack and Python ===

<syntaxhighlight lang="bash">
$ module purge
$ module load 2024
$ module load Python/3.12.3
</syntaxhighlight>

=== 2.2 — Create the virtual environment (in Lustre scratch) ===

<syntaxhighlight lang="bash">
$ mkdir $myScratch/hpcCourse; cd $_
$ python -m venv $myScratch/hpcCourse/venv
$ source $myScratch/hpcCourse/venv/bin/activate
</syntaxhighlight>

{{Note|<code>cd $_</code> reuses the last argument of the previous command — here, the directory you just created.}}

=== 2.3 — Check you are using the venv's Python ===

<syntaxhighlight lang="bash">
$ which python # should point inside .../hpcCourse/venv/bin
$ python -V # should report Python 3.12.3
</syntaxhighlight>

=== 2.4 — Install the required libraries ===

<syntaxhighlight lang="bash">
$ pip install matplotlib pandas datetime
$ pip freeze # for checking what is installed
</syntaxhighlight>

== Part 3 — The Launcher Script <code>weather.sh</code> ==

=== Tasks ===

# Copy the Python script into your course folder:
#: <syntaxhighlight lang="bash" inline>cp /lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py $myScratch/hpcCourse/</syntaxhighlight>
# Write a bash script called '''<code>weather.sh</code>''' that loads the modules and activates the virtual environment you created, and make it executable.
# The script should end with the run command below. '''Hint:''' use variables for paths.
#: <syntaxhighlight lang="bash" inline>python $myScratch/hpcCourse/weer_vanaf_2000.py 240</syntaxhighlight>
#: (<code>240</code> is the argument passed to the analyser — the number of months of data to plot, i.e. ~20 years.)

=== Example solution — <code>weather.sh</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
# weather.sh — load the environment and run the Weather Data Analyser

# Load the software stack and Python
module purge
module load 2024
module load Python/3.12.3

# Activate the virtual environment
source "$myScratch/hpcCourse/venv/bin/activate"

# Run the analyser (240 = months of data to plot, ~20 years)
python "$myScratch/hpcCourse/weer_vanaf_2000.py" 240
</syntaxhighlight>

Make it executable:

<syntaxhighlight lang="bash">
$ chmod +x $myScratch/hpcCourse/weather.sh
</syntaxhighlight>

== Part 4 — Find the Job Requirements ==

Before submitting a batch job you need to know how much memory it needs. We start with a deliberately small allocation and increase it until the job stops crashing.

=== Task ===

Find the job requirements (e.g. memory) of the Python script using <code>sinteractive</code>. Increase the amount of RAM until the job no longer crashes.

<syntaxhighlight lang="bash">
$ sinteractive --mem 100M --time=0-0:10 -c 1
$ cd $myScratch/hpcCourse
$ ./weather.sh
</syntaxhighlight>

If the job is killed (out-of-memory), exit the interactive session, re-launch <code>sinteractive</code> with a larger <code>--mem</code> value (e.g. <code>200M</code>, <code>500M</code>, <code>1G</code>, …) and run again. Repeat until <code>weather.sh</code> completes cleanly. '''Note the smallest <code>--mem</code> value that works''' — you will use it in Step 3.

== Part 5 — Write and Submit a SLURM Script ==

=== Tasks ===

# '''Write''' a SLURM script for the job using the resource requirements you determined in Step 2. Label it '''<code>weather.slurm</code>'''.
# '''Submit''' the job with <code>sbatch</code>.
# '''Check the status''' of your job and cancel it if needed.
# '''Check the figures''' generated by the Python script.

{{Note|'''Tip:''' a SLURM template is available at <code>/lustre/shared/hpcCourses/Basics/script_slurm.sh</code>}}

=== Example solution — <code>weather.slurm</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=weather
#SBATCH --mem=<RAM-from-Step-2> # e.g. 500M — the value you found in Step 2
#SBATCH --time=0-0:10
#SBATCH --cpus-per-task=1
#SBATCH --output=weather-%j.out
#SBATCH --error=weather-%j.err

cd "$myScratch/hpcCourse"
./weather.sh
</syntaxhighlight>

=== Submit, monitor, and collect results ===

<syntaxhighlight lang="bash">
# Submit the job
$ sbatch weather.slurm

# Check the status of your jobs
$ squeue --me

# Cancel a job if needed (replace <jobid> with the ID from squeue / sbatch)
$ scancel <jobid>

# Once finished, inspect the output and the generated figures
$ ls $myScratch/hpcCourse
$ cat weather-<jobid>.out
</syntaxhighlight>

The analyser writes its figures into your course folder — open them to confirm the run succeeded.

== Checklist ==

* <code>echo $myScratch</code> prints your scratch path and <code>cd $myScratch</code> works
* <code>cds</code> / <code>cdb</code> / <code>cdn</code> aliases jump to the right folders
* Virtual environment activates and <code>which python</code> points inside it
* <code>weather.sh</code> is executable and runs the analyser interactively
* You know the minimum <code>--mem</code> the job needs (from Step 2)
* <code>weather.slurm</code> submits with <code>sbatch</code> and the figures are generated

You are now ready for the '''HPC Advanced''' course.

HPC Advanced/Preparation Exercise

2026-06-16T13:55:10Z

Honfi001: /* 1.1 — Set up the variables */

= HPC Basics Refresher — Weather Data Analyser =

{{Note|'''Who is this for?''' This is a self-paced preparation exercise for people who are attending the '''HPC Advanced''' course but did '''not''' attend '''HPC Basics'''. Working through it end to end means you arrive at the advanced course already comfortable with environment variables, virtual environments, launcher scripts, interactive jobs, and SLURM batch submission on '''Anunna'''.}}

'''The application:''' a small Python script that loads weather data from a Dutch weather station and draws some figures.

Source: <code>/lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py</code>

By the end you will have:

* Set up handy Lustre path variables (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) and aliases
* Created a Python virtual environment on Lustre scratch
* Written an executable launcher script (<code>weather.sh</code>)
* Measured the job's memory requirement interactively
* Written and submitted a SLURM batch script (<code>weather.slurm</code>) and collected the figures

''Conventions in this document:'' lines starting with <code>$</code> are commands you type at the shell (do not type the <code>$</code>). <code><nowiki><placeholder></nowiki></code> means ''substitute your own value''.

== Part 1 — Environment Variables & Aliases ==

The Anunna <code>getLustreDir</code> helper writes your personal Lustre directories (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) into your environment so you never have to type long paths again.

=== 1.1 — Set up the variables ===

<syntaxhighlight lang="bash">
# Provided it does not exist
# Create an empty ~/.bash_aliases if you do not already have one
$ touch ~/.bash_aliases

# Load the anunna module
$ module load anunna

# Inspect what getLustreDir does
$ getLustreDir -h

# Append the export lines to your ~/.bash_aliases
$ getLustreDir -e >> ~/.bash_aliases

# Check the result (cat, less, more or nano all work)
$ cat ~/.bash_aliases

# Reload your environment so the new variables take effect
$ source ~/.bashrc
</syntaxhighlight>

=== 1.2 — Test the variables ===

'''Test 1 — does this print your scratch folder location?'''

<syntaxhighlight lang="bash">
$ echo $myScratch
</syntaxhighlight>

'''Test 2 — does this take you to your scratch folder?'''

<syntaxhighlight lang="bash">
$ cd $myScratch
</syntaxhighlight>

If both work, your variables are live.

=== 1.3 — Bonus: convenience aliases ===

Add these lines to your <code>~/.bash_aliases</code> and reload your environment again:

<syntaxhighlight lang="bash">
alias cds="cd $myScratch"
alias cdb="cd $myBkp"
alias cdn="cd $myNoBkp"
</syntaxhighlight>

{{Note|'''Important:''' the alias lines must come '''after''' the variable definitions in <code>~/.bash_aliases</code>, because they reference those variables.}}

If your aliases still do not work after <code>source ~/.bashrc</code>, your <code>~/.bashrc</code> may not be sourcing <code>~/.bash_aliases</code> at all. Add this block to the '''end''' of your <code>~/.bashrc</code>:

<syntaxhighlight lang="bash">
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
</syntaxhighlight>

Then <code>source ~/.bashrc</code> once more.

== Part 2 (Step 0) — Python Virtual Environment ==

We isolate the analyser's dependencies in a virtual environment living on Lustre scratch.

=== 2.1 — Load the software stack and Python ===

<syntaxhighlight lang="bash">
$ module purge
$ module load 2024
$ module load Python/3.12.3
</syntaxhighlight>

=== 2.2 — Create the virtual environment (in Lustre scratch) ===

<syntaxhighlight lang="bash">
$ mkdir $myScratch/hpcCourse; cd $_
$ python -m venv $myScratch/hpcCourse/venv
$ source $myScratch/hpcCourse/venv/bin/activate
</syntaxhighlight>

{{Note|<code>cd $_</code> reuses the last argument of the previous command — here, the directory you just created.}}

=== 2.3 — Check you are using the venv's Python ===

<syntaxhighlight lang="bash">
$ which python # should point inside .../hpcCourse/venv/bin
$ python -V # should report Python 3.12.3
</syntaxhighlight>

=== 2.4 — Install the required libraries ===

<syntaxhighlight lang="bash">
$ pip install matplotlib pandas datetime
$ pip freeze # for checking what is installed
</syntaxhighlight>

== Part 3 (Step 1) — The Launcher Script <code>weather.sh</code> ==

=== Tasks ===

# Copy the Python script into your course folder:
#: <syntaxhighlight lang="bash" inline>cp /lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py $myScratch/hpcCourse/</syntaxhighlight>
# Write a bash script called '''<code>weather.sh</code>''' that loads the modules and activates the virtual environment you created, and make it executable.
# The script should end with the run command below. '''Hint:''' use variables for paths.
#: <syntaxhighlight lang="bash" inline>python $myScratch/hpcCourse/weer_vanaf_2000.py 240</syntaxhighlight>
#: (<code>240</code> is the argument passed to the analyser — the number of months of data to plot, i.e. ~20 years.)

=== Example solution — <code>weather.sh</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
# weather.sh — load the environment and run the Weather Data Analyser

# Load the software stack and Python
module purge
module load 2024
module load Python/3.12.3

# Activate the virtual environment
source "$myScratch/hpcCourse/venv/bin/activate"

# Run the analyser (240 = months of data to plot, ~20 years)
python "$myScratch/hpcCourse/weer_vanaf_2000.py" 240
</syntaxhighlight>

Make it executable:

<syntaxhighlight lang="bash">
$ chmod +x $myScratch/hpcCourse/weather.sh
</syntaxhighlight>

== Part 4 (Step 2) — Find the Job Requirements ==

Before submitting a batch job you need to know how much memory it needs. We start with a deliberately small allocation and increase it until the job stops crashing.

=== Task ===

Find the job requirements (e.g. memory) of the Python script using <code>sinteractive</code>. Increase the amount of RAM until the job no longer crashes.

<syntaxhighlight lang="bash">
$ sinteractive --mem 100M --time=0-0:10 -c 1
$ cd $myScratch/hpcCourse
$ ./weather.sh
</syntaxhighlight>

If the job is killed (out-of-memory), exit the interactive session, re-launch <code>sinteractive</code> with a larger <code>--mem</code> value (e.g. <code>200M</code>, <code>500M</code>, <code>1G</code>, …) and run again. Repeat until <code>weather.sh</code> completes cleanly. '''Note the smallest <code>--mem</code> value that works''' — you will use it in Step 3.

== Part 5 (Step 3) — Write and Submit a SLURM Script ==

=== Tasks ===

# '''Write''' a SLURM script for the job using the resource requirements you determined in Step 2. Label it '''<code>weather.slurm</code>'''.
# '''Submit''' the job with <code>sbatch</code>.
# '''Check the status''' of your job and cancel it if needed.
# '''Check the figures''' generated by the Python script.

{{Note|'''Tip:''' a SLURM template is available at <code>/lustre/shared/hpcCourses/Basics/script_slurm.sh</code>}}

=== Example solution — <code>weather.slurm</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=weather
#SBATCH --mem=<RAM-from-Step-2> # e.g. 500M — the value you found in Step 2
#SBATCH --time=0-0:10
#SBATCH --cpus-per-task=1
#SBATCH --output=weather-%j.out
#SBATCH --error=weather-%j.err

cd "$myScratch/hpcCourse"
./weather.sh
</syntaxhighlight>

=== Submit, monitor, and collect results ===

<syntaxhighlight lang="bash">
# Submit the job
$ sbatch weather.slurm

# Check the status of your jobs
$ squeue --me

# Cancel a job if needed (replace <jobid> with the ID from squeue / sbatch)
$ scancel <jobid>

# Once finished, inspect the output and the generated figures
$ ls $myScratch/hpcCourse
$ cat weather-<jobid>.out
</syntaxhighlight>

The analyser writes its figures into your course folder — open them to confirm the run succeeded.

== Checklist ==

* <code>echo $myScratch</code> prints your scratch path and <code>cd $myScratch</code> works
* <code>cds</code> / <code>cdb</code> / <code>cdn</code> aliases jump to the right folders
* Virtual environment activates and <code>which python</code> points inside it
* <code>weather.sh</code> is executable and runs the analyser interactively
* You know the minimum <code>--mem</code> the job needs (from Step 2)
* <code>weather.slurm</code> submits with <code>sbatch</code> and the figures are generated

You are now ready for the '''HPC Advanced''' course.

HPC Advanced/Preparation Exercise

2026-06-16T13:48:31Z

Honfi001: Created page with "= HPC Basics Refresher — Weather Data Analyser = {{Note|'''Who is this for?''' This is a self-paced preparation exercise for people who are attending the '''HPC Advanced''' course but did '''not''' attend '''HPC Basics'''. Working through it end to end means you arrive at the advanced course already comfortable with environment variables, virtual environments, launcher scripts, interactive jobs, and SLURM batch submission on '''Anunna'''.}} '''The application:''' a s..."

= HPC Basics Refresher — Weather Data Analyser =

{{Note|'''Who is this for?''' This is a self-paced preparation exercise for people who are attending the '''HPC Advanced''' course but did '''not''' attend '''HPC Basics'''. Working through it end to end means you arrive at the advanced course already comfortable with environment variables, virtual environments, launcher scripts, interactive jobs, and SLURM batch submission on '''Anunna'''.}}

'''The application:''' a small Python script that loads weather data from a Dutch weather station and draws some figures.

Source: <code>/lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py</code>

By the end you will have:

* Set up handy Lustre path variables (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) and aliases
* Created a Python virtual environment on Lustre scratch
* Written an executable launcher script (<code>weather.sh</code>)
* Measured the job's memory requirement interactively
* Written and submitted a SLURM batch script (<code>weather.slurm</code>) and collected the figures

''Conventions in this document:'' lines starting with <code>$</code> are commands you type at the shell (do not type the <code>$</code>). <code><nowiki><placeholder></nowiki></code> means ''substitute your own value''.

== Part 1 — Environment Variables & Aliases ==

The Anunna <code>getLustreDir</code> helper writes your personal Lustre directories (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) into your environment so you never have to type long paths again.

=== 1.1 — Set up the variables ===

<syntaxhighlight lang="bash">
# Create an empty ~/.bash_aliases if you do not already have one
$ touch ~/.bash_aliases

# Load the anunna module (this provides getLustreDir)
$ module load anunna

# Inspect what getLustreDir does
$ getLustreDir -h

# Append the export lines to your ~/.bash_aliases
$ getLustreDir -e >> ~/.bash_aliases

# Check the result (cat, less, more or nano all work)
$ cat ~/.bash_aliases

# Reload your environment so the new variables take effect
$ source ~/.bashrc
</syntaxhighlight>

=== 1.2 — Test the variables ===

'''Test 1 — does this print your scratch folder location?'''

<syntaxhighlight lang="bash">
$ echo $myScratch
</syntaxhighlight>

'''Test 2 — does this take you to your scratch folder?'''

<syntaxhighlight lang="bash">
$ cd $myScratch
</syntaxhighlight>

If both work, your variables are live.

=== 1.3 — Bonus: convenience aliases ===

Add these lines to your <code>~/.bash_aliases</code> and reload your environment again:

<syntaxhighlight lang="bash">
alias cds="cd $myScratch"
alias cdb="cd $myBkp"
alias cdn="cd $myNoBkp"
</syntaxhighlight>

{{Note|'''Important:''' the alias lines must come '''after''' the variable definitions in <code>~/.bash_aliases</code>, because they reference those variables.}}

If your aliases still do not work after <code>source ~/.bashrc</code>, your <code>~/.bashrc</code> may not be sourcing <code>~/.bash_aliases</code> at all. Add this block to the '''end''' of your <code>~/.bashrc</code>:

<syntaxhighlight lang="bash">
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
</syntaxhighlight>

Then <code>source ~/.bashrc</code> once more.

== Part 2 (Step 0) — Python Virtual Environment ==

We isolate the analyser's dependencies in a virtual environment living on Lustre scratch.

=== 2.1 — Load the software stack and Python ===

<syntaxhighlight lang="bash">
$ module purge
$ module load 2024
$ module load Python/3.12.3
</syntaxhighlight>

=== 2.2 — Create the virtual environment (in Lustre scratch) ===

<syntaxhighlight lang="bash">
$ mkdir $myScratch/hpcCourse; cd $_
$ python -m venv $myScratch/hpcCourse/venv
$ source $myScratch/hpcCourse/venv/bin/activate
</syntaxhighlight>

{{Note|<code>cd $_</code> reuses the last argument of the previous command — here, the directory you just created.}}

=== 2.3 — Check you are using the venv's Python ===

<syntaxhighlight lang="bash">
$ which python # should point inside .../hpcCourse/venv/bin
$ python -V # should report Python 3.12.3
</syntaxhighlight>

=== 2.4 — Install the required libraries ===

<syntaxhighlight lang="bash">
$ pip install matplotlib pandas datetime
$ pip freeze # for checking what is installed
</syntaxhighlight>

{{Note|'''Note on <code>datetime</code>:''' <code>datetime</code> is part of the Python '''standard library''' — it ships with the interpreter and does not actually need to be installed. The analyser imports the built-in module. The line above is kept for parity with the course slides; you only truly need <code>matplotlib</code> and <code>pandas</code>.}}

== Part 3 (Step 1) — The Launcher Script <code>weather.sh</code> ==

=== Tasks ===

# Copy the Python script into your course folder:
#: <syntaxhighlight lang="bash" inline>cp /lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py $myScratch/hpcCourse/</syntaxhighlight>
# Write a bash script called '''<code>weather.sh</code>''' that loads the modules and activates the virtual environment you created, and make it executable.
# The script should end with the run command below. '''Hint:''' use variables for paths.
#: <syntaxhighlight lang="bash" inline>python $myScratch/hpcCourse/weer_vanaf_2000.py 240</syntaxhighlight>
#: (<code>240</code> is the argument passed to the analyser — the number of months of data to plot, i.e. ~20 years.)

=== Example solution — <code>weather.sh</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
# weather.sh — load the environment and run the Weather Data Analyser

# Load the software stack and Python
module purge
module load 2024
module load Python/3.12.3

# Activate the virtual environment
source "$myScratch/hpcCourse/venv/bin/activate"

# Run the analyser (240 = months of data to plot, ~20 years)
python "$myScratch/hpcCourse/weer_vanaf_2000.py" 240
</syntaxhighlight>

Make it executable:

<syntaxhighlight lang="bash">
$ chmod +x $myScratch/hpcCourse/weather.sh
</syntaxhighlight>

== Part 4 (Step 2) — Find the Job Requirements ==

Before submitting a batch job you need to know how much memory it needs. We start with a deliberately small allocation and increase it until the job stops crashing.

=== Task ===

Find the job requirements (e.g. memory) of the Python script using <code>sinteractive</code>. Increase the amount of RAM until the job no longer crashes.

<syntaxhighlight lang="bash">
$ sinteractive --mem 100M --time=0-0:10 -c 1
$ cd $myScratch/hpcCourse
$ ./weather.sh
</syntaxhighlight>

If the job is killed (out-of-memory), exit the interactive session, re-launch <code>sinteractive</code> with a larger <code>--mem</code> value (e.g. <code>200M</code>, <code>500M</code>, <code>1G</code>, …) and run again. Repeat until <code>weather.sh</code> completes cleanly. '''Note the smallest <code>--mem</code> value that works''' — you will use it in Step 3.

== Part 5 (Step 3) — Write and Submit a SLURM Script ==

=== Tasks ===

# '''Write''' a SLURM script for the job using the resource requirements you determined in Step 2. Label it '''<code>weather.slurm</code>'''.
# '''Submit''' the job with <code>sbatch</code>.
# '''Check the status''' of your job and cancel it if needed.
# '''Check the figures''' generated by the Python script.

{{Note|'''Tip:''' a SLURM template is available at <code>/lustre/shared/hpcCourses/Basics/script_slurm.sh</code>}}

=== Example solution — <code>weather.slurm</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=weather
#SBATCH --mem=<RAM-from-Step-2> # e.g. 500M — the value you found in Step 2
#SBATCH --time=0-0:10
#SBATCH --cpus-per-task=1
#SBATCH --output=weather-%j.out
#SBATCH --error=weather-%j.err

cd "$myScratch/hpcCourse"
./weather.sh
</syntaxhighlight>

=== Submit, monitor, and collect results ===

<syntaxhighlight lang="bash">
# Submit the job
$ sbatch weather.slurm

# Check the status of your jobs
$ squeue --me

# Cancel a job if needed (replace <jobid> with the ID from squeue / sbatch)
$ scancel <jobid>

# Once finished, inspect the output and the generated figures
$ ls $myScratch/hpcCourse
$ cat weather-<jobid>.out
</syntaxhighlight>

The analyser writes its figures into your course folder — open them to confirm the run succeeded.

== Checklist ==

* <code>echo $myScratch</code> prints your scratch path and <code>cd $myScratch</code> works
* <code>cds</code> / <code>cdb</code> / <code>cdn</code> aliases jump to the right folders
* Virtual environment activates and <code>which python</code> points inside it
* <code>weather.sh</code> is executable and runs the analyser interactively
* You know the minimum <code>--mem</code> the job needs (from Step 2)
* <code>weather.slurm</code> submits with <code>sbatch</code> and the figures are generated

You are now ready for the '''HPC Advanced''' course.

Linux Basic/Linux Self Assessment

2026-06-11T09:35:21Z

Honfi001:

= Linux Basics — Self-Assessment =

This page lets you verify that you have mastered the material from the [[Linux Basics Course]]. Work through the parts in order on '''Anunna'''. The final challenge produces a report file that proves you completed every step — submit (or keep) that file as evidence.

'''Prerequisites:''' an Anunna account and the course data at <code>/lustre/shared/hpcCourses/shell-lesson-data.zip</code>.

'''Rules:'''
* Do everything from the command line. No file-manager GUIs, no OnDemand file browser (the shell is fine).
* If you get stuck, refer back to the [[:File:WUR Linux Basics Course 2026 Feb.pdf|course slides]], use <code>man</code>, <code>--help</code>, or the [https://swcarpentry.github.io/shell-novice/ Software Carpentry shell course] — that is part of the skill.

== Part 1 — Connect and orient ==

# Connect to Anunna via SSH.
# Confirm where you are with <code>pwd</code>. You should be in your home directory (<code>/home/WUR/yourusername</code>).

== Part 2 — Build the workspace (directories and navigation) ==

# In your home directory, create this directory tree '''with a single command''':
#: <pre>~/linux_test/
#: ├── data/
#: ├── scripts/
#: └── results/</pre>
#: ''Hint: one <code>mkdir</code> flag makes parent directories for you.''
# Change into <code>~/linux_test/data</code> using a '''relative''' path from your home directory.
# Go back to your home directory using only <code>cd</code> shortcuts (no full paths).

== Part 3 — Work with files ==

# Copy the course archive into your home directory and extract it:
#: <syntaxhighlight lang="bash">
cp /lustre/shared/hpcCourses/shell-lesson-data.zip ~/
unzip ~/shell-lesson-data.zip -d ~/
</syntaxhighlight>
# Find the file <code>unicorn.dat</code> inside the extracted data and '''copy''' it to <code>~/linux_test/data/sample.dat</code>.
# Inside <code>~/linux_test/data</code>, create 12 empty files named <code>run_01.log</code> through <code>run_12.log</code> '''with a single command''' (brace ranges).
# Delete '''only''' the even-numbered log files using wildcards/ranges — in as few commands as you can. Check with <code>ls</code> that exactly 6 odd-numbered files remain.

== Part 4 — Pipes, filters, redirects ==

# Count how many lines in <code>sample.dat</code> contain the pattern <code>TACCGGACAA</code>, using <code>grep</code> piped into <code>wc</code>.
# Redirect that number into <code>~/linux_test/results/pattern_count.txt</code> (it should contain a single number, nothing else).

== Part 5 — Environment variables ==

# Create an environment variable holding your full name and '''export''' it:
#: <syntaxhighlight lang="bash">
export REPORT_AUTHOR="Your Name"
</syntaxhighlight>
# Verify it exists with <code>echo $REPORT_AUTHOR</code> and find it in the output of <code>env</code>.
# ''Question to yourself: why is <code>export</code> needed here? You will find out in the final challenge — your script runs as a child process, and only exported variables reach it.''

== Part 6 — Final challenge: the report script ==

Using a command-line text editor (<code>nano</code> is what the course taught, but <code>vim</code> or <code>emacs</code> are equally fine), create the file <code>~/linux_test/scripts/generate_report.sh</code> with exactly this behaviour. Write it yourself — the skeleton below shows the required output labels, you fill in the commands.

The script must:

* start with the correct '''interpreter header''' for bash,
* write all output to <code>~/linux_test/results/report.txt</code> (overwriting any previous report),
* produce one labelled line per check, in this order:

{| class="wikitable"
! Label !! Content !! What it proves
|-
| <code>USER:</code> || output of <code>whoami</code> || you ran it yourself
|-
| <code>DATE:</code> || output of <code>date</code> || when
|-
| <code>AUTHOR:</code> || value of <code>$REPORT_AUTHOR</code> || you understand <code>export</code>
|-
| <code>SHEBANG:</code> || first line of the script itself (<code>head -n 1 "$0"</code>) || interpreter header present
|-
| <code>PERMS:</code> || permission octal of the script itself (<code>stat -c '%a' "$0"</code>) || you made it executable
|-
| <code>LOGFILES:</code> || number of <code>.log</code> files in <code>~/linux_test/data</code> || Part 3 globbing done
|-
| <code>PATTERN:</code> || contents of <code>~/linux_test/results/pattern_count.txt</code> || Part 4 done
|}

A starting skeleton (the first two lines are done for you):

<syntaxhighlight lang="bash">
#!/bin/bash

REPORT=~/linux_test/results/report.txt

echo "USER: $(whoami)" > "$REPORT"
echo "DATE: $(date)" >> "$REPORT"
# ... your turn: AUTHOR, SHEBANG, PERMS, LOGFILES, PATTERN ...
</syntaxhighlight>

''Mind the difference between <code>></code> and <code>>></code> — only the first line should overwrite.''

=== Run it ===

# Make the script executable for yourself with <code>chmod</code>.
# Change into <code>~/linux_test/scripts</code> and run it with a '''relative''' path:
#: <syntaxhighlight lang="bash">
cd ~/linux_test/scripts
./generate_report.sh
</syntaxhighlight>
# Run it once more from your home directory using the '''full''' path. Both must work.
# ''Bonus question: why does plain <code>generate_report.sh</code> (no <code>./</code>) fail? Which environment variable is responsible?''

=== Check yourself ===

Display the report:

<syntaxhighlight lang="bash">
cat ~/linux_test/results/report.txt
</syntaxhighlight>

You pass if '''all''' of the following hold:

* <code>AUTHOR:</code> shows your name (not empty — if it is empty, you forgot <code>export</code>);
* <code>SHEBANG:</code> shows <code>#!/bin/bash</code>;
* <code>PERMS:</code> starts with a <code>7</code> (e.g. <code>744</code>, <code>755</code>) — owner has read, write '''and''' execute;
* <code>LOGFILES:</code> is <code>6</code>;
* <code>PATTERN:</code> is a number greater than <code>0</code>.

If any line is wrong, fix the corresponding part and re-run the script — that is the whole point of scripts.

== For instructors ==

A submitted <code>report.txt</code> is verifiable at a glance, or in bulk:

<syntaxhighlight lang="bash">
grep -E '^(AUTHOR: .+|SHEBANG: #!/bin/bash|PERMS: 7..|LOGFILES: 6|PATTERN: [1-9])' report.txt | wc -l
# Expect: 5
</syntaxhighlight>

Asking students to also submit the script itself catches copy-pasted reports: the <code>SHEBANG:</code> and <code>PERMS:</code> lines must be consistent with the actual file.

[[Category:Courses]]
[[Category:Linux]]

Linux Basic/Linux Self Assessment

2026-06-11T09:34:55Z

Honfi001:

= Linux Basics — Self-Assessment =

This page lets you verify that you have mastered the material from the [[Linux Basics Course]]. Work through the parts in order on '''Anunna'''. The final challenge produces a report file that proves you completed every step — submit (or keep) that file as evidence.

'''Prerequisites:''' an Anunna account and the course data at <code>/lustre/shared/hpcCourses/shell-lesson-data.zip</code>.

'''Rules:'''
* Do everything from the command line. No file-manager GUIs, no OnDemand file browser (the shell is fine).
* If you get stuck, refer back to the [[:File:WUR Linux Basics Course 2026 Feb.pdf|course slides]] use <code>man</code>, <code>--help</code>, or the [https://swcarpentry.github.io/shell-novice/ Software Carpentry shell course] — that is part of the skill.

== Part 1 — Connect and orient ==

# Connect to Anunna via SSH.
# Confirm where you are with <code>pwd</code>. You should be in your home directory (<code>/home/WUR/yourusername</code>).

== Part 2 — Build the workspace (directories and navigation) ==

# In your home directory, create this directory tree '''with a single command''':
#: <pre>~/linux_test/
#: ├── data/
#: ├── scripts/
#: └── results/</pre>
#: ''Hint: one <code>mkdir</code> flag makes parent directories for you.''
# Change into <code>~/linux_test/data</code> using a '''relative''' path from your home directory.
# Go back to your home directory using only <code>cd</code> shortcuts (no full paths).

== Part 3 — Work with files ==

# Copy the course archive into your home directory and extract it:
#: <syntaxhighlight lang="bash">
cp /lustre/shared/hpcCourses/shell-lesson-data.zip ~/
unzip ~/shell-lesson-data.zip -d ~/
</syntaxhighlight>
# Find the file <code>unicorn.dat</code> inside the extracted data and '''copy''' it to <code>~/linux_test/data/sample.dat</code>.
# Inside <code>~/linux_test/data</code>, create 12 empty files named <code>run_01.log</code> through <code>run_12.log</code> '''with a single command''' (brace ranges).
# Delete '''only''' the even-numbered log files using wildcards/ranges — in as few commands as you can. Check with <code>ls</code> that exactly 6 odd-numbered files remain.

== Part 4 — Pipes, filters, redirects ==

# Count how many lines in <code>sample.dat</code> contain the pattern <code>TACCGGACAA</code>, using <code>grep</code> piped into <code>wc</code>.
# Redirect that number into <code>~/linux_test/results/pattern_count.txt</code> (it should contain a single number, nothing else).

== Part 5 — Environment variables ==

# Create an environment variable holding your full name and '''export''' it:
#: <syntaxhighlight lang="bash">
export REPORT_AUTHOR="Your Name"
</syntaxhighlight>
# Verify it exists with <code>echo $REPORT_AUTHOR</code> and find it in the output of <code>env</code>.
# ''Question to yourself: why is <code>export</code> needed here? You will find out in the final challenge — your script runs as a child process, and only exported variables reach it.''

== Part 6 — Final challenge: the report script ==

Using a command-line text editor (<code>nano</code> is what the course taught, but <code>vim</code> or <code>emacs</code> are equally fine), create the file <code>~/linux_test/scripts/generate_report.sh</code> with exactly this behaviour. Write it yourself — the skeleton below shows the required output labels, you fill in the commands.

The script must:

* start with the correct '''interpreter header''' for bash,
* write all output to <code>~/linux_test/results/report.txt</code> (overwriting any previous report),
* produce one labelled line per check, in this order:

{| class="wikitable"
! Label !! Content !! What it proves
|-
| <code>USER:</code> || output of <code>whoami</code> || you ran it yourself
|-
| <code>DATE:</code> || output of <code>date</code> || when
|-
| <code>AUTHOR:</code> || value of <code>$REPORT_AUTHOR</code> || you understand <code>export</code>
|-
| <code>SHEBANG:</code> || first line of the script itself (<code>head -n 1 "$0"</code>) || interpreter header present
|-
| <code>PERMS:</code> || permission octal of the script itself (<code>stat -c '%a' "$0"</code>) || you made it executable
|-
| <code>LOGFILES:</code> || number of <code>.log</code> files in <code>~/linux_test/data</code> || Part 3 globbing done
|-
| <code>PATTERN:</code> || contents of <code>~/linux_test/results/pattern_count.txt</code> || Part 4 done
|}

A starting skeleton (the first two lines are done for you):

<syntaxhighlight lang="bash">
#!/bin/bash

REPORT=~/linux_test/results/report.txt

echo "USER: $(whoami)" > "$REPORT"
echo "DATE: $(date)" >> "$REPORT"
# ... your turn: AUTHOR, SHEBANG, PERMS, LOGFILES, PATTERN ...
</syntaxhighlight>

''Mind the difference between <code>></code> and <code>>></code> — only the first line should overwrite.''

=== Run it ===

# Make the script executable for yourself with <code>chmod</code>.
# Change into <code>~/linux_test/scripts</code> and run it with a '''relative''' path:
#: <syntaxhighlight lang="bash">
cd ~/linux_test/scripts
./generate_report.sh
</syntaxhighlight>
# Run it once more from your home directory using the '''full''' path. Both must work.
# ''Bonus question: why does plain <code>generate_report.sh</code> (no <code>./</code>) fail? Which environment variable is responsible?''

=== Check yourself ===

Display the report:

<syntaxhighlight lang="bash">
cat ~/linux_test/results/report.txt
</syntaxhighlight>

You pass if '''all''' of the following hold:

* <code>AUTHOR:</code> shows your name (not empty — if it is empty, you forgot <code>export</code>);
* <code>SHEBANG:</code> shows <code>#!/bin/bash</code>;
* <code>PERMS:</code> starts with a <code>7</code> (e.g. <code>744</code>, <code>755</code>) — owner has read, write '''and''' execute;
* <code>LOGFILES:</code> is <code>6</code>;
* <code>PATTERN:</code> is a number greater than <code>0</code>.

If any line is wrong, fix the corresponding part and re-run the script — that is the whole point of scripts.

== For instructors ==

A submitted <code>report.txt</code> is verifiable at a glance, or in bulk:

<syntaxhighlight lang="bash">
grep -E '^(AUTHOR: .+|SHEBANG: #!/bin/bash|PERMS: 7..|LOGFILES: 6|PATTERN: [1-9])' report.txt | wc -l
# Expect: 5
</syntaxhighlight>

Asking students to also submit the script itself catches copy-pasted reports: the <code>SHEBANG:</code> and <code>PERMS:</code> lines must be consistent with the actual file.

[[Category:Courses]]
[[Category:Linux]]

Linux Basic/Linux Self Assessment

2026-06-11T09:34:18Z

Honfi001:

= Linux Basics — Self-Assessment =

This page lets you verify that you have mastered the material from the [[Linux Basics Course]]. Work through the parts in order on '''Anunna'''. The final challenge produces a report file that proves you completed every step — submit (or keep) that file as evidence.

'''Prerequisites:''' an Anunna account and the course data at <code>/lustre/shared/hpcCourses/shell-lesson-data.zip</code>.

'''Rules:'''
* Do everything from the command line. No file-manager GUIs, no OnDemand file browser (the shell is fine).
* If you get stuck, refer back to the [[course slides]] use <code>man</code>, <code>--help</code>, or the [https://swcarpentry.github.io/shell-novice/ Software Carpentry shell course] — that is part of the skill.

== Part 1 — Connect and orient ==

# Connect to Anunna via SSH.
# Confirm where you are with <code>pwd</code>. You should be in your home directory (<code>/home/WUR/yourusername</code>).

== Part 2 — Build the workspace (directories and navigation) ==

# In your home directory, create this directory tree '''with a single command''':
#: <pre>~/linux_test/
#: ├── data/
#: ├── scripts/
#: └── results/</pre>
#: ''Hint: one <code>mkdir</code> flag makes parent directories for you.''
# Change into <code>~/linux_test/data</code> using a '''relative''' path from your home directory.
# Go back to your home directory using only <code>cd</code> shortcuts (no full paths).

== Part 3 — Work with files ==

# Copy the course archive into your home directory and extract it:
#: <syntaxhighlight lang="bash">
cp /lustre/shared/hpcCourses/shell-lesson-data.zip ~/
unzip ~/shell-lesson-data.zip -d ~/
</syntaxhighlight>
# Find the file <code>unicorn.dat</code> inside the extracted data and '''copy''' it to <code>~/linux_test/data/sample.dat</code>.
# Inside <code>~/linux_test/data</code>, create 12 empty files named <code>run_01.log</code> through <code>run_12.log</code> '''with a single command''' (brace ranges).
# Delete '''only''' the even-numbered log files using wildcards/ranges — in as few commands as you can. Check with <code>ls</code> that exactly 6 odd-numbered files remain.

== Part 4 — Pipes, filters, redirects ==

# Count how many lines in <code>sample.dat</code> contain the pattern <code>TACCGGACAA</code>, using <code>grep</code> piped into <code>wc</code>.
# Redirect that number into <code>~/linux_test/results/pattern_count.txt</code> (it should contain a single number, nothing else).

== Part 5 — Environment variables ==

# Create an environment variable holding your full name and '''export''' it:
#: <syntaxhighlight lang="bash">
export REPORT_AUTHOR="Your Name"
</syntaxhighlight>
# Verify it exists with <code>echo $REPORT_AUTHOR</code> and find it in the output of <code>env</code>.
# ''Question to yourself: why is <code>export</code> needed here? You will find out in the final challenge — your script runs as a child process, and only exported variables reach it.''

== Part 6 — Final challenge: the report script ==

Using a command-line text editor (<code>nano</code> is what the course taught, but <code>vim</code> or <code>emacs</code> are equally fine), create the file <code>~/linux_test/scripts/generate_report.sh</code> with exactly this behaviour. Write it yourself — the skeleton below shows the required output labels, you fill in the commands.

The script must:

* start with the correct '''interpreter header''' for bash,
* write all output to <code>~/linux_test/results/report.txt</code> (overwriting any previous report),
* produce one labelled line per check, in this order:

{| class="wikitable"
! Label !! Content !! What it proves
|-
| <code>USER:</code> || output of <code>whoami</code> || you ran it yourself
|-
| <code>DATE:</code> || output of <code>date</code> || when
|-
| <code>AUTHOR:</code> || value of <code>$REPORT_AUTHOR</code> || you understand <code>export</code>
|-
| <code>SHEBANG:</code> || first line of the script itself (<code>head -n 1 "$0"</code>) || interpreter header present
|-
| <code>PERMS:</code> || permission octal of the script itself (<code>stat -c '%a' "$0"</code>) || you made it executable
|-
| <code>LOGFILES:</code> || number of <code>.log</code> files in <code>~/linux_test/data</code> || Part 3 globbing done
|-
| <code>PATTERN:</code> || contents of <code>~/linux_test/results/pattern_count.txt</code> || Part 4 done
|}

A starting skeleton (the first two lines are done for you):

<syntaxhighlight lang="bash">
#!/bin/bash

REPORT=~/linux_test/results/report.txt

echo "USER: $(whoami)" > "$REPORT"
echo "DATE: $(date)" >> "$REPORT"
# ... your turn: AUTHOR, SHEBANG, PERMS, LOGFILES, PATTERN ...
</syntaxhighlight>

''Mind the difference between <code>></code> and <code>>></code> — only the first line should overwrite.''

=== Run it ===

# Make the script executable for yourself with <code>chmod</code>.
# Change into <code>~/linux_test/scripts</code> and run it with a '''relative''' path:
#: <syntaxhighlight lang="bash">
cd ~/linux_test/scripts
./generate_report.sh
</syntaxhighlight>
# Run it once more from your home directory using the '''full''' path. Both must work.
# ''Bonus question: why does plain <code>generate_report.sh</code> (no <code>./</code>) fail? Which environment variable is responsible?''

=== Check yourself ===

Display the report:

<syntaxhighlight lang="bash">
cat ~/linux_test/results/report.txt
</syntaxhighlight>

You pass if '''all''' of the following hold:

* <code>AUTHOR:</code> shows your name (not empty — if it is empty, you forgot <code>export</code>);
* <code>SHEBANG:</code> shows <code>#!/bin/bash</code>;
* <code>PERMS:</code> starts with a <code>7</code> (e.g. <code>744</code>, <code>755</code>) — owner has read, write '''and''' execute;
* <code>LOGFILES:</code> is <code>6</code>;
* <code>PATTERN:</code> is a number greater than <code>0</code>.

If any line is wrong, fix the corresponding part and re-run the script — that is the whole point of scripts.

== For instructors ==

A submitted <code>report.txt</code> is verifiable at a glance, or in bulk:

<syntaxhighlight lang="bash">
grep -E '^(AUTHOR: .+|SHEBANG: #!/bin/bash|PERMS: 7..|LOGFILES: 6|PATTERN: [1-9])' report.txt | wc -l
# Expect: 5
</syntaxhighlight>

Asking students to also submit the script itself catches copy-pasted reports: the <code>SHEBANG:</code> and <code>PERMS:</code> lines must be consistent with the actual file.

[[Category:Courses]]
[[Category:Linux]]

Linux Basic/Linux Self Assessment

2026-06-11T09:32:26Z

Honfi001:

= Linux Basics — Self-Assessment =

This page lets you verify that you have mastered the material from the [[Linux Basics Course]]. Work through the parts in order on '''Anunna'''. The final challenge produces a report file that proves you completed every step — submit (or keep) that file as evidence.

'''Prerequisites:''' an Anunna account and the course data at <code>/lustre/shared/hpcCourses/shell-lesson-data.zip</code>.

'''Rules:'''
* Do everything from the command line. No file-manager GUIs, no OnDemand file browser (the shell is fine).
* If you get stuck, use <code>man</code>, <code>--help</code>, or the [https://swcarpentry.github.io/shell-novice/ Software Carpentry shell course] — that is part of the skill.

== Part 1 — Connect and orient ==

# Connect to Anunna via SSH.
# Confirm where you are with <code>pwd</code>. You should be in your home directory (<code>/home/WUR/yourusername</code>).

== Part 2 — Build the workspace (directories and navigation) ==

# In your home directory, create this directory tree '''with a single command''':
#: <pre>~/linux_test/
#: ├── data/
#: ├── scripts/
#: └── results/</pre>
#: ''Hint: one <code>mkdir</code> flag makes parent directories for you.''
# Change into <code>~/linux_test/data</code> using a '''relative''' path from your home directory.
# Go back to your home directory using only <code>cd</code> shortcuts (no full paths).

== Part 3 — Work with files ==

# Copy the course archive into your home directory and extract it:
#: <syntaxhighlight lang="bash">
cp /lustre/shared/hpcCourses/shell-lesson-data.zip ~/
unzip ~/shell-lesson-data.zip -d ~/
</syntaxhighlight>
# Find the file <code>unicorn.dat</code> inside the extracted data and '''copy''' it to <code>~/linux_test/data/sample.dat</code>.
# Inside <code>~/linux_test/data</code>, create 12 empty files named <code>run_01.log</code> through <code>run_12.log</code> '''with a single command''' (brace ranges).
# Delete '''only''' the even-numbered log files using wildcards/ranges — in as few commands as you can. Check with <code>ls</code> that exactly 6 odd-numbered files remain.

== Part 4 — Pipes, filters, redirects ==

# Count how many lines in <code>sample.dat</code> contain the pattern <code>TACCGGACAA</code>, using <code>grep</code> piped into <code>wc</code>.
# Redirect that number into <code>~/linux_test/results/pattern_count.txt</code> (it should contain a single number, nothing else).

== Part 5 — Environment variables ==

# Create an environment variable holding your full name and '''export''' it:
#: <syntaxhighlight lang="bash">
export REPORT_AUTHOR="Your Name"
</syntaxhighlight>
# Verify it exists with <code>echo $REPORT_AUTHOR</code> and find it in the output of <code>env</code>.
# ''Question to yourself: why is <code>export</code> needed here? You will find out in the final challenge — your script runs as a child process, and only exported variables reach it.''

== Part 6 — Final challenge: the report script ==

Using a command-line text editor (<code>nano</code> is what the course taught, but <code>vim</code> or <code>emacs</code> are equally fine), create the file <code>~/linux_test/scripts/generate_report.sh</code> with exactly this behaviour. Write it yourself — the skeleton below shows the required output labels, you fill in the commands.

The script must:

* start with the correct '''interpreter header''' for bash,
* write all output to <code>~/linux_test/results/report.txt</code> (overwriting any previous report),
* produce one labelled line per check, in this order:

{| class="wikitable"
! Label !! Content !! What it proves
|-
| <code>USER:</code> || output of <code>whoami</code> || you ran it yourself
|-
| <code>DATE:</code> || output of <code>date</code> || when
|-
| <code>AUTHOR:</code> || value of <code>$REPORT_AUTHOR</code> || you understand <code>export</code>
|-
| <code>SHEBANG:</code> || first line of the script itself (<code>head -n 1 "$0"</code>) || interpreter header present
|-
| <code>PERMS:</code> || permission octal of the script itself (<code>stat -c '%a' "$0"</code>) || you made it executable
|-
| <code>LOGFILES:</code> || number of <code>.log</code> files in <code>~/linux_test/data</code> || Part 3 globbing done
|-
| <code>PATTERN:</code> || contents of <code>~/linux_test/results/pattern_count.txt</code> || Part 4 done
|}

A starting skeleton (the first two lines are done for you):

<syntaxhighlight lang="bash">
#!/bin/bash

REPORT=~/linux_test/results/report.txt

echo "USER: $(whoami)" > "$REPORT"
echo "DATE: $(date)" >> "$REPORT"
# ... your turn: AUTHOR, SHEBANG, PERMS, LOGFILES, PATTERN ...
</syntaxhighlight>

''Mind the difference between <code>></code> and <code>>></code> — only the first line should overwrite.''

=== Run it ===

# Make the script executable for yourself with <code>chmod</code>.
# Change into <code>~/linux_test/scripts</code> and run it with a '''relative''' path:
#: <syntaxhighlight lang="bash">
cd ~/linux_test/scripts
./generate_report.sh
</syntaxhighlight>
# Run it once more from your home directory using the '''full''' path. Both must work.
# ''Bonus question: why does plain <code>generate_report.sh</code> (no <code>./</code>) fail? Which environment variable is responsible?''

=== Check yourself ===

Display the report:

<syntaxhighlight lang="bash">
cat ~/linux_test/results/report.txt
</syntaxhighlight>

You pass if '''all''' of the following hold:

* <code>AUTHOR:</code> shows your name (not empty — if it is empty, you forgot <code>export</code>);
* <code>SHEBANG:</code> shows <code>#!/bin/bash</code>;
* <code>PERMS:</code> starts with a <code>7</code> (e.g. <code>744</code>, <code>755</code>) — owner has read, write '''and''' execute;
* <code>LOGFILES:</code> is <code>6</code>;
* <code>PATTERN:</code> is a number greater than <code>0</code>.

If any line is wrong, fix the corresponding part and re-run the script — that is the whole point of scripts.

== For instructors ==

A submitted <code>report.txt</code> is verifiable at a glance, or in bulk:

<syntaxhighlight lang="bash">
grep -E '^(AUTHOR: .+|SHEBANG: #!/bin/bash|PERMS: 7..|LOGFILES: 6|PATTERN: [1-9])' report.txt | wc -l
# Expect: 5
</syntaxhighlight>

Asking students to also submit the script itself catches copy-pasted reports: the <code>SHEBANG:</code> and <code>PERMS:</code> lines must be consistent with the actual file.

[[Category:Courses]]
[[Category:Linux]]

Linux Basic

2026-06-11T09:24:11Z

Honfi001: Added a link to the self-assessment test

Are you interested in using the High Performance Computing (HPC) Cluster but lack basic Linux knowledge? Join our free 5-hour course on Linux fundamentals for working on the HPC. You will learn the essential Linux commands and skills needed to efficiently navigate the system and automate repetitive tasks.

The course is organized by the Anunna team and SRF, with contributions from WUR scientists experienced in working with the HPC.

'''Prerequisites:'''

Please bring your laptop so you can actively participate and complete the hands-on exercises during the workshop.
See [[2026 Course dates]] for more information on scheduling and registration details.

==== Self-study material: ====
If you are not able to attend the course, you can make use of the following resources

* [[:File:WUR Linux Basics Course 2026 Feb.pdf|Latest Slides]]: Slides of the latest course session offered at thw WUR
* [https://labex.io/linuxjourney Linux Journey]: Great site for self-paced learning about linux. It offers free virtual machines for training
* [https://www.linux-path.com/en Linux Path]: Fork of Linux journey, has additional curated
* [https://carpentries.org Software Carpentry]: Community of free lessons on many topics. For linux click Learn > lessons > Unix shell.

=== Self-assessment ===
The material of this course is a pre-requirement to the HPC and container courses offered by the WUR. In order to attend them we require you do a self-assessment test of your knowledge of the linux command line.

* [[Linux Basic/Linux Self Assessment]]

Linux Basic/Linux Self Assessment

2026-06-11T09:17:54Z

Honfi001: Created page with "= Linux Basics — Self-Assessment = This page lets you verify that you have mastered the material from the Linux Basics Course. Work through the parts in order on '''Anunna'''. The final challenge produces a report file that proves you completed every step — submit (or keep) that file as evidence. '''Prerequisites:''' an Anunna account and the course data at <code>/lustre/shared/hpcCourses/shell-lesson-data.zip</code>. '''Rules:''' * Do everything from the comm..."

= Linux Basics — Self-Assessment =

This page lets you verify that you have mastered the material from the [[Linux Basics Course]]. Work through the parts in order on '''Anunna'''. The final challenge produces a report file that proves you completed every step — submit (or keep) that file as evidence.

'''Prerequisites:''' an Anunna account and the course data at <code>/lustre/shared/hpcCourses/shell-lesson-data.zip</code>.

'''Rules:'''
* Do everything from the command line. No file-manager GUIs, no OnDemand file browser.
* If you get stuck, use <code>man</code>, <code>--help</code>, or the [https://swcarpentry.github.io/shell-novice/ Software Carpentry shell course] — that is part of the skill.

== Part 1 — Connect and orient ==

# Connect to Anunna via SSH.
# Confirm where you are with <code>pwd</code>. You should be in your home directory (<code>/home/WUR/yourusername</code>).

== Part 2 — Build the workspace (directories and navigation) ==

# In your home directory, create this directory tree '''with a single command''':
#: <pre>~/linux_test/
#: ├── data/
#: ├── scripts/
#: └── results/</pre>
#: ''Hint: one <code>mkdir</code> flag makes parent directories for you.''
# Change into <code>~/linux_test/data</code> using a '''relative''' path from your home directory.
# Go back to your home directory using only <code>cd</code> shortcuts (no full paths).

== Part 3 — Work with files ==

# Copy the course archive into your home directory and extract it:
#: <syntaxhighlight lang="bash">
cp /lustre/shared/hpcCourses/shell-lesson-data.zip ~/
unzip ~/shell-lesson-data.zip -d ~/
</syntaxhighlight>
# Find the file <code>unicorn.dat</code> inside the extracted data and '''copy''' it to <code>~/linux_test/data/sample.dat</code>.
# Inside <code>~/linux_test/data</code>, create 12 empty files named <code>run_01.log</code> through <code>run_12.log</code> '''with a single command''' (brace ranges).
# Delete '''only''' the even-numbered log files using wildcards/ranges — in as few commands as you can. Check with <code>ls</code> that exactly 6 odd-numbered files remain.

== Part 4 — Pipes, filters, redirects ==

# Count how many lines in <code>sample.dat</code> contain the pattern <code>TACCGGACAA</code>, using <code>grep</code> piped into <code>wc</code>.
# Redirect that number into <code>~/linux_test/results/pattern_count.txt</code> (it should contain a single number, nothing else).

== Part 5 — Environment variables ==

# Create an environment variable holding your full name and '''export''' it:
#: <syntaxhighlight lang="bash">
export REPORT_AUTHOR="Your Name"
</syntaxhighlight>
# Verify it exists with <code>echo $REPORT_AUTHOR</code> and find it in the output of <code>env</code>.
# ''Question to yourself: why is <code>export</code> needed here? You will find out in the final challenge — your script runs as a child process, and only exported variables reach it.''

== Part 6 — Final challenge: the report script ==

Using a command-line text editor (<code>nano</code> is what the course taught, but <code>vim</code> or <code>emacs</code> are equally fine), create the file <code>~/linux_test/scripts/generate_report.sh</code> with exactly this behaviour. Write it yourself — the skeleton below shows the required output labels, you fill in the commands.

The script must:

* start with the correct '''interpreter header''' for bash,
* write all output to <code>~/linux_test/results/report.txt</code> (overwriting any previous report),
* produce one labelled line per check, in this order:

{| class="wikitable"
! Label !! Content !! What it proves
|-
| <code>USER:</code> || output of <code>whoami</code> || you ran it yourself
|-
| <code>DATE:</code> || output of <code>date</code> || when
|-
| <code>AUTHOR:</code> || value of <code>$REPORT_AUTHOR</code> || you understand <code>export</code>
|-
| <code>SHEBANG:</code> || first line of the script itself (<code>head -n 1 "$0"</code>) || interpreter header present
|-
| <code>PERMS:</code> || permission octal of the script itself (<code>stat -c '%a' "$0"</code>) || you made it executable
|-
| <code>LOGFILES:</code> || number of <code>.log</code> files in <code>~/linux_test/data</code> || Part 3 globbing done
|-
| <code>PATTERN:</code> || contents of <code>~/linux_test/results/pattern_count.txt</code> || Part 4 done
|}

A starting skeleton (the first two lines are done for you):

<syntaxhighlight lang="bash">
#!/bin/bash

REPORT=~/linux_test/results/report.txt

echo "USER: $(whoami)" > "$REPORT"
echo "DATE: $(date)" >> "$REPORT"
# ... your turn: AUTHOR, SHEBANG, PERMS, LOGFILES, PATTERN ...
</syntaxhighlight>

''Mind the difference between <code>></code> and <code>>></code> — only the first line should overwrite.''

=== Run it ===

# Make the script executable for yourself with <code>chmod</code>.
# Change into <code>~/linux_test/scripts</code> and run it with a '''relative''' path:
#: <syntaxhighlight lang="bash">
cd ~/linux_test/scripts
./generate_report.sh
</syntaxhighlight>
# Run it once more from your home directory using the '''full''' path. Both must work.
# ''Bonus question: why does plain <code>generate_report.sh</code> (no <code>./</code>) fail? Which environment variable is responsible?''

=== Check yourself ===

Display the report:

<syntaxhighlight lang="bash">
cat ~/linux_test/results/report.txt
</syntaxhighlight>

You pass if '''all''' of the following hold:

* <code>AUTHOR:</code> shows your name (not empty — if it is empty, you forgot <code>export</code>);
* <code>SHEBANG:</code> shows <code>#!/bin/bash</code>;
* <code>PERMS:</code> starts with a <code>7</code> (e.g. <code>744</code>, <code>755</code>) — owner has read, write '''and''' execute;
* <code>LOGFILES:</code> is <code>6</code>;
* <code>PATTERN:</code> is a number greater than <code>0</code>.

If any line is wrong, fix the corresponding part and re-run the script — that is the whole point of scripts.

== For instructors ==

A submitted <code>report.txt</code> is verifiable at a glance, or in bulk:

<syntaxhighlight lang="bash">
grep -E '^(AUTHOR: .+|SHEBANG: #!/bin/bash|PERMS: 7..|LOGFILES: 6|PATTERN: [1-9])' report.txt | wc -l
# Expect: 5
</syntaxhighlight>

Asking students to also submit the script itself catches copy-pasted reports: the <code>SHEBANG:</code> and <code>PERMS:</code> lines must be consistent with the actual file.

[[Category:Courses]]
[[Category:Linux]]

SACCT

2026-04-14T05:33:28Z

Honfi001:

The command [http://slurm.schedmd.com/sacct.html sacct] is meant to show your usage of Anunna.

For instance at the login screen the command: sacct -a gives the following result:

<syntaxhighlight lang="bash">
user001@login200:~$ sacct -a
</syntaxhighlight>[[File:sacct capture.jpg]]

It is a standard view to see all the users jobs of that day.
Better is to modify this command with special parameters to see what you are using from Anunna.<syntaxhighlight lang="bash">
user001@login200:~$ sacct -P -X --delimiter=',' -S 2026-01-01 -E 2026-02-01 \
--format=comment%15,User,Partition%20,JobID,JobName,ncpus,nnodes,NodeList,Start,alloccpus,cputime%12,cputimeraw,state > usage_report_$(date -I).csv

</syntaxhighlight>The csv-file can be uploaded in Excel. This gives you the next overview.
Cputimeraw is the time of cpu-usage in seconds. The parameters "-S 2026-01-01 -E 2026-02-01" give you the start date and the end date of period you want to present.

[[File:sacct uitgebreid.jpg]]

SACCT

2026-04-14T05:32:15Z

Honfi001:

The command [http://slurm.schedmd.com/sacct.html sacct] is meant to show your usage of Anunna.

For instance at the login screen the command: sacct -a gives the following result:

'''[test001@nfs01 ~]$ sacct -a'''

[[File:sacct capture.jpg]]

It is a standard view to see all the users jobs of that day.
Better is to modify this command with special parameters to see what you are using from Anunna.<syntaxhighlight lang="bash">
user001@login200:~$ sacct -P -X --delimiter=',' -S 2026-01-01 -E 2026-02-01 \
--format=comment%15,User,Partition%20,JobID,JobName,ncpus,nnodes,NodeList,Start,alloccpus,cputime%12,cputimeraw,state > usage_report_$(date -I).csv

</syntaxhighlight>The csv-file can be uploaded in Excel. This gives you the next overview.
Cputimeraw is the time of cpu-usage in seconds. The parameters "-S 2026-01-01 -E 2026-02-01" give you the start date and the end date of period you want to present.

[[File:sacct uitgebreid.jpg]]

Tutorials

2026-03-25T10:37:06Z

Honfi001: /* Apptainer */

== Tutorials ==

=== Modules ===
[[Create Your Own Modules]]

=== Apptainer ===

* [[Tutorials/Apptainer-Introduction|Introduction]]
* [[Tutorials/Apptainer-PullingImages|Pulling Images]]
* [[Tutorials/Apptainer-FakerootAndOverlays|Fakeroot and overlays]]
* [[Tutorials/Apptainer-FakerootAndSandbox|Fakeroot and sandbox]]
* [[Tutorials/Apptainer-DefinitionFiles|Definition Files]]
* [[Tutorials/Apptainer-GPUs|GPUs]]

Tutorials

2026-03-25T10:36:07Z

Honfi001:

== Tutorials ==

=== Modules ===
[[Create Your Own Modules]]

=== Apptainer ===

* Introduction
* [[Tutorials/Apptainer-PullingImages|Pulling Images]]
* [[Tutorials/Apptainer-FakerootAndOverlays|Fakeroot and overlays]]
* [[Tutorials/Apptainer-FakerootAndSandbox|Fakeroot and sandbox]]
* [[Tutorials/Apptainer-DefinitionFiles|Definition Files]]
* [[Tutorials/Apptainer-GPUs|GPUs]]

Tutorials/Apptainer-GPUs

2026-03-25T10:33:38Z

Honfi001:

= Running Apptainer with GPUs =

Apptainer can pass through GPU hardware from the host into a container, allowing you to run GPU-accelerated workloads (such as deep learning inference or training) inside a fully contained environment. This page covers how to use both NVIDIA and AMD GPUs on the Anunna cluster.

'''Important:''' Before you begin, make sure the following are in place:

* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory.
* Set your Apptainer cache to Lustre, since these images can get quite large:

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/Apptainer
</syntaxhighlight>

== How GPU Passthrough Works ==

Apptainer does not include GPU drivers inside the container. Instead, it '''binds''' the GPU drivers and libraries from the host system into the container at runtime. This means:

* The container must include software built for the correct GPU framework (CUDA for NVIDIA, ROCm for AMD).
* The host must have the matching GPU drivers installed (which Anunna already has on the GPU nodes).
* You tell Apptainer to enable GPU access using a flag: <code>--nv</code> for NVIDIA or <code>--rocm</code> for AMD.

== NVIDIA GPUs (<code>gpu</code> partition) ==

To use NVIDIA GPUs, you need to:

# Request a node on the <code>gpu</code> partition.
# Use the <code>--nv</code> flag when running your container.

The <code>--nv</code> flag tells Apptainer to:
* Make the <code>/dev/nvidiaX</code> device entries available inside the container.
* Locate and bind the CUDA libraries from the host into the container.
* Set <code>LD_LIBRARY_PATH</code> so the container uses the host's GPU libraries.

=== Example: Transcribing Audio with Whisper (NVIDIA) ===

[https://github.com/openai/whisper OpenAI Whisper] is a speech recognition model that benefits greatly from GPU acceleration. Let's build a container that runs Whisper on an NVIDIA GPU.

==== The Definition File ====

Create a file called <code>whisper_nvidia.def</code>:

<syntaxhighlight lang="bash">Bootstrap: docker
From: pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel

%post
apt-get update
apt-get install -y ffmpeg git
pip install openai-whisper
apt-get clean

%environment
export LC_ALL=C

%runscript
exec whisper --device cuda"$@"

%help
Whisper speech recognition container (NVIDIA GPU).
Usage: apptainer run --nv whisper_nvidia.sif <audio_file> [options]</syntaxhighlight>

A quick walkthrough of what is happening here:

{| class="wikitable"
|-
! Section !! Purpose
|-
| <code>Bootstrap / From</code> || Uses the official PyTorch Docker image, which already includes Python, PyTorch, CUDA, and cuDNN.
|-
| <code>%post</code> || Installs <code>ffmpeg</code> (required by Whisper for audio decoding), <code>git</code>, and the Whisper package itself.
|-
| <code>%environment</code> || Sets the locale to avoid encoding warnings.
|-
| <code>%runscript</code> || Makes <code>whisper</code> the default command, passing through any arguments.
|-
| <code>%help</code> || Provides usage information (accessible via <code>apptainer run-help</code>).
|}

==== Building the Image ====

Request a compute node and build:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer build whisper_nvidia.sif whisper_nvidia.def
</syntaxhighlight>

==== Running Whisper ====

Once built, run Whisper on the sample audio file. First, request a GPU node:

<syntaxhighlight lang="bash">
srun --partition=gpu --gres=gpu:1 --pty bash
</syntaxhighlight>

Then load Apptainer and run:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer run --nv whisper_nvidia.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model base
apptainer run --nv whisper_nvidia.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model large

</syntaxhighlight>

The <code>--nv</code> flag is what makes the GPU visible to the container. Without it, PyTorch would not detect any CUDA devices and Whisper would fall back to CPU (much slower).

You can verify GPU access from inside the container with:

<syntaxhighlight lang="bash">
apptainer exec --nv whisper_nvidia.sif python -c "import torch; print(torch.cuda.is_available())"
</syntaxhighlight>

This should print <code>True</code>.

== AMD GPUs (<code>gpu_amd</code> partition) ==

To use AMD GPUs, you need to:

# Request a node on the <code>gpu_amd</code> partition.
# Use the <code>--rocm</code> flag when running your container.

The <code>--rocm</code> flag tells Apptainer to:
* Make the <code>/dev/dri/</code> and <code>/dev/kfd</code> device entries available inside the container.
* Locate and bind the ROCm libraries from the host into the container.
* Set <code>LD_LIBRARY_PATH</code> so the container uses the host's ROCm libraries.

=== Example: Transcribing Audio with Whisper (AMD) ===

The same Whisper workflow, but using an AMD GPU with the ROCm stack.

==== The Definition File ====

Create a file called <code>whisper_amd.def</code>:

<syntaxhighlight lang="bash">
Bootstrap: docker
From: rocm/pytorch:rocm7.2_ubuntu24.04_py3.13_pytorch_release_2.10.0

%post
apt-get update
apt-get install -y ffmpeg git
pip install openai-whisper
apt-get clean

%environment
export LC_ALL=C

%runscript
exec whisper --device cuda "$@"

%help
Whisper speech recognition container (AMD ROCm GPU).
Usage: apptainer run --rocm whisper_amd.sif <audio_file> [options] --model <model_size>
</syntaxhighlight>

The structure is identical to the NVIDIA version. The only difference is the base image: instead of <code>pytorch/pytorch</code> (which includes CUDA), we use <code>rocm/pytorch</code> (which includes ROCm). PyTorch's API is the same regardless of the backend — <code>torch.cuda.is_available()</code> returns <code>True</code> on ROCm as well, since ROCm maps onto the CUDA API.

==== Building the Image ====

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer build whisper_amd.sif whisper_amd.def
</syntaxhighlight>

==== Running Whisper ====

Request an AMD GPU node:

<syntaxhighlight lang="bash">
srun --partition=gpu_amd --gres=gpu:1 --pty bash
</syntaxhighlight>

Then load Apptainer and run:

<syntaxhighlight lang="bash">module reset
module load utilities Apptainer
apptainer run --rocm whisper_amd.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model base
apptainer run --rocm whisper_amd.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model large </syntaxhighlight>

Note that even though we are on an AMD GPU, the <code>--device cuda</code> flag is correct. PyTorch's ROCm backend uses the same <code>cuda</code> device name for compatibility.

Verify GPU access:

<syntaxhighlight lang="bash">
apptainer exec --rocm whisper_amd.sif python -c "import torch; print(torch.cuda.is_available())"
</syntaxhighlight>

== Summary ==

{| class="wikitable"
|-
! !! NVIDIA !! AMD
|-
| '''Partition''' || <code>gpu</code> || <code>gpu_amd</code>
|-
| '''Apptainer flag''' || <code>--nv</code> || <code>--rocm</code>
|-
| '''Base image''' || <code>pytorch/pytorch</code> (includes CUDA) || <code>rocm/pytorch</code> (includes ROCm)
|-
| '''PyTorch device''' || <code>--device cuda</code> || <code>--device cuda</code> (same API)
|-
| '''Host devices bound''' || <code>/dev/nvidiaX</code> || <code>/dev/dri/</code>, <code>/dev/kfd</code>
|}

The key takeaway: the only things that change between NVIDIA and AMD are the '''base container image''', the '''Apptainer flag''', and the '''Slurm partition'''. Your application code stays the same.

Tutorials/Apptainer-GPUs

2026-03-25T09:42:06Z

Honfi001: Created page with "= Running Apptainer with GPUs = Apptainer can pass through GPU hardware from the host into a container, allowing you to run GPU-accelerated workloads (such as deep learning inference or training) inside a fully contained environment. This page covers how to use both NVIDIA and AMD GPUs on the Anunna cluster. '''Important:''' Before you begin, make sure the following are in place: * Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scrat..."

= Running Apptainer with GPUs =

Apptainer can pass through GPU hardware from the host into a container, allowing you to run GPU-accelerated workloads (such as deep learning inference or training) inside a fully contained environment. This page covers how to use both NVIDIA and AMD GPUs on the Anunna cluster.

'''Important:''' Before you begin, make sure the following are in place:

* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory.
* Set your Apptainer cache to Lustre:

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/Apptainer
</syntaxhighlight>

== How GPU Passthrough Works ==

Apptainer does not include GPU drivers inside the container. Instead, it '''binds''' the GPU drivers and libraries from the host system into the container at runtime. This means:

* The container must include software built for the correct GPU framework (CUDA for NVIDIA, ROCm for AMD).
* The host must have the matching GPU drivers installed (which Anunna already has on the GPU nodes).
* You tell Apptainer to enable GPU access using a flag: <code>--nv</code> for NVIDIA or <code>--rocm</code> for AMD.

== NVIDIA GPUs (<code>gpu</code> partition) ==

To use NVIDIA GPUs, you need to:

# Request a node on the <code>gpu</code> partition.
# Use the <code>--nv</code> flag when running your container.

The <code>--nv</code> flag tells Apptainer to:
* Make the <code>/dev/nvidiaX</code> device entries available inside the container.
* Locate and bind the CUDA libraries from the host into the container.
* Set <code>LD_LIBRARY_PATH</code> so the container uses the host's GPU libraries.

=== Example: Transcribing Audio with Whisper (NVIDIA) ===

[https://github.com/openai/whisper OpenAI Whisper] is a speech recognition model that benefits greatly from GPU acceleration. Let's build a container that runs Whisper on an NVIDIA GPU.

==== The Definition File ====

Create a file called <code>whisper_nvidia.def</code>:

<syntaxhighlight lang="bash">
Bootstrap: docker
From: pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel

%post
apt-get update
apt-get install -y ffmpeg git
pip install openai-whisper
apt-get clean

%environment
export LC_ALL=C

%runscript
exec whisper "$@"

%help
Whisper speech recognition container (NVIDIA GPU).
Usage: apptainer run --nv whisper_nvidia.sif <audio_file> [options]
</syntaxhighlight>

A quick walkthrough of what is happening here:

{| class="wikitable"
|-
! Section !! Purpose
|-
| <code>Bootstrap / From</code> || Uses the official PyTorch Docker image, which already includes Python, PyTorch, CUDA, and cuDNN.
|-
| <code>%post</code> || Installs <code>ffmpeg</code> (required by Whisper for audio decoding), <code>git</code>, and the Whisper package itself.
|-
| <code>%environment</code> || Sets the locale to avoid encoding warnings.
|-
| <code>%runscript</code> || Makes <code>whisper</code> the default command, passing through any arguments.
|-
| <code>%help</code> || Provides usage information (accessible via <code>apptainer run-help</code>).
|}

==== Building the Image ====

Request a compute node and build:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer build whisper_nvidia.sif whisper_nvidia.def
</syntaxhighlight>

==== Running Whisper ====

Once built, run Whisper on the sample audio file. First, request a GPU node:

<syntaxhighlight lang="bash">
srun --partition=gpu --gres=gpu:1 --pty bash
</syntaxhighlight>

Then load Apptainer and run:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer run --nv whisper_nvidia.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model base --device cuda
</syntaxhighlight>

The <code>--nv</code> flag is what makes the GPU visible to the container. Without it, PyTorch would not detect any CUDA devices and Whisper would fall back to CPU (much slower).

You can verify GPU access from inside the container with:

<syntaxhighlight lang="bash">
apptainer exec --nv whisper_nvidia.sif python -c "import torch; print(torch.cuda.is_available())"
</syntaxhighlight>

This should print <code>True</code>.

== AMD GPUs (<code>gpu_amd</code> partition) ==

To use AMD GPUs, you need to:

# Request a node on the <code>gpu_amd</code> partition.
# Use the <code>--rocm</code> flag when running your container.

The <code>--rocm</code> flag tells Apptainer to:
* Make the <code>/dev/dri/</code> and <code>/dev/kfd</code> device entries available inside the container.
* Locate and bind the ROCm libraries from the host into the container.
* Set <code>LD_LIBRARY_PATH</code> so the container uses the host's ROCm libraries.

=== Example: Transcribing Audio with Whisper (AMD) ===

The same Whisper workflow, but using an AMD GPU with the ROCm stack.

==== The Definition File ====

Create a file called <code>whisper_amd.def</code>:

<syntaxhighlight lang="bash">
Bootstrap: docker
From: rocm/pytorch:rocm6.2.4_ubuntu22.04_py3.10_pytorch_release_2.5.0

%post
apt-get update
apt-get install -y ffmpeg git
pip install openai-whisper
apt-get clean

%environment
export LC_ALL=C

%runscript
exec whisper "$@"

%help
Whisper speech recognition container (AMD ROCm GPU).
Usage: apptainer run --rocm whisper_amd.sif <audio_file> [options]
</syntaxhighlight>

The structure is identical to the NVIDIA version. The only difference is the base image: instead of <code>pytorch/pytorch</code> (which includes CUDA), we use <code>rocm/pytorch</code> (which includes ROCm). PyTorch's API is the same regardless of the backend — <code>torch.cuda.is_available()</code> returns <code>True</code> on ROCm as well, since ROCm maps onto the CUDA API.

==== Building the Image ====

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer build whisper_amd.sif whisper_amd.def
</syntaxhighlight>

==== Running Whisper ====

Request an AMD GPU node:

<syntaxhighlight lang="bash">
srun --partition=gpu_amd --gres=gpu:1 --pty bash
</syntaxhighlight>

Then load Apptainer and run:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer run --rocm whisper_amd.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model base --device cuda
</syntaxhighlight>

Note that even though we are on an AMD GPU, the <code>--device cuda</code> flag is correct. PyTorch's ROCm backend uses the same <code>cuda</code> device name for compatibility.

Verify GPU access:

<syntaxhighlight lang="bash">
apptainer exec --rocm whisper_amd.sif python -c "import torch; print(torch.cuda.is_available())"
</syntaxhighlight>

== Summary ==

{| class="wikitable"
|-
! !! NVIDIA !! AMD
|-
| '''Partition''' || <code>gpu</code> || <code>gpu_amd</code>
|-
| '''Apptainer flag''' || <code>--nv</code> || <code>--rocm</code>
|-
| '''Base image''' || <code>pytorch/pytorch</code> (includes CUDA) || <code>rocm/pytorch</code> (includes ROCm)
|-
| '''PyTorch device''' || <code>--device cuda</code> || <code>--device cuda</code> (same API)
|-
| '''Host devices bound''' || <code>/dev/nvidiaX</code> || <code>/dev/dri/</code>, <code>/dev/kfd</code>
|}

The key takeaway: the only things that change between NVIDIA and AMD are the '''base container image''', the '''Apptainer flag''', and the '''Slurm partition'''. Your application code stays the same.

Tutorials/Apptainer-Introduction

2026-03-25T09:23:54Z

Honfi001: /* apptainer pull */

= Introduction to Apptainer =

== What is Apptainer? ==

Apptainer (formerly known as Singularity) is a container platform designed for High Performance Computing (HPC) environments. If you have heard of Docker, Apptainer solves a similar problem — it lets you package an application together with all of its dependencies (libraries, tools, configuration) into a single portable unit called a '''container'''. The key difference is that Apptainer was built from the ground up to run safely on shared multi-user systems like HPC clusters, where Docker cannot be used because it requires root privileges that would pose a security risk.

Some important differences between Apptainer and Docker:

{| class="wikitable"
|-
! !! Apptainer !! Docker
|-
| '''Privileges''' || Runs as a normal user — no root access needed || Requires root or membership in the <code>docker</code> group, which is effectively root
|-
| '''Image format''' || Single portable <code>.sif</code> file || Layered images stored in a daemon's internal storage
|-
| '''Integration with host''' || Seamless access to host filesystems, GPUs, and network by default || Isolated by default; explicit configuration needed to share resources
|-
| '''Designed for''' || HPC, scientific computing, shared multi-user systems || Cloud, microservices, development environments
|-
| '''Docker compatibility''' || Can pull and run Docker images directly || N/A
|}

=== Why Apptainer for Research? ===

Two features make Apptainer particularly valuable for scientific work:

'''Portability:''' An Apptainer container is a single <code>.sif</code> file. You can build it on your laptop, copy it to the cluster, share it with a collaborator, or archive it alongside a publication. It will run the same way everywhere Apptainer is installed.

'''Reproducibility:''' A container captures the exact software environment used for an analysis — down to the specific library versions. This means an experiment can be repeated months or years later with the same environment, even if the host system has changed. When combined with a definition file (a text recipe that describes how the container was built), every step is documented and reproducible.

== Loading Apptainer ==

On the Anunna HPC cluster, Apptainer is available in the <code>utilities</code> module bucket. Load it with:

<syntaxhighlight lang="bash">
module reset
module load utilities
module load Apptainer
</syntaxhighlight>

You can verify it is loaded by running:

<syntaxhighlight lang="bash">
apptainer --version
</syntaxhighlight>

== Best Practices ==

=== Set Your Cache Directory ===

Apptainer downloads container layers to a local cache before building SIF files. By default this cache lives under your home directory (<code>$HOME/.apptainer/cache</code>), which can quickly exceed your home quota — especially with large images.

Change the cache location to your scratch space by setting the <code>APPTAINER_CACHEDIR</code> variable. You will need to create the directory first:

<syntaxhighlight lang="bash">
mkdir -p $myScratch/Apptainer
export APPTAINER_CACHEDIR=$myScratch/Apptainer
</syntaxhighlight>

Consider adding the <code>export</code> line to your <code>~/.bashrc</code> so it is set automatically in every session.

=== Do Not Run Apptainer on the Login Nodes ===

Building and running containers can be resource-intensive. Always request a compute node (e.g. via <code>srun</code>) before running Apptainer commands. The login nodes are shared and should not be used for heavy workloads.

=== Do Not Store Images in Your Home Directory ===

SIF files can easily be hundreds of megabytes or even several gigabytes in size. Store them on Lustre (e.g. in your scratch or project space) rather than in your home directory to avoid quota issues. Running images can be I/O intensive and this is not suitable for the home directory.

== Basic Commands ==

Apptainer has five core commands you will use regularly:

=== <code>apptainer pull</code> ===

Downloads a container image from a remote registry and saves it as a <code>.sif</code> file.

<syntaxhighlight lang="bash">
apptainer pull ubuntu.sif docker://ubuntu:18.04
</syntaxhighlight>

=== <code>apptainer build</code> ===

The build command is used for converting images from one format to another. It can create a SIF from a definition file, convert a sandbox to a SIF, or convert between formats.

<syntaxhighlight lang="bash">
# Build from a definition file
apptainer build myimage.sif myrecipe.def

# Convert a sandbox directory to a SIF
apptainer build myimage.sif /path/to/sandbox/
</syntaxhighlight>

=== <code>apptainer exec</code> ===

Runs a single specific command inside a container and returns the output.

<syntaxhighlight lang="bash">
apptainer exec ubuntu.sif cat /etc/os-release
</syntaxhighlight>

=== <code>apptainer shell</code> ===

Opens an interactive shell inside the container, allowing you to explore and run commands as if you had logged into a different machine.

<syntaxhighlight lang="bash">
apptainer shell ubuntu.sif
</syntaxhighlight>

=== <code>apptainer run</code> ===

Executes the container's default action (its '''runscript'''). What this does depends on how the container was built.

<syntaxhighlight lang="bash">
apptainer run myimage.sif
</syntaxhighlight>

=== At a Glance ===

{| class="wikitable"
|-
! Command !! Purpose
|-
| <code>apptainer pull</code> || Download an image from a registry
|-
| <code>apptainer build</code> || Build an image from a definition file or sandbox
|-
| <code>apptainer exec</code> || Run a single command inside a container
|-
| <code>apptainer shell</code> || Open an interactive shell inside a container
|-
| <code>apptainer run</code> || Run the container's default action
|}

Tutorials/Apptainer-Introduction

2026-03-25T09:18:12Z

Honfi001: Created page with "= Introduction to Apptainer = == What is Apptainer? == Apptainer (formerly known as Singularity) is a container platform designed for High Performance Computing (HPC) environments. If you have heard of Docker, Apptainer solves a similar problem — it lets you package an application together with all of its dependencies (libraries, tools, configuration) into a single portable unit called a '''container'''. The key difference is that Apptainer was built from the ground..."

= Introduction to Apptainer =

== What is Apptainer? ==

Apptainer (formerly known as Singularity) is a container platform designed for High Performance Computing (HPC) environments. If you have heard of Docker, Apptainer solves a similar problem — it lets you package an application together with all of its dependencies (libraries, tools, configuration) into a single portable unit called a '''container'''. The key difference is that Apptainer was built from the ground up to run safely on shared multi-user systems like HPC clusters, where Docker cannot be used because it requires root privileges that would pose a security risk.

Some important differences between Apptainer and Docker:

{| class="wikitable"
|-
! !! Apptainer !! Docker
|-
| '''Privileges''' || Runs as a normal user — no root access needed || Requires root or membership in the <code>docker</code> group, which is effectively root
|-
| '''Image format''' || Single portable <code>.sif</code> file || Layered images stored in a daemon's internal storage
|-
| '''Integration with host''' || Seamless access to host filesystems, GPUs, and network by default || Isolated by default; explicit configuration needed to share resources
|-
| '''Designed for''' || HPC, scientific computing, shared multi-user systems || Cloud, microservices, development environments
|-
| '''Docker compatibility''' || Can pull and run Docker images directly || N/A
|}

=== Why Apptainer for Research? ===

Two features make Apptainer particularly valuable for scientific work:

'''Portability:''' An Apptainer container is a single <code>.sif</code> file. You can build it on your laptop, copy it to the cluster, share it with a collaborator, or archive it alongside a publication. It will run the same way everywhere Apptainer is installed.

'''Reproducibility:''' A container captures the exact software environment used for an analysis — down to the specific library versions. This means an experiment can be repeated months or years later with the same results, even if the host system has changed. When combined with a definition file (a text recipe that describes how the container was built), every step is documented and reproducible.

== Loading Apptainer ==

On the Anunna HPC cluster, Apptainer is available in the <code>utilities</code> module bucket. Load it with:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>

You can verify it is loaded by running:

<syntaxhighlight lang="bash">
apptainer --version
</syntaxhighlight>

== Best Practices ==

=== Set Your Cache Directory ===

Apptainer downloads container layers to a local cache before building SIF files. By default this cache lives under your home directory (<code>$HOME/.apptainer/cache</code>), which can quickly exceed your home quota — especially with large images.

Change the cache location to your scratch space by setting the <code>APPTAINER_CACHEDIR</code> variable. You will need to create the directory first:

<syntaxhighlight lang="bash">
mkdir -p $myScratch/Apptainer
export APPTAINER_CACHEDIR=$myScratch/Apptainer
</syntaxhighlight>

Consider adding the <code>export</code> line to your <code>~/.bashrc</code> so it is set automatically in every session.

=== Do Not Run Apptainer on the Login Nodes ===

Building and running containers can be resource-intensive. Always request a compute node (e.g. via <code>srun</code>) before running Apptainer commands. The login nodes are shared and should not be used for heavy workloads.

=== Do Not Store Images in Your Home Directory ===

SIF files can easily be hundreds of megabytes or even several gigabytes in size. Store them on Lustre (e.g. in your scratch or project space) rather than in your home directory to avoid quota issues.

== Basic Commands ==

Apptainer has five core commands you will use regularly:

=== <code>apptainer pull</code> ===

Downloads a container image from a remote registry and saves it as a <code>.sif</code> file.

<syntaxhighlight lang="bash">
apptainer pull ubuntu.sif docker://ubuntu:24.04
</syntaxhighlight>

=== <code>apptainer build</code> ===

The build command is used for converting images from one format to another. It can create a SIF from a definition file, convert a sandbox to a SIF, or convert between formats.

<syntaxhighlight lang="bash">
# Build from a definition file
apptainer build myimage.sif myrecipe.def

# Convert a sandbox directory to a SIF
apptainer build myimage.sif /path/to/sandbox/
</syntaxhighlight>

=== <code>apptainer exec</code> ===

Runs a single specific command inside a container and returns the output.

<syntaxhighlight lang="bash">
apptainer exec ubuntu.sif cat /etc/os-release
</syntaxhighlight>

=== <code>apptainer shell</code> ===

Opens an interactive shell inside the container, allowing you to explore and run commands as if you had logged into a different machine.

<syntaxhighlight lang="bash">
apptainer shell ubuntu.sif
</syntaxhighlight>

=== <code>apptainer run</code> ===

Executes the container's default action (its '''runscript'''). What this does depends on how the container was built.

<syntaxhighlight lang="bash">
apptainer run myimage.sif
</syntaxhighlight>

=== At a Glance ===

{| class="wikitable"
|-
! Command !! Purpose
|-
| <code>apptainer pull</code> || Download an image from a registry
|-
| <code>apptainer build</code> || Build an image from a definition file or sandbox
|-
| <code>apptainer exec</code> || Run a single command inside a container
|-
| <code>apptainer shell</code> || Open an interactive shell inside a container
|-
| <code>apptainer run</code> || Run the container's default action
|}

Main Page

2026-03-25T06:56:50Z

Honfi001: Add tutorials section and edited the heading of the courses section

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [https://www.wur.nl/en/show/supercomputer-anunna-opens-up-more-opportunities-for-data-storage-and-artificial-intelligence-applications.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= About =

* [[History of the Cluster|Historical information on the startup of Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Note that the use of Anunna is not free of charge.

= Our Courses and Tutorials =
The Anunna team organizes HPC courses three times a year to strengthen basic & more advanced skills and enable users to make the most effective use of our facility.

* [[Linux Basic]]
* [[HPC Basic]]
* [[HPC Advanced]]
* [[2026 Course dates]]
* [[Tutorials]]

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Using Anunna for courses (mainly jupyter notebooks) ==
* [[steps_for_courses | Steps involved to run a course on Anunna]]

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Software =
* [[Modules]]
* [[Apptainer]]
* [[Python]]
* [[R]]
* [[Julia]]

== Browser apps ==
This page provides an overview of the GUI-based applications available Anunna, including background information and practical guidance on how to access and use interactive desktops and graphical tools directly from your web browser.

* [[General overview|General Overview]]
* [[Anunna Shell Access]]
* [[Using the File browser|File Browser]]
* [[Jupyter|Featured Apps]]
* [[Jupyter]]
* [[RStudio|R Studio]]
* [[Linux desktop|Linux Desktop]]
* [[Requesting new software|Requesting New Software]]

== Command-line Software ==

==== Cluster Scheduler ====
Anunna uses Slurm as job scheduler.
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

==== [[Globally installed software]] ====

==== [[ABGC_modules |ABGC specific modules]] ====

==== Installation of software by users ====
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Aliases and local variables]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Locale_settings]] (how numbers and dates are displayed)

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User: prins089 | Fons Prinsen (Wageningen UR, FB-IT, C&PS)]] is responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Bioinformatics_tips_tricks_workflows |Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]
* [[Old_binaries | Running older binaries on the updated OS]]
* [[locale_settings | How to change language settings for yourself]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Tutorials/Apptainer-FakerootAndOverlays

2026-03-25T06:54:45Z

Honfi001:

= Apptainer Overlays: Modifying Container Images =

'''Important:''' Before you begin, make sure the following are in place:

* You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler).
* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast.
* Set your Apptainer cache to Lustre as well. Add this to your session (or your <code>.bashrc</code>):

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/apptainer_cache
</syntaxhighlight>

== Getting Started ==

Load the required modules:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>

== Pulling the Base Image ==

Let's pull an Ubuntu 24.04 image from Docker Hub:

<syntaxhighlight lang="bash">
apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04
</syntaxhighlight>

This will download and convert the image into a SIF file in your current directory.

== The Problem: SIF Images Are Read-Only ==

Now let's try to update the package list inside the container:

<syntaxhighlight lang="bash">
apptainer exec ubuntu_24.04.sif apt update
</syntaxhighlight>

This will fail. You will see errors about not being able to write to package lists and lock files. This is because SIF files are '''read-only''' by design — you cannot modify them directly.

So how do we install software into a container? This is where '''overlays''' and '''fakeroot''' come in.

== What is fakeroot? ==

On an HPC system you do not have root (administrator) privileges. But many operations inside a container — like installing packages with <code>apt</code> — expect to be run as root.

The <code>--fakeroot</code> flag (or <code>-f</code>) tells Apptainer to make it ''look like'' you are running as root inside the container, even though you are not actually root on the host system. This is enough to trick package managers like <code>apt</code> into working.

== What is an Overlay? ==

An overlay is a writable layer that sits on top of a read-only SIF image. Think of it like putting a sheet of tracing paper over a printed page — you can write on the tracing paper without changing the original page underneath.

Any changes you make (installed packages, new files, config changes) are stored in the overlay, leaving the original SIF image untouched.

== Creating and Using an Overlay ==

Let's create a temporary directory to use as our overlay:

<syntaxhighlight lang="bash">
ov=$(mktemp -d)
</syntaxhighlight>

This creates a temporary directory somewhere under <code>/tmp</code> and stores the path in the variable <code>$ov</code>.

Now let's try that <code>apt update</code> again, this time with <code>--fakeroot</code> and <code>--overlay</code>:

<syntaxhighlight lang="bash">
apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update
</syntaxhighlight>

This time it should work. Apptainer is running you as fake root, and any writes are going into the overlay directory instead of trying to modify the read-only SIF.

== Installing Software via the Overlay ==

Now let's install <code>neofetch</code> (a tool that displays system information) into the container through the overlay:

<syntaxhighlight lang="bash">
apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch
</syntaxhighlight>

The <code>-y</code> flag tells <code>apt</code> to answer "yes" to all prompts automatically.

Once this completes, neofetch is installed — but only in the overlay. The original <code>ubuntu_24.04.sif</code> is still untouched.

The takeaway tis that the an overlay enables a read-only image to be "modified" at runtime. This enables the use of applications that dowlooad data, e.g. databases at runtime.

== Cleaning Up ==

Don't forget to remove the temporary overlay directory when you're done:

<syntaxhighlight lang="bash">
rm -rf $ov
</syntaxhighlight>

== Summary ==

{| class="wikitable"
|-
! Step !! Command
|-
| Pull base image || <code>apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04</code>
|-
| Create overlay directory || <code>ov=$(mktemp -d)</code>
|-
| Update packages (with overlay + fakeroot) || <code>apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update</code>
|-
| Install software || <code>apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch</code>
|-
| Clean up overlay || <code>rm -rf $ov</code>
|}

Tutorials

2026-03-12T13:18:36Z

Honfi001:

== Tutorials ==

=== Modules ===
[[Create Your Own Modules]]

=== Apptainer ===

* [[Tutorials/Apptainer-PullingImages|Pulling Images]]
* [[Tutorials/Apptainer-FakerootAndOverlays|Fakeroot and overlays]]
* [[Tutorials/Apptainer-FakerootAndSandbox|Fakeroot and sandbox]]
* [[Tutorials/Apptainer-DefinitionFiles|Definition Files]]

Tutorials/Apptainer-DefinitionFiles

2026-03-12T12:32:44Z

Honfi001: Created page with "= Apptainer Definition Files: Building Containers from a Recipe = '''Important:''' Before you begin, make sure the following are in place: * You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler). * Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota..."

= Apptainer Definition Files: Building Containers from a Recipe =

'''Important:''' Before you begin, make sure the following are in place:

* You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler).
* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast.
* Set your Apptainer cache to Lustre as well. Add this to your session (or your <code>.bashrc</code>):

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/apptainer_cache
</syntaxhighlight>

== Getting Started ==

Load the required modules:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>

== What is a Definition File? ==

In the previous tutorials we modified containers by hand — either through overlays or by entering a sandbox and running commands interactively. That works, but it has a problem: if someone else (or future you) wants to know exactly what was changed, there is no record of the steps taken.

A '''definition file''' (sometimes called a "def file" or "recipe") solves this. It is a plain text file that describes, step by step, how to build a container from scratch. Think of it like a cooking recipe: it lists the base ingredients and every instruction needed to produce the final result. Anyone with the same definition file can reproduce the exact same container.

== The Definition File ==

Create a file called <code>cowsay.def</code> with the following contents:

<syntaxhighlight lang="bash">
Bootstrap: docker
From: ubuntu:24.04

%post
apt-get update
apt-get install -y cowsay fortune-mod
apt-get clean

%environment
export PATH=/usr/games:${PATH}

%runscript
fortune | cowsay
exec "$@"
</syntaxhighlight>

Let's walk through each section.

=== <code>Bootstrap</code> and <code>From</code> (the Header) ===

<syntaxhighlight lang="bash">
Bootstrap: docker
From: ubuntu:24.04
</syntaxhighlight>

The header tells Apptainer where to get the base image to start from. <code>Bootstrap: docker</code> means we are pulling from a Docker/OCI registry, and <code>From: ubuntu:24.04</code> specifies which image. This is equivalent to the <code>apptainer pull docker://ubuntu:24.04</code> command we used in the earlier tutorials.

=== <code>%post</code> ===

<syntaxhighlight lang="bash">
%post
apt-get update
apt-get install -y cowsay fortune-mod
apt-get clean
</syntaxhighlight>

The <code>%post</code> section contains commands that run '''inside the container at build time''', as root. This is where you install software, download data, compile code, or make any other modifications to the container's filesystem. It is the equivalent of what we did interactively inside the sandbox shell.

=== <code>%environment</code> ===

<syntaxhighlight lang="bash">
%environment
export PATH=/usr/games:${PATH}
</syntaxhighlight>

The <code>%environment</code> section defines environment variables that are set every time the container runs. Here we add <code>/usr/games</code> to the PATH so that <code>cowsay</code> and <code>fortune</code> can be found. This replaces the manual <code>echo</code> to <code>90-environment.sh</code> that we did in the sandbox tutorial.

=== <code>%runscript</code> ===

<syntaxhighlight lang="bash">
%runscript
fortune | cowsay
exec "$@"
</syntaxhighlight>

The <code>%runscript</code> section defines the default action when someone uses <code>apptainer run</code> on the container. This replaces the manual edit we made to the runscript file in the sandbox tutorial. The <code>exec "$@"</code> line at the end allows the container to accept additional arguments if needed.

== Building the Image ==

With the definition file in place, build the SIF image:

<syntaxhighlight lang="bash">
apptainer build cowsay.sif cowsay.def
</syntaxhighlight>

Apptainer will pull the base Ubuntu image, run the <code>%post</code> commands to install the software, set up the environment and runscript, and package everything into <code>cowsay.sif</code>.

== Running It ==

<syntaxhighlight lang="bash">
apptainer run cowsay.sif
</syntaxhighlight>

You should see a random fortune inside a cow speech bubble:

<syntaxhighlight lang="text">
________________________________________
/ You will be awarded some great honor. \
\ /
----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
</syntaxhighlight>

The same result as the sandbox tutorial — but this time, every step is documented in a single file that anyone can inspect and reproduce.

== Summary ==

{| class="wikitable"
|-
! Step !! Command
|-
| Create the definition file || Write <code>cowsay.def</code> (see above)
|-
| Build the image || <code>apptainer build cowsay.sif cowsay.def</code>
|-
| Run it || <code>apptainer run cowsay.sif</code>
|}

=== Definition File Sections at a Glance ===

{| class="wikitable"
|-
! Section !! Purpose
|-
| <code>Bootstrap</code> / <code>From</code> || Where to get the base image (the starting point)
|-
| <code>%post</code> || Commands to run inside the container at build time (install software, configure things)
|-
| <code>%environment</code> || Environment variables set every time the container runs
|-
| <code>%runscript</code> || The default command executed by <code>apptainer run</code>
|}

Tutorials/Apptainer-FakerootAndSandbox

2026-03-12T12:22:43Z

Honfi001: Created page with "= Apptainer Sandbox: Modifying Containers Interactively = '''Important:''' Before you begin, make sure the following are in place: * You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler). * Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast. *..."

= Apptainer Sandbox: Modifying Containers Interactively =

'''Important:''' Before you begin, make sure the following are in place:

* You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler).
* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast.
* Set your Apptainer cache to Lustre as well. Add this to your session (or your <code>.bashrc</code>):

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/apptainer_cache
</syntaxhighlight>

== Getting Started ==

Load the required modules:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>

== What is a Sandbox? ==

In the previous tutorial we used overlays to make changes on top of a read-only SIF image. Another approach is to convert the image into a '''sandbox''' — a regular directory on disk that contains the full filesystem of the container. Because it is just a directory, you can write to it directly.

This makes it easy to interactively install software, edit config files, and generally tinker with the container before converting it back into a portable SIF file.

== Pulling as a Sandbox ==

First, create a temporary directory and pull the Ubuntu 24.04 image directly as a sandbox:

<syntaxhighlight lang="bash">
sandbox=$(mktemp -d)
apptainer pull $sandbox/ubuntu --sandbox docker://ubuntu:24.04
</syntaxhighlight>

Instead of producing a single <code>.sif</code> file, this creates a directory at <code>$sandbox/ubuntu</code> that contains the entire container filesystem. You can <code>ls</code> it to see the familiar Linux directory structure:

<syntaxhighlight lang="bash">
ls $sandbox/ubuntu
</syntaxhighlight>

You should see directories like <code>bin</code>, <code>etc</code>, <code>usr</code>, <code>var</code>, and so on — just like a normal Linux root filesystem.

== Entering the Sandbox ==

Now let's open an interactive shell inside the sandbox:

<syntaxhighlight lang="bash">
apptainer shell --containall --writable --fakeroot $sandbox/ubuntu
</syntaxhighlight>

We are using three flags here, each doing something important:

{| class="wikitable"
|-
! Flag !! What it does
|-
| <code>--containall</code> (or <code>-C</code>) || Fully isolates the container from the host system. Your home directory, host environment variables, and host filesystems are '''not''' mounted into the container. This gives you a clean environment.
|-
| <code>--writable</code> (or <code>-w</code>) || Allows you to make changes to the container's filesystem. Without this flag, even a sandbox would be mounted read-only.
|-
| <code>--fakeroot</code> (or <code>-f</code>) || Makes it look like you are running as root inside the container. This is needed because package managers like <code>apt</code> expect root privileges to install software.
|}

Your prompt should change to <code>Apptainer></code>, indicating you are now inside the container.

== Installing Software ==

From inside the container, update the package lists and install <code>nano</code>, <code>cowsay</code> and <code>fortune-mod</code>:

<syntaxhighlight lang="bash">
Apptainer> apt-get update
Apptainer> apt-get install -y nano cowsay fortune-mod
</syntaxhighlight>

Once the installation finishes, clean up the apt cache to keep the image small:

<syntaxhighlight lang="bash">
Apptainer> apt-get clean
</syntaxhighlight>

== Fixing the PATH ==

In Ubuntu, <code>cowsay</code> and <code>fortune</code> are installed into <code>/usr/games/</code>, which is '''not''' in the default PATH for non-interactive shells. If we were to build a SIF from this sandbox right now, running <code>fortune</code> or <code>cowsay</code> would fail with a "command not found" error.

We need to add <code>/usr/games</code> to the container's PATH. Apptainer reads environment settings from files in <code>/.singularity.d/env/</code>. Run this command inside the container:

<syntaxhighlight lang="bash">
Apptainer> echo 'export PATH=/usr/games:${PATH}' >> /.singularity.d/env/90-environment.sh
</syntaxhighlight>

This appends a line to the environment script that ensures <code>/usr/games</code> is on the PATH every time the container runs.

== Editing the Runscript ==

The '''runscript''' is the command that gets executed when you use <code>apptainer run</code> on the container. It lives at <code>/.singularity.d/runscript</code>. Let's edit it so that our container does something fun by default.

Open the runscript with nano:

<syntaxhighlight lang="bash">
Apptainer> nano /.singularity.d/runscript
</syntaxhighlight>

You should see something like:

<syntaxhighlight lang="bash">
#!/bin/sh
OCI_ENTRYPOINT=''
# ...various lines...
exec "$@"
</syntaxhighlight>

Add <code>fortune | cowsay</code> on the line just '''above''' the final <code>exec "$@"</code>, so the end of the file looks like this:

<syntaxhighlight lang="bash">
fortune | cowsay
exec "$@"
</syntaxhighlight>

Save and exit nano (<code>Ctrl+O</code>, <code>Enter</code>, <code>Ctrl+X</code>).

Now exit the container:

<syntaxhighlight lang="bash">
Apptainer> exit
</syntaxhighlight>

== Building the Final SIF Image ==

Convert the sandbox back into a portable SIF file:

<syntaxhighlight lang="bash">
apptainer build fortune.sif $sandbox/ubuntu
</syntaxhighlight>

This compresses the entire sandbox directory into a single <code>fortune.sif</code> file.

== Running It ==

Now for the moment of truth:

<syntaxhighlight lang="bash">
apptainer run fortune.sif
</syntaxhighlight>

You should see a random fortune displayed inside a speech bubble from a friendly cow. Something like:

<syntaxhighlight lang="text">
________________________________________
/ You will be awarded some great honor. \
\ /
----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
</syntaxhighlight>

Every time you run it, you get a different fortune.

== Cleaning Up ==

Remove the temporary sandbox directory:

<syntaxhighlight lang="bash">
rm -rf $sandbox
</syntaxhighlight>

== Summary ==

{| class="wikitable"
|-
! Step !! Command
|-
| Create temp directory || <code>sandbox=$(mktemp -d)</code>
|-
| Pull as sandbox || <code>apptainer pull $sandbox/ubuntu --sandbox docker://ubuntu:24.04</code>
|-
| Enter sandbox || <code>apptainer shell --containall --writable --fakeroot $sandbox/ubuntu</code>
|-
| Install software || <code>apt-get update && apt-get install -y nano cowsay fortune-mod && apt-get clean</code>
|-
| Fix PATH || <code>echo 'export PATH=/usr/games:${PATH}' >> /.singularity.d/env/90-environment.sh</code>
|-
| Edit runscript || <code>nano /.singularity.d/runscript</code>
|-
| Exit container || <code>exit</code>
|-
| Build SIF from sandbox || <code>apptainer build fortune.sif $sandbox/ubuntu</code>
|-
| Run it || <code>apptainer run fortune.sif</code>
|-
| Clean up || <code>rm -rf $sandbox</code>
|}

Tutorials/Apptainer-FakerootAndOverlays

2026-03-12T10:00:23Z

Honfi001: /* What is --fakeroot? */

= Apptainer Overlays: Modifying Container Images =

'''Important:''' Before you begin, make sure the following are in place:

* You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler).
* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast.
* Set your Apptainer cache to Lustre as well. Add this to your session (or your <code>.bashrc</code>):

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/apptainer_cache
</syntaxhighlight>

== Getting Started ==

Load the required modules:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>

== Pulling the Base Image ==

Let's pull an Ubuntu 24.04 image from Docker Hub:

<syntaxhighlight lang="bash">
apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04
</syntaxhighlight>

This will download and convert the image into a SIF file in your current directory.

== The Problem: SIF Images Are Read-Only ==

Now let's try to update the package list inside the container:

<syntaxhighlight lang="bash">
apptainer exec ubuntu_24.04.sif apt update
</syntaxhighlight>

This will fail. You will see errors about not being able to write to package lists and lock files. This is because SIF files are '''read-only''' by design — you cannot modify them directly.

So how do we install software into a container? This is where '''overlays''' and '''fakeroot''' come in.

== What is fakeroot? ==

On an HPC system you do not have root (administrator) privileges. But many operations inside a container — like installing packages with <code>apt</code> — expect to be run as root.

The <code>--fakeroot</code> flag (or <code>-f</code>) tells Apptainer to make it ''look like'' you are running as root inside the container, even though you are not actually root on the host system. This is enough to trick package managers like <code>apt</code> into working.

== What is an Overlay? ==

An overlay is a writable layer that sits on top of a read-only SIF image. Think of it like putting a sheet of tracing paper over a printed page — you can write on the tracing paper without changing the original page underneath.

Any changes you make (installed packages, new files, config changes) are stored in the overlay, leaving the original SIF image untouched.

== Creating and Using an Overlay ==

Let's create a temporary directory to use as our overlay:

<syntaxhighlight lang="bash">
ov=$(mktemp -d)
</syntaxhighlight>

This creates a temporary directory somewhere under <code>/tmp</code> and stores the path in the variable <code>$ov</code>.

Now let's try that <code>apt update</code> again, this time with <code>--fakeroot</code> and <code>--overlay</code>:

<syntaxhighlight lang="bash">
apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update
</syntaxhighlight>

This time it should work. Apptainer is running you as fake root, and any writes are going into the overlay directory instead of trying to modify the read-only SIF.

== Installing Software via the Overlay ==

Now let's install <code>neofetch</code> (a tool that displays system information) into the container through the overlay:

<syntaxhighlight lang="bash">
apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch
</syntaxhighlight>

The <code>-y</code> flag tells <code>apt</code> to answer "yes" to all prompts automatically.

Once this completes, neofetch is installed — but only in the overlay. The original <code>ubuntu_24.04.sif</code> is still untouched.

== Building a New Image from the Original + Overlay ==

Right now your changes only exist in that temporary overlay directory. If you were to delete it or reboot, those changes would be gone. To make your modifications permanent, you can build a new SIF image that bakes in the overlay changes:

<syntaxhighlight lang="bash">
apptainer build --fakeroot ubuntu_neofetch.sif --overlay $ov ubuntu_24.04.sif
</syntaxhighlight>

This takes the original <code>ubuntu_24.04.sif</code>, applies everything in the overlay on top of it, and produces a brand new file: <code>ubuntu_neofetch.sif</code>.

== Testing the New Image ==

Let's verify that neofetch is available in our new image:

<syntaxhighlight lang="bash">
apptainer exec ubuntu_neofetch.sif neofetch
</syntaxhighlight>

You should see the familiar Ubuntu logo and system information output. Neofetch is now permanently part of this new image — no overlay or fakeroot needed to run it.

== Cleaning Up ==

Don't forget to remove the temporary overlay directory when you're done:

<syntaxhighlight lang="bash">
rm -rf $ov
</syntaxhighlight>

== Summary ==

{| class="wikitable"
|-
! Step !! Command
|-
| Pull base image || <code>apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04</code>
|-
| Create overlay directory || <code>ov=$(mktemp -d)</code>
|-
| Update packages (with overlay + fakeroot) || <code>apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update</code>
|-
| Install software || <code>apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch</code>
|-
| Build new image from original + overlay || <code>apptainer build --fakeroot ubuntu_neofetch.sif --overlay $ov ubuntu_24.04.sif</code>
|-
| Test the new image || <code>apptainer exec ubuntu_neofetch.sif neofetch</code>
|-
| Clean up overlay || <code>rm -rf $ov</code>
|}

Tutorials/Apptainer-FakerootAndOverlays

2026-03-12T09:59:42Z

Honfi001: Created page with "= Apptainer Overlays: Modifying Container Images = '''Important:''' Before you begin, make sure the following are in place: * You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler). * Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast. * Set you..."

= Apptainer Overlays: Modifying Container Images =

'''Important:''' Before you begin, make sure the following are in place:

* You are running on a '''compute node''', not a login node. Request an interactive session first (e.g. via <code>srun</code> or your scheduler).
* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast.
* Set your Apptainer cache to Lustre as well. Add this to your session (or your <code>.bashrc</code>):

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/apptainer_cache
</syntaxhighlight>

== Getting Started ==

Load the required modules:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>

== Pulling the Base Image ==

Let's pull an Ubuntu 24.04 image from Docker Hub:

<syntaxhighlight lang="bash">
apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04
</syntaxhighlight>

This will download and convert the image into a SIF file in your current directory.

== The Problem: SIF Images Are Read-Only ==

Now let's try to update the package list inside the container:

<syntaxhighlight lang="bash">
apptainer exec ubuntu_24.04.sif apt update
</syntaxhighlight>

This will fail. You will see errors about not being able to write to package lists and lock files. This is because SIF files are '''read-only''' by design — you cannot modify them directly.

So how do we install software into a container? This is where '''overlays''' and '''fakeroot''' come in.

== What is <code>--fakeroot</code>? ==

On an HPC system you do not have root (administrator) privileges. But many operations inside a container — like installing packages with <code>apt</code> — expect to be run as root.

The <code>--fakeroot</code> flag (or <code>-f</code>) tells Apptainer to make it ''look like'' you are running as root inside the container, even though you are not actually root on the host system. This is enough to trick package managers like <code>apt</code> into working.

== What is an Overlay? ==

An overlay is a writable layer that sits on top of a read-only SIF image. Think of it like putting a sheet of tracing paper over a printed page — you can write on the tracing paper without changing the original page underneath.

Any changes you make (installed packages, new files, config changes) are stored in the overlay, leaving the original SIF image untouched.

== Creating and Using an Overlay ==

Let's create a temporary directory to use as our overlay:

<syntaxhighlight lang="bash">
ov=$(mktemp -d)
</syntaxhighlight>

This creates a temporary directory somewhere under <code>/tmp</code> and stores the path in the variable <code>$ov</code>.

Now let's try that <code>apt update</code> again, this time with <code>--fakeroot</code> and <code>--overlay</code>:

<syntaxhighlight lang="bash">
apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update
</syntaxhighlight>

This time it should work. Apptainer is running you as fake root, and any writes are going into the overlay directory instead of trying to modify the read-only SIF.

== Installing Software via the Overlay ==

Now let's install <code>neofetch</code> (a tool that displays system information) into the container through the overlay:

<syntaxhighlight lang="bash">
apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch
</syntaxhighlight>

The <code>-y</code> flag tells <code>apt</code> to answer "yes" to all prompts automatically.

Once this completes, neofetch is installed — but only in the overlay. The original <code>ubuntu_24.04.sif</code> is still untouched.

== Building a New Image from the Original + Overlay ==

Right now your changes only exist in that temporary overlay directory. If you were to delete it or reboot, those changes would be gone. To make your modifications permanent, you can build a new SIF image that bakes in the overlay changes:

<syntaxhighlight lang="bash">
apptainer build --fakeroot ubuntu_neofetch.sif --overlay $ov ubuntu_24.04.sif
</syntaxhighlight>

This takes the original <code>ubuntu_24.04.sif</code>, applies everything in the overlay on top of it, and produces a brand new file: <code>ubuntu_neofetch.sif</code>.

== Testing the New Image ==

Let's verify that neofetch is available in our new image:

<syntaxhighlight lang="bash">
apptainer exec ubuntu_neofetch.sif neofetch
</syntaxhighlight>

You should see the familiar Ubuntu logo and system information output. Neofetch is now permanently part of this new image — no overlay or fakeroot needed to run it.

== Cleaning Up ==

Don't forget to remove the temporary overlay directory when you're done:

<syntaxhighlight lang="bash">
rm -rf $ov
</syntaxhighlight>

== Summary ==

{| class="wikitable"
|-
! Step !! Command
|-
| Pull base image || <code>apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04</code>
|-
| Create overlay directory || <code>ov=$(mktemp -d)</code>
|-
| Update packages (with overlay + fakeroot) || <code>apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update</code>
|-
| Install software || <code>apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch</code>
|-
| Build new image from original + overlay || <code>apptainer build --fakeroot ubuntu_neofetch.sif --overlay $ov ubuntu_24.04.sif</code>
|-
| Test the new image || <code>apptainer exec ubuntu_neofetch.sif neofetch</code>
|-
| Clean up overlay || <code>rm -rf $ov</code>
|}

Tutorials

2026-03-12T09:58:49Z

Honfi001:

== Tutorials ==

=== Modules ===
[[Create Your Own Modules]]

=== Apptainer ===

* [[Tutorials/Apptainer-PullingImages|Pulling Images]]
* Fakeroot and overlays

Tutorials

2026-03-12T09:14:20Z

Honfi001: /* Apptainer */

== Tutorials ==

=== Modules ===
[[Create Your Own Modules]]

=== Apptainer ===

* [[Tutorials/Apptainer-PullingImages|Pulling Images]]

Tutorials

2026-03-12T09:13:07Z

Honfi001: /* Apptainer */

== Tutorials ==

=== Modules ===
[[Create Your Own Modules]]

=== Apptainer ===

* [[Apptainer-PullingImages|Pulling Images]]

Tutorials/Apptainer-PullingImages

2026-03-12T09:11:44Z

Honfi001:

= Pulling Apptainer Images =

== Getting Started ==

First, grab an interactive session in a compute node.

Then make sure your module environment is clean and Apptainer is loaded:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>Also, please make sure to store image and in your lustre folders.

== Your First Pull ==

Let's start by pulling a simple "hello world" container image:

<syntaxhighlight lang="bash">
apptainer pull ./hello-world.sif shub://vsoch/hello-world
</syntaxhighlight>

That is a lot happening in one line, so let's break it down piece by piece:

{| class="wikitable"
|-
! Part !! What it means
|-
| <code>apptainer pull</code> || The Apptainer command to download a container image.
|-
| <code>./hello-world.sif</code> || The output filename. This is where Apptainer will save the downloaded image. The <code>.sif</code> extension stands for '''Singularity Image Format''' — Apptainer's own container format. The <code>./</code> means "save it in my current directory".
|-
| <code>shub://vsoch/hello-world</code> || The source to pull from. <code>shub://</code> tells Apptainer to look at [https://singularity-hub.org Singularity Hub], a public registry for container images. <code>vsoch/hello-world</code> is the user and image name, much like a path on GitHub.
|}

Once the command finishes, you will have a file called <code>hello-world.sif</code> in your current directory. You can verify this:

<syntaxhighlight lang="bash">
ls -lh hello-world.sif
</syntaxhighlight>

This <code>.sif</code> file is your container — a single, portable file that contains an entire operating system and software environment. You can copy it, move it, or share it just like any other file.

== Running the Container ==

The simplest way to use a container is with <code>apptainer run</code>. This executes whatever default action the container was built to perform:

<syntaxhighlight lang="bash">
apptainer run ./hello-world.sif
</syntaxhighlight>

You should see a friendly output from the container. That is it — you just ran software inside a container.

== Opening a Shell Inside the Container ==

Instead of running the container's default action, you can open an interactive shell inside it. This lets you look around and explore the container's environment as if you had logged into a different machine.

<syntaxhighlight lang="bash">
apptainer shell ./hello-world.sif
</syntaxhighlight>

You will notice your prompt changes (usually to <code>Apptainer></code>), indicating that you are now "inside" the container. Type <code>exit</code> when you want to leave.

=== The Container is Not Fully Isolated (by Default) ===

Here is something important that surprises many beginners: by default, Apptainer does '''not''' fully isolate the container from the host system. Your home directory and parts of the host filesystem are still visible inside the container.

Try this while inside the shell:

<syntaxhighlight lang="bash">
apptainer shell ./hello-world.sif
</syntaxhighlight>

Then, from inside the container:

<syntaxhighlight lang="bash">
Apptainer> mkdir ~/test-from-container
Apptainer> ls ~/test-from-container
Apptainer> exit
</syntaxhighlight>

Now, back on the host:

<syntaxhighlight lang="bash">
ls ~/test-from-container
</syntaxhighlight>

The directory is there — on your '''real''' filesystem. The container was able to create it because your home directory was mounted inside the container by default. This is by design: it makes it easy to work with your own files, but it means the container is not completely separate from the host.

=== Full Isolation with <code>--containall</code> ===

If you want the container to be fully isolated from the host system — no access to your home directory, no shared environment variables, no host filesystems — use the <code>--containall</code> flag (or its short form <code>-C</code>):

<syntaxhighlight lang="bash">
apptainer shell --containall ./hello-world.sif
</syntaxhighlight>

Now try the same thing from inside:

<syntaxhighlight lang="bash">
Apptainer> mkdir ~/test-isolated
Apptainer> exit
</syntaxhighlight>

Back on the host:

<syntaxhighlight lang="bash">
ls ~/test-isolated
</syntaxhighlight>

This time, the directory does '''not''' exist on your host. The container was completely isolated — anything it created lived only inside the container's temporary environment and was discarded when you exited.

Use <code>--containall</code> (or <code>-C</code>) when you want a clean, reproducible environment that does not interfere with (or depend on) the host system.

== Running a Specific Command with <code>exec</code> ==

Sometimes you do not want a full interactive shell — you just want to run a single command inside the container and get the output. That is what <code>apptainer exec</code> is for.

For example, let's check which operating system is inside this container:

<syntaxhighlight lang="bash">
apptainer exec ./hello-world.sif cat /etc/lsb-release
</syntaxhighlight>

You should see output similar to:

<syntaxhighlight lang="text">
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.6 LTS"
</syntaxhighlight>

The container is running '''Ubuntu 14.04''' — regardless of what operating system your host machine is running. This is the power of containers: you can run software built for a completely different OS environment without any conflicts.

The pattern for <code>exec</code> is:

<code>apptainer exec ''[options]'' ''container.sif'' ''command'' ''[arguments]''</code>

== Summary ==

You now know the three main ways to interact with a pulled Apptainer container:

{| class="wikitable"
|-
! Command !! What it does
|-
| <code>apptainer run container.sif</code> || Runs the container's default action
|-
| <code>apptainer shell container.sif</code> || Opens an interactive shell inside the container
|-
| <code>apptainer exec container.sif ''command''</code> || Runs a single specific command inside the container
|}

And one important flag to remember:

{| class="wikitable"
|-
! Flag !! What it does
|-
| <code>--containall</code> (or <code>-C</code>) || Fully isolates the container from the host (no shared home directory, no host environment)
|}

Tutorials/Apptainer-PullingImages

2026-03-12T09:09:41Z

Honfi001: Add message about storing image in lustre

= Pulling Apptainer Images =

== Getting Started ==

First, make sure your module environment is clean and Apptainer is loaded:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
</syntaxhighlight>Also, please make sure to store image and in your lustre folders.

== Your First Pull ==

Let's start by pulling a simple "hello world" container image:

<syntaxhighlight lang="bash">
apptainer pull ./hello-world.sif shub://vsoch/hello-world
</syntaxhighlight>

That is a lot happening in one line, so let's break it down piece by piece:

{| class="wikitable"
|-
! Part !! What it means
|-
| <code>apptainer pull</code> || The Apptainer command to download a container image.
|-
| <code>./hello-world.sif</code> || The output filename. This is where Apptainer will save the downloaded image. The <code>.sif</code> extension stands for '''Singularity Image Format''' — Apptainer's own container format. The <code>./</code> means "save it in my current directory".
|-
| <code>shub://vsoch/hello-world</code> || The source to pull from. <code>shub://</code> tells Apptainer to look at [https://singularity-hub.org Singularity Hub], a public registry for container images. <code>vsoch/hello-world</code> is the user and image name, much like a path on GitHub.
|}

Once the command finishes, you will have a file called <code>hello-world.sif</code> in your current directory. You can verify this:

<syntaxhighlight lang="bash">
ls -lh hello-world.sif
</syntaxhighlight>

This <code>.sif</code> file is your container — a single, portable file that contains an entire operating system and software environment. You can copy it, move it, or share it just like any other file.

== Running the Container ==

The simplest way to use a container is with <code>apptainer run</code>. This executes whatever default action the container was built to perform:

<syntaxhighlight lang="bash">
apptainer run ./hello-world.sif
</syntaxhighlight>

You should see a friendly output from the container. That is it — you just ran software inside a container.

== Opening a Shell Inside the Container ==

Instead of running the container's default action, you can open an interactive shell inside it. This lets you look around and explore the container's environment as if you had logged into a different machine.

<syntaxhighlight lang="bash">
apptainer shell ./hello-world.sif
</syntaxhighlight>

You will notice your prompt changes (usually to <code>Apptainer></code>), indicating that you are now "inside" the container. Type <code>exit</code> when you want to leave.

=== The Container is Not Fully Isolated (by Default) ===

Here is something important that surprises many beginners: by default, Apptainer does '''not''' fully isolate the container from the host system. Your home directory and parts of the host filesystem are still visible inside the container.

Try this while inside the shell:

<syntaxhighlight lang="bash">
apptainer shell ./hello-world.sif
</syntaxhighlight>

Then, from inside the container:

<syntaxhighlight lang="bash">
Apptainer> mkdir ~/test-from-container
Apptainer> ls ~/test-from-container
Apptainer> exit
</syntaxhighlight>

Now, back on the host:

<syntaxhighlight lang="bash">
ls ~/test-from-container
</syntaxhighlight>

The directory is there — on your '''real''' filesystem. The container was able to create it because your home directory was mounted inside the container by default. This is by design: it makes it easy to work with your own files, but it means the container is not completely separate from the host.

=== Full Isolation with <code>--containall</code> ===

If you want the container to be fully isolated from the host system — no access to your home directory, no shared environment variables, no host filesystems — use the <code>--containall</code> flag (or its short form <code>-C</code>):

<syntaxhighlight lang="bash">
apptainer shell --containall ./hello-world.sif
</syntaxhighlight>

Now try the same thing from inside:

<syntaxhighlight lang="bash">
Apptainer> mkdir ~/test-isolated
Apptainer> exit
</syntaxhighlight>

Back on the host:

<syntaxhighlight lang="bash">
ls ~/test-isolated
</syntaxhighlight>

This time, the directory does '''not''' exist on your host. The container was completely isolated — anything it created lived only inside the container's temporary environment and was discarded when you exited.

Use <code>--containall</code> (or <code>-C</code>) when you want a clean, reproducible environment that does not interfere with (or depend on) the host system.

== Running a Specific Command with <code>exec</code> ==

Sometimes you do not want a full interactive shell — you just want to run a single command inside the container and get the output. That is what <code>apptainer exec</code> is for.

For example, let's check which operating system is inside this container:

<syntaxhighlight lang="bash">
apptainer exec ./hello-world.sif cat /etc/lsb-release
</syntaxhighlight>

You should see output similar to:

<syntaxhighlight lang="text">
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.6 LTS"
</syntaxhighlight>

The container is running '''Ubuntu 14.04''' — regardless of what operating system your host machine is running. This is the power of containers: you can run software built for a completely different OS environment without any conflicts.

The pattern for <code>exec</code> is:

<code>apptainer exec ''[options]'' ''container.sif'' ''command'' ''[arguments]''</code>

== Summary ==

You now know the three main ways to interact with a pulled Apptainer container:

{| class="wikitable"
|-
! Command !! What it does
|-
| <code>apptainer run container.sif</code> || Runs the container's default action
|-
| <code>apptainer shell container.sif</code> || Opens an interactive shell inside the container
|-
| <code>apptainer exec container.sif ''command''</code> || Runs a single specific command inside the container
|}

And one important flag to remember:

{| class="wikitable"
|-
! Flag !! What it does
|-
| <code>--containall</code> (or <code>-C</code>) || Fully isolates the container from the host (no shared home directory, no host environment)
|}

Main Page/Tutorials/Apptainer

2026-03-11T13:55:51Z

Honfi001: Created page with "This page contains a list of tutorials related to the usage of apptainer in Anunna."

This page contains a list of tutorials related to the usage of apptainer in Anunna.

Main Page/Tutorials

2026-03-11T13:55:02Z

Honfi001: Created page with " =Tutorials= This page contains an index of various tutorials specific to the Anunna supercomputer."

=Tutorials=

This page contains an index of various tutorials specific to the Anunna supercomputer.

Apptainer

2026-03-11T13:46:58Z

Honfi001:

Apptainer is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.

Apptainer is a fork of singularity and, thus, also supports singularity commands and workflows.

==Apptainer on Anunna==

Since Apptainer is not dependent on specific compilers and toolchains, it is now place inside the '''utilities''' bucket. Inside this bucket, there are two options:

* Apptainer
* Apptainer-compat

The regular '''Apptainer''' module loads the version compiled in Anunna. This is more suitable to containers built with more recent operating systems. For most projects this will be enough.

However, some projects still use some very old Linux versions and through errors related to missing versions of GLIBC when running in Anunna. That is where '''Apptainer-compat''' comes in, this is a package distributed by the Fedora Project EPEL (Extra Packages for Enterprise Linux) repositories. This package is built with a broad range of versions of GLIBC in mind, so will likely work with older images like ubuntu 22.04 or older.

One can load apptainer by running the following commands

<pre>module load utilities
module load Apptainer</pre>

==== Initial Setup ====
Apptainer will cache SIF container images generated from remote sources, and any OCI/docker layers used to create them. The cache is created at <code>$HOME/.apptainer/cache</code> by default. The location of the cache can be changed by setting the <code>APPTAINER_CACHEDIR</code> environment variable.

Since there is a limited amount of space available at <code>$HOME</code>, the Apptainer cache can quickly fill your quota. So it is recommended that you set the variable <code>APPTAINER_CACHEDIR</code> to a location ion lustre by edition your <code>$HOME/.bash_aliases</code> file. An example is provided at [[Aliases and local variables]]

Jupyter

2026-02-25T18:10:23Z

Honfi001: New jupyter page in OOD

JupyterLab is the latest web-based interactive development environment for notebooks, code, and data. Its flexible interface allows users to configure and arrange workflows in data science, scientific computing, computational journalism, and machine learning. A modular design invites extensions to expand and enrich functionality.

Traditionally, jupyter has been offered at Anunna via [https://notebook.anunna.wur.nl notebook.anunna.wur.nl], though OOD makes it possible to deploy Jupyter based off the environment modules from Anunna. Adding greater stability and flexibility. Jupyter is listed in the featured apps speed-dial and upon clicking on it, the user is presented with a form where they can select their preferences and allocate resources for their session
[[File:Jupyter-form-20260225.png|thumb|695x695px|Resource Allocation form for Jupyter Session]]

=== Resource Allocation Form ===

* '''Python version:''' version of python the default kernel will be running
* '''Session Type:''' traditional notebooks interface or the newer lab interface. It is also possible to change the interface from within the active Jupyter session.
* '''Hours:''' Number of hours allocated to the session, max 4 hours
* '''Number of CPUs:''' Number of CPU cores allocated for the session
* '''Amount of Memory:''' RAM allocated for the session
* '''Number of GPUs:''' 0 denotes no GPU is used, a non-zero number will switch the partition from main to the gpu partition
* '''Comment:''' label or project number that will show up in billing.
* '''Slurm Options:''' slurm flags that may override settings above

==== Connecting to an active session ====
After the resources and preferences have been defined, the user can simply click on the launch button. Which will submit a job at Anunna and take them to the My Interactive Sessions page where they can connect to their session.

Once the session is ready a "Connect to Jupyter " button will appear which will allow the user to (re)connect to the corresponding active session.

The card also has a red "Cancel" button where the user may terminate their session.

File:Jupyter-form-20260225.png

2026-02-25T18:01:19Z

Honfi001:

OOD jupyter form

Shell Access

2026-02-25T14:25:35Z

Honfi001:

Anunna Shell Access is a web based bash shell terminal emulator available at the Anunna apps portal.

This application provides quick access to a bash session in one of the login nodes.

User's are able to select a theme for their terminal session on the top right corner of the screen.

When a session times out, just refresh your browser tab and it should reconnect.

Since this runs in a browser, users should mind that some bash keyboard shortcuts may not work properly, e.g. ctrl + w

Shell Access

2026-02-25T14:23:56Z

Honfi001: Created page with "Anunna Shell Access is a web based bash shell terminal emulator available at the Anunna apps portal. This application provides quick access to a bash session in one of the login nodes. User's are able to select a theme for their terminal session on the top right corner of the screen. When a session times out, just refresh your browser tab and it should reconnect. Since this runs in a browser, users should mind that some keyboard shortcuts may not work properly, e..."

Anunna Shell Access is a web based bash shell terminal emulator available at the Anunna apps portal.

This application provides quick access to a bash session in one of the login nodes.

User's are able to select a theme for their terminal session on the top right corner of the screen.

When a session times out, just refresh your browser tab and it should reconnect.

Since this runs in a browser, users should mind that some keyboard shortcuts may not work properly, e.g. ctrl + w

Main Page

2026-02-25T14:16:18Z

Honfi001:

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [https://www.wur.nl/en/show/supercomputer-anunna-opens-up-more-opportunities-for-data-storage-and-artificial-intelligence-applications.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= About =

* [[History of the Cluster|Historical information on the startup of Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Note that the use of Anunna is not free of charge.

= Our Courses =
The Anunna team organizes HPC courses three times a year to strengthen basic & more advanced skills and enable users to make the most effective use of our facility.

== [[Linux Basic]] ==

== [[HPC Basic]] ==

== [[HPC Advanced]] ==

== [[2026 Course dates]] ==

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Using Anunna for courses (mainly jupyter notebooks) ==
* [[steps_for_courses | Steps involved to run a course on Anunna]]

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Software =
* [[Modules]]
* [[Apptainer]]
* [[Python]]
* [[R]]
* [[Julia]]

== Browser apps ==
This page provides an overview of the GUI-based applications available Anunna, including background information and practical guidance on how to access and use interactive desktops and graphical tools directly from your web browser.

* [[General overview|General Overview]]
* [[Anunna Shell Access]]
* [[Using the File browser|File Browser]]
* [[Jupyter|Featured Apps]]
* [[Jupyter]]
* [[RStudio|R Studio]]
* [[Linux desktop|Linux Desktop]]
* [[Requesting new software|Requesting New Software]]

== Command-line Software ==

==== Cluster Scheduler ====
Anunna uses Slurm as job scheduler.
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

==== [[Globally installed software]] ====

==== [[ABGC_modules |ABGC specific modules]] ====

==== Installation of software by users ====
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Aliases and local variables]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Locale_settings]] (how numbers and dates are displayed)

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User: prins089 | Fons Prinsen (Wageningen UR, FB-IT, C&PS)]] is responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Bioinformatics_tips_tricks_workflows |Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]
* [[Old_binaries | Running older binaries on the updated OS]]
* [[locale_settings | How to change language settings for yourself]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

File Browser

2026-02-25T14:12:08Z

Honfi001: Basic file browser information

Open Ondemand provides a web based file browser to Anunna, which provides a convenient and easy interface for managing fiiles.

By default the file browser points to the user's home directory, but shortcuts to lustre and archive are provided in the left side bar. Alterntively users can use the change directory button and type a location they would like to see displayed

* Upload files
* Download files
* Create files and directories
* Delete files and directories
* Move files and directories
* Edit files

These features, including editing feature can be acessed by using the dot menu next to the file name. The editor is very basic and only suitable for editing one file at a time.

Note that, despite having the option to upload and download files, this file browser is not suitable for files larges than a couple of gigabytes.

Main Page

2026-02-25T13:30:50Z

Honfi001: /* Browser apps */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [https://www.wur.nl/en/show/supercomputer-anunna-opens-up-more-opportunities-for-data-storage-and-artificial-intelligence-applications.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= About =

* [[History of the Cluster|Historical information on the startup of Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Note that the use of Anunna is not free of charge.

= Our Courses =
The Anunna team organizes HPC courses three times a year to strengthen basic & more advanced skills and enable users to make the most effective use of our facility.

== [[Linux Basic]] ==

== [[HPC Basic]] ==

== [[HPC Advanced]] ==

== [[2026 Course dates]] ==

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Using Anunna for courses (mainly jupyter notebooks) ==
* [[steps_for_courses | Steps involved to run a course on Anunna]]

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Software =
* [[Modules]]
* [[Apptainer]]
* [[Python]]
* [[R]]
* [[Julia]]

== Browser apps ==
This page provides an overview of the GUI-based applications available Anunna, including background information and practical guidance on how to access and use interactive desktops and graphical tools directly from your web browser.

* [[General overview|General Overview]]
* [[Using the Shell|Anunna Shell Access]]
* [[Using the File browser|File Browser]]
* [[Jupyter|Featured Apps]]
* [[Jupyter]]
* [[RStudio|R Studio]]
* [[Linux desktop|Linux Desktop]]
* [[Requesting new software|Requesting New Software]]

== Command-line Software ==

==== Cluster Scheduler ====
Anunna uses Slurm as job scheduler.
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

==== [[Globally installed software]] ====

==== [[ABGC_modules |ABGC specific modules]] ====

==== Installation of software by users ====
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Aliases and local variables]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Locale_settings]] (how numbers and dates are displayed)

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User: prins089 | Fons Prinsen (Wageningen UR, FB-IT, C&PS)]] is responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Bioinformatics_tips_tricks_workflows |Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]
* [[Old_binaries | Running older binaries on the updated OS]]
* [[locale_settings | How to change language settings for yourself]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

RStudio

2026-02-25T12:17:12Z

Honfi001: Rstudio

The '''RStudio integrated development environment (IDE''') is a set of tools built to help you be more productive with R and Python.
[[File:RStudio-form-20260225.png|thumb|595x595px|Resource Allocation Form]]
RStudio is built with the aid of EasyBuild and makes use of the relevant R modules available at Anunna

Rstudio can be found both in the All Applications Menu at the navigation bar, and as a button in the Featured Apps speed dial. By clicking in either option, the user will be taken to a form where they will be able to allocate resources for their RStudio session.

=== Resource Allocation form ===
The resource allocation form allows the use to select the following options

* R version
* Partition: main by default but the GPU partition is available
* Number of Hours: Max 4 hours
* Number of CPUs: CPU cores available
* Memory : Amount of RAM available
* Number of GPUs: Zero by default, this need to be switched to, at least 1
** Note that the GPU partition must be selected
* Comment: Project number that will show up at your bill
* Slurm options: Slurm flag overrides.

=== Connecting to an active session ===
Once the form has been filled, the use can then click on the Launch button. This cause the screen to switch to the interactive sessions page where a new session will be listed.

Once the session is ready a "Connect to RStudio Server" button will appear which will allow the user to (re)connect to the corresponding active session.

The card also has a red "Cancel" button where the user may terminate their session.
[[File:RStudio-connect-20260225.png|center|thumb|509x509px|RStudio active session card at My Interactive Sessions]]

File:RStudio-connect-20260225.png

2026-02-25T12:15:04Z

Honfi001:

RStudio connection card

File:RStudio-form-20260225.png

2026-02-25T11:59:48Z

Honfi001:

Rstudio resource allocation form