View source for HPC Advanced/Preparation Exercise

= HPC Basics Refresher — Weather Data Analyser =

{{Note|'''Who is this for?''' This is a self-paced preparation exercise for people who are attending the '''HPC Advanced''' course but did '''not''' attend '''HPC Basics'''. Working through it end to end means you arrive at the advanced course already comfortable with environment variables, virtual environments, launcher scripts, interactive jobs, and SLURM batch submission on '''Anunna'''.}}

'''The application:''' a small Python script that loads weather data from a Dutch weather station and draws some figures.

Source: <code>/lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py</code>

By the end you will have:

* Set up handy Lustre path variables (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) and aliases
* Created a Python virtual environment on Lustre scratch
* Written an executable launcher script (<code>weather.sh</code>)
* Measured the job's memory requirement interactively
* Written and submitted a SLURM batch script (<code>weather.slurm</code>) and collected the figures

''Conventions in this document:'' lines starting with <code>$</code> are commands you type at the shell (do not type the <code>$</code>). <code><nowiki><placeholder></nowiki></code> means ''substitute your own value''.

== Part 1 — Environment Variables & Aliases ==

The Anunna <code>getLustreDir</code> helper writes your personal Lustre directories (<code>$myScratch</code>, <code>$myBkp</code>, <code>$myNoBkp</code>) into your environment so you never have to type long paths again.

=== 1.1 — Set up the variables ===

<syntaxhighlight lang="bash">
# Provided it does not exist
# Create an empty ~/.bash_aliases if you do not already have one
$ touch ~/.bash_aliases

# Load the anunna module 
$ module load anunna

# Inspect what getLustreDir does
$ getLustreDir -h

# Append the export lines to your ~/.bash_aliases
$ getLustreDir -e >> ~/.bash_aliases

# Check the result (cat, less, more or nano all work)
$ cat ~/.bash_aliases

# Reload your environment so the new variables take effect
$ source ~/.bashrc
</syntaxhighlight>

=== 1.2 — Test the variables ===

'''Test 1 — does this print your scratch folder location?'''

<syntaxhighlight lang="bash">
$ echo $myScratch
</syntaxhighlight>

'''Test 2 — does this take you to your scratch folder?'''

<syntaxhighlight lang="bash">
$ cd $myScratch
</syntaxhighlight>

If both work, your variables are live.

=== 1.3 — Bonus: convenience aliases ===

Add these lines to your <code>~/.bash_aliases</code> and reload your environment again:

<syntaxhighlight lang="bash">
alias cds="cd $myScratch"
alias cdb="cd $myBkp"
alias cdn="cd $myNoBkp"
</syntaxhighlight>

{{Note|'''Important:''' the alias lines must come '''after''' the variable definitions in <code>~/.bash_aliases</code>, because they reference those variables.}}

If your aliases still do not work after <code>source ~/.bashrc</code>, your <code>~/.bashrc</code> may not be sourcing <code>~/.bash_aliases</code> at all. Add this block to the '''end''' of your <code>~/.bashrc</code>:

<syntaxhighlight lang="bash">
if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi
</syntaxhighlight>

Then <code>source ~/.bashrc</code> once more.

== Part 2 — Python Virtual Environment ==

We isolate the analyser's dependencies in a virtual environment living on Lustre scratch.

=== 2.1 — Load the software stack and Python ===

<syntaxhighlight lang="bash">
$ module purge
$ module load 2024
$ module load Python/3.12.3
</syntaxhighlight>

=== 2.2 — Create the virtual environment (in Lustre scratch) ===

<syntaxhighlight lang="bash">
$ mkdir $myScratch/hpcCourse; cd $_
$ python -m venv $myScratch/hpcCourse/venv
$ source $myScratch/hpcCourse/venv/bin/activate
</syntaxhighlight>

{{Note|<code>cd $_</code> reuses the last argument of the previous command — here, the directory you just created.}}

=== 2.3 — Check you are using the venv's Python ===

<syntaxhighlight lang="bash">
$ which python      # should point inside .../hpcCourse/venv/bin
$ python -V          # should report Python 3.12.3
</syntaxhighlight>

=== 2.4 — Install the required libraries ===

<syntaxhighlight lang="bash">
$ pip install matplotlib pandas datetime
$ pip freeze          # for checking what is installed
</syntaxhighlight>

== Part 3 — The Launcher Script <code>weather.sh</code> ==

=== Tasks ===

# Copy the Python script into your course folder:
#: <syntaxhighlight lang="bash" inline>cp /lustre/shared/hpcCourses/Basics/weer_vanaf_2000.py $myScratch/hpcCourse/</syntaxhighlight>
# Write a bash script called '''<code>weather.sh</code>''' that loads the modules and activates the virtual environment you created, and make it executable.
# The script should end with the run command below. '''Hint:''' use variables for paths.
#: <syntaxhighlight lang="bash" inline>python $myScratch/hpcCourse/weer_vanaf_2000.py 240</syntaxhighlight>
#: (<code>240</code> is the argument passed to the analyser — the number of months of data to plot, i.e. ~20 years.)

=== Example solution — <code>weather.sh</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
# weather.sh — load the environment and run the Weather Data Analyser

# Load the software stack and Python
module purge
module load 2024
module load Python/3.12.3

# Activate the virtual environment
source "$myScratch/hpcCourse/venv/bin/activate"

# Run the analyser (240 = ID of this specific weather station)
python "$myScratch/hpcCourse/weer_vanaf_2000.py" 240
</syntaxhighlight>

Make it executable:

<syntaxhighlight lang="bash">
$ chmod +x $myScratch/hpcCourse/weather.sh
</syntaxhighlight>

== Part 4 — Find the Job Requirements ==

Before submitting a batch job you need to know how much memory it needs. We start with a deliberately small allocation and increase it until the job stops crashing.

=== Task ===

Find the job requirements (e.g. memory) of the Python script using <code>sinteractive</code>. Increase the amount of RAM until the job no longer crashes.

<syntaxhighlight lang="bash">
$ sinteractive --mem 100M --time=0-0:10 -c 1
$ cd $myScratch/hpcCourse
$ ./weather.sh
</syntaxhighlight>

If the job is killed (out-of-memory), exit the interactive session, re-launch <code>sinteractive</code> with a larger <code>--mem</code> value (e.g. <code>200M</code>, <code>500M</code>, <code>1G</code>, …) and run again. Repeat until <code>weather.sh</code> completes cleanly. '''Note the smallest <code>--mem</code> value that works''' — you will use it in Step 3.

== Part 5 — Write and Submit a SLURM Script ==

=== Tasks ===

# '''Write''' a SLURM script for the job using the resource requirements you determined in Step 2. Label it '''<code>weather.slurm</code>'''.
# '''Submit''' the job with <code>sbatch</code>.
# '''Check the status''' of your job and cancel it if needed, though probably not since the jobs are very short.
# '''Check the figures''' generated by the Python script.

{{Note|'''Tip:''' a SLURM template is available at <code>/lustre/shared/hpcCourses/Basics/script_slurm.sh</code>}}

=== Example solution — <code>weather.slurm</code> ===

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=weather
#SBATCH --mem=<RAM-from-Step-2>     # e.g. 500M — the value you found in Step 2
#SBATCH --time=0-0:10
#SBATCH --cpus-per-task=1
#SBATCH --output=weather-%j.out
#SBATCH --error=weather-%j.err

cd "$myScratch/hpcCourse"
./weather.sh
</syntaxhighlight>

=== Submit, monitor, and collect results ===

<syntaxhighlight lang="bash">
# Submit the job
$ sbatch weather.slurm

# Check the status of your jobs
$ squeue --me

# Cancel a job if needed (replace <jobid> with the ID from squeue / sbatch)
$ scancel <jobid>

# Once finished, inspect the output and the generated figures
$ ls $myScratch/hpcCourse
$ cat weather-<jobid>.out
</syntaxhighlight>

The analyser writes its figures into your course folder — open them to confirm the run succeeded.

== Checklist ==

* <code>echo $myScratch</code> prints your scratch path and <code>cd $myScratch</code> works
* <code>cds</code> / <code>cdb</code> / <code>cdn</code> aliases jump to the right folders
* Virtual environment activates and <code>which python</code> points inside it
* <code>weather.sh</code> is executable and runs the analyser interactively
* You know the minimum <code>--mem</code> the job needs (from Step 2)
* <code>weather.slurm</code> submits with <code>sbatch</code> and the figures are generated

You are now ready for the '''HPC Advanced''' course.