HPCwiki - User contributions [en]

Best-Practices

2026-08-28T16:03:14Z

Haars001: /* Jobs */

{{DISPLAYTITLE:Best Practices}}

== Login Node ==

* Do not run analyses or jobs in the login node.

== Home folder ($HOME, ~) ==

* Do not store datasets or large files in your home directory. You have a quota of 200 GB.
* Do not install software that you intend to run in jobs in your $HOME . Install your software in lustre.
* Do not point the logs of your jobs into your $HOME, specially when running array jobs. If not defined slurm will point them to the location you run the sbatch command.
* Do not edit or let software edit your ~/.bashrc or ~/.bash_profile. If something goes wrong you will be locked out.

== Jobs ==

*If you have jobs that are very much alike, please use [[Array Jobs]]. This will make your life a lot easier.

2026-08-28T15:25:43Z

2026-08-28T12:35:43Z

Haars001:

= Anunna Roadmap: How We Decide What to Work On =
The Anunna HPC team has a lot of ideas, wishes, requests and opportunities to work on. At the same time, our capacity is limited. Besides improving and changing Anunna, we also need to keep the day-to-day HPC service running.

This means we need to make deliberate choices about what we work on, when we work on it, and what we do not work on.

Our roadmap is the way we make these choices together with our users and key users.

== Why do we work with a roadmap? ==
There are several reasons why having a clear roadmap is important:

* There is a lot of work to do. If everything is important, nothing is important. We need to make choices and focus our effort where it creates the most value.
* Our capacity is limited. The HPC team is responsible not only for improvements and changes, but also for the day-to-day operation and support of Anunna.
* We are still learning. The Anunna team is continuously learning what works best for our users and for the team.
* Wishes and demands come in different stages of maturity. Some ideas are well understood and ready to act on, while others are still "green" and need more exploration. The roadmap process helps us distinguish between them.
* Every decision to do something is also a decision not to do something else. By making priorities explicit, we make these trade-offs visible.
* Our users should help decide what matters most. The HPC team has technical expertise, but our clients and users are best positioned to tell us where improvements will have the greatest value for their work.

The roadmap is therefore not simply a list of things the HPC team wants to do. It is a shared prioritisation process between the HPC team and the people who use Anunna.

== How does the roadmap process work? ==
We work with a six-month roadmap cycle. Twice a year, representatives of our users come together with the HPC team to discuss and prioritize the work for the coming half year.

The process starts well before the roadmap session itself.

=== 1. Build the longlist ===
We start with a longlist of User Stories.

A User Story describes a need or desired outcome from a user's perspective:<blockquote>As a [type of user],

I want [something],

so that [benefit or outcome].</blockquote>For example:<blockquote>As a researcher, I want to be able to monitor my jobs more easily, so that I can quickly identify jobs that are not progressing as expected.</blockquote>User Stories can cover both small improvements and larger topics. They can be written from any relevant perspective: researcher, data manager, software developer, HPC support, system administrator or another stakeholder.

The important thing is that the story clearly describes who needs something, what they need, and why it matters.

=== 2. Prepare and prioritise the User Stories ===
Before a User Story is considered for the roadmap, we try to make sure that we understand it sufficiently well.

When preparing a story, we keep several questions in mind:

* What is the expected value? What business, research or user value would this deliver?
* What risk does it reduce or what opportunity does it enable?
* Why now? Is there a reason this needs to happen soon rather than later?
* Are there dependencies? Does this work depend on another change being completed first, or does other work depend on it?

This does not mean that every User Story needs a detailed business case. The purpose is to make the reasoning behind prioritization visible and understandable.

At this stage, the longlist can still be quite large.

=== 3. Vote: what matters most? ===
During the bi-annual roadmap session, every representative receives the complete longlist of User Stories.

Each representative reads through the stories and selects '''four''' User Stories that are most relevant from their perspective.

The voting is deliberately done from the perspective of the people represented in the session. Different users may value different things, and that is useful information.

The goal is not to find a mathematically perfect ranking. The goal is to identify the topics that have the strongest relevance across our user community.

Based on the voting, we reduce the longlist to a shortlist of approximately 10–12 User Stories.

=== 4. High-level estimation ===
The shortlist is then further refined by the HPC team. This is also done during the bi-annual roadmap session.

For each User Story, the team makes a high-level estimate of the effort and complexity involved. This is not intended to be a detailed project plan or a commitment to a precise number of hours.

The purpose is to understand the relationship between:

value × urgency × risk/opportunity × effort × dependencies

The HPC team does the technical estimation, while the client representatives observe the process and are encouraged to ask questions and provide feedback.

This is an important part of the process: the HPC team brings the technical perspective, while the users bring the perspective of value and impact.

== What comes out of the roadmap session? ==
At the end of the roadmap session, we have a clear and predictable set of User Stories for the coming six months.

These stories form the roadmap that the Anunna team will focus on.

The intention is to work through the selected User Stories one by one, rather than constantly switching between many competing priorities. This allows the team to focus its capacity and increases the likelihood that we actually finish the things we start.

The roadmap is therefore also a way of saying:<blockquote>"These are the things we believe will deliver the most value right now, and these are the things we will focus on."</blockquote>It does not mean that nothing else can happen during the six months. Operational incidents, security issues, urgent technical problems and genuinely time-critical opportunities may require us to change course.

However, such changes should be visible and discussed rather than happening simply because the latest request happened to arrive first.

== How can users influence the roadmap? ==
The most direct way to influence the roadmap is through the key-user group.

Key users help us understand what is important to the people using Anunna and bring different perspectives into the roadmap process. They participate in the bi-annual roadmap sessions and help determine which User Stories deserve attention.

You do not need to be an HPC expert to contribute. In fact, a key part of the role is bringing the user perspective into the conversation.

If you have an idea or a problem that you believe should be addressed, it can become a User Story and enter the longlist. Being on the longlist does not automatically mean that it will be implemented; it means that the idea is visible and can be considered as part of the prioritization process.

== Want to join the key-user group? ==
Would you like to help shape the future of Anunna and have a voice in deciding what the HPC team should focus on?

Please reach out to [mailto::alexander.vanittersum@wur.nl Alexander van Ittersum] and let him know that you would like to be part of the Anunna key-user group.

Your perspective helps us make better choices about where our limited capacity can create the most value for the Anunna community.

== See also ==

* [[Mission and Governance]]
* [[Maintenance and Management]]

Roadmap

Uv

2026-08-28T09:15:41Z

Haars001:

{{DISPLAYTITLE:uv}}

uv is a tool for installing Python packages and for managing the environments they live in. It does the same work as <code>pip</code> and <code>venv</code>, and it works it a great deal faster.

This page explains what uv is, how to load it on Anunna, where it keeps your files, and how to use it for everyday work. It also explains what uv is '''not''' the right tool for: the packages it installs are not built for our hardware, so when the speed of your calculation matters, the Python modules remain the better choice.

== Introduction ==

=== What uv is ===

uv is a single program that covers work you would otherwise need several separate tools for. On Anunna it is available as a module, already set up for the cluster.

=== What uv does ===

{| class="wikitable"
! uv can !! Otherwise you would use
|-
| Install Python packages || <code>pip</code>
|-
| Create and manage environments || <code>venv</code>
|-
| Download other versions of Python || a Python module, if one exists
|}

=== What it is faster at ===

Installing packages, and building the environments that hold them. uv produces the same result as <code>pip</code> — the very same packages, from the very same place — in a fraction of the time.

That is worth more here than it would be on your own computer. Installing an environment means writing tens of thousands of small files onto a filesystem that everybody on the cluster shares, so finishing sooner is better for you and better for everyone else.

=== What it is not faster at ===

Running your code.

The packages uv installs are general-purpose builds, made to work on any machine in the world. They are exactly what <code>pip</code> would have given you. The software we provide as modules is compiled for the precise processors in each generation of Anunna's nodes, and that is where the speed of a calculation comes from. If your work is heavy, use the modules — see [[Python/Modules]].

The same applies to Python itself. uv is able to download a Python for you, but it is a general-purpose build too. The module is set up to use the Python you have loaded rather than fetching one.

=== What we have already set up ===

Left alone, uv keeps its downloads and environments in your home directory, which is small, backed up, and slower than Lustre. The Anunna module points all of it at Lustre instead, and there is nothing you need to do to make that happen. The next two sections cover loading the module and where your files end up.

== Loading uv ==

uv lives in the '''2025''' bucket:

<syntaxhighlight lang="bash">
module load 2025 uv
</syntaxhighlight>

Loading uv also loads Python 3.13.1, so you do not need to load a Python module first.

Software from different buckets should not be mixed, so anything you use alongside uv should come from the 2025 bucket as well. See [[Environment Modules#Software buckets]].

To check that it worked:

<syntaxhighlight lang="bash">
uv --version
</syntaxhighlight>

A second command, <code>uvx</code>, arrives together with <code>uv</code>. It is described in [[#Other things uv can do]].

== Where uv keeps your files ==

uv accumulates three kinds of thing as you use it: the packages it has downloaded, any Python versions it has fetched, and the tools you have installed with it. On Anunna the module places all of them on Lustre, under a folder called <code>uv</code> in your nobackup location — <code>$myNoBkp/uv</code>. Nobackup is the right tier because everything in there can be downloaded again.

The module does this by setting a handful of variables for you:

{| class="wikitable"
! Variable !! What it points at
|-
| <code>UV_DIR</code> || The folder that holds all of the others. This is the only one you might want to change.
|-
| <code>UV_CACHE_DIR</code> || Packages uv has downloaded. Normally the largest of them.
|-
| <code>UV_PYTHON_INSTALL_DIR</code> || Python versions uv has downloaded.
|-
| <code>UV_TOOL_DIR</code> || Environments for tools installed with <code>uv tool install</code>.
|-
| <code>UV_TOOL_BIN_DIR</code> || The programs those tools provide. The module has already added this to your <code>PATH</code>.
|-
| <code>UV_PYTHON_BIN_DIR</code> || Shortcuts to downloaded Python versions.
|}

=== Putting them somewhere else ===

Every folder in that table sits inside <code>UV_DIR</code>, so setting that one variable moves all of them. Set it '''before''' loading the module:

<syntaxhighlight lang="bash">
export UV_DIR=$myScratch/uv
module load 2025 uv
</syntaxhighlight>

To see where uv is actually working, ask:

<syntaxhighlight lang="bash">
echo $UV_DIR
</syntaxhighlight>

Use that rather than <code>$myNoBkp</code>, which is only set once you have added it to your shell yourself — see [[Installing Personal Software#Aliases and local variables]].

=== Keeping it from growing ===

The cache is meant to grow; that is part of what makes uv fast. It still counts against your Lustre quota, so clear it out from time to time:

<syntaxhighlight lang="bash">
uv cache prune # remove entries nothing is using
uv cache clean # remove everything
</syntaxhighlight>

== Creating an environment and installing packages ==

=== Never build an environment on a login node ===

Building an environment writes thousands of small files at once. A login node is shared by every user on the cluster, so that work is felt by all of them rather than only by you. Ask for an interactive session on a compute node and work there:

<syntaxhighlight lang="bash">
sinteractive -c 4 --mem 16G --time 120
</syntaxhighlight>

See [[Interactive Jobs]] for the options <code>sinteractive</code> accepts.

=== Creating an environment ===

<syntaxhighlight lang="bash">
module load 2025 uv
uv venv $myNoBkp/PythonEnv/my_env
</syntaxhighlight>

Environments belong on Lustre rather than in your home directory, and nobackup is the right tier because an environment can always be built again from the list of packages in it.

If you run <code>uv venv</code> without giving a path, uv creates a folder named <code>.venv</code> in whatever directory you happen to be in. Giving the path yourself is clearer, and it keeps your environments together in one place.

=== Using it ===

Activate the environment whenever you want it:

<syntaxhighlight lang="bash">
source $myNoBkp/PythonEnv/my_env/bin/activate
</syntaxhighlight>

Its name then appears at the front of your prompt:

<syntaxhighlight lang="text">
(my_env) user001@login200:~$
</syntaxhighlight>

Install packages into it:

<syntaxhighlight lang="bash">
uv pip install numpy pandas
</syntaxhighlight>

Leave it again with <code>deactivate</code>.

What uv builds is an ordinary virtual environment. <code>pip</code> still works inside it if you prefer, and it can be turned into a Jupyter kernel exactly as [[Python/Modules#Jupyter kernels]] describes.

=== The commands you already know ===

{| class="wikitable"
! If you know this !! Use this
|-
| <code>python -m venv my_env</code> || <code>uv venv my_env</code>
|-
| <code>source my_env/bin/activate</code> || unchanged
|-
| <code>pip install numpy</code> || <code>uv pip install numpy</code>
|-
| <code>pip list</code> || <code>uv pip list</code>
|}

== Other things uv can do ==

=== Running a tool without installing it ===

<code>uvx</code> fetches a command-line program, runs it, and does not leave an environment behind:

<syntaxhighlight lang="bash">
uvx ruff check .
</syntaxhighlight>

=== Installing a tool you use often ===

<syntaxhighlight lang="bash">
uv tool install ruff
</syntaxhighlight>

The program lands in your uv folder, which the module has already added to your <code>PATH</code>, so you can run it straight away and in any later session.

=== Using a different version of Python ===

If you need a version of Python we do not provide as a module, uv can fetch one:

<syntaxhighlight lang="bash">
uv python install 3.12
uv venv --python 3.12 $myNoBkp/PythonEnv/py312
</syntaxhighlight>

Remember that these are general-purpose builds, like the packages. They are convenient, and they are the right choice when you need a specific version for compatibility — but if the speed of your calculation matters, a Python module is the better starting point.

'''Do not run <code>uv python update-shell</code>.''' It writes a fixed path into your <code>~/.bashrc</code> and <code>~/.bash_profile</code>. That applies in every shell you ever open, whether or not the module is loaded, and it stops being correct the moment your uv folder moves.

== Using uv in a job script ==

Build the environment '''before''' you submit the job, not inside it. Create it once in an interactive session, as above, and let the job simply use it. An environment built inside the job is rebuilt on every run, and when many jobs start at the same time they all write to the filesystem at once.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --time=1:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G

module load 2025 uv
source $myNoBkp/PythonEnv/my_env/bin/activate

python myscript.py
</syntaxhighlight>

Both the <code>module load</code> and the <code>source</code> have to be inside the script. A job does not inherit the shell you submitted it from, so anything you loaded by hand beforehand is not there when the job runs. This is the most common reason a script that worked interactively fails under <code>sbatch</code>.

See [[Batch Jobs]] for what else can go in the <code>#SBATCH</code> lines.

== FAQ ==

; <code>module load uv</code> says there is no such module.
: The bucket has to be loaded first. Use <code>module load 2025 uv</code>. See [[Environment Modules#Software buckets]].

; Where did uv put my files?
: <code>echo $UV_DIR</code> gives the folder they are all under. To hear it from uv itself, <code>uv cache dir</code> and <code>uv tool dir</code>.

; uv warned that it could not find a Lustre directory.
: The module could not work out where your Lustre folders are, so uv has fallen back to your home directory — which will eventually fill up. Choose a location yourself with <code>export UV_DIR=$myNoBkp/uv</code> and then load the module again.

; My home directory is still filling up, even though uv uses Lustre.
: If you used uv before this module existed, an old cache is probably still sitting in <code>~/.cache/uv</code>. uv no longer looks there, so its own cleaning commands cannot reach it. Check its size with <code>du -sh ~/.cache/uv</code> and delete it by hand if you no longer need it.

; Can I still use <code>pip</code> inside a uv environment?
: Yes. It is an ordinary virtual environment and <code>pip</code> behaves normally in it.

== See also ==

* [[Python]]
* [[Python/Modules]]
* [[Python/Conda]]
* [[Environment Modules]]
* [[Installing Personal Software]]
* [[Interactive Jobs]]
* [[Batch Jobs]]
* [[Quotas]]

== External links ==

* [https://docs.astral.sh/uv/ uv documentation]
* [https://github.com/astral-sh/uv uv on GitHub]

Python/Modules

2026-08-28T07:52:10Z

Haars001:

{{DISPLAYTITLE:Python Modules}}
= Modules =
The Python Environment Modules are the only officially supported Python distributions in the HPC. These are compiled for each specific architecture of the HPC and hence are likely going to be more performant than the versions obtained via Mamba or UV.

Anunna provides one Python version per [[Environment Modules | module bucket]], plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module:

* '''2023''': <code>Python/3.11.3</code>
* '''2024''': <code>Python/3.12.3</code>
* '''2024''': <code>Python/3.13.1</code>

So, for instance, loading Python 3.12.3
<syntaxhighlight lang="bash">
module load 2024
module load Python/3.12.3
</syntaxhighlight>

The bundle module <code>Python-bundle-PyPI</code> adds many frequently-used packages on top of the base interpreter. Use <code>module key <package></code> to find which bundle contains a package you need (see [[Environment Modules#Searching by keyword | searching modules by keyword]]).

For packages not in a module, the two recommended routes are a '''virtual environment''' built on a Python module (below), or '''Miniforge''' for a self-contained conda/mamba setup. The use of Anaconda is discouraged on Anunna — its default channels carry licensing restrictions and the full distribution is heavy; Miniforge is the lighter, unrestricted alternative.

== Virtual environments ==

A virtual environment is a self-contained directory holding a specific Python and its packages, so one project's dependencies cannot clash with another's. Python's built-in <code>venv</code> module is the simplest way to make one on top of a Python module.

First load the Python version you want:

<syntaxhighlight lang="bash">
module load 2024
module load Python/3.12.3
</syntaxhighlight>

Then create the environment in a location of your choosing. The example uses <code>$myNobackup/PythonEnv</code> — <code>$myNobackup</code> is your Lustre nobackup location, set in your <code>~/.bash_aliases</code> (see [[Installing Personal Software#Aliases and local variables | Aliases and local variables]]). Keeping environments on Lustre rather than your home directory avoids filling your home quota, and the nobackup tier is the right choice because an environment can always be recreated from scratch and so does not need backing up.

<syntaxhighlight lang="bash">
python -m venv $myNobackup/PythonEnv/my_env
</syntaxhighlight>

Activate it whenever you want to use it:

<syntaxhighlight lang="bash">
source $myNobackup/PythonEnv/my_env/bin/activate
</syntaxhighlight>

Once active, the environment name appears as a prefix in your prompt:

<syntaxhighlight lang="text">
(my_env) user001@login200:~$
</syntaxhighlight>

Install packages with <code>pip</code> while the environment is active; they go into the environment, not your home directory:

<syntaxhighlight lang="bash">
pip install -U numpy pandas matplotlib
</syntaxhighlight>

Leave the environment with <code>deactivate</code>.

== Jupyter kernels ==

Once your virtual environment has been setup, you can can access it in jupyter, either via the apps or the notebooks page by setting up a '''kernel'''. A kernel is nothing more than a json file with details for jupyter on where to find your Python environment. First things first, with your Python modules loaded activate your virtual environment from the example before.

<syntaxhighlight lang="bash">source $myNobackup/PythonEnv/my_env/bin/activate</syntaxhighlight>The install [https://pypi.org/project/ipykernel/ ipyKernel]<syntaxhighlight lang="bash">(my_env) user001@login200:~$ pip install -U ipykernel</syntaxhighlight>This is the only dependency that Python needs for creating kernels, you can then run <syntaxhighlight lang="bash">(my_env) user001@login200:~$ python -m ipykernel install --user --name=myenv --display-name="Python (myenv)"</syntaxhighlight>This command will create a folder labelled <code>myenv</code> containing a <code>kernel.json</code> file inside of <code>$HOME/.local/share/jupyter/kernels/</code> , which jupyter watches. If your goal is just to create a kernel for yourself, then you are done. The kernel should be visible to you both inside of the jupyter instances, either the ones accessed via the [https://apps.anunna.wur.nl anuna apps portal] or the [https://notebook.anunna.wur.nl notebooks page]. Thus, the kernel will not be visible or accessible to other users.

The --name=myenv flag is entirely optional, it is the directory in which the <code>kernel.json</code> file is generated. If not used, python will simply label it <code>python3</code> . The use of this flag is recommended, specially when working with multiple kernels to avoid overwriting previously created kernels.

=== Shared Kernels ===
Whether for a course or collaboration, sometimes it is convenient to share a kernel with other users. This is possible and the procedure is a variation of the previous one, with just some additional requirements

* The virtual environment you create needs to be in a shared location , e.g. /lutre/shared
* The virtual environment folder needs to be visiible and accessible to other users i.e. it must have read and execute permissions, 755 is recommended

Like before load the bucket and its corresponding Python version and create your project folder and go there<syntaxhighlight lang="bash">module load 2025
module load Python/3.13.1
mkdir /lustre/shared/MyProject
cd /lustre/shared/MyProject

</syntaxhighlight>Then create your virtual environemnt, activate it and install [https://pypi.org/project/ipykernel/ ipyKernel]<syntaxhighlight lang="bash">python -m venv ./project_env
source /lustre/shared/MyProject/project_env/bin/activate
pip install -U ipykernel</syntaxhighlight>

Here is where things differ, inside the activate kernel run the command<syntaxhighlight lang="bash">
python -m ipykernel install --sys-prefix --name=myproject --display-name="Python (project_env)"
</syntaxhighlight>Note that now the command uses the flag --sys-prefix . This flag sets the installation path of the kernel '''inside''' the virtual environment's directory, in this case, the kernel.json file will be generate in <code>/lustre/shared/MyProject/project_env/share/jupyter/kernels/myproject</code>.

While the kernel has been created, jupyter still will not be able to see it. In order to be able to see and access the kernel you would need to either copy it to your folder<syntaxhighlight lang="bash">
MYPROJ=/lustre/shared/MyProject
cp -r $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/
</syntaxhighlight>or link it<syntaxhighlight lang="bash">
MYPROJ=/lustre/shared/MyProject
ln -s $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/
</syntaxhighlight>Any users that run this last step should gain access to the kernel from jupyter. Though note, that only users that have write permissions to virtual environment will be able to install and remove modules from it.

<div style="background-color: #e7f3fe; border-left: 6px solid #2196F3; padding: 10px; margin-bottom: 15px;">
'''Note:''' When using custom kernels do not run <code>!pip</code>, just run <code>pip</code> without the <code>!</code>
</div>

== See also ==

* [[Environment Modules]]
* [[Installing Personal Software]]
* [[Jupyter]]
* [[Python]]
* [[R]]
* [[Apptainer]]

== External links ==

* [https://docs.python.org/3/library/venv.html Python venv documentation]
* [https://github.com/conda-forge/miniforge Miniforge]
* [https://conda-forge.org/ conda-forge]

Quotas

2026-08-14T11:34:26Z

Haars001:

Each storage area on Anunna has limits on how much you can store. This page collects the quotas that apply and how to check your usage.

== Home and shared directories ==

Each user has a 200 GB quota on their [[Home Directory|home directory]] (<code>/home</code>) combined with data they store under <code>/shared</code>.

== Lustre ==

Currently there are no enforced quotas on the lustre filesystem
== Checking your usage ==

To check your usage for your home directory, do this:

<code>quota -u -f /home</code>

To check your usage for the lustre filesystem, do this:

<code>lfs quota -h /lustre/<br /></code>

== Requesting more space ==

If you need more storage than your current quota allows, contact the HPC team through [[Support]].
== See also ==
* [[Storage Systems Overview]]
* [[Home Directory]]
* [[Compute Storage]]
* [[Backup Policy]]

Workflow Engines (Snakemake, Nextflow)

2026-07-14T14:44:22Z

Haars001:

Workflow engines let you describe a multi-step analysis as a set of rules — which steps depend on which, and how to run each — and then execute the whole pipeline reproducibly, submitting the individual steps to the scheduler for you. The two most common on Anunna are [https://snakemake.github.io/ Snakemake] and [https://www.nextflow.io/ Nextflow].

Using a workflow engine has real advantages on an HPC cluster: steps run as SLURM jobs with the right resources, only the parts that need to run are rerun, and the same pipeline can be shared and reproduced by others.

== Snakemake ==

Snakemake describes a pipeline as a set of rules in a <code>Snakefile</code>. It can submit each rule's work to SLURM and manage the dependencies between steps.

=== Set up ===

Snakemake is usually installed in a conda environment. If you do not have conda/Miniforge yet, see [[Python]]. Create an environment containing Snakemake and your pipeline's dependencies:

<syntaxhighlight lang="bash">
conda create --name my-pipeline --file requirements.txt
conda activate my-pipeline
pip install snakemake-executor-plugin-slurm # or add to requirements.txt
</syntaxhighlight>

Giving the environment the same name as the pipeline makes it easy to find later.

=== SLURM profile ===

To let Snakemake submit jobs to SLURM, create a profile. Make a directory for it:

<syntaxhighlight lang="bash">
mkdir -p ~/.config/snakemake/my-pipeline
</syntaxhighlight>

and create a <code>config.yaml</code> inside it that tells Snakemake how to submit jobs, for example:

<syntaxhighlight lang="yaml">
executor: slurm
jobs: 10
software-deployment-method: conda
default-resources:
slurm_partition: "main" # your Anunna partition
slurm_account: "your_account"
runtime: 60 # minutes
mem_mb: 16000
cpus_per_task: 16
# per-rule overrides:
# set-resources:
# big_rule:
# mem_mb: 64000
# runtime: 240
</syntaxhighlight>

Adjust the resources (time, memory, cores) to what your rules need.
=== Configure and run ===

Open the pipeline's own <code>config.yaml</code> and set the input and output paths, keeping the variable names already in the file:

<syntaxhighlight lang="yaml">
OUTDIR: /path/to/output
READS_DIR: /path/to/reads/
ASSEMBLY: /path/to/assembly
PREFIX: output_name
</syntaxhighlight>

Because pipelines can take a long time, run Snakemake inside a persistent session ([https://linuxize.com/post/how-to-use-linux-screen/ screen] or tmux) so it keeps running if your connection drops. First do a dry run to check what will happen:

<syntaxhighlight lang="bash">
snakemake -np
</syntaxhighlight>

If the steps and commands look right, run the pipeline with your profile:

<syntaxhighlight lang="bash">
snakemake --profile my-pipeline
</syntaxhighlight>

The jobs are submitted to SLURM and you can follow the progress in your terminal and with the usual tools — see [[Monitoring Jobs]].

== Nextflow ==

[https://www.nextflow.io/ Nextflow] is another widely used workflow engine, popular in bioinformatics (for example the nf-core pipelines).



== See also ==
* [[Python]]
* [[Environment Modules]]
* [[Monitoring Jobs]]
* [[Scheduler Overview (Slurm)]]
* [https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html SLURM executor plugin]
* [https://snakemake.readthedocs.io/en/stable/executing/cli.html Profiles / CLI reference]

Remote Access (VPN, Gateway)

2026-07-14T14:35:08Z

Haars001:

Some ways of reaching Anunna require you to be on the WUR network (WURnet). From off campus, you connect through the WUR VPN.

== When you need the VPN ==

* [[SSH Access|SSH]] and the [[Portal Overview|Apps Portal]] are reachable directly over the internet, though access may be restricted to certain IP ranges (see [[SSH Access]]).
* Some services are only available inside WURnet — for example the [[Data Transfer Methods|CIFS/Samba mounts]]. From off campus, connect to the WUR VPN first.

== Connecting to the WUR VPN ==

The WUR VPN is provided by WUR IT, not by the Anunna team.

The documentation on that lives [https://support.wur.nl/esc/en/working-at-a-non-wur-location/vpn-what-is-it-and-how-do-you-request-the?id=kb_article&sysparm_article=KB0010763&table=kb_knowledge&searchTerm=vpn here].

For students who are doing their thesis, ask your supervisor to create a ticket with the team that is responsible for the VPN to request accces.
== See also ==
* [[SSH Access]]
* [[Data Transfer Methods]]
* [[How to Get Help]]

Resource Allocation Requests

2026-07-14T13:51:24Z

Haars001: