Environment Modules: Difference between revisions

From HPCwiki
Jump to navigation Jump to search
Phase 1 § 5 P1.5.2: merge Modules + Environment Modules + Using environment modules into the canonical Environment Modules page. Base content from Modules (Lmod/buckets), salvaged switching+conflicts concept, dropped pre-Lmod stale content. (via update-page on MediaWiki MCP Server)
 
(3 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Anunna provides software through '''environment modules''', managed with [https://lmod.readthedocs.io/ Lmod]. A module configures your shell — and the environment of your jobs — so that a chosen application and the right version of its dependencies are available. Because each module lives in its own self-contained tree, modules let several otherwise-conflicting programs and several versions of the same program coexist on the cluster without clashing.
== Software buckets ==
Modules on Anunna are organised into '''buckets'''. Each bucket is a snapshot of a particular compiler-and-toolchain generation; all the software in a bucket is built with the same compiler. This matters because mixing software built with different compilers in one job can cause conflicts, errors, or silently wrong results — so keeping a job's software within a single bucket keeps it consistent.
The buckets currently available are:
* '''legacy''' — old software that is no longer maintained or updated but is still used in active research.
* '''2023''' — software built with the 2023 compilers and toolchain. One version of each package.
* '''2024''' — software built with the 2024 compilers and toolchain. One version of each package.
* '''utilities''' — software that does not depend on a specific compiler or toolchain (for example [[Apptainer]]).
* '''groups''' — module files contributed by groups inside and outside WUR.
* '''GPU''' — CUDA, cuDNN, and related packages that are independent of toolchains.
A bucket has to be loaded before its modules become visible. To use the 2023 bucket:
<syntaxhighlight lang="bash">
module load 2023
</syntaxhighlight>
After this, <code>module avail</code> shows the expanded list of modules from that bucket.
The modules are built with [https://easybuild.io/ EasyBuild], which uses publicly shared recipes called easyconfigs. Their [https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs repository] is a good place to check what software is available upstream.
== Requesting modules ==
If a module you need is not available, submit a software request at [https://ideas.anunna.wur.nl https://ideas.anunna.wur.nl]. There you can follow the progress of your request and upvote requests from other users — the more upvotes, the sooner it is prioritised.
== Listing modules ==
Three commands list the available modules in increasing detail. <code>overview</code> gives a top-level view — just the software names and how many versions exist. <code>avail</code> lists the individual versions. <code>spider</code> gives a verbose list with a description of each.
<syntaxhighlight lang="bash">
module overview
module avail
module spider
</syntaxhighlight>
== Searching for modules ==
The same three commands search when you pass a module name as an argument, again at increasing levels of detail:
<syntaxhighlight lang="bash">
module overview <nameOfModule>
module avail <nameOfModule>
module spider <nameOfModule>
</syntaxhighlight>
=== Searching by keyword ===
You can also search for a keyword inside modules with <code>module key</code>. This is useful for finding which module contains a specific Python or R extension — there are bundle modules for both languages that list their extensions, and Lmod searches the module descriptions too.
<syntaxhighlight lang="bash">
module key <keyword>
</syntaxhighlight>
For example, to find which module provides the R package <code>terra</code>, first load a bucket, then search:
<syntaxhighlight lang="bash">
module load 2023
module key terra
</syntaxhighlight>
which yields:
<syntaxhighlight lang="text">
The following modules match your search criteria: "terra"
--------------------------------------------------------------------------------------------------------
  R-bundle-CRAN: R-bundle-CRAN/2023.12-foss-2023a
    Bundle of R packages from CRAN
--------------------------------------------------------------------------------------------------------
</syntaxhighlight>
So you would load <code>R-bundle-CRAN/2023.12-foss-2023a</code> to get the <code>terra</code> package.


== Loading modules ==
== Loading modules ==
Availability of (different versions of) software can be checked by the following command:
  module avail


In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
Load a module by name:
   module load python/3.3.3
 
<syntaxhighlight lang="bash">
module load <moduleName>
</syntaxhighlight>
 
For example, to load Python from the 2023 bucket:
 
<syntaxhighlight lang="bash">
module load 2023
module load Python/3.11.3
</syntaxhighlight>
 
'''Specify the version''' you load, as in the example. It is good practice for consistency and reproducibility — if you omit the version, Lmod picks whatever the current default is, and that default can change over time. Naming the exact version in a submit script also turns the script into documentation of the environment it ran in. When you load a module, its dependencies are loaded automatically.
 
=== Listing loaded modules ===
 
<syntaxhighlight lang="bash">
module list
</syntaxhighlight>
 
Following the example above, after loading the 2023 bucket and <code>Python/3.11.3</code>, the list shows the module plus the dependencies pulled in with it:
 
<syntaxhighlight lang="text">
user001@login201:~$ module list
 
Currently Loaded Modules:
  1) slurm/24.05.1              (S)  5) binutils/2.40-GCCcore-12.3.0    9) Tcl/8.6.13-GCCcore-12.3.0    13) OpenSSL/1.1
  2) 2023                            6) bzip2/1.0.8-GCCcore-12.3.0      10) SQLite/3.42.0-GCCcore-12.3.0  14) Python/3.11.3-GCCcore-12.3.0
  3) GCCcore/12.3.0                  7) ncurses/6.4-GCCcore-12.3.0      11) XZ/5.4.2-GCCcore-12.3.0
  4) zlib/1.2.13-GCCcore-12.3.0      8) libreadline/8.2-GCCcore-12.3.0  12) libffi/3.4.4-GCCcore-12.3.0
 
  Where:
  S:  Module is Sticky, requires --force to unload or purge
</syntaxhighlight>
 
The <code>slurm</code> module is loaded by default and marked '''Sticky''' (<code>S</code>) — it survives a normal unload or purge.
 
== Removing and switching modules ==
 
Unload a single module:
 
<syntaxhighlight lang="bash">
module unload Python/3.11.3
</syntaxhighlight>
 
This unloads only the named module, not its dependencies. To swap one module for another in a single step:
 
<syntaxhighlight lang="bash">
module switch <oldModule> <newModule>
</syntaxhighlight>
 
Some modules refuse to load alongside others — for example, two Java modules cannot be loaded at once. Attempting it produces a conflict message:
 
<syntaxhighlight lang="text">
Module 'foo/2' conflicts with the currently loaded module(s) 'foo/1'
</syntaxhighlight>
 
When you see this, unload or switch rather than trying to stack the two modules.
 
To clear the whole environment, use <code>purge</code>:
 
<syntaxhighlight lang="bash">
module purge
</syntaxhighlight>
 
Sticky modules such as <code>slurm</code> survive a plain purge:
 
<syntaxhighlight lang="text">
user001@login201:~$ module purge
The following modules were not unloaded:
   (Use "module --force purge" to unload all):
 
  1) slurm/24.05.1
</syntaxhighlight>
 
Putting <code>module purge</code> (or <code>module reset</code>) near the top of a job script is good practice: it clears anything that might have been loaded by mistake, so the job starts from a known-clean environment.


== See also ==
== See also ==
* [[B4F_cluster | B4F Cluster]]
 
* [[Installing Personal Software]]
* [[Batch Jobs]]
* [[Python]]
* [[R]]
* [[Apptainer]]


== External links ==
== External links ==
* http://modules.sourceforge.net
 
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules
* [https://lmod.readthedocs.io/ Lmod documentation]
* [https://easybuild.io/ EasyBuild]
* [https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs EasyBuild easyconfigs repository]

Latest revision as of 13:20, 16 June 2026

Anunna provides software through environment modules, managed with Lmod. A module configures your shell — and the environment of your jobs — so that a chosen application and the right version of its dependencies are available. Because each module lives in its own self-contained tree, modules let several otherwise-conflicting programs and several versions of the same program coexist on the cluster without clashing.

Software buckets

Modules on Anunna are organised into buckets. Each bucket is a snapshot of a particular compiler-and-toolchain generation; all the software in a bucket is built with the same compiler. This matters because mixing software built with different compilers in one job can cause conflicts, errors, or silently wrong results — so keeping a job's software within a single bucket keeps it consistent.

The buckets currently available are:

  • legacy — old software that is no longer maintained or updated but is still used in active research.
  • 2023 — software built with the 2023 compilers and toolchain. One version of each package.
  • 2024 — software built with the 2024 compilers and toolchain. One version of each package.
  • utilities — software that does not depend on a specific compiler or toolchain (for example Apptainer).
  • groups — module files contributed by groups inside and outside WUR.
  • GPU — CUDA, cuDNN, and related packages that are independent of toolchains.

A bucket has to be loaded before its modules become visible. To use the 2023 bucket:

module load 2023

After this, module avail shows the expanded list of modules from that bucket.

The modules are built with EasyBuild, which uses publicly shared recipes called easyconfigs. Their repository is a good place to check what software is available upstream.

Requesting modules

If a module you need is not available, submit a software request at https://ideas.anunna.wur.nl. There you can follow the progress of your request and upvote requests from other users — the more upvotes, the sooner it is prioritised.

Listing modules

Three commands list the available modules in increasing detail. overview gives a top-level view — just the software names and how many versions exist. avail lists the individual versions. spider gives a verbose list with a description of each.

module overview
module avail
module spider

Searching for modules

The same three commands search when you pass a module name as an argument, again at increasing levels of detail:

module overview <nameOfModule>
module avail <nameOfModule>
module spider <nameOfModule>

Searching by keyword

You can also search for a keyword inside modules with module key. This is useful for finding which module contains a specific Python or R extension — there are bundle modules for both languages that list their extensions, and Lmod searches the module descriptions too.

module key <keyword>

For example, to find which module provides the R package terra, first load a bucket, then search:

module load 2023
module key terra

which yields:

The following modules match your search criteria: "terra"
--------------------------------------------------------------------------------------------------------

  R-bundle-CRAN: R-bundle-CRAN/2023.12-foss-2023a
    Bundle of R packages from CRAN

--------------------------------------------------------------------------------------------------------

So you would load R-bundle-CRAN/2023.12-foss-2023a to get the terra package.

Loading modules

Load a module by name:

module load <moduleName>

For example, to load Python from the 2023 bucket:

module load 2023
module load Python/3.11.3

Specify the version you load, as in the example. It is good practice for consistency and reproducibility — if you omit the version, Lmod picks whatever the current default is, and that default can change over time. Naming the exact version in a submit script also turns the script into documentation of the environment it ran in. When you load a module, its dependencies are loaded automatically.

Listing loaded modules

module list

Following the example above, after loading the 2023 bucket and Python/3.11.3, the list shows the module plus the dependencies pulled in with it:

user001@login201:~$ module list

Currently Loaded Modules:
  1) slurm/24.05.1              (S)   5) binutils/2.40-GCCcore-12.3.0     9) Tcl/8.6.13-GCCcore-12.3.0     13) OpenSSL/1.1
  2) 2023                             6) bzip2/1.0.8-GCCcore-12.3.0      10) SQLite/3.42.0-GCCcore-12.3.0  14) Python/3.11.3-GCCcore-12.3.0
  3) GCCcore/12.3.0                   7) ncurses/6.4-GCCcore-12.3.0      11) XZ/5.4.2-GCCcore-12.3.0
  4) zlib/1.2.13-GCCcore-12.3.0       8) libreadline/8.2-GCCcore-12.3.0  12) libffi/3.4.4-GCCcore-12.3.0

  Where:
   S:  Module is Sticky, requires --force to unload or purge

The slurm module is loaded by default and marked Sticky (S) — it survives a normal unload or purge.

Removing and switching modules

Unload a single module:

module unload Python/3.11.3

This unloads only the named module, not its dependencies. To swap one module for another in a single step:

module switch <oldModule> <newModule>

Some modules refuse to load alongside others — for example, two Java modules cannot be loaded at once. Attempting it produces a conflict message:

Module 'foo/2' conflicts with the currently loaded module(s) 'foo/1'

When you see this, unload or switch rather than trying to stack the two modules.

To clear the whole environment, use purge:

module purge

Sticky modules such as slurm survive a plain purge:

user001@login201:~$ module purge
The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) slurm/24.05.1

Putting module purge (or module reset) near the top of a job script is good practice: it clears anything that might have been loaded by mistake, so the job starts from a known-clean environment.

See also