Environment Modules: Difference between revisions

From HPCwiki
Jump to navigation Jump to search
Dawes0011 (talk | contribs)
No edit summary
Phase 1 § 5 P1.5.2: merge Modules + Environment Modules + Using environment modules into the canonical Environment Modules page. Base content from Modules (Lmod/buckets), salvaged switching+conflicts concept, dropped pre-Lmod stale content. (via update-page on MediaWiki MCP Server)
 
Line 1: Line 1:
== Preface ==
Anunna provides software through '''environment modules''', managed with [https://lmod.readthedocs.io/ Lmod]. A module configures your shell — and the environment of your jobs — so that a chosen application and the right version of its dependencies are available. Because each module lives in its own self-contained tree, modules let several otherwise-conflicting programs and several versions of the same program coexist on the cluster without clashing.
Environment modules [http://modules.sourceforge.net] are a smart way to provide interchangeable blocks of executables and reproducible environments for use in an HPC. It's also the only way to provide simultaneous versions of the same software without collisions, as each module is housed entirely in its own subfolder structure.


== Using modules ==
== Software buckets ==
The module executable is automatically provided to you upon login. Most users have some modules automatically loaded as well; to see these, use


<nowiki>module list</nowiki>
Modules on Anunna are organised into '''buckets'''. Each bucket is a snapshot of a particular compiler-and-toolchain generation; all the software in a bucket is built with the same compiler. This matters because mixing software built with different compilers in one job can cause conflicts, errors, or silently wrong results — so keeping a job's software within a single bucket keeps it consistent.


You should be able to see which modules are loaded.
The buckets currently available are:


One of the most important modules to load is 'shared' - this is Anunna specific, as it will extend the MODULEPATH environment variable to use modules present in /shared as well as /cm/local/ . Without this, many modules will not be available to you.
* '''legacy''' — old software that is no longer maintained or updated but is still used in active research.
* '''2023''' — software built with the 2023 compilers and toolchain. One version of each package.
* '''2024''' — software built with the 2024 compilers and toolchain. One version of each package.
* '''utilities''' — software that does not depend on a specific compiler or toolchain (for example [[Apptainer]]).
* '''groups''' — module files contributed by groups inside and outside WUR.
* '''GPU''' — CUDA, cuDNN, and related packages that are independent of toolchains.
 
A bucket has to be loaded before its modules become visible. To use the 2023 bucket:
 
<syntaxhighlight lang="bash">
module load 2023
</syntaxhighlight>
 
After this, <code>module avail</code> shows the expanded list of modules from that bucket.
 
The modules are built with [https://easybuild.io/ EasyBuild], which uses publicly shared recipes called easyconfigs. Their [https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs repository] is a good place to check what software is available upstream.
 
== Requesting modules ==
 
If a module you need is not available, submit a software request at [https://ideas.anunna.wur.nl https://ideas.anunna.wur.nl]. There you can follow the progress of your request and upvote requests from other users — the more upvotes, the sooner it is prioritised.
 
== Listing modules ==
 
Three commands list the available modules in increasing detail. <code>overview</code> gives a top-level view — just the software names and how many versions exist. <code>avail</code> lists the individual versions. <code>spider</code> gives a verbose list with a description of each.
 
<syntaxhighlight lang="bash">
module overview
module avail
module spider
</syntaxhighlight>
 
== Searching for modules ==
 
The same three commands search when you pass a module name as an argument, again at increasing levels of detail:
 
<syntaxhighlight lang="bash">
module overview <nameOfModule>
module avail <nameOfModule>
module spider <nameOfModule>
</syntaxhighlight>
 
=== Searching by keyword ===
 
You can also search for a keyword inside modules with <code>module key</code>. This is useful for finding which module contains a specific Python or R extension — there are bundle modules for both languages that list their extensions, and Lmod searches the module descriptions too.
 
<syntaxhighlight lang="bash">
module key <keyword>
</syntaxhighlight>
 
For example, to find which module provides the R package <code>terra</code>, first load a bucket, then search:
 
<syntaxhighlight lang="bash">
module load 2023
module key terra
</syntaxhighlight>
 
which yields:
 
<syntaxhighlight lang="text">
The following modules match your search criteria: "terra"
--------------------------------------------------------------------------------------------------------
 
  R-bundle-CRAN: R-bundle-CRAN/2023.12-foss-2023a
    Bundle of R packages from CRAN
 
--------------------------------------------------------------------------------------------------------
</syntaxhighlight>
 
So you would load <code>R-bundle-CRAN/2023.12-foss-2023a</code> to get the <code>terra</code> package.


== Loading modules ==
== Loading modules ==
Availability of (different versions of) software can be checked by the following command:
<nowiki>module avail</nowiki>


For example, you should be able to find the basic module slurm. This provides the path to the sbatch, srun, etc. executables for job submission. To load this, simply:
Load a module by name:
  <nowiki>module load slurm</nowiki>
 
<syntaxhighlight lang="bash">
module load <moduleName>
</syntaxhighlight>
 
For example, to load Python from the 2023 bucket:
 
<syntaxhighlight lang="bash">
module load 2023
module load Python/3.11.3
</syntaxhighlight>
 
'''Specify the version''' you load, as in the example. It is good practice for consistency and reproducibility — if you omit the version, Lmod picks whatever the current default is, and that default can change over time. Naming the exact version in a submit script also turns the script into documentation of the environment it ran in. When you load a module, its dependencies are loaded automatically.
 
=== Listing loaded modules ===
 
<syntaxhighlight lang="bash">
module list
</syntaxhighlight>
 
Following the example above, after loading the 2023 bucket and <code>Python/3.11.3</code>, the list shows the module plus the dependencies pulled in with it:
 
<syntaxhighlight lang="text">
user001@login201:~$ module list
 
Currently Loaded Modules:
  1) slurm/24.05.1              (S)  5) binutils/2.40-GCCcore-12.3.0    9) Tcl/8.6.13-GCCcore-12.3.0    13) OpenSSL/1.1
  2) 2023                            6) bzip2/1.0.8-GCCcore-12.3.0      10) SQLite/3.42.0-GCCcore-12.3.0  14) Python/3.11.3-GCCcore-12.3.0
  3) GCCcore/12.3.0                  7) ncurses/6.4-GCCcore-12.3.0      11) XZ/5.4.2-GCCcore-12.3.0
  4) zlib/1.2.13-GCCcore-12.3.0      8) libreadline/8.2-GCCcore-12.3.0  12) libffi/3.4.4-GCCcore-12.3.0
 
  Where:
  S: Module is Sticky, requires --force to unload or purge
</syntaxhighlight>
 
The <code>slurm</code> module is loaded by default and marked '''Sticky''' (<code>S</code>) — it survives a normal unload or purge.
 
== Removing and switching modules ==
 
Unload a single module:
 
<syntaxhighlight lang="bash">
module unload Python/3.11.3
</syntaxhighlight>


And you will see that it will automatically load the latest version - no need to write out any further.
This unloads only the named module, not its dependencies. To swap one module for another in a single step:


Many of the hand-installed programs have a path such as:
<syntaxhighlight lang="bash">
<nowiki>hdf5/gcc/64/1.8.14</nowiki>
module switch <oldModule> <newModule>
Which translates into:
</syntaxhighlight>
<nowiki>SOFTWARE/COMPILER/BITS/VERSION</nowiki>


You can elect to load this to various levels:
Some modules refuse to load alongside others — for example, two Java modules cannot be loaded at once. Attempting it produces a conflict message:


<nowiki>module load hdf5              # loads the latest version, not caring for compiler
<syntaxhighlight lang="text">
module load hdf5/gcc          # loads the latest gcc-compiled version, not caring for 32/64 bits (default 64)
Module 'foo/2' conflicts with the currently loaded module(s) 'foo/1'
module load hdf5/gcc/64        # loads the latest 64-bit gcc-compiled version
</syntaxhighlight>
module load hdf5/gcc/64/1.8.14 # loads this specific version of hdf5</nowiki>


This allows your job scripts to either automatically be upgraded when the latest executables are installed, or elect to use only one specific version of a piece of code.
When you see this, unload or switch rather than trying to stack the two modules.


== Switching modules ==
To clear the whole environment, use <code>purge</code>:


If you want to remove a module, simply
<syntaxhighlight lang="bash">
<nowiki>module unload module/1</nowiki>
module purge
This will remove the executable path from your environment. It'll also follow the same logic as above, i.e. you can unload all loaded slurm modules independent of version by just unloading the base module name. You can then load up a new one. You can do this in one command, with:
</syntaxhighlight>
<nowiki>module switch module/1 module/2</nowiki>


Some modules will not allow themselves to be loaded when another one is loaded, for instance, for sanity reasons it's not possible to load two java modules at the same time. Trying to do this will give:
Sticky modules such as <code>slurm</code> survive a plain purge:


<nowiki>Module 'module/2' conflicts with the currently loaded module(s) 'module/1'</nowiki>
<syntaxhighlight lang="text">
If you're seeing this, you must unload or switch your modules rather than overloading them:
user001@login201:~$ module purge
<nowiki>module switch module/2</nowiki>
The following modules were not unloaded:
This works if both modules have the same base path.
  (Use "module --force purge" to unload all):


  1) slurm/24.05.1
</syntaxhighlight>
Putting <code>module purge</code> (or <code>module reset</code>) near the top of a job script is good practice: it clears anything that might have been loaded by mistake, so the job starts from a known-clean environment.
== See also ==
* [[Installing Personal Software]]
* [[Batch Jobs]]
* [[Python]]
* [[R]]
* [[Apptainer]]


== External links ==
== External links ==
* http://modules.sourceforge.net
 
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules
* [https://lmod.readthedocs.io/ Lmod documentation]
* [https://easybuild.io/ EasyBuild]
* [https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs EasyBuild easyconfigs repository]

Latest revision as of 13:20, 16 June 2026

Anunna provides software through environment modules, managed with Lmod. A module configures your shell — and the environment of your jobs — so that a chosen application and the right version of its dependencies are available. Because each module lives in its own self-contained tree, modules let several otherwise-conflicting programs and several versions of the same program coexist on the cluster without clashing.

Software buckets

Modules on Anunna are organised into buckets. Each bucket is a snapshot of a particular compiler-and-toolchain generation; all the software in a bucket is built with the same compiler. This matters because mixing software built with different compilers in one job can cause conflicts, errors, or silently wrong results — so keeping a job's software within a single bucket keeps it consistent.

The buckets currently available are:

  • legacy — old software that is no longer maintained or updated but is still used in active research.
  • 2023 — software built with the 2023 compilers and toolchain. One version of each package.
  • 2024 — software built with the 2024 compilers and toolchain. One version of each package.
  • utilities — software that does not depend on a specific compiler or toolchain (for example Apptainer).
  • groups — module files contributed by groups inside and outside WUR.
  • GPU — CUDA, cuDNN, and related packages that are independent of toolchains.

A bucket has to be loaded before its modules become visible. To use the 2023 bucket:

module load 2023

After this, module avail shows the expanded list of modules from that bucket.

The modules are built with EasyBuild, which uses publicly shared recipes called easyconfigs. Their repository is a good place to check what software is available upstream.

Requesting modules

If a module you need is not available, submit a software request at https://ideas.anunna.wur.nl. There you can follow the progress of your request and upvote requests from other users — the more upvotes, the sooner it is prioritised.

Listing modules

Three commands list the available modules in increasing detail. overview gives a top-level view — just the software names and how many versions exist. avail lists the individual versions. spider gives a verbose list with a description of each.

module overview
module avail
module spider

Searching for modules

The same three commands search when you pass a module name as an argument, again at increasing levels of detail:

module overview <nameOfModule>
module avail <nameOfModule>
module spider <nameOfModule>

Searching by keyword

You can also search for a keyword inside modules with module key. This is useful for finding which module contains a specific Python or R extension — there are bundle modules for both languages that list their extensions, and Lmod searches the module descriptions too.

module key <keyword>

For example, to find which module provides the R package terra, first load a bucket, then search:

module load 2023
module key terra

which yields:

The following modules match your search criteria: "terra"
--------------------------------------------------------------------------------------------------------

  R-bundle-CRAN: R-bundle-CRAN/2023.12-foss-2023a
    Bundle of R packages from CRAN

--------------------------------------------------------------------------------------------------------

So you would load R-bundle-CRAN/2023.12-foss-2023a to get the terra package.

Loading modules

Load a module by name:

module load <moduleName>

For example, to load Python from the 2023 bucket:

module load 2023
module load Python/3.11.3

Specify the version you load, as in the example. It is good practice for consistency and reproducibility — if you omit the version, Lmod picks whatever the current default is, and that default can change over time. Naming the exact version in a submit script also turns the script into documentation of the environment it ran in. When you load a module, its dependencies are loaded automatically.

Listing loaded modules

module list

Following the example above, after loading the 2023 bucket and Python/3.11.3, the list shows the module plus the dependencies pulled in with it:

user001@login201:~$ module list

Currently Loaded Modules:
  1) slurm/24.05.1              (S)   5) binutils/2.40-GCCcore-12.3.0     9) Tcl/8.6.13-GCCcore-12.3.0     13) OpenSSL/1.1
  2) 2023                             6) bzip2/1.0.8-GCCcore-12.3.0      10) SQLite/3.42.0-GCCcore-12.3.0  14) Python/3.11.3-GCCcore-12.3.0
  3) GCCcore/12.3.0                   7) ncurses/6.4-GCCcore-12.3.0      11) XZ/5.4.2-GCCcore-12.3.0
  4) zlib/1.2.13-GCCcore-12.3.0       8) libreadline/8.2-GCCcore-12.3.0  12) libffi/3.4.4-GCCcore-12.3.0

  Where:
   S:  Module is Sticky, requires --force to unload or purge

The slurm module is loaded by default and marked Sticky (S) — it survives a normal unload or purge.

Removing and switching modules

Unload a single module:

module unload Python/3.11.3

This unloads only the named module, not its dependencies. To swap one module for another in a single step:

module switch <oldModule> <newModule>

Some modules refuse to load alongside others — for example, two Java modules cannot be loaded at once. Attempting it produces a conflict message:

Module 'foo/2' conflicts with the currently loaded module(s) 'foo/1'

When you see this, unload or switch rather than trying to stack the two modules.

To clear the whole environment, use purge:

module purge

Sticky modules such as slurm survive a plain purge:

user001@login201:~$ module purge
The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) slurm/24.05.1

Putting module purge (or module reset) near the top of a job script is good practice: it clears anything that might have been loaded by mistake, so the job starts from a known-clean environment.

See also