Environment Modules
Preface
Environment modules [1] are a smart way to provide interchangeable blocks of executables and reproducible environments for use in an HPC. It's also the only way to provide simultaneous versions of the same software without collisions, as each module is housed entirely in its own subfolder structure. This is in heavy use on the AgroGenomics HPC.
Using modules
The module executable is automatically provided to you upon login. Most users have some modules automatically loaded as well; to see these, use
module list
You should be able to see which modules are loaded.
One of the most important modules to load is 'shared' - this is HPCAG specific, as it will extend the MODULEPATH environment variable to use modules present in /cm/shared as well as /cm/local/ . Without this, many modules will not be available to you.
Loading modules
Availability of (different versions of) software can be checked by the following command:
module avail
For example, you should be able to find the basic module slurm. This provides the path to the sbatch, srun, etc. executables for job submission. To load this, simply:
module load slurm
And you will see that it will automatically load the latest version - no need to write out any further.
Many of the hand-installed programs have a path such as:
hdf5/gcc/64/1.8.14
Which translates into:
SOFTWARE/COMPILER/BITS/VERSION
You can elect to load this to various levels:
module load hdf5 # loads the latest version, not caring for compiler module load hdf5/gcc # loads the latest gcc-compiled version, not caring for 32/64 bits (default 64) module load hdf5/gcc/64 # loads the latest 64-bit gcc-compiled version module load hdf5/gcc/64/1.8.14 # loads this specific version of hdf5
This allows your job scripts to either automatically be upgraded when the latest executables are installed, or elect to use only one specific version of a piece of code.
Switching modules
If you want to remove a module, simply
module unload module/1
This will remove the executable path from your environment. It'll also follow the same logic as above, i.e. you can unload all loaded slurm modules independent of version by just unloading the base module name. You can then load up a new one. You can do this in one command, with:
module switch module/1 module/2
Some modules will not allow themselves to be loaded when another one is loaded, for instance, for sanity reasons it's not possible to load two java modules at the same time. Trying to do this will give:
Module 'module/2' conflicts with the currently loaded module(s) 'module/1'
If you're seeing this, you must unload or switch your modules rather than overloading them:
module switch module/2
This works if both modules have the same base path.