Using environment modules
Environment modules are a simple way to allow multiple potentially clashing programs to coexist on a large shared machine such as an HPC. It allows a user to specify exactly which programs are loaded, and even which version of each program, whilst simultaneously allowing the administrator the ability to automatically configure the appropriate environment variables for the system itself.
Upon logging in to the HPC, you should find that when you do:
module list
You will see something like this: <source lang='bash'> -bash-4.1$ module list Currently Loaded Modulefiles:
1) shared 2) slurm/2.5.7
</source>
This is a list of all loaded modules in your shell session. To get a list of all available modules, simply
module available
And this will show you the (very exhaustive) list of modules on the HPC:
<source lang='bash'> -bash-4.1$ module avail
/cm/shared/modulefiles ----------------------------
acml/gcc/64/5.3.1 netcdf/gcc/64/4.1.3 acml/gcc/fma4/5.3.1 netcdf/gcc/64/4.3.0 acml/gcc/mp/64/5.3.1 netcdf/gcc/64/4.3.2 acml/gcc/mp/fma4/5.3.1 netcdf/gcc/64/4.3.3 acml/gcc-int64/64/5.3.1 netcdf/gcc/64/4.3.3.1 acml/gcc-int64/fma4/5.3.1 netcdf/intel/64/4.1.3 ... </source>
Let's look at each of these module names. Each module is named for the application it provides, plus a subfolder of which compiler it was compiled with (if compiled), the number of address bits or options (if compiled), and the version.
If you want to see a list for a specific module, you can
module avail netcdf
And the complete list of versions will be shown.
To load a module, simply
module load foo
And the most recent version of module foo will automatically be loaded. If foo is compiled, it will automatically select the gcc version. If you want to specify a certain version, then
module load foo/gcc/64/1.0.0
Will load foo version 1, compiled with gcc. Be advised that this may not always work, as some modules are not compatible with each other, but a message will be shown if this is the case. Additionally, some modules will automatically load other modules with them for them to operate.
If you want to remove a module that you've loaded, then
module unload foo
Will remove all module foo's loaded.
Example
Consider this simple python3 script that should calculate Pi to 1 million digits: <source lang='python'> from decimal import * D=Decimal getcontext().prec=10000000 p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411)) print(str(p)[:10000002]) </source>
Loading modules
This script will not run at all in the default 2.4 version of Python on the cluster. In order for this script to run you must use Python3. To do this, first list all versions of Python: <source> -bash-4.1$ module avail python
/cm/shared/modulefiles ----------------------------
python/2.7.6 python/3.3.3 python/3.4.2 </source>
Then you can load the specific version you need:
module load python/3.3.3
Now you have access to the executable python3.
See also
- B4F Cluster
- Lx6 and Lx7 compute nodes
- Environment Modules
- Control R environment using modules
- Create a shortcut for the ssh log-in command
- Installing R packages locally