Python/Modules

Modules

The Python Environment Modules are the only officially supported Python distributions in the HPC. These are compiled for each specific architecture of the HPC and hence are likely going to be more performant than the versions obtained via Mamba or UV.

Anunna provides one Python version per module bucket, plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module:

• 2023: Python/3.11.3
• 2024: Python/3.12.3
• 2024: Python/3.13.1

So, for instance, loading Python 3.12.3

module load 2024
module load Python/3.12.3

The bundle module Python-bundle-PyPI adds many frequently-used packages on top of the base interpreter. Use module key <package> to find which bundle contains a package you need (see searching modules by keyword).

For packages not in a module, the two recommended routes are a virtual environment built on a Python module (below), or Miniforge for a self-contained conda/mamba setup. The use of Anaconda is discouraged on Anunna — its default channels carry licensing restrictions and the full distribution is heavy; Miniforge is the lighter, unrestricted alternative.

Virtual environments

A virtual environment is a self-contained directory holding a specific Python and its packages, so one project's dependencies cannot clash with another's. Python's built-in venv module is the simplest way to make one on top of a Python module.

First load the Python version you want:

module load 2024
module load Python/3.12.3

Then create the environment in a location of your choosing. The example uses $myNobackup/PythonEnv — $myNobackup is your Lustre nobackup location, set in your ~/.bash_aliases (see Aliases and local variables). Keeping environments on Lustre rather than your home directory avoids filling your home quota, and the nobackup tier is the right choice because an environment can always be recreated from scratch and so does not need backing up.

python -m venv $myNobackup/PythonEnv/my_env

Activate it whenever you want to use it:

source $myNobackup/PythonEnv/my_env/bin/activate

Once active, the environment name appears as a prefix in your prompt:

(my_env) user001@login200:~$

Install packages with pip while the environment is active; they go into the environment, not your home directory:

pip install -U numpy pandas matplotlib

Leave the environment with deactivate.

Jupyter kernels

Once your virtual environment has been setup, you can can access it in jupyter, either via the apps or the notebooks page by setting up a kernel. A kernel is nothing more than a json file with details for jupyter on where to find your Python environment. First things first, with your Python modules loaded activate your virtual environment from the example before.

source $myNobackup/PythonEnv/my_env/bin/activate

The install ipyKernel

(my_env) user001@login200:~$ pip install -U ipykernel

This is the only dependency that Python needs for creating kernels, you can then run

(my_env) user001@login200:~$ python -m ipykernel install --user --name=myenv --display-name="Python (myenv)"

This command will create a folder labelled myenv containing a kernel.json file inside of $HOME/.local/share/jupyter/kernels/ , which jupyter watches. If your goal is just to create a kernel for yourself, then you are done. The kernel should be visible to you both inside of the jupyter instances, either the ones accessed via the anuna apps portal or the notebooks page. Thus, the kernel will not be visible or accessible to other users.

The --name=myenv flag is entirely optional, it is the directory in which the kernel.json file is generated. If not used, python will simply label it python3 . The use of this flag is recommended, specially when working with multiple kernels to avoid overwriting previously created kernels.

Shared Kernels

Whether for a course or collaboration, sometimes it is convenient to share a kernel with other users. This is possible and the procedure is a variation of the previous one, with just some additional requirements

• The virtual environment you create needs to be in a shared location , e.g. /lutre/shared
• The virtual environment folder needs to be visiible and accessible to other users i.e. it must have read and execute permissions, 755 is recommended

Like before load the bucket and its corresponding Python version and create your project folder and go there

module load 2025
module load Python/3.13.1
mkdir /lustre/shared/MyProject
cd /lustre/shared/MyProject

Then create your virtual environemnt, activate it and install ipyKernel

python -m venv ./project_env
source /lustre/shared/MyProject/project_env/bin/activate
pip install -U ipykernel

Here is where things differ, inside the activate kernel run the command

python -m ipykernel install --sys-prefix --name=myproject --display-name="Python (project_env)"

Note that now the command uses the flag --sys-prefix . This flag sets the installation path of the kernel inside the virtual environment's directory, in this case, the kernel.json file will be generate in /lustre/shared/MyProject/project_env/share/jupyter/kernels/myproject.

While the kernel has been created, jupyter still will not be able to see it. In order to be able to see and access the kernel you would need to either copy it to your folder

MYPROJ=/lustre/shared/MyProject
cp -r $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/

or link it

MYPROJ=/lustre/shared/MyProject
ln -s $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/

Any users that run this last step should gain access to the kernel from jupyter. Though note, that only users that have write permissions to virtual environment will be able to install and remove modules from it.

Note: When using custom kernels do not run !pip, just run pip without the !

See also

• Environment Modules
• Installing Personal Software
• Jupyter
• Python
• R
• Apptainer

External links

• Python venv documentation
• Miniforge
• conda-forge