Python/Modules: Difference between revisions

From HPCwiki
Jump to navigation Jump to search
← Older editNewer edit →
Honfi001 (talk | contribs)
Bureaucrats, Administrators
240 edits
VisualWikitext
No edit summary
No edit summary
Line 54: Line 54:


Leave the environment with <code>deactivate</code>.
Leave the environment with <code>deactivate</code>.
== Miniforge (conda / mamba) ==
[https://github.com/conda-forge/miniforge Miniforge] is a minimal installer that gives you the <code>conda</code> and <code>mamba</code> package managers preconfigured to use the community [https://conda-forge.org/ conda-forge] channel. <code>mamba</code> is a fast drop-in replacement for <code>conda</code>. This is the recommended way to use conda-style environments on Anunna; it avoids Anaconda's licensing restrictions.
Download and run the installer, pointing it at a location with room (your Lustre nobackup space, not your home directory):
<syntaxhighlight lang="bash">
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh -b -p $myNobackup/miniforge3
</syntaxhighlight>
'''Do not run <code>conda init</code>''' on Anunna — it writes startup code into your <code>~/.bashrc</code> that runs on every login and can interfere with the module system. Instead, activate Miniforge only when you need it:
<syntaxhighlight lang="bash">
source $myNobackup/miniforge3/bin/activate
</syntaxhighlight>
Then create and use environments with <code>mamba</code>:
<syntaxhighlight lang="bash">
mamba create -n myenv python=3.12 numpy pandas
mamba activate myenv
</syntaxhighlight>


== Jupyter kernels ==
== Jupyter kernels ==


To use one of your environments inside [[Jupyter]], register it as a kernel.
Once your virtual environment has been setup, you can can access it in jupyter, either via the apps or the notebooks page by setting up a kernel. A kernel is nothing more than a json file with details for jupyter on where to find your Python environment. First things first, with your Python modules loaded activate your virtual environment from the example before.  
 
=== From a virtual environment ===
 
A <code>venv</code> kernel needs a wrapper script, because the kernel is launched without your normal shell environment and so cannot load modules by itself. First, with the environment active, install <code>ipykernel</code> and generate the kernel:
 
<syntaxhighlight lang="bash">
module load 2024
module load Python/3.12.3
source $myNobackup/PythonEnv/my_env/bin/activate
pip install ipykernel
python -m ipykernel install --user --name=my_env_kernel
</syntaxhighlight>
 
The kernel is written to <code>~/.local/share/jupyter/kernels/</code>, which Jupyter watches. On its own it will not work, because it cannot find the modules — so write a wrapper script that loads them. Save this as, for example, <code>$HOME/wrap.sh</code>:
 
<syntaxhighlight lang="bash">
#!/bin/bash -l
 
module reset
module load 2024
module load Python/3.12.3


exec $myNobackup/PythonEnv/my_env/bin/python -m ipykernel_launcher "$@"
<syntaxhighlight lang="bash">source $myNobackup/PythonEnv/my_env/bin/activate</syntaxhighlight>The install [https://pypi.org/project/ipykernel/ ipyKernel]<syntaxhighlight lang="bash">(my_env) user001@login200:~$ pip install -U ipykernel</syntaxhighlight>This is the only dependency that Python needs for creating kernels, you can then run <syntaxhighlight lang="bash">(my_env) user001@login200:~$ python -m ipykernel install --user --name=myenv --display-name="Python (myenv)"</syntaxhighlight>This command will create a folder labelled <code>myenv</code>  containing a <code>kernel.json</code> file inside of  <code>$HOME/.local/share/jupyter/kernels/</code> , which jupyter watches. If your goal is just to create a kernel for yourself, then you are done. The kernel should be visible to you both inside of the jupyter instances, either the ones accessed via the [https://apps.anunna.wur.nl anuna apps portal] or the [https://notebook.anunna.wur.nl notebooks page].  Thus, the kernel will not be visible or accessible to other users.
</syntaxhighlight>


The <code>#!/bin/bash -l</code> line starts a login shell, which loads Lmod and sources your <code>~/.bash_aliases</code> (so <code>$myNobackup</code> is defined). Make the wrapper executable with <code>chmod +x $HOME/wrap.sh</code>.
The --name=myenv flag is entirely optional, it is the directory in which the <code>kernel.json</code> file is generated. If not used, python will simply label it <code>python3</code> . The use of this flag is recommended, specially when working with multiple kernels to avoid overwriting previously created kernels.  


Finally point the kernel at the wrapper by editing <code>~/.local/share/jupyter/kernels/my_env_kernel/kernel.json</code>:
=== Shared Kernels ===
Whether for a course or collaboration, sometimes it is convenient to share a kernel with other users. This is possible and the procedure is a variation of the previous one, with just some additional requirements


<syntaxhighlight lang="json">
* The virtual environment you create needs to be in a shared location , e.g. /lutre/shared
{
* The virtual environment folder needs to be visiible and accessible to other users i.e. it must have read and execute permissions, 755 is recommended
"argv": [
  "/home/WUR/user001/wrap.sh",
  "-f",
  "{connection_file}"
],
"display_name": "Python my_env",
"language": "python",
"metadata": {
  "debugger": true
}
}
</syntaxhighlight>


The only difference from a plain kernel file is that <code>argv</code> points at the wrapper script instead of the Python executable directly.


=== From a conda / mamba environment ===
Like before load the  bucket and its corresponding Python version and create your project folder and go there<syntaxhighlight lang="bash">module load 2025
module load Python/3.13.1
mkdir /lustre/shared/MyProject
cd /lustre/shared/MyProject


A conda or mamba environment is simpler, because the environment is self-contained. With Miniforge active and your environment created, install <code>ipykernel</code> into it and register the kernel:
</syntaxhighlight>Then create your virtual environemnt, activate it and install [https://pypi.org/project/ipykernel/ ipyKernel]<syntaxhighlight lang="bash">python -m venv ./project_env
source /lustre/shared/MyProject/project_env/bin/activate
pip install -U ipykernel</syntaxhighlight>


<syntaxhighlight lang="bash">
Here is where things differ, inside the activate kernel run the command<syntaxhighlight lang="bash">
mamba create -y -n kernel_test python=3 ipykernel
python -m ipykernel install --sys-prefix --name=myproject --display-name="Python (project_env)"
mamba activate kernel_test
</syntaxhighlight>Note that now the command uses the flag  --sys-prefix . This flag sets the installation path of the kernel '''inside''' the virtual environment's directory, in this case, the kernel.json file will be generate in  <code>/lustre/shared/MyProject/project_env/share/jupyter/kernels/myproject</code>.
python -m ipykernel install --user --name kernel_test
</syntaxhighlight>


To remove the kernel and environment again:


<syntaxhighlight lang="bash">
While the kernel has been created, jupyter still will not be able to see it. In order to be able to see and access the kernel you would need to either copy it to your  folder<syntaxhighlight lang="bash">
jupyter kernelspec uninstall kernel_test
MYPROJ=/lustre/shared/MyProject
mamba deactivate
cp -r $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/
mamba remove -y -n kernel_test --all
</syntaxhighlight>or link it<syntaxhighlight lang="bash">
</syntaxhighlight>
MYPROJ=/lustre/shared/MyProject
ln -s $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/
</syntaxhighlight>Any users that run this last step should gain access to the kernel from jupyter. Though note, that only users that have write permissions to the kernel file will be able to modify


== See also ==
== See also ==

Revision as of 14:56, 12 August 2026

Modules

The Python Environment Modules are the only officially supported Python distributions in the HPC. These are compiled for each specific architecture of the HPC and hence are likely going to be more performant than the versions obtained via Mamba or UV.

Anunna provides one Python version per module bucket, plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module:

  • 2023: Python/3.11.3
  • 2024: Python/3.12.3
  • 2024: Python/3.13.1


module load 2024
module load Python/3.12.3

The bundle module Python-bundle-PyPI adds many frequently-used packages on top of the base interpreter. Use module key <package> to find which bundle contains a package you need (see searching modules by keyword).

For packages not in a module, the two recommended routes are a virtual environment built on a Python module (below), or Miniforge for a self-contained conda/mamba setup. The use of Anaconda is discouraged on Anunna — its default channels carry licensing restrictions and the full distribution is heavy; Miniforge is the lighter, unrestricted alternative.

Virtual environments

A virtual environment is a self-contained directory holding a specific Python and its packages, so one project's dependencies cannot clash with another's. Python's built-in venv module is the simplest way to make one on top of a Python module.

First load the Python version you want: 

module load 2024
module load Python/3.12.3

Then create the environment in a location of your choosing. The example uses $myNobackup/PythonEnv — $myNobackup is your Lustre nobackup location, set in your ~/.bash_aliases (see Aliases and local variables). Keeping environments on Lustre rather than your home directory avoids filling your home quota, and the nobackup tier is the right choice because an environment can always be recreated from scratch and so does not need backing up. 

python -m venv $myNobackup/PythonEnv/my_env

Activate it whenever you want to use it: 

source $myNobackup/PythonEnv/my_env/bin/activate

Once active, the environment name appears as a prefix in your prompt: 

(my_env) user001@login200:~$

Install packages with pip while the environment is active; they go into the environment, not your home directory: 

pip install -U numpy pandas matplotlib

Leave the environment with deactivate.

Jupyter kernels

Once your virtual environment has been setup, you can can access it in jupyter, either via the apps or the notebooks page by setting up a kernel. A kernel is nothing more than a json file with details for jupyter on where to find your Python environment. First things first, with your Python modules loaded activate your virtual environment from the example before. 

source $myNobackup/PythonEnv/my_env/bin/activate

The install ipyKernel

(my_env) user001@login200:~$ pip install -U ipykernel

This is the only dependency that Python needs for creating kernels, you can then run 

(my_env) user001@login200:~$ python -m ipykernel install --user --name=myenv --display-name="Python (myenv)"

This command will create a folder labelled myenv containing a kernel.json file inside of $HOME/.local/share/jupyter/kernels/ , which jupyter watches. If your goal is just to create a kernel for yourself, then you are done. The kernel should be visible to you both inside of the jupyter instances, either the ones accessed via the anuna apps portal or the notebooks page. Thus, the kernel will not be visible or accessible to other users.

The --name=myenv flag is entirely optional, it is the directory in which the kernel.json file is generated. If not used, python will simply label it python3 . The use of this flag is recommended, specially when working with multiple kernels to avoid overwriting previously created kernels.

Shared Kernels

Whether for a course or collaboration, sometimes it is convenient to share a kernel with other users. This is possible and the procedure is a variation of the previous one, with just some additional requirements

  • The virtual environment you create needs to be in a shared location , e.g. /lutre/shared
  • The virtual environment folder needs to be visiible and accessible to other users i.e. it must have read and execute permissions, 755 is recommended


Like before load the bucket and its corresponding Python version and create your project folder and go there

module load 2025
module load Python/3.13.1
mkdir /lustre/shared/MyProject
cd /lustre/shared/MyProject

Then create your virtual environemnt, activate it and install ipyKernel

python -m venv ./project_env
source /lustre/shared/MyProject/project_env/bin/activate
pip install -U ipykernel

Here is where things differ, inside the activate kernel run the command

python -m ipykernel install --sys-prefix --name=myproject --display-name="Python (project_env)"

Note that now the command uses the flag --sys-prefix . This flag sets the installation path of the kernel inside the virtual environment's directory, in this case, the kernel.json file will be generate in /lustre/shared/MyProject/project_env/share/jupyter/kernels/myproject.


While the kernel has been created, jupyter still will not be able to see it. In order to be able to see and access the kernel you would need to either copy it to your folder

MYPROJ=/lustre/shared/MyProject
cp -r $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/

or link it

MYPROJ=/lustre/shared/MyProject
ln -s $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/

Any users that run this last step should gain access to the kernel from jupyter. Though note, that only users that have write permissions to the kernel file will be able to modify

See also

Retrieved from "https://wiki.anunna.wur.nl/index.php?title=Python/Modules&oldid=3045"