Python/Modules - Revision history

Haars001 at 07:52, 28 August 2026

2026-08-28T07:52:10Z

← Older revision		Revision as of 07:52, 28 August 2026
Line 58:		Line 58:
	== Jupyter kernels ==		== Jupyter kernels ==

	Once your virtual environment has been setup, you can can access it in jupyter, either via the apps or the notebooks page by setting up a kernel. A kernel is nothing more than a json file with details for jupyter on where to find your Python environment. First things first, with your Python modules loaded activate your virtual environment from the example before.		Once your virtual environment has been setup, you can can access it in jupyter, either via the apps or the notebooks page by setting up a '''kernel'''. A kernel is nothing more than a json file with details for jupyter on where to find your Python environment. First things first, with your Python modules loaded activate your virtual environment from the example before.

	<syntaxhighlight lang="bash">source $myNobackup/PythonEnv/my_env/bin/activate</syntaxhighlight>The install [https://pypi.org/project/ipykernel/ ipyKernel]<syntaxhighlight lang="bash">(my_env) user001@login200:~$ pip install -U ipykernel</syntaxhighlight>This is the only dependency that Python needs for creating kernels, you can then run <syntaxhighlight lang="bash">(my_env) user001@login200:~$ python -m ipykernel install --user --name=myenv --display-name="Python (myenv)"</syntaxhighlight>This command will create a folder labelled <code>myenv</code> containing a <code>kernel.json</code> file inside of <code>$HOME/.local/share/jupyter/kernels/</code> , which jupyter watches. If your goal is just to create a kernel for yourself, then you are done. The kernel should be visible to you both inside of the jupyter instances, either the ones accessed via the [https://apps.anunna.wur.nl anuna apps portal] or the [https://notebook.anunna.wur.nl notebooks page]. Thus, the kernel will not be visible or accessible to other users.		<syntaxhighlight lang="bash">source $myNobackup/PythonEnv/my_env/bin/activate</syntaxhighlight>The install [https://pypi.org/project/ipykernel/ ipyKernel]<syntaxhighlight lang="bash">(my_env) user001@login200:~$ pip install -U ipykernel</syntaxhighlight>This is the only dependency that Python needs for creating kernels, you can then run <syntaxhighlight lang="bash">(my_env) user001@login200:~$ python -m ipykernel install --user --name=myenv --display-name="Python (myenv)"</syntaxhighlight>This command will create a folder labelled <code>myenv</code> containing a <code>kernel.json</code> file inside of <code>$HOME/.local/share/jupyter/kernels/</code> , which jupyter watches. If your goal is just to create a kernel for yourself, then you are done. The kernel should be visible to you both inside of the jupyter instances, either the ones accessed via the [https://apps.anunna.wur.nl anuna apps portal] or the [https://notebook.anunna.wur.nl notebooks page]. Thus, the kernel will not be visible or accessible to other users.
Line 69:		Line 69:
	* The virtual environment you create needs to be in a shared location , e.g. /lutre/shared		* The virtual environment you create needs to be in a shared location , e.g. /lutre/shared
	* The virtual environment folder needs to be visiible and accessible to other users i.e. it must have read and execute permissions, 755 is recommended		* The virtual environment folder needs to be visiible and accessible to other users i.e. it must have read and execute permissions, 755 is recommended


	Like before load the bucket and its corresponding Python version and create your project folder and go there<syntaxhighlight lang="bash">module load 2025		Like before load the bucket and its corresponding Python version and create your project folder and go there<syntaxhighlight lang="bash">module load 2025

Honfi001: /* See also */

2026-08-17T14:59:54Z

Honfi001 at 14:49, 17 August 2026

2026-08-17T14:49:19Z

← Older revision		Revision as of 14:49, 17 August 2026
Line 1:		Line 1:
			{{DISPLAYTITLE:Python Modules}}
	= Modules =		= Modules =
	The Python Environment Modules are the only officially supported Python distributions in the HPC. These are compiled for each specific architecture of the HPC and hence are likely going to be more performant than the versions obtained via Mamba or UV.		The Python Environment Modules are the only officially supported Python distributions in the HPC. These are compiled for each specific architecture of the HPC and hence are likely going to be more performant than the versions obtained via Mamba or UV.

Honfi001 at 15:07, 12 August 2026

2026-08-12T15:07:10Z

← Older revision		Revision as of 15:07, 12 August 2026
Line 8:		Line 8:
	* '''2024''': <code>Python/3.13.1</code>		* '''2024''': <code>Python/3.13.1</code>

			So, for instance, loading Python 3.12.3
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	module load 2024		module load 2024

Honfi001: /* Jupyter kernels */

2026-08-12T15:06:07Z

Jupyter kernels

← Older revision		Revision as of 15:06, 12 August 2026
Line 90:		Line 90:
	MYPROJ=/lustre/shared/MyProject		MYPROJ=/lustre/shared/MyProject
	ln -s $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/		ln -s $MYPROJ/project_env/share/jupyter/kernels/myproject $HOME/.local/share/jupyter/kernels/
	</syntaxhighlight>Any users that run this last step should gain access to the kernel from jupyter. Though note, that only users that have write permissions to ~~the kernel file~~ will be able to ~~modify~~		</syntaxhighlight>Any users that run this last step should gain access to the kernel from jupyter. Though note, that only users that have write permissions to virtual environment will be able to install and remove modules from it.

			<div style="background-color: #e7f3fe; border-left: 6px solid #2196F3; padding: 10px; margin-bottom: 15px;">
			'''Note:''' When using custom kernels do not run <code>!pip</code>, just run <code>pip</code> without the <code>!</code>
			</div>

	== See also ==		== See also ==

Honfi001 at 14:56, 12 August 2026

2026-08-12T14:56:26Z

Show changes

Honfi001 at 11:56, 12 August 2026

2026-08-12T11:56:02Z

← Older revision		Revision as of 11:56, 12 August 2026
Line 1:		Line 1:
	= Modules =		= Modules =
			The Python Environment Modules are the only officially supported Python distributions in the HPC. These are compiled for each specific architecture of the HPC and hence are likely going to be more performant than the versions obtained via Mamba or UV.

	Anunna provides one Python version per [[Environment Modules \| module bucket]], plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module:		Anunna provides one Python version per [[Environment Modules \| module bucket]], plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module:

Honfi001: Created page with "= Modules = Anunna provides one Python version per module bucket, plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module: * '''2023''': `Python/3.11.3` * '''2024''': `Python/3.12.3` * '''2024''': `Python/3.13.1` module load 2024 module load Python/3.12.3 The bundle module `Python-bundle-PyPI` adds ma..."

2026-08-12T11:52:28Z

Created page with "= Modules = Anunna provides one Python version per module bucket, plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module: * '''2023''': <code>Python/3.11.3</code> * '''2024''': <code>Python/3.12.3</code> * '''2024''': <code>Python/3.13.1</code> <syntaxhighlight lang="bash"> module load 2024 module load Python/3.12.3 </syntaxhighlight> The bundle module <code>Python-bundle-PyPI</code> adds ma..."

New page

= Modules =

Anunna provides one Python version per [[Environment Modules | module bucket]], plus bundle modules that carry a curated set of common extensions. Load a bucket, then the Python module:

* '''2023''': <code>Python/3.11.3</code>
* '''2024''': <code>Python/3.12.3</code>
* '''2024''': <code>Python/3.13.1</code>

<syntaxhighlight lang="bash">
module load 2024
module load Python/3.12.3
</syntaxhighlight>

The bundle module <code>Python-bundle-PyPI</code> adds many frequently-used packages on top of the base interpreter. Use <code>module key <package></code> to find which bundle contains a package you need (see [[Environment Modules#Searching by keyword | searching modules by keyword]]).

For packages not in a module, the two recommended routes are a '''virtual environment''' built on a Python module (below), or '''Miniforge''' for a self-contained conda/mamba setup. The use of Anaconda is discouraged on Anunna — its default channels carry licensing restrictions and the full distribution is heavy; Miniforge is the lighter, unrestricted alternative.

== Virtual environments ==

A virtual environment is a self-contained directory holding a specific Python and its packages, so one project's dependencies cannot clash with another's. Python's built-in <code>venv</code> module is the simplest way to make one on top of a Python module.

First load the Python version you want:

<syntaxhighlight lang="bash">
module load 2024
module load Python/3.12.3
</syntaxhighlight>

Then create the environment in a location of your choosing. The example uses <code>$myNobackup/PythonEnv</code> — <code>$myNobackup</code> is your Lustre nobackup location, set in your <code>~/.bash_aliases</code> (see [[Installing Personal Software#Aliases and local variables | Aliases and local variables]]). Keeping environments on Lustre rather than your home directory avoids filling your home quota, and the nobackup tier is the right choice because an environment can always be recreated from scratch and so does not need backing up.

<syntaxhighlight lang="bash">
python -m venv $myNobackup/PythonEnv/my_env
</syntaxhighlight>

Activate it whenever you want to use it:

<syntaxhighlight lang="bash">
source $myNobackup/PythonEnv/my_env/bin/activate
</syntaxhighlight>

Once active, the environment name appears as a prefix in your prompt:

<syntaxhighlight lang="text">
(my_env) user001@login200:~$
</syntaxhighlight>

Install packages with <code>pip</code> while the environment is active; they go into the environment, not your home directory:

<syntaxhighlight lang="bash">
pip install -U numpy pandas matplotlib
</syntaxhighlight>

Leave the environment with <code>deactivate</code>.

== Miniforge (conda / mamba) ==

[https://github.com/conda-forge/miniforge Miniforge] is a minimal installer that gives you the <code>conda</code> and <code>mamba</code> package managers preconfigured to use the community [https://conda-forge.org/ conda-forge] channel. <code>mamba</code> is a fast drop-in replacement for <code>conda</code>. This is the recommended way to use conda-style environments on Anunna; it avoids Anaconda's licensing restrictions.

Download and run the installer, pointing it at a location with room (your Lustre nobackup space, not your home directory):

<syntaxhighlight lang="bash">
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh -b -p $myNobackup/miniforge3
</syntaxhighlight>

'''Do not run <code>conda init</code>''' on Anunna — it writes startup code into your <code>~/.bashrc</code> that runs on every login and can interfere with the module system. Instead, activate Miniforge only when you need it:

<syntaxhighlight lang="bash">
source $myNobackup/miniforge3/bin/activate
</syntaxhighlight>

Then create and use environments with <code>mamba</code>:

<syntaxhighlight lang="bash">
mamba create -n myenv python=3.12 numpy pandas
mamba activate myenv
</syntaxhighlight>

== Jupyter kernels ==

To use one of your environments inside [[Jupyter]], register it as a kernel.

=== From a virtual environment ===

A <code>venv</code> kernel needs a wrapper script, because the kernel is launched without your normal shell environment and so cannot load modules by itself. First, with the environment active, install <code>ipykernel</code> and generate the kernel:

<syntaxhighlight lang="bash">
module load 2024
module load Python/3.12.3
source $myNobackup/PythonEnv/my_env/bin/activate
pip install ipykernel
python -m ipykernel install --user --name=my_env_kernel
</syntaxhighlight>

The kernel is written to <code>~/.local/share/jupyter/kernels/</code>, which Jupyter watches. On its own it will not work, because it cannot find the modules — so write a wrapper script that loads them. Save this as, for example, <code>$HOME/wrap.sh</code>:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module reset
module load 2024
module load Python/3.12.3

exec $myNobackup/PythonEnv/my_env/bin/python -m ipykernel_launcher "$@"
</syntaxhighlight>

The <code>#!/bin/bash -l</code> line starts a login shell, which loads Lmod and sources your <code>~/.bash_aliases</code> (so <code>$myNobackup</code> is defined). Make the wrapper executable with <code>chmod +x $HOME/wrap.sh</code>.

Finally point the kernel at the wrapper by editing <code>~/.local/share/jupyter/kernels/my_env_kernel/kernel.json</code>:

<syntaxhighlight lang="json">
{
"argv": [
"/home/WUR/user001/wrap.sh",
"-f",
"{connection_file}"
],
"display_name": "Python my_env",
"language": "python",
"metadata": {
"debugger": true
}
}
</syntaxhighlight>

The only difference from a plain kernel file is that <code>argv</code> points at the wrapper script instead of the Python executable directly.

=== From a conda / mamba environment ===

A conda or mamba environment is simpler, because the environment is self-contained. With Miniforge active and your environment created, install <code>ipykernel</code> into it and register the kernel:

<syntaxhighlight lang="bash">
mamba create -y -n kernel_test python=3 ipykernel
mamba activate kernel_test
python -m ipykernel install --user --name kernel_test
</syntaxhighlight>

To remove the kernel and environment again:

<syntaxhighlight lang="bash">
jupyter kernelspec uninstall kernel_test
mamba deactivate
mamba remove -y -n kernel_test --all
</syntaxhighlight>

== See also ==

* [[Environment Modules]]
* [[Installing Personal Software]]
* [[Jupyter]]
* [[R]]
* [[Apptainer]]

== External links ==

* [https://docs.python.org/3/library/venv.html Python venv documentation]
* [https://github.com/conda-forge/miniforge Miniforge]
* [https://conda-forge.org/ conda-forge]

← Older revision		Revision as of 14:59, 17 August 2026
Line 102:		Line 102:
	* [[Installing Personal Software]]		* [[Installing Personal Software]]
	* [[Jupyter]]		* [[Jupyter]]
			* [[Python]]
	* [[R]]		* [[R]]
	* [[Apptainer]]		* [[Apptainer]]