Python/Conda - Revision history

Haars001: added bit about converting the initialisation code to a function

2026-08-28T09:25:48Z

added bit about converting the initialisation code to a function

← Older revision		Revision as of 09:25, 28 August 2026
Line 36:		Line 36:
	\| '''Where packages come from''' \|\| Anaconda's own repository. \|\| conda-forge and bioconda.		\| '''Where packages come from''' \|\| Anaconda's own repository. \|\| conda-forge and bioconda.
	\|-		\|-
	\| '''Cost''' \|\| A paid licence is required for large organisations. \|\| Free and open source.		\| '''Cost''' \|\| A paid licence is required for large organisations like WUR. \|\| Free and open source.
	\|}		\|}

Line 61:		Line 61:
	* '''They change your shell without asking.''' The <code>conda init</code> command writes a block into your <code>~/.bashrc</code>. That block is then read again by job scripts and by interactive sessions, and it can leak into containers, causing failures that look nothing like a conda problem.		* '''They change your shell without asking.''' The <code>conda init</code> command writes a block into your <code>~/.bashrc</code>. That block is then read again by job scripts and by interactive sessions, and it can leak into containers, causing failures that look nothing like a conda problem.

	* '''Anaconda additionally has a licensing problem.''' Its 2024 terms require a paid licence for organisations of 200 or more people, and the free exemption covers teaching rather than research. Other HPC centres have already stopped providing it.		* '''Anaconda additionally has a BIG licensing problem.''' Its 2024 terms require a paid licence for organisations of 200 or more people, and the free exemption covers teaching rather than research. Other HPC centres have already stopped providing it.

	== What to use instead ==		== What to use instead ==
Line 83:		Line 83:
	[[Python/Modules#Virtual environments]] covers this in full. The one thing to remember here is that the environment belongs on Lustre — under <code>$myNobackup</code> — and not in your home directory.		[[Python/Modules#Virtual environments]] covers this in full. The one thing to remember here is that the environment belongs on Lustre — under <code>$myNobackup</code> — and not in your home directory.

	For a faster alternative to <code>pip</code> when installing into such an environment, the <code>uv</code> module is available in the 2025 bucket.		For a faster alternative to <code>pip</code> when installing into such an environment, the <code>uv</code> module is available in the 2025 bucket, see [[Uv\|here]].

	=== A ready-made container ===		=== A ready-made container ===
Line 99:		Line 99:
	=== When none of these work ===		=== When none of these work ===

	Some software is published only as a conda package and has no module, no Python-only equivalent and no container. That is a real situation, and the rest of this page is for it.		Some software is published only as a conda package and has no module, no Python-only equivalent and no container. That is a real situation, and the rest of this page is for this situation.

	== If you still need conda ==		== If you still need conda ==

	Everything below is aimed at avoiding the ~~problems~~ described above. The two that matter most are keeping conda out of your <code>~/.bashrc</code> and never building an environment on a login node.		Everything below is aimed at avoiding the issues described above. The two that matter most are keeping conda out of your <code>~/.bashrc</code> and never building an environment on a login node.

	=== Use Miniforge, not Anaconda ===		=== Use Miniforge, not Anaconda ===
Line 133:		Line 133:

	If you have run <code>conda init</code> in the past, open <code>~/.bashrc</code> and delete everything between the lines <code># >>> conda initialize >>></code> and <code># <<< conda initialize <<<</code>, including those two lines.		If you have run <code>conda init</code> in the past, open <code>~/.bashrc</code> and delete everything between the lines <code># >>> conda initialize >>></code> and <code># <<< conda initialize <<<</code>, including those two lines.

			Another option if you do want to keep that code block available to you is to convert it to a function in <code>~/.bashrc</code>, like this:
			cdi ()
			{
			# >>> conda initialize >>>
			< original conda initialisation code >
			# <<< conda initialize <<<
			}
			Then you can use <code>cdi</code> to load that code when in an interactive shell

	=== Keep the package cache off your home directory ===		=== Keep the package cache off your home directory ===

Honfi001 at 14:33, 17 August 2026

2026-08-17T14:33:16Z

← Older revision		Revision as of 14:33, 17 August 2026
Line 72:		Line 72:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	module key blast		module key blast
	</syntaxhighlight>		</syntaxhighlight>

Honfi001 at 14:32, 17 August 2026

2026-08-17T14:32:37Z

← Older revision		Revision as of 14:32, 17 August 2026
Line 72:		Line 72:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	module key blast		module key blast
	</syntaxhighlight>		</syntaxhighlight>

Honfi001 at 14:25, 17 August 2026

2026-08-17T14:25:48Z

← Older revision		Revision as of 14:25, 17 August 2026
Line 1:		Line 1:
	{{DISPLAYTITLE:Anaconda}}		{{DISPLAYTITLE:Anaconda}}

	Conda is a tool for installing scientific software and keeping it in self-contained folders called environments. You will meet it most often in bioinformatics, data science and machine learning, where a great deal of software is published this way.		Conda is a tool for installing scientific software and keeping it in self-contained folders called environments. You will meet it most often in bioinformatics, data science and machine learning, where a great deal of software is published this way.

Honfi001 at 14:00, 17 August 2026

2026-08-17T14:00:05Z

← Older revision		Revision as of 14:00, 17 August 2026
Line 1:		Line 1:
			{{DISPLAYTITLE:Anaconda}}
	Conda is a tool for installing scientific software and keeping it in self-contained folders called environments. You will meet it most often in bioinformatics, data science and machine learning, where a great deal of software is published this way.		Conda is a tool for installing scientific software and keeping it in self-contained folders called environments. You will meet it most often in bioinformatics, data science and machine learning, where a great deal of software is published this way.

Honfi001 at 13:55, 17 August 2026

2026-08-17T13:55:35Z

← Older revision		Revision as of 13:55, 17 August 2026
Line 126:		Line 126:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	source $~~myNobackup~~/miniforge3/etc/profile.d/conda.sh		source $myNoBkp/miniforge3/etc/profile.d/conda.sh
	conda activate myenv		conda activate myenv
	</syntaxhighlight>		</syntaxhighlight>
Line 137:		Line 137:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	export CONDA_PKGS_DIRS=$~~myNobackup~~/conda/pkgs		export CONDA_PKGS_DIRS=$myNoBkp/conda/pkgs
	export CONDA_ENVS_DIRS=$~~myNobackup~~/conda/envs		export CONDA_ENVS_DIRS=$myNoBkp/conda/envs
	</syntaxhighlight>		</syntaxhighlight>

Line 152:		Line 152:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	source $~~myNobackup~~/miniforge3/etc/profile.d/conda.sh		source $myNoBkp/miniforge3/etc/profile.d/conda.sh
	mamba create -n myenv python=3.12 numpy pandas		mamba create -n myenv python=3.12 numpy pandas
	</syntaxhighlight>		</syntaxhighlight>

Honfi001: Created page with "Conda is a tool for installing scientific software and keeping it in self-contained folders called environments. You will meet it most often in bioinformatics, data science and machine learning, where a great deal of software is published this way. This page explains what Anaconda and Mamba are, why we do not recommend them on Anunna, what to use instead, and — if you have no alternative — how to use conda here with the least trouble. == Introduction == === What i..."

2026-08-17T13:42:12Z

Created page with "Conda is a tool for installing scientific software and keeping it in self-contained folders called environments. You will meet it most often in bioinformatics, data science and machine learning, where a great deal of software is published this way. This page explains what Anaconda and Mamba are, why we do not recommend them on Anunna, what to use instead, and — if you have no alternative — how to use conda here with the least trouble. == Introduction == === What i..."

New page

Conda is a tool for installing scientific software and keeping it in self-contained folders called environments. You will meet it most often in bioinformatics, data science and machine learning, where a great deal of software is published this way.

This page explains what Anaconda and Mamba are, why we do not recommend them on Anunna, what to use instead, and — if you have no alternative — how to use conda here with the least trouble.

== Introduction ==

=== What is Anaconda ===

Anaconda is a Python distribution made by a company, Anaconda Inc. It arrives as one large download containing Python, the <code>conda</code> command and several hundred scientific packages, already installed and ready to use. It is several gigabytes in size.

It was designed for a personal computer: you install it once and nearly everything you might need is already there. That convenience is why it became so popular.

=== What is Mamba ===

Mamba is a free, open-source tool that does the same job as Anaconda's <code>conda</code> command. It is not a bundle — it installs only the packages you ask for, and nothing else.

Mamba comes from the [https://conda-forge.org/ conda-forge] community. You normally get it by installing [https://github.com/conda-forge/miniforge Miniforge], which provides both <code>conda</code> and <code>mamba</code>. It gives you access to conda-forge and to [https://bioconda.github.io/ bioconda], the repository where most bioinformatics software is published.

=== What they have in common ===

Both build '''environments''' — a self-contained folder with its own Python and its own libraries. You can keep several side by side, each holding different versions of the same software, without them interfering with one another.

Both also install much more than Python packages. <code>pip</code> installs Python code and nothing else, whereas conda and mamba can install the things that code depends on as well: R, compilers, C libraries and command-line programs. That is the main reason people choose them.

Underneath, they are the same system. They use the same package format and the same repositories, so an environment built by one can be used by the other, and the commands are identical — <code>create</code>, <code>install</code>, <code>activate</code>. Moving from one to the other is not a migration; nothing needs converting.

=== How they differ ===

{| class="wikitable"
! !! Anaconda !! Mamba
|-
| '''What you get''' || Python and several hundred packages, whether you wanted them or not. Several gigabytes. || Only the packages you ask for.
|-
| '''Where packages come from''' || Anaconda's own repository. || conda-forge and bioconda.
|-
| '''Cost''' || A paid licence is required for large organisations. || Free and open source.
|}

Speed used to be a clear difference too, as mamba worked out dependencies far faster than conda did. Since conda version 23.10 the two share the same machinery for this, so the gap is much smaller than it once was.

On licensing, Anaconda changed its terms in 2024 and they may not permit research use at an institution the size of WUR — see [https://www.anaconda.com/blog/update-on-anacondas-terms-of-service-for-academia-and-research Anaconda's own statement on academia and research].

== Why they are not recommended in the cluster ==

Both tools do run on Anunna. Neither is a good fit for it. The reasons below apply to Anaconda and Mamba alike, except the last one.

* '''Neither is built for our hardware.''' Conda packages — from Anaconda and from conda-forge equally — are general-purpose builds that have to work on any machine in the world. The software we provide as modules is compiled for the exact processors in each generation of Anunna's nodes. If you care about speed and efficiency, use the modules in the buckets: see [[Python/Modules]].

* '''They bring their own copies of software the cluster already provides.''' Ask for one package and you may also receive compilers, maths libraries, CUDA and MPI as dependencies. These sit alongside Anunna's own versions and can be picked up instead of them, which produces results that are hard to explain.

* '''Their MPI is unlikely to use Anunna's fast network.''' A program that spreads work across several nodes needs MPI built against the cluster's high-speed interconnect. The MPI that comes out of a conda environment is a generic build, so a multi-node job may fall back to ordinary networking and run far slower, or fail to start at all. If your work runs on more than one node, use the MPI modules.

* '''They assume system libraries live where they do on a laptop.''' Those assumptions often do not hold here, and the result is a job that fails part-way through with an error that points nowhere useful.

* '''They fill your home directory.''' A single environment is tens of thousands of small files. Your home directory is small, it is backed up, and it sits on a filesystem measured at roughly ninety times slower than Lustre for large reads and writes. See [[Quotas]] and [[Storage Systems Overview]].

* '''Installing them slows the cluster down for everybody.''' Working out and unpacking an environment writes thousands of small files at once. On a login node, which is shared by every user, that is felt by everyone rather than only by you.

* '''They change your shell without asking.''' The <code>conda init</code> command writes a block into your <code>~/.bashrc</code>. That block is then read again by job scripts and by interactive sessions, and it can leak into containers, causing failures that look nothing like a conda problem.

* '''Anaconda additionally has a licensing problem.''' Its 2024 terms require a paid licence for organisations of 200 or more people, and the free exemption covers teaching rather than research. Other HPC centres have already stopped providing it.

== What to use instead ==

In order — try the first before the second, and so on.

=== Check whether it is already a module ===

Most widely used scientific software is already installed on Anunna and compiled for our hardware. Search for it by name before installing anything yourself:

<syntaxhighlight lang="bash">
module key blast
</syntaxhighlight>

<code>module key</code> searches every bucket at once, so you do not need to load one first. See [[Environment Modules]].

=== A virtual environment on a Python module ===

If you only need Python packages, a virtual environment built on one of our Python modules is the right tool. It is smaller, faster and simpler than a conda environment, and it uses the Python we compiled for this machine.

[[Python/Modules#Virtual environments]] covers this in full. The one thing to remember here is that the environment belongs on Lustre — under <code>$myNobackup</code> — and not in your home directory.

For a faster alternative to <code>pip</code> when installing into such an environment, the <code>uv</code> module is available in the 2025 bucket.

=== A ready-made container ===

Almost every package on bioconda is also published as a prebuilt container image, so you can run the software without building an environment at all. This is often the quickest route for bioinformatics software.

<syntaxhighlight lang="bash">
module load utilities Apptainer
apptainer pull https://depot.galaxyproject.org/singularity/multiqc:1.29--pyhdfd78af_0
apptainer run multiqc_1.29--pyhdfd78af_0.sif multiqc --help
</syntaxhighlight>

Browse [https://depot.galaxyproject.org/singularity/ the image list] for the tool and version you want. See [[Apptainer]], and set <code>APPTAINER_CACHEDIR</code> to a location on Lustre first, as that page describes.

=== When none of these work ===

Some software is published only as a conda package and has no module, no Python-only equivalent and no container. That is a real situation, and the rest of this page is for it.

== If you still need conda ==

Everything below is aimed at avoiding the problems described above. The two that matter most are keeping conda out of your <code>~/.bashrc</code> and never building an environment on a login node.

=== Use Miniforge, not Anaconda ===

[https://github.com/conda-forge/miniforge Miniforge] is a small installer from the conda-forge community. It gives you both <code>conda</code> and <code>mamba</code>, it uses conda-forge instead of Anaconda's repository, and it carries no licence conditions.

=== Install it on Lustre, not in your home directory ===

Start an interactive session first — see below — then:

<syntaxhighlight lang="bash">
cd $myNobackup
curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh -b -p $myNobackup/miniforge3
</syntaxhighlight>

<code>-p</code> chooses where it goes. <code>-b</code> installs without asking questions '''and without touching your shell configuration''', which is what you want.

<code>$myNobackup</code> is your Lustre location without backup, which is the right tier here: an environment can always be rebuilt from its list of packages, so it does not need backing up. If you have not set that variable yet, see [[Installing Personal Software#Aliases and local variables]].

=== Do not let it change your shell ===

Never run <code>conda init</code>. Load conda explicitly instead, whenever you need it:

<syntaxhighlight lang="bash">
source $myNobackup/miniforge3/etc/profile.d/conda.sh
conda activate myenv
</syntaxhighlight>

If you have run <code>conda init</code> in the past, open <code>~/.bashrc</code> and delete everything between the lines <code># >>> conda initialize >>></code> and <code># <<< conda initialize <<<</code>, including those two lines.

=== Keep the package cache off your home directory ===

Conda keeps a cache of downloaded packages and a record of your environments. Point both at Lustre by adding these lines to your <code>~/.bash_aliases</code>:

<syntaxhighlight lang="bash">
export CONDA_PKGS_DIRS=$myNobackup/conda/pkgs
export CONDA_ENVS_DIRS=$myNobackup/conda/envs
</syntaxhighlight>

=== Never build an environment on a login node ===

Creating an environment is exactly the kind of work that disturbs the shared filesystem for other users. Ask for an interactive session on a compute node and do it there:

<syntaxhighlight lang="bash">
sinteractive -c 4 --mem 16G --time 120
</syntaxhighlight>

Then create the environment. Listing the packages in a file makes it repeatable:

<syntaxhighlight lang="bash">
source $myNobackup/miniforge3/etc/profile.d/conda.sh
mamba create -n myenv python=3.12 numpy pandas
</syntaxhighlight>

See [[Interactive Jobs]] for the options <code>sinteractive</code> accepts.

=== Using an environment in a job script ===

Load conda and activate the environment inside the script itself. Note the first line: use plain <code>#!/bin/bash</code>, not <code>#!/bin/bash -i</code>, so the script does not read your shell configuration.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --time=1:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G

source $myNobackup/miniforge3/etc/profile.d/conda.sh
conda activate myenv

python myscript.py
</syntaxhighlight>

See [[Batch Jobs]].

=== Cleaning up ===

Environments are large and easy to forget. Check what you have from time to time, remove what you no longer use, and empty the download cache:

<syntaxhighlight lang="bash">
conda env list
conda env remove -n myenv
conda clean --all
</syntaxhighlight>

== See also ==

* [[Python]]
* [[Python/Modules]]
* [[Environment Modules]]
* [[Apptainer]]
* [[Interactive Jobs]]
* [[Batch Jobs]]
* [[Installing Personal Software]]
* [[Quotas]]

== External links ==

* [https://github.com/conda-forge/miniforge Miniforge]
* [https://conda-forge.org/ conda-forge]
* [https://bioconda.github.io/ bioconda]
* [https://depot.galaxyproject.org/singularity/ Prebuilt bioconda container images]
* [https://www.anaconda.com/blog/update-on-anacondas-terms-of-service-for-academia-and-research Anaconda's terms of service for academia and research]