Workflow Engines (Snakemake, Nextflow) - Revision history

Honfi001: /* Pitfalls */

2026-07-22T18:55:35Z

Pitfalls

← Older revision		Revision as of 18:55, 22 July 2026
Line 72:		Line 72:

	==== Pitfalls ====		==== Pitfalls ====
	If you run your pipelines with <code>--use-conda</code>, please make sure to point the variable SNAKEMAKE_CONDA_PREFIX to a directory in lustre. For ~~instance~~ make sure this line is in your <code>~/.bash_aliases</code>. <syntaxhighlight lang="bash">		If you run your pipelines with <code>--use-conda</code>, please make sure to point the variable <code>SNAKEMAKE_CONDA_PREFIX</code> to a directory in lustre. For example make sure this line is in your <code>~/.bash_aliases</code>. <syntaxhighlight lang="bash">
	export SNAKEMAKE_CONDA_PREFIX=$myScratch/snakemake/conda		export SNAKEMAKE_CONDA_PREFIX=$myScratch/snakemake/conda
	</syntaxhighlight>		</syntaxhighlight>

Honfi001 at 18:54, 22 July 2026

2026-07-22T18:54:32Z

← Older revision		Revision as of 18:54, 22 July 2026
Line 16:		Line 16:
	</syntaxhighlight>If you need additional python packages installed. Condiser creating a virtual environment.		</syntaxhighlight>If you need additional python packages installed. Condiser creating a virtual environment.

	~~==== Pitfalls ====~~
	If you run your pipelines with <code>--use-conda</code>, please make sure to point the variable SNAKEMAKE_CONDA_PREFIX to a directory in lustre. For instance make sure this line is in your <code>~/.bash_aliases</code>. <syntaxhighlight lang="bash">
	~~export SNAKEMAKE_CONDA_PREFIX=$myScratch/snakemake/conda~~
	~~</syntaxhighlight>~~

	=== SLURM profile ===		=== SLURM profile ===
Line 73:		Line 70:

	The jobs are submitted to SLURM and you can follow the progress in your terminal and with the usual tools — see [[Monitoring Jobs]].		The jobs are submitted to SLURM and you can follow the progress in your terminal and with the usual tools — see [[Monitoring Jobs]].

			==== Pitfalls ====
			If you run your pipelines with <code>--use-conda</code>, please make sure to point the variable SNAKEMAKE_CONDA_PREFIX to a directory in lustre. For instance make sure this line is in your <code>~/.bash_aliases</code>. <syntaxhighlight lang="bash">
			export SNAKEMAKE_CONDA_PREFIX=$myScratch/snakemake/conda
			</syntaxhighlight>

	== Nextflow ==		== Nextflow ==

	Nextflow is another widely used workflow engine, popular in bioinformatics (for example the [https://nf-co.re/ nf-core] pipelines). It describes a pipeline as a set of processes connected by data channels, and it can submit each process to SLURM just like Snakemake does.		Nextflow is another widely used workflow engine, popular in bioinformatics (for example the [https://nf-co.re/ nf-core] pipelines). It describes a pipeline as a set of processes connected by data channels, and it can submit each process to SLURM just like Snakemake does.


	=== Set up ===		=== Set up ===

Honfi001: /* Set up */

2026-07-22T18:53:34Z

Set up

← Older revision		Revision as of 18:53, 22 July 2026
Line 15:		Line 15:
	module load snakemake/9.14.0		module load snakemake/9.14.0
	</syntaxhighlight>If you need additional python packages installed. Condiser creating a virtual environment.		</syntaxhighlight>If you need additional python packages installed. Condiser creating a virtual environment.

			==== Pitfalls ====
			If you run your pipelines with <code>--use-conda</code>, please make sure to point the variable SNAKEMAKE_CONDA_PREFIX to a directory in lustre. For instance make sure this line is in your <code>~/.bash_aliases</code>. <syntaxhighlight lang="bash">
			export SNAKEMAKE_CONDA_PREFIX=$myScratch/snakemake/conda
			</syntaxhighlight>

	=== SLURM profile ===		=== SLURM profile ===

Honfi001: add snakemake module load info.

2026-07-22T16:46:31Z

add snakemake module load info.

← Older revision		Revision as of 16:46, 22 July 2026
Line 9:		Line 9:
	=== Set up ===		=== Set up ===

	Snakemake is ~~usually installed~~ in ~~a conda environment. If you do not have conda/Miniforge yet~~, ~~see [[Python]]. Create an environment containing Snakemake and your pipeline's dependencies:~~		Snakemake is available in through our modules, for instance, in the 2025 bucket

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	~~conda create --name my-pipeline --file requirements.txt~~		module load 2025
	~~conda activate my-pipeline~~		module load snakemake/9.14.0
	~~pip install~~ snakemake~~-executor-plugin-slurm # or add to requirements~~.~~txt~~		</syntaxhighlight>If you need additional python packages installed. Condiser creating a virtual environment.
	</syntaxhighlight>

	~~Giving the~~ environment ~~the same name as the pipeline makes it easy to find later~~.

	=== SLURM profile ===		=== SLURM profile ===
Line 94:		Line 91:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	module load utilities Apptainer		module load utilities
			module load Apptainer
	</syntaxhighlight>		</syntaxhighlight>

	=== Keep Nextflow out of your home directory ===		=== Keep Nextflow out of your home directory ===

	By default Nextflow and Apptainer store their caches in <code>$HOME</code>. On Anunna this will fill your home quota very quickly: nf-core pipelines alone download several gigabytes of reference files and container images. Set these environment variables — for example in your <code>~/.~~bashrc~~</code> — to point them at fast, large storage instead:		By default Nextflow and Apptainer store their caches in <code>$HOME</code>. On Anunna this will fill your home quota very quickly: nf-core pipelines alone download several gigabytes of reference files and container images. Set these environment variables — for example in your <code>~/.bash_aliases</code> — to point them at fast, large storage instead:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 109:		Line 107:
	* <code>NXF_APPTAINER_CACHEDIR</code> — where container images pulled by Nextflow are stored. Images are single large files and add up fast. Never point this at <code>$HOME</code>.		* <code>NXF_APPTAINER_CACHEDIR</code> — where container images pulled by Nextflow are stored. Images are single large files and add up fast. Never point this at <code>$HOME</code>.

	~~Neither variable points~~ to ~~your home directory after this step — log out and back~~ in~~, or <code>source ~/.bashrc</code>, to apply.~~		Make sure both variables point to a location in lustr

	=== SLURM configuration ===		=== SLURM configuration ===

Honfi001: /* Nextflow */

2026-07-22T16:09:45Z

Nextflow

@@ Line 74: / Line 74: @@
 == Nextflow ==
-[https://www.nextflow.io/ Nextflow] is another widely used workflow engine, popular in bioinformatics (for example the nf-core pipelines).
+Nextflow is another widely used workflow engine, popular in bioinformatics (for example the [https://nf-co.re/ nf-core] pipelines). It describes a pipeline as a set of processes connected by data channels, and it can submit each process to SLURM just like Snakemake does.
 <!-- TODO: add a Nextflow section for Anunna — how to load or install Nextflow, the SLURM executor configuration (nextflow.config: process.executor = 'slurm'), and a minimal example. -->

Haars001 at 14:44, 14 July 2026

2026-07-14T14:44:22Z

← Older revision		Revision as of 14:44, 14 July 2026
Line 14:		Line 14:
	conda create --name my-pipeline --file requirements.txt		conda create --name my-pipeline --file requirements.txt
	conda activate my-pipeline		conda activate my-pipeline
			pip install snakemake-executor-plugin-slurm # or add to requirements.txt
	</syntaxhighlight>		</syntaxhighlight>

Line 29:		Line 30:

	<syntaxhighlight lang="yaml">		<syntaxhighlight lang="yaml">
			executor: slurm
	jobs: 10		jobs: 10
	~~cluster~~: "~~sbatch -t 1~~:0:~~0 --mem=~~16000 -c 16 -~~-job-name={~~rule} -~~-output=logs_slurm/{rule}.out --error=logs_slurm/{rule}.err"~~		software-deployment-method: conda
	~~use-conda~~: ~~true~~		default-resources:
			slurm_partition: "main" # your Anunna partition
			slurm_account: "your_account"
			runtime: 60 # minutes
			mem_mb: 16000
			cpus_per_task: 16
			# per-rule overrides:
			# set-resources:
			# big_rule:
			# mem_mb: 64000
			# runtime: 240
	</syntaxhighlight>		</syntaxhighlight>

	Adjust the resources (time, memory, cores) to what your rules need.		Adjust the resources (time, memory, cores) to what your rules need.

	<!-- TODO: confirm the recommended way to run Snakemake on the current cluster. The cluster-command profile shown here is the older (Snakemake <8) style; Snakemake 8+ uses the SLURM executor plugin (--executor slurm). Document whichever is installed/recommended on Anunna. -->

	=== Configure and run ===		=== Configure and run ===

Line 74:		Line 83:
	* [[Monitoring Jobs]]		* [[Monitoring Jobs]]
	* [[Scheduler Overview (Slurm)]]		* [[Scheduler Overview (Slurm)]]
			* [https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html SLURM executor plugin]
			* [https://snakemake.readthedocs.io/en/stable/executing/cli.html Profiles / CLI reference]

Haars001: IA migration §8: rewrite — remove personal/B4F specifics, fix HTML-encoded brackets, point conda to Python, add Nextflow section + executor TODO (via update-page on MediaWiki MCP Server)

2026-06-18T14:14:11Z

IA migration §8: rewrite — remove personal/B4F specifics, fix HTML-encoded brackets, point conda to Python, add Nextflow section + executor TODO (via update-page on MediaWiki MCP Server)

Show changes

Haars001: Haars0011 moved page Running Snakemake pipelines to Workflow Engines (Snakemake, Nextflow): IA migration §8: rename Running Snakemake pipelines → Workflow Engines (Snakemake, Nextflow) (leaving redirect) (via move-page on MediaWiki MCP Server)

2026-06-18T14:13:54Z

Haars0011 moved page Running Snakemake pipelines to Workflow Engines (Snakemake, Nextflow): IA migration §8: rename Running Snakemake pipelines → Workflow Engines (Snakemake, Nextflow) (leaving redirect) (via move-page on MediaWiki MCP Server)

← Older revision	Revision as of 14:13, 18 June 2026
(No difference)

Moiti001: added info on how to install miniconda

2022-01-05T11:25:11Z

added info on how to install miniconda

← Older revision		Revision as of 11:25, 5 January 2022
Line 28:		Line 28:

	Install <code>conda</code> if you don’t have it		Install <code>conda</code> if you don’t have it
			''Update 05/01/2022:''<br />
			Here I show how to install miniconda in a linux system<br />
			[https://docs.conda.io/en/latest/miniconda.html Download installer]<br />
			[https://conda.io/projects/conda/en/latest/user-guide/install/index.html Installation instructions]

			# Download the installer to your home directory. Choose the version according to your operating system. You can right click the link, copy and download with

			<pre>wget <link></pre>
			At the time of writing this update, for me it would be:

			<pre>wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh</pre>
			To install miniconda, run:

			<pre>bash <installer name></pre>
			installer name could be <code>Miniconda3-latest-Linux-x86_64.sh</code>

			Set up the conda channels in this order:

			<pre>conda config --add channels defaults
			conda config --add channels bioconda
			conda config --add channels conda-forge</pre>

	=== Create conda environment ===		=== Create conda environment ===

Moiti001 at 12:57, 8 October 2021

2021-10-08T12:57:02Z

← Older revision		Revision as of 12:57, 8 October 2021
Line 22:		Line 22:
	Go to <code>/lustre/nobackup/WUR/ABGC/shared/PIPELINES/</code> and choose which pipeline you want to use.		Go to <code>/lustre/nobackup/WUR/ABGC/shared/PIPELINES/</code> and choose which pipeline you want to use.

	<pre>cp <pipeline directory> <directory where you want to save it to></pre>		<pre>cp -r <pipeline directory> <directory where you want to save it to></pre>
	First you’ll need to do some set up. Go to the pipeline’s directory.		First you’ll need to do some set up. Go to the pipeline’s directory.