CWL: Difference between revisions
No edit summary |
No edit summary |
||
| Line 11: | Line 11: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
module load 2025 | module load 2025 | ||
module load cwltool/3.3 | module load cwltool/3.3 #cwltool/3.3.20261106131234-foss-2025a | ||
cwltool --version | cwltool --version | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Latest revision as of 16:21, 16 November 2026
CWL (cwltool)
The Common Workflow Language (CWL) is a little different from Snakemake and Nextflow: it is an open standard for describing tools and workflows in YAML files, rather than a single program. Many engines can run the same CWL file. On Anunna you can use cwltool, the reference implementation.
One difference matters more than any other. cwltool does not submit steps to SLURM. It runs every step of the workflow inside the job you start it from, on that one node. So you size a single job for the whole workflow, the way you would for a script, instead of letting the engine submit a job per step.
Set up
cwltool is available from the 2025 bucket:
module load 2025
module load cwltool/3.3 #cwltool/3.3.20261106131234-foss-2025a
cwltool --version
Most published CWL tools run inside containers, so load Apptainer too:
module load utilities
module load Apptainer
Containers: Apptainer, not Docker
CWL tools usually name a Docker image to run in (a DockerRequirement), and cwltool tries to use Docker by default. Docker is not available on Anunna. Add --singularity to every run, and cwltool will pull the same image and run it with Apptainer instead.
If your tools are installed as modules and you do not want containers at all, use --no-container. The tools then have to be on your PATH, so load their modules first.
Keep images out of your home directory
Container images are large single files. Point cwltool and Apptainer at fast, large storage before you run anything, for example in your ~/.bash_aliases:
export CWL_SINGULARITY_CACHE=/path/to/scratch/.cwl-images
export APPTAINER_CACHEDIR=/path/to/scratch/.apptainer-cache
CWL_SINGULARITY_CACHE— where cwltool keeps the images it pulls. If you do not set it, cwltool puts them in a temporary directory and downloads every image again on every run.APPTAINER_CACHEDIR— Apptainer's own download cache. By default it is in$HOME. Never point either variable at$HOME.
Run a workflow
Check that the workflow file is valid before you queue it. This catches typos and broken links between steps in seconds, instead of after a wait in the queue:
cwltool --validate workflow.cwl
Then run it as an ordinary batch job:
#!/bin/bash
#SBATCH --job-name=my-cwl-run
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=04:00:00
module load 2025
module load cwltool/3.3.20261106131234-foss-2025a
module load utilities
module load Apptainer
cwltool --singularity --parallel \
--outdir /path/to/output \
--cachedir /path/to/scratch/cwl-cache \
workflow.cwl inputs.yml
--parallelruns independent steps at the same time. cwltool counts the CPUs your job was given, but it judges memory by what the whole node has free, not by your--mem. If your steps are memory-hungry, ask for enough memory for the steps that run together, or limit how much runs at once with--parallel-max.--cachedirkeeps the results of finished steps. If the run stops, fix the cause and run the same command again: steps that already finished are reused instead of repeated.- The final outputs are copied to
--outdir. cwltool also prints a JSON summary of them at the end.
Going further
This section is an overview, not a full primer. The CWL user guide teaches the language, and the cwltool documentation lists every option. If you need each step submitted to SLURM as its own job, you need a different CWL engine such as Toil, which is not installed yet. Submit a request at ideas.anunna.wur.nl if you need it.
If you use CWL on Anunna and have tips to share, please add them here.