View source for Tutorials/Apptainer-GPUs

= Running Apptainer with GPUs =

Apptainer can pass through GPU hardware from the host into a container, allowing you to run GPU-accelerated workloads (such as deep learning inference or training) inside a fully contained environment. This page covers how to use both NVIDIA and AMD GPUs on the Anunna cluster.

'''Important:''' Before you begin, make sure the following are in place:

* Your <code>.sif</code> image files should be stored on '''Lustre''' (e.g. in your scratch space), not in your home directory.
* Set your Apptainer cache to Lustre, since these images can get quite large:

<syntaxhighlight lang="bash">
export APPTAINER_CACHEDIR=$myScratch/Apptainer
</syntaxhighlight>

== How GPU Passthrough Works ==

Apptainer does not include GPU drivers inside the container. Instead, it '''binds''' the GPU drivers and libraries from the host system into the container at runtime. This means:

* The container must include software built for the correct GPU framework (CUDA for NVIDIA, ROCm for AMD).
* The host must have the matching GPU drivers installed (which Anunna already has on the GPU nodes).
* You tell Apptainer to enable GPU access using a flag: <code>--nv</code> for NVIDIA or <code>--rocm</code> for AMD.

== NVIDIA GPUs (<code>gpu</code> partition) ==

To use NVIDIA GPUs, you need to:

# Request a node on the <code>gpu</code> partition.
# Use the <code>--nv</code> flag when running your container.

The <code>--nv</code> flag tells Apptainer to:
* Make the <code>/dev/nvidiaX</code> device entries available inside the container.
* Locate and bind the CUDA libraries from the host into the container.
* Set <code>LD_LIBRARY_PATH</code> so the container uses the host's GPU libraries.

=== Example: Transcribing Audio with Whisper (NVIDIA) ===

[https://github.com/openai/whisper OpenAI Whisper] is a speech recognition model that benefits greatly from GPU acceleration. Let's build a container that runs Whisper on an NVIDIA GPU.

==== The Definition File ====

Create a file called <code>whisper_nvidia.def</code>:

<syntaxhighlight lang="bash">Bootstrap: docker
From: pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel

%post
    apt-get update
    apt-get install -y ffmpeg git
    pip install openai-whisper
    apt-get clean

%environment
    export LC_ALL=C

%runscript
    exec whisper --device cuda"$@"

%help
    Whisper speech recognition container (NVIDIA GPU).
    Usage: apptainer run --nv whisper_nvidia.sif <audio_file> [options]</syntaxhighlight>

A quick walkthrough of what is happening here:

{| class="wikitable"
|-
! Section !! Purpose
|-
| <code>Bootstrap / From</code> || Uses the official PyTorch Docker image, which already includes Python, PyTorch, CUDA, and cuDNN.
|-
| <code>%post</code> || Installs <code>ffmpeg</code> (required by Whisper for audio decoding), <code>git</code>, and the Whisper package itself.
|-
| <code>%environment</code> || Sets the locale to avoid encoding warnings.
|-
| <code>%runscript</code> || Makes <code>whisper</code> the default command, passing through any arguments.
|-
| <code>%help</code> || Provides usage information (accessible via <code>apptainer run-help</code>).
|}

==== Building the Image ====

Request a compute node and build:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer build whisper_nvidia.sif whisper_nvidia.def
</syntaxhighlight>

==== Running Whisper ====

Once built, run Whisper on the sample audio file. First, request a GPU node:

<syntaxhighlight lang="bash">
srun --partition=gpu --gres=gpu:1 --pty bash
</syntaxhighlight>

Then load Apptainer and run:

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer run --nv whisper_nvidia.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model base
apptainer run --nv whisper_nvidia.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model large

</syntaxhighlight>

The <code>--nv</code> flag is what makes the GPU visible to the container. Without it, PyTorch would not detect any CUDA devices and Whisper would fall back to CPU (much slower).

You can verify GPU access from inside the container with:

<syntaxhighlight lang="bash">
apptainer exec --nv whisper_nvidia.sif python -c "import torch; print(torch.cuda.is_available())"
</syntaxhighlight>

This should print <code>True</code>.

== AMD GPUs (<code>gpu_amd</code> partition) ==

To use AMD GPUs, you need to:

# Request a node on the <code>gpu_amd</code> partition.
# Use the <code>--rocm</code> flag when running your container.

The <code>--rocm</code> flag tells Apptainer to:
* Make the <code>/dev/dri/</code> and <code>/dev/kfd</code> device entries available inside the container.
* Locate and bind the ROCm libraries from the host into the container.
* Set <code>LD_LIBRARY_PATH</code> so the container uses the host's ROCm libraries.

=== Example: Transcribing Audio with Whisper (AMD) ===

The same Whisper workflow, but using an AMD GPU with the ROCm stack.

==== The Definition File ====

Create a file called <code>whisper_amd.def</code>:

<syntaxhighlight lang="bash">
Bootstrap: docker
From: rocm/pytorch:rocm7.2_ubuntu24.04_py3.13_pytorch_release_2.10.0


%post
    apt-get update
    apt-get install -y ffmpeg git
    pip install openai-whisper
    apt-get clean

%environment
    export LC_ALL=C

%runscript
    exec whisper --device cuda "$@"

%help
    Whisper speech recognition container (AMD ROCm GPU).
    Usage: apptainer run --rocm whisper_amd.sif <audio_file> [options]  --model <model_size>
</syntaxhighlight>

The structure is identical to the NVIDIA version. The only difference is the base image: instead of <code>pytorch/pytorch</code> (which includes CUDA), we use <code>rocm/pytorch</code> (which includes ROCm). PyTorch's API is the same regardless of the backend — <code>torch.cuda.is_available()</code> returns <code>True</code> on ROCm as well, since ROCm maps onto the CUDA API.

==== Building the Image ====

<syntaxhighlight lang="bash">
module reset
module load utilities Apptainer
apptainer build whisper_amd.sif whisper_amd.def
</syntaxhighlight>

==== Running Whisper ====

Request an AMD GPU node:

<syntaxhighlight lang="bash">
srun --partition=gpu_amd --gres=gpu:1 --pty bash
</syntaxhighlight>

Then load Apptainer and run:

<syntaxhighlight lang="bash">module reset
module load utilities Apptainer
apptainer run --rocm whisper_amd.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model base  
apptainer run --rocm whisper_amd.sif /lustre/shared/hpcCourses/Whisper/audio.mp3 --model large  </syntaxhighlight>

Note that even though we are on an AMD GPU, the <code>--device cuda</code> flag is correct. PyTorch's ROCm backend uses the same <code>cuda</code> device name for compatibility.

Verify GPU access:

<syntaxhighlight lang="bash">
apptainer exec --rocm whisper_amd.sif python -c "import torch; print(torch.cuda.is_available())"
</syntaxhighlight>

== Summary ==

{| class="wikitable"
|-
! !! NVIDIA !! AMD
|-
| '''Partition''' || <code>gpu</code> || <code>gpu_amd</code>
|-
| '''Apptainer flag''' || <code>--nv</code> || <code>--rocm</code>
|-
| '''Base image''' || <code>pytorch/pytorch</code> (includes CUDA) || <code>rocm/pytorch</code> (includes ROCm)
|-
| '''PyTorch device''' || <code>--device cuda</code> || <code>--device cuda</code> (same API)
|-
| '''Host devices bound''' || <code>/dev/nvidiaX</code> || <code>/dev/dri/</code>, <code>/dev/kfd</code>
|}

The key takeaway: the only things that change between NVIDIA and AMD are the '''base container image''', the '''Apptainer flag''', and the '''Slurm partition'''. Your application code stays the same.