Tutorials/Apptainer-FakerootAndOverlays

Apptainer Overlays: Modifying Container Images

Important: Before you begin, make sure the following are in place:

• You are running on a compute node, not a login node. Request an interactive session first (e.g. via srun or your scheduler).
• Your .sif image files should be stored on Lustre (e.g. in your scratch space), not in your home directory. SIF files can be large and will eat through your home quota fast.
• Set your Apptainer cache to Lustre as well. Add this to your session (or your .bashrc):

export APPTAINER_CACHEDIR=$myScratch/apptainer_cache

Getting Started

Load the required modules:

module reset
module load utilities Apptainer

Pulling the Base Image

Let's pull an Ubuntu 24.04 image from Docker Hub:

apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04

This will download and convert the image into a SIF file in your current directory.

The Problem: SIF Images Are Read-Only

Now let's try to update the package list inside the container:

apptainer exec ubuntu_24.04.sif apt update

This will fail. You will see errors about not being able to write to package lists and lock files. This is because SIF files are read-only by design — you cannot modify them directly.

So how do we install software into a container? This is where overlays and fakeroot come in.

What is fakeroot?

On an HPC system you do not have root (administrator) privileges. But many operations inside a container — like installing packages with apt — expect to be run as root.

The --fakeroot flag (or -f) tells Apptainer to make it look like you are running as root inside the container, even though you are not actually root on the host system. This is enough to trick package managers like apt into working.

What is an Overlay?

An overlay is a writable layer that sits on top of a read-only SIF image. Think of it like putting a sheet of tracing paper over a printed page — you can write on the tracing paper without changing the original page underneath.

Any changes you make (installed packages, new files, config changes) are stored in the overlay, leaving the original SIF image untouched.

Creating and Using an Overlay

Let's create a temporary directory to use as our overlay:

ov=$(mktemp -d)

This creates a temporary directory somewhere under /tmp and stores the path in the variable $ov.

Now let's try that apt update again, this time with --fakeroot and --overlay:

apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update

This time it should work. Apptainer is running you as fake root, and any writes are going into the overlay directory instead of trying to modify the read-only SIF.

Installing Software via the Overlay

Now let's install neofetch (a tool that displays system information) into the container through the overlay:

apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch

The -y flag tells apt to answer "yes" to all prompts automatically.

Once this completes, neofetch is installed — but only in the overlay. The original ubuntu_24.04.sif is still untouched.

Building a New Image from the Original + Overlay

Right now your changes only exist in that temporary overlay directory. If you were to delete it or reboot, those changes would be gone. To make your modifications permanent, you can build a new SIF image that bakes in the overlay changes:

apptainer build --fakeroot ubuntu_neofetch.sif --overlay $ov ubuntu_24.04.sif

This takes the original ubuntu_24.04.sif, applies everything in the overlay on top of it, and produces a brand new file: ubuntu_neofetch.sif.

Testing the New Image

Let's verify that neofetch is available in our new image:

apptainer exec ubuntu_neofetch.sif neofetch

You should see the familiar Ubuntu logo and system information output. Neofetch is now permanently part of this new image — no overlay or fakeroot needed to run it.

Cleaning Up

Don't forget to remove the temporary overlay directory when you're done:

rm -rf $ov

Summary

Step	Command
Pull base image	apptainer pull ./ubuntu_24.04.sif docker://ubuntu:24.04
Create overlay directory	ov=$(mktemp -d)
Update packages (with overlay + fakeroot)	apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt update
Install software	apptainer exec --fakeroot --overlay $ov ubuntu_24.04.sif apt install -y neofetch
Build new image from original + overlay	apptainer build --fakeroot ubuntu_neofetch.sif --overlay $ov ubuntu_24.04.sif
Test the new image	apptainer exec ubuntu_neofetch.sif neofetch
Clean up overlay	rm -rf $ov