Uv: Difference between revisions

From HPCwiki
Jump to navigation Jump to search
Created page with "{{DISPLAYTITLE:uv}} uv is a tool for installing Python packages and for managing the environments they live in. It does the same work as <code>pip</code> and <code>venv</code>, and it does it a great deal faster. This page explains what uv is, how to load it on Anunna, where it keeps your files, and how to use it for everyday work. It also explains what uv is '''not''' the right tool for: the packages it installs are not built for our hardware, so when the speed of you..."
 
mNo edit summary
 
Line 1: Line 1:
{{DISPLAYTITLE:uv}}
{{DISPLAYTITLE:uv}}


uv is a tool for installing Python packages and for managing the environments they live in. It does the same work as <code>pip</code> and <code>venv</code>, and it does it a great deal faster.
uv is a tool for installing Python packages and for managing the environments they live in. It does the same work as <code>pip</code> and <code>venv</code>, and it works it a great deal faster.


This page explains what uv is, how to load it on Anunna, where it keeps your files, and how to use it for everyday work. It also explains what uv is '''not''' the right tool for: the packages it installs are not built for our hardware, so when the speed of your calculation matters, the Python modules remain the better choice.
This page explains what uv is, how to load it on Anunna, where it keeps your files, and how to use it for everyday work. It also explains what uv is '''not''' the right tool for: the packages it installs are not built for our hardware, so when the speed of your calculation matters, the Python modules remain the better choice.

Latest revision as of 09:15, 28 August 2026


uv is a tool for installing Python packages and for managing the environments they live in. It does the same work as pip and venv, and it works it a great deal faster.

This page explains what uv is, how to load it on Anunna, where it keeps your files, and how to use it for everyday work. It also explains what uv is not the right tool for: the packages it installs are not built for our hardware, so when the speed of your calculation matters, the Python modules remain the better choice.

Introduction

What uv is

uv is a single program that covers work you would otherwise need several separate tools for. On Anunna it is available as a module, already set up for the cluster.

What uv does

uv can Otherwise you would use
Install Python packages pip
Create and manage environments venv
Download other versions of Python a Python module, if one exists

What it is faster at

Installing packages, and building the environments that hold them. uv produces the same result as pip — the very same packages, from the very same place — in a fraction of the time.

That is worth more here than it would be on your own computer. Installing an environment means writing tens of thousands of small files onto a filesystem that everybody on the cluster shares, so finishing sooner is better for you and better for everyone else.

What it is not faster at

Running your code.

The packages uv installs are general-purpose builds, made to work on any machine in the world. They are exactly what pip would have given you. The software we provide as modules is compiled for the precise processors in each generation of Anunna's nodes, and that is where the speed of a calculation comes from. If your work is heavy, use the modules — see Python/Modules.

The same applies to Python itself. uv is able to download a Python for you, but it is a general-purpose build too. The module is set up to use the Python you have loaded rather than fetching one.

What we have already set up

Left alone, uv keeps its downloads and environments in your home directory, which is small, backed up, and slower than Lustre. The Anunna module points all of it at Lustre instead, and there is nothing you need to do to make that happen. The next two sections cover loading the module and where your files end up.

Loading uv

uv lives in the 2025 bucket:

module load 2025 uv

Loading uv also loads Python 3.13.1, so you do not need to load a Python module first.

Software from different buckets should not be mixed, so anything you use alongside uv should come from the 2025 bucket as well. See Environment Modules#Software buckets.

To check that it worked:

uv --version

A second command, uvx, arrives together with uv. It is described in #Other things uv can do.

Where uv keeps your files

uv accumulates three kinds of thing as you use it: the packages it has downloaded, any Python versions it has fetched, and the tools you have installed with it. On Anunna the module places all of them on Lustre, under a folder called uv in your nobackup location — $myNoBkp/uv. Nobackup is the right tier because everything in there can be downloaded again.

The module does this by setting a handful of variables for you:

Variable What it points at
UV_DIR The folder that holds all of the others. This is the only one you might want to change.
UV_CACHE_DIR Packages uv has downloaded. Normally the largest of them.
UV_PYTHON_INSTALL_DIR Python versions uv has downloaded.
UV_TOOL_DIR Environments for tools installed with uv tool install.
UV_TOOL_BIN_DIR The programs those tools provide. The module has already added this to your PATH.
UV_PYTHON_BIN_DIR Shortcuts to downloaded Python versions.

Putting them somewhere else

Every folder in that table sits inside UV_DIR, so setting that one variable moves all of them. Set it before loading the module:

export UV_DIR=$myScratch/uv
module load 2025 uv

To see where uv is actually working, ask:

echo $UV_DIR

Use that rather than $myNoBkp, which is only set once you have added it to your shell yourself — see Installing Personal Software#Aliases and local variables.

Keeping it from growing

The cache is meant to grow; that is part of what makes uv fast. It still counts against your Lustre quota, so clear it out from time to time:

uv cache prune      # remove entries nothing is using
uv cache clean      # remove everything

Creating an environment and installing packages

Never build an environment on a login node

Building an environment writes thousands of small files at once. A login node is shared by every user on the cluster, so that work is felt by all of them rather than only by you. Ask for an interactive session on a compute node and work there:

sinteractive -c 4 --mem 16G --time 120

See Interactive Jobs for the options sinteractive accepts.

Creating an environment

module load 2025 uv
uv venv $myNoBkp/PythonEnv/my_env

Environments belong on Lustre rather than in your home directory, and nobackup is the right tier because an environment can always be built again from the list of packages in it.

If you run uv venv without giving a path, uv creates a folder named .venv in whatever directory you happen to be in. Giving the path yourself is clearer, and it keeps your environments together in one place.

Using it

Activate the environment whenever you want it:

source $myNoBkp/PythonEnv/my_env/bin/activate

Its name then appears at the front of your prompt:

(my_env) user001@login200:~$

Install packages into it:

uv pip install numpy pandas

Leave it again with deactivate.

What uv builds is an ordinary virtual environment. pip still works inside it if you prefer, and it can be turned into a Jupyter kernel exactly as Python/Modules#Jupyter kernels describes.

The commands you already know

If you know this Use this
python -m venv my_env uv venv my_env
source my_env/bin/activate unchanged
pip install numpy uv pip install numpy
pip list uv pip list

Other things uv can do

Running a tool without installing it

uvx fetches a command-line program, runs it, and does not leave an environment behind:

uvx ruff check .

Installing a tool you use often

uv tool install ruff

The program lands in your uv folder, which the module has already added to your PATH, so you can run it straight away and in any later session.

Using a different version of Python

If you need a version of Python we do not provide as a module, uv can fetch one:

uv python install 3.12
uv venv --python 3.12 $myNoBkp/PythonEnv/py312

Remember that these are general-purpose builds, like the packages. They are convenient, and they are the right choice when you need a specific version for compatibility — but if the speed of your calculation matters, a Python module is the better starting point.

Do not run uv python update-shell. It writes a fixed path into your ~/.bashrc and ~/.bash_profile. That applies in every shell you ever open, whether or not the module is loaded, and it stops being correct the moment your uv folder moves.

Using uv in a job script

Build the environment before you submit the job, not inside it. Create it once in an interactive session, as above, and let the job simply use it. An environment built inside the job is rebuilt on every run, and when many jobs start at the same time they all write to the filesystem at once.

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --time=1:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G

module load 2025 uv
source $myNoBkp/PythonEnv/my_env/bin/activate

python myscript.py

Both the module load and the source have to be inside the script. A job does not inherit the shell you submitted it from, so anything you loaded by hand beforehand is not there when the job runs. This is the most common reason a script that worked interactively fails under sbatch.

See Batch Jobs for what else can go in the #SBATCH lines.

FAQ

module load uv says there is no such module.
The bucket has to be loaded first. Use module load 2025 uv. See Environment Modules#Software buckets.
Where did uv put my files?
echo $UV_DIR gives the folder they are all under. To hear it from uv itself, uv cache dir and uv tool dir.
uv warned that it could not find a Lustre directory.
The module could not work out where your Lustre folders are, so uv has fallen back to your home directory — which will eventually fill up. Choose a location yourself with export UV_DIR=$myNoBkp/uv and then load the module again.
My home directory is still filling up, even though uv uses Lustre.
If you used uv before this module existed, an old cache is probably still sitting in ~/.cache/uv. uv no longer looks there, so its own cleaning commands cannot reach it. Check its size with du -sh ~/.cache/uv and delete it by hand if you no longer need it.
Can I still use pip inside a uv environment?
Yes. It is an ordinary virtual environment and pip behaves normally in it.

See also