Jump to content

OpenFOAM

From HPCwiki

OpenFOAM is a free, open-source toolbox for computational fluid dynamics (CFD). It simulates how fluids flow, mix, heat up and react, and it also covers related problems such as solid stress and electromagnetics. It is not a single program but a large collection of them: solvers that run the simulation, and utilities that prepare a case beforehand (meshing, setting initial conditions, splitting it across processors) and process the results afterwards.

This page covers the practical side of using OpenFOAM on Anunna: which versions are installed, how to load them, and how to set up your cases so they don't create an enormous number of files. That last point matters more for OpenFOAM than for most software, and a few settings make a large difference. How to actually run a case as a Slurm job will be covered in a separate tutorial.

Two families of OpenFOAM

Before anything else, it helps to know that there are two separate OpenFOAM projects sharing the same name and the same history. They split many years ago and have developed independently since:

OpenFOAM Foundation ESI-OpenCFD
Website openfoam.org openfoam.com
Version numbers Whole numbers: 6, 11, 12 … Year and month: v2312, v2406 …
On Anunna 12 v2406

The two are not interchangeable. Solver names, dictionary keywords and the layout of the tutorials differ between them, and they keep drifting further apart. For example, from version 11 onwards the Foundation release runs most simulations through a single foamRun program with a solver module (foamRun -solver incompressibleFluid), where older releases and the ESI line use separate programs such as simpleFoam. A case set up for one family will usually not run on the other without changes, and sometimes not even on a different version of the same family.

The practical rule: find out which family and version your case, tutorial or colleague uses, and load that one. When you start something new, begin from the tutorials that ship with the version you loaded. Once OpenFOAM is loaded (see Loading OpenFOAM) they are in $FOAM_TUTORIALS, so you can copy one into your own scratch space and work from there.

What a case looks like

OpenFOAM keeps everything about a simulation, which it calls a case, in an ordinary directory of plain text files. A freshly set-up case has three folders:

Folder What it holds
0/ The starting state: one file per field (velocity U, pressure p, temperature T …) with its initial and boundary conditions
constant/ Things that do not change during the run: the mesh (in constant/polyMesh/) and physical properties
system/ How to run: time stepping and output in controlDict, numerical schemes and solver settings, and how to split the case across processors

As the simulation runs, OpenFOAM writes a new directory for every time it saves results (0.1/, 0.2/, 0.3/ …), each with its own file for every field. Run the case in parallel and, by default, each processor gets its own processor0/, processor1/ … directory containing its own copy of all of that.

Each file is small, but there are a great many of them, and the number multiplies quickly. A case that started with a dozen files can end with tens of thousands. This is the reason for the Keeping file counts down section further down this page.

Available versions

Software on Anunna is grouped into buckets, and all the OpenFOAM versions live in the 2023 bucket:

Module Family Released When to choose it
OpenFOAM/12-foss-2023a Foundation July 2024 The usual choice for new work on the openfoam.org side
OpenFOAM/v2406-foss-2023a ESI-OpenCFD June 2024 Your cases, tutorials or colleagues use openfoam.com

Both were built with the same toolchain, foss/2023a: GCC 12.3.0 as the compiler and OpenMPI 4.1.5 for running in parallel. Loading OpenFOAM loads these for you. If you want to know more about the MPI side, see OpenMPI.

ParaView, the program most people use to look at OpenFOAM results, comes along automatically: loading either version also loads ParaView/5.11.2-foss-2023a.

This table is accurate at the time of writing. To see what is installed right now, ask the module system directly. module spider searches every bucket, so you do not need to load one first:

module spider OpenFOAM

Need a different version?

The 2023 bucket is in maintenance mode: what is there stays, but no new software is added to it. If you need a version of OpenFOAM that is not listed above, submit a software request at ideas.anunna.wur.nl. A new version would be installed in a newer bucket, not in 2023.

Loading OpenFOAM

Getting OpenFOAM ready takes three lines, always in this order. The examples on this page assume the bash shell, which is the default on Anunna:

module load 2023                     # 1. the bucket
module load OpenFOAM/12-foss-2023a   # 2. the version you want
source $FOAM_BASH                    # 3. OpenFOAM's own environment
  1. The bucket. Nothing in a bucket can be loaded until the bucket itself is.
  2. The module. This puts OpenFOAM's files on the system and loads what it depends on (the compiler, OpenMPI and ParaView). Swap in v2406-foss-2023a for the ESI version.
  3. OpenFOAM's environment. OpenFOAM comes with its own setup script, and the module tells you where it is in the $FOAM_BASH variable. Sourcing it sets up the paths to the solvers and utilities and variables such as $FOAM_TUTORIALS. Do not skip this step. The module alone is not enough, and without it commands like blockMesh are simply not found.

Checking it worked

Two quick commands confirm that you have the version you meant to load and that its programs are found:

echo $WM_PROJECT_VERSION    # prints the version, for example 12
which blockMesh             # prints a path ending in .../bin/blockMesh

If which prints nothing at all, the source $FOAM_BASH line has not been run in this shell.

Where to keep your cases

Once loaded, OpenFOAM suggests a working directory for you, $FOAM_RUN, and offers a run shortcut to go there. Do not use it. It points inside your home directory (~/OpenFOAM/<user>-<version>/run), and your home directory is small, backed up, and not meant for simulation data. See Shared Storage for what each filesystem is for.

Keep your cases on Lustre instead, under $myScratch. If you have not set up that variable yet, Part 1 of HPC Advanced/Preparation Exercise shows you how. For example, to make a folder for your OpenFOAM work and copy in a tutorial case to start from:

mkdir -p $myScratch/openfoam
cp -r $FOAM_TUTORIALS/incompressibleFluid/pitzDaily $myScratch/openfoam/
cd $myScratch/openfoam/pitzDaily

That tutorial path is the one for version 12. In v2406 the same case is at $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily, another small sign of how the two families differ.

Setting up and copying cases on the login node is fine. Running the solvers is not: that belongs in a Slurm job on a compute node, which the upcoming OpenFOAM tutorial will cover. And whichever directory you work in, the advice in Keeping file counts down applies.

Keep it out of your .bashrc

It is tempting to put these three lines in your ~/.bashrc so OpenFOAM is always ready. Please don't. They would then run on every login, slow down every new shell, and interfere when you switch OpenFOAM versions or use other software built with a different toolchain. Run them at the start of the session where you need OpenFOAM, and put them at the top of every job script that uses it.

Keeping file counts down

This is the most important section on this page. OpenFOAM's default settings can turn one simulation into hundreds of thousands of small files, and a few changes to your case keep that number small without losing anything you need.

Why file counts matter

Every file and every directory on Lustre uses one inode, the entry the filesystem keeps to record that the file exists. It doesn't matter how small the file is: a 1 KB file costs as much bookkeeping as a 1 GB one. All of these entries are kept on a shared metadata server, which has a limited number of them and handles every file lookup for every user on Anunna. Millions of tiny files make the whole filesystem slower for everyone, not just for you.

OpenFOAM is especially prone to this because of how it writes results (see What a case looks like). By default, every processor writes its own file for every field at every saved time. A worked example:

Default (uncollated) Collated
Parallel run on 128 processors, 100 saved times, 6 fields 128 × 100 × 6 ≈ 77,000 files 100 × 6 ≈ 600 files

Same simulation, same results, over a hundred times fewer files.

Anunna does not yet limit how many files each user may have on Lustre, but per-user quotas are planned. Getting into these habits now keeps the filesystem fast for everyone, and means nothing breaks when the limits arrive. See Quotas for the current state.

Write results less often

How often OpenFOAM saves results is set in your case's system/controlDict. The tutorials ship with settings meant for learning, not for production runs, so it is worth looking at these lines in every case you run:

writeInterval    0.01;     // how often to save (meaning depends on writeControl)
purgeWrite       0;        // 0 = keep every saved time; N = keep only the last N
writeFormat      ascii;    // ascii or binary
writeCompression off;      // off or on
  • writeInterval controls how often results are saved. Choose it for what you will actually look at afterwards, not "as often as possible, just in case". Saving every 10 time units instead of every 1 means ten times fewer files. What the number means (seconds of simulated time, or number of time steps) depends on the writeControl line just above it.
  • purgeWrite keeps only the most recent saved times and deletes older ones as the run goes on. For a steady-state run, where only the final answer matters, purgeWrite 2; keeps the last two saves, which is enough to restart from. For a transient run where you need the history, leave it at 0 and use function objects (see below) to record the history instead.
  • writeFormat binary; and writeCompression on; make each file smaller and faster to write. To be clear: they reduce the size of your data, not the number of files. They are still worth turning on, but they are not a substitute for the other settings in this section.

Collate parallel output

For every parallel run on Anunna, use the collated file handler. Instead of one processor0/, processor1/ … directory per processor, each with its own set of files, the collated handler gathers everything into a single processors128/ directory (the number is how many processors you used), with one file per field per saved time. This is where the 77,000 versus 600 in the table above comes from.

Both versions on Anunna support it. The simplest way to switch it on is to add this to your case's system/controlDict, so that every OpenFOAM command run on that case uses it automatically:

OptimisationSwitches
{
    fileHandler collated;
}

Alternatively, add -fileHandler collated to each command, for example decomposePar -fileHandler collated. That is easy to forget on one command and end up with a mix, so the controlDict setting is the better habit.

For very large runs, writing everything through a single process can become a bottleneck. You can then let a few processes share the writing, for example one per node, by setting the FOAM_IORANKS variable in your job script to the list of processes that should write:

export FOAM_IORANKS='(0 64)'   # example: processes 0 and 64 each write for their group

The output is then split into a few processors… directories, one per writing group: still a handful of files rather than thousands. For most runs you do not need this.

If you already have a case decomposed the old way, you can convert it to collated with foamFormatConvert, run in parallel inside a job:

mpirun -np ${SLURM_NTASKS} foamFormatConvert -parallel -fileHandler collated

This writes the new processors… directory but does not remove the old processor0/, processor1/ … directories, so for a moment you have both. Once you have checked that the collated copy works (for example with checkMesh -parallel), delete the old ones with rm -r processor[0-9]*. Note that the pattern has a digit after processor, which keeps the new processors… directory safe.

Save only what you need

Often you don't need the whole flow field at every saved time. You need a few numbers: the force on a wall, the pressure at a point, the flow rate through an outlet. OpenFOAM's function objects calculate these while the simulation runs and append them to one small file per quantity, instead of you saving complete fields and working the numbers out afterwards.

Every version ships ready-made function objects, including probes (values at chosen points), forces (forces and moments on surfaces) and surfaceFieldValue (averages, totals or flow rates over a surface). To see the full list for your version:

foamPostProcess -list    # version 12
postProcess -list        # version v2406

The syntax for adding them to a case differs between the two OpenFOAM families, so follow the tutorials and documentation for your version. Combined with a large writeInterval, function objects let you keep the full time history of what you care about while saving complete fields only occasionally.

Clean up when a run is finished

A finished case usually contains a lot it no longer needs.

  • Processor directories. If you reconstruct the results into a single case with reconstructPar (or reconstructPar -latestTime for only the final time), the per-processor data is a complete duplicate. Once you have checked the reconstructed results, delete it (rm -r processor*). With the collated handler there is only one processors… directory to remove.
  • Saved times you don't need. foamListTimes lists the saved times in a case, and with -rm it deletes them. It deletes every time it lists, so always run it without -rm first and narrow it down with -time:
foamListTimes -time '0.1:0.9'        # 1. see which times would be affected
foamListTimes -time '0.1:0.9' -rm    # 2. then delete exactly those
Add -processor to do the same inside an uncollated case's processor* directories.
  • Finished cases you want to keep. Pack the whole case into a single archive file, which takes one inode instead of thousands, check that the archive is complete, and only then remove the directory:
tar -czf pitzDaily.tar.gz pitzDaily
tar -tzf pitzDaily.tar.gz > /dev/null && rm -r pitzDaily

Check your own file count

To count the files and directories in a single case:

find $myScratch/openfoam/pitzDaily | wc -l

To count everything you own in your scratch folder, use Lustre's own lfs find, which is gentler on the metadata server than plain find. It can still take a while if you have a lot of files:

lfs find $myScratch --user $USER | wc -l

If either number runs into the hundreds of thousands, the sections above are the place to start. Once per-user limits are introduced, lfs quota will show your usage directly; see Quotas.

See also

On this wiki:

Need software that is not installed? Submit a request at ideas.anunna.wur.nl.

OpenFOAM's own documentation: