Jump to content

How to keep the number of files down in your analysis

From HPCwiki

While doing your analysis, you might need a lot of files as input, or your analysis creates a lot of files as output.

Using a lot of small files either as input or output from Lustre will slow down your analysis, as each read or write means a metadata action, which are relatively slow.

Filesystems like big files, leading to less metadata actions.

To see how much files you are using on Lustre, do this:

lfs quota -h -u $USER /lustre

Anything above a million should give you pause.

Some options to handle this are:

Lots of generated files

If your analysis produces a lot of files, try to store them locally on the node where your analysis runs (in /tmp), and as part of your analysis, pack them into a single file, e.g.:

mksquashfs <dir> $myNoBkp/data.sqfs -no-xattrs

Then you can read the files with this:

squashfuse $myNoBkp/data.sqfs $myNoBkp/data_folder/

A zipfile is also an option, but don't compress by default, unless you know your data is very compressable, as this will otherwise slow things down:

zip --compression-method store --recurse-paths $myNoBkp/data.zip data-folder/

Python and R can natively handle zip files.

Don't forget to clean up once the data is safely stored on Lustre

Lots of input files

For this, the flow of generated files can be used, but in reverse. First extract the files locally on the machine, and then do your analysis.

If you are more in control yourself, you can use a file format that is better suited for the input type of your data, like zip/Parquet instead of many CSVs/HFD5/sqlite/batching R targets/Zarr.

Some programs/libraries have options to work with bundled data natively, use them if possible.