Best-Practices

Anunna is a shared machine. A few hundred people use the same login nodes, the same filesystems and the same scheduler, so most of the habits below exist because breaking one costs someone else time as well as you — and usually you first.

Each entry is a rule and the reason for it. Where the reason needs more than a line, it links to the page that gives it properly.

Environment

Your shell environment is read on every login and by every job you submit. A broken one follows you everywhere, including into jobs that were queued before you broke it.

• Do not edit ~/.bashrc or ~/.bash_profile, and do not let software edit them either. These files gate your login. If one of them breaks, every new session breaks with it and you are locked out of the cluster until someone fixes it for you.
• Read what an installer proposes to do before you let it run. Plenty of tools append to ~/.bashrc as part of their setup, without making much noise about it.
• Keep your own aliases and variables in ~/.bash_aliases rather than in the login files themselves. See Installing Personal Software.
• Wrap anything more involved than an alias in a module of your own. A module is loaded when you want it and unloaded when you do not, which a line in ~/.bashrc is not. See Environment Modules.
• Set up the variables that point at your Lustre folders — $myScratch, $myBkp and $myNoBkp — so that you are not typing long paths by hand or guessing at them in a job script. HPC Advanced/Preparation Exercise Part 1 walks through it using the getLustreDir helper.

Login nodes

The login nodes are a staging area, not a workplace. They are shared by everyone who is logged in, and they have no scheduler protecting them: a single heavy command there is felt by every other user on that node.

• Use a login node for editing files, submitting jobs and checking on them. That is what it is for. See Login Nodes.
• Run the actual work in a job on a compute node, however short you expect it to be. See Batch Jobs.
• Ask for an interactive job when you need a shell to work in — compiling, installing software, debugging, exploring a dataset. You get a real allocation and you are not competing with everyone else's editors. See Interactive Jobs.

Home directory (/home)

Your home directory is small, backed up, and not built for job data. It holds a 200 GB quota.

• Do not store datasets or other large files here. You will hit the quota, and jobs fail in confusing ways when a write is refused partway through.
• Periodically check the size of your ~/.local and ~/.cache folders. Some applications, like rstudio, store gigabytes of data there. You can use environment variables to point them at a different location and avoid going over your quota
• Do not install software here if you intend to run it in jobs. Install it on Lustre instead, where the compute nodes can read it at speed. See Installing Personal Software.
• Do not point job output or error logs here, especially for array jobs: a thousand tasks writing logs into a 200 GB backed-up filesystem is the fastest way to fill it.

Lustre (/lustre)

Lustre is a parallel filesystem. It is very good at moving large amounts of data quickly, and much weaker at metadata operations — anything that asks a question about a file rather than reading its contents, such as opening it, listing a directory or checking whether a path exists. Work with the grain of that and Lustre is fast; work against it and it is slow for you and for everyone else on the cluster.

• Avoid directories holding large numbers of small files. The cost is in the per-file bookkeeping, not in the bytes.
• Do metadata-heavy work on the compute node's local /tmp, or in /dev/shm, and copy the result back to Lustre when it is done. Unpacking an archive, building software and running a package manager all fall into this category. See Setting TMPDIR.
• Know whether the folder you are writing to is backed up, before you rely on it being there. Lustre is split into backed-up and non-backed-up areas, and your variables are named after that split. Backup Policy and Data Lifecycle Policy are the authority on what is kept and for how long; do not infer it from a folder name.

Jobs

• Control where your job's output and error files go, and make sure they do not land in your home directory.
• Be careful with array jobs that write to Lustre. Thousands of tasks hammering the filesystem at once will slow Lustre down, and your own jobs are among the ones that suffer. Keep the number of tasks that can run at once to a couple of hundred — --array=1-10000%200 rather than --array=1-10000. See Array Jobs and Embarassinly Parallel Workflows.
• Test at small scale before you submit at large scale. One task that works tells you far more than ten thousand that fail identically.

Educational courses

• Ask for help with your course as early as you can, and at least a month in advance. That leaves room to customise the setup for what you are teaching and to test it before a room full of students meets it.
• Do not modify your students' ~/.bashrc or ~/.bash_profile. Students may also need to use the HPC for other courses or thesis work. Make any environment changes through a module the students load. See Environment Modules and Hints for courses.

See also

• Installing Personal Software — where to put your own software, and the ~/.bash_aliases recipe
• Environment Modules — loading software, and writing a module of your own
• Batch Jobs and Interactive Jobs — getting work onto a compute node
• Shared Storage and Quotas — what each filesystem is for, and how much of it you have
• HPC Advanced/Preparation Exercise — the Lustre variables and aliases, set up step by step
• Backup Policy and Data Lifecycle Policy — what is kept, and for how long
• Dos and Don'ts and Data storage best practices — related guidance, in more detail on their own subjects