View source for Installing Personal Software

This page covers the personal touches that make Anunna comfortable to work on: shell configuration, aliases, environment variables, a custom temporary directory, and a workaround for running older binaries. None of this is required to use the cluster, but most regular users set up at least some of it.

For installing actual software, the recommended routes are language- or tool-specific:

* '''Python''' — virtual environments and Jupyter kernels: see [[Python]].
* '''R''' — local package libraries: see [[R]].
* '''Containers''' — anything else, packaged reproducibly: see [[Apptainer]].
* '''Cluster-wide modules''' — request a shared install: see [[Environment Modules]].

== Shell configuration files ==

Your shell reads a couple of files when it starts:

* <code>~/.bashrc</code> — read by interactive non-login shells. This is the usual place for aliases and most environment variables.
* <code>~/.bash_profile</code> — read by login shells. Often it just sources <code>~/.bashrc</code> so both kinds of shell behave the same.

Keeping your aliases and variables in a separate <code>~/.bash_aliases</code> file (sourced from <code>~/.bashrc</code>) keeps things tidy. Add this to <code>~/.bashrc</code> so the file is picked up:

<syntaxhighlight lang="bash">
if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi
</syntaxhighlight>

== Aliases and local variables ==

An '''alias''' is a short name for a longer command; a '''local variable''' stores a value (such as a path) you reuse often. Defining a handful of both saves a lot of typing.

The example below is a complete <code>~/.bash_aliases</code>. It defines convenience aliases plus three variables that point at your personal directories on the [[Filesystems | Lustre filesystem]] — <code>$myScratch</code>, <code>$myBackup</code>, and <code>$myNobackup</code>. Other pages (for example [[Apptainer]]) assume these variables exist, so this is the recommended place to set them.

<syntaxhighlight lang="bash">
#!/bin/bash

# Preferred editor
editor='nano'   # or vim

# Edit / source the bash config files
alias ebc="${editor} ${HOME}/.bashrc"
alias sbc="source ${HOME}/.bashrc"
alias eba="${editor} ${HOME}/.bash_aliases"
alias sba="source ${HOME}/.bash_aliases"

# ls shortcuts
alias ll="ls -alth"

# Personal Lustre locations.
# The 'affiliation' depends on your group:
#   WUR researchers in a commitment group:  affiliation=WUR/<group>
#   External users:                         affiliation=<organisation>
# Set this before the exports below — leaving it empty will break the paths.
affiliation=

export myScratch=/lustre/scratch/$affiliation/$USER
export myBackup=/lustre/backup/$affiliation/$USER
export myNobackup=/lustre/nobackup/$affiliation/$USER

# Jump to those locations
alias cds="cd $myScratch"
alias cdb="cd $myBackup"
alias cdn="cd $myNobackup"

# Keep the Apptainer cache off your home quota (see Apptainer page)
export APPTAINER_CACHEDIR=$myScratch/.apptainer-cache
</syntaxhighlight>

After editing the file, either log in again or run <code>source ~/.bash_aliases</code> to apply the changes to your current shell.

== Environment variables ==

Beyond the variables above, two common tweaks go in <code>~/.bashrc</code>.

Add a personal <code>bin</code> directory to your <code>PATH</code> so your own scripts run by name:

<syntaxhighlight lang="bash">
export PATH=$PATH:$HOME/bin
</syntaxhighlight>

Keep more shell history:

<syntaxhighlight lang="bash">
# Lines kept in the history file on disk
export HISTFILESIZE=100000
# Lines kept in memory for the current session
export HISTSIZE=100000
</syntaxhighlight>

== Custom temporary directory (TMPDIR) ==

Many programs write intermediate or temporary files, sometimes without telling you — <code>sort</code>, for instance, needs a lot of scratch space for large inputs. By default they use <code>/tmp</code>. Every compute node has a large local <code>/tmp</code> (around 400 GB), but it is shared, and a job that fills it can cause other users' jobs to fail with anything from a crash to silently wrong output.

To send temporary files somewhere with more room, set the <code>TMPDIR</code> environment variable to a directory in your scratch space. Scratch is a good choice because it is periodically tidied up, so stray temporary files do not linger.

First create the directory:

<syntaxhighlight lang="bash">
mkdir -p $myScratch/tmp
</syntaxhighlight>

Then set <code>TMPDIR</code> in <code>~/.bash_profile</code> or <code>~/.bashrc</code>:

<syntaxhighlight lang="bash">
export TMPDIR=$myScratch/tmp
</syntaxhighlight>

It takes effect on your next login; to apply it to the current shell immediately, <code>source</code> the file you edited.

=== Java applications ===

Some compiled applications — Java programs in particular, and some C/C++ binaries — ignore <code>TMPDIR</code>. For Java, set the temporary directory through <code>_JAVA_OPTIONS</code> instead:

<syntaxhighlight lang="bash">
export _JAVA_OPTIONS=-Djava.io.tmpdir=$myScratch/tmp
</syntaxhighlight>

This is not strictly a temporary-directory variable; it passes a runtime option to the JVM that happens to set the temporary directory.

== Running older binaries ==

Most Linux binaries depend on shared libraries loaded by the operating system at runtime. As the cluster's OS is upgraded, some of those libraries change or disappear, and an older binary may then fail with errors about <code>glibc</code> or missing symbols.

The libraries from previous OS builds are kept on the shared filesystem so that older binaries can still find them. The <code>LD_LIBRARY_PATH</code> environment variable tells the linker where to look for libraries, and a module — <code>sl7-libs</code> — sets it up for you. It is already a prerequisite of most older module installs, so usually you do not need to do anything.

If you have compiled something yourself that is not a module and it fails this way, load the module by hand:

<syntaxhighlight lang="bash">
module load legacy sl7-libs/main
</syntaxhighlight>Be aware that this bad practice, if you really need to keep old software running, you are better off putting it inside a container. See the [[Apptainer]] page

=== Older binaries in a Jupyter kernel ===

A custom [[Jupyter]] kernel cannot load a module, because the kernel is started from a <code>kernel.json</code> file rather than a shell. Instead, set <code>LD_LIBRARY_PATH</code> directly in the kernel's environment block:

<syntaxhighlight lang="json">
{
 "language": "python",
 "argv": [
  "/path/to/my/venv/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "display_name": "myvenv",
 "env": {
  "LD_LIBRARY_PATH": "/usr/lib:/usr/lib64:/usr/lib/x86_64-linux-gnu:/shared/legacyapps/sl7-libs/lib:/shared/legacyapps/sl7-libs/lib64"
 }
}
</syntaxhighlight>

When editing JSON, remember that every entry in an object is separated by a comma and only double quotes are allowed. Check that the file is valid by piping it through <code>jq</code>:

<syntaxhighlight lang="bash">
cat ~/.local/share/jupyter/kernels/mykernel/kernel.json | jq
</syntaxhighlight>

If <code>jq</code> prints your JSON back, it parses correctly.

== See also ==

* [[Environment Modules]]
* [[Locale settings]]
* [[Python]]
* [[R]]
* [[Apptainer]]
* [[Filesystems]]