HPCwiki - User contributions [en]

Old binaries

2023-06-19T09:27:57Z

Dawes001:

On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed.

This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications.

However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no longer be automatically available. This will make older installs fail to execute, giving, for instance, errors related to glibc, or missing symbols.

Thankfully, the entire library from previous OS builds is made available on the shared filesystem, granting older binaries the ability to access them whilst also running on a younger OS.

There is a special environment variable: <code>LD_LIBRARY_PATH</code> which tells the linker (ld) where extra to go find libraries. We use this extensively in the underlying module system on Anunna to control which libraries can be used. One such module, <code>sl7_libs</code> automatically adds the older libraries in to older binaries, and has been added as a prerequisite for most older installs.

If you've compiled something yourself that isn't a module, you may find yourself needing this module. Simply:

<code>
module load sl7-libs/main
</code>

To add this path.

== Jupyter ==

This may also extend to custom-built kernels on Jupyter. Here, however, you can't add this module, as instantiating a kernel is done though <code>kernel.json</code> instead of bash.

To add the needed libraries, you need to modify your notebook to look like the following:
<code>
{
"language": "python",
"argv": [
"/path/to/my/venv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "myvenv",
"env" : {"LD_LIBRARY_PATH": "/usr/lib:/usr/lib64:/usr/lib/x86_64-linux-gnu:/shared/legacyapps/sl7-libs/lib:/shared/legacyapps/sl7-libs/lib64" }
}
</code>

When editing JSON, please be mindful that all elements in an object must be separated by commas, and that only doublequotes may be used.

To test your JSON, try the following:

<code>
cat ~/.local.share/jupyter/kernels/mykernel/kernel.json | jq
</code>

If <code>jq</code> repeats your JSON, then it can parse it and you are good to go.

Old binaries

2023-06-19T09:27:26Z

Dawes001:

On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed.

This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications.

However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no longer be automatically available. This will make older installs fail to execute, giving, for instance, errors related to glibc, or missing symbols.

Thankfully, the entire library from previous OS builds is made available on the shared filesystem, granting older binaries the ability to access them whilst also running on a younger OS.

There is a special environment variable: <code>LD_LIBRARY_PATH</code> which tells the linker (ld) where extra to go find libraries. We use this extensively in the underlying module system on Anunna to control which libraries can be used. One such module, <code>sl7_libs</code> automatically adds the older libraries in to older binaries, and has been added as a prerequisite for most older installs.

If you've compiled something yourself that isn't a module, you may find yourself needing this module. Simply:
<code>
module load sl7-libs/main
</code>

To add this path.

== Jupyter ==

This may also extend to custom-built kernels on Jupyter. Here, however, you can't add this module, as instantiating a kernel is done though <code>kernel.json</code> instead of bash.

To add the needed libraries, you need to modify your notebook to look like the following:
<code>
{
"language": "python",
"argv": [
"/path/to/my/venv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "myvenv",
"env" : {"LD_LIBRARY_PATH": "/usr/lib:/usr/lib64:/usr/lib/x86_64-linux-gnu:/shared/legacyapps/sl7-libs/lib:/shared/legacyapps/sl7-libs/lib64" }
}
</code>

When editing JSON, please be mindful that all elements in an object must be separated by commas, and that only doublequotes may be used.

To test your JSON, try the following:

<code>
cat ~/.local.share/jupyter/kernels/mykernel/kernel.json | jq
</code>

If <code>jq</code> repeats your JSON, then it can parse it and you are good to go.

Old binaries

2023-06-19T09:27:09Z

Dawes001: Created page with "On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed. This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications. However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no l..."

On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed.

This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications.

However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no longer be automatically available. This will make older installs fail to execute, giving, for instance, errors related to glibc, or missing symbols.

Thankfully, the entire library from previous OS builds is made available on the shared filesystem, granting older binaries the ability to access them whilst also running on a younger OS.

There is a special environment variable: <code>LD_LIBRARY_PATH</code> which tells the linker (ld) where extra to go find libraries. We use this extensively in the underlying module system on Anunna to control which libraries can be used. One such module, <code>sl7_libs</code> automatically adds the older libraries in to older binaries, and has been added as a prerequisite for most older installs.

If you've compiled something yourself that isn't a module, you may find yourself needing this module. Simply:
<code>
module load sl7-libs/main
</code>

To add this path.

== Jupyter ==

This may also extend to custom-built kernels on Jupyter. Here, however, you can't add this module, as instantiating a kernel is done though <code>kernel.json</code> instead of bash.

To add the needed libraries, you need to modify your notebook to look like the following:
<code>
{
"language": "python",
"argv": [
"/path/to/my/venv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "myvenv",
"env" : {"LD_LIBRARY_PATH": "/usr/lib:/usr/lib64:/usr/lib/x86_64-linux-gnu:/shared/legacyapps/sl7-libs/lib:/shared/legacyapps/sl7-libs/lib64" }
}
</code>

When editing JSON, please be mindful that all elements in an object must be separated by commas, and that only doublequotes may be used.

To test your JSON, try the following:
<code>
cat ~/.local.share/jupyter/kernels/mykernel/kernel.json | jq
</code>

If <code>jq</code> repeats your JSON, then it can parse it and you are good to go.

Main Page

2023-06-19T09:11:41Z

Dawes001: /* Miscellaneous */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]
* [[Old_binaries | Running older binaries on the updated OS]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2023-06-19T09:11:27Z

Dawes001: /* Miscellaneous */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]
* [[Old_binaries | running older binaries on the updated OS]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Storage Systems Overview

2023-06-16T09:04:11Z

Dawes001:

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from nfs01. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of two major parts (and some minor):
* /lustre/backup - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/nobackup - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recivered.
* /lustre/shared - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

And additionally:
* /lustre/scratch - A separated, low resilience filesystem. Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* /archive - an archive mount only accessible from the login nodes. Files here are sent to the Isilon for deeper storage. The cost of storing data here is much less than on the Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via snapshot, and there is a separated backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<pre>
#SBATCH --tmp=<required space>
</pre>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Setting up Python virtualenv

2023-06-16T09:02:11Z

Dawes001:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<pre>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</pre>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<pre>
source newenv/bin/activate
</pre>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<pre>
pip install numpy
</pre>
Similarly, installing packages from source works exactly the same as usual.
<pre>
python setup.py install
</pre>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<pre>
deactivate
</pre>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

(an alternative way to the manual way (using conda) is described [[Using conda to install a new kernel into your notebook|here ]])

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"/home/myhome/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/shared/apps/spark/my-spark-version"
},
"argv": [
"/home/myhome/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<pre>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</pre>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Performance Optimization/Multiple nodes (MPI)

2023-06-16T09:01:59Z

Dawes001:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<pre>
module list
</pre>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

Performance Optimization/Multiple nodes (MPI)

2023-06-16T09:01:42Z

Dawes001:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<source lang='bash'>
module list
</source>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

ABGC/JBrowse

2023-06-16T09:01:30Z

Dawes001:

=== Typical commands used to set up a JBrowse ===

Author: Martijn Derks

* JBrowse is available for multiple species:
** https://jbrowse.hpcagrogenomics.wur.nl/pig/
** https://jbrowse.hpcagrogenomics.wur.nl/chicken/
** https://jbrowse.hpcagrogenomics.wur.nl/cattle/
** https://jbrowse.hpcagrogenomics.wur.nl/turkey/
** https://jbrowse.hpcagrogenomics.wur.nl/Cyprinus_carpio/
* Users are free to add usefull commands to this tutorial

=== Install JBrowse ===

Download the latest JBrowse here: http://jbrowse.org/

Make a directory in <code>/shared/apps/jbrowse/</code> for your species of interested (e.g. <code>mkdir Cyprinus_carpio</code>). Move the downloaded JBrowse source files there. All further procedures detailed in this Wiki page assume working from that directory (NOTE: if your species of interest is already there, contact the maintainer of that JBrowse instance).
Run the setup script to install perl dependencies and required modules

<pre>
unzip JBrowse-1.12.0.zip
mv JBrowse-1.12.0/* $PWD
./setup.sh
</pre>

=== Add reference sequence ===

Example code for chicken genome

<pre>
bin/prepare-refseqs.pl --fasta /lustre/nobackup/WUR/ABGC/shared/public_data_store/genomes/chicken/Ensembl74/Gallus_gallus.Galgal4.74.dna.toplevel.fa
</pre>

To remove tracks use following command:

<pre>
bin/remove-track.pl -D --trackLabel 'trackname'
</pre>

===Add annotation files (GFF/BED)===

Data can be downloaded from the Ensembl FTP site: http://www.ensembl.org/info/data/ftp/index.html

Add gene features:

<pre>
bin/flatfile-to-json.pl --key "Genes" --type gene --config '{ "category": "GalGal4.83 Annotation" }' --trackLabel Genes --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</pre>

Add corresponding transcripts:

<pre>
bin/flatfile-to-json.pl --key "Transcripts" --className transcript --subfeatureClasses '{"exon": "exon", "CDS": "CDS", "five_prime_UTR": "five_prime_UTR", "three_prime_UTR": "three_prime_UTR"}' --config '{ "category": "GalGal4.83 Annotation" }' --type transcript --trackLabel Transcripts --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</pre>

===Alignment tracks (BAM)===

You can load single BAM-files by following command:
<pre>
bin/add-bam-track --label <label> --bam_url <url>
</pre>

To load multiple BAM files present in a certain directory use:
<pre>

for bam in /<dir>*.bam; do
ln -s $bam track_symlinks/ ## Make symlinks from the BAM files
ln -s $bam.bai track_symlinks/ ## Make symlinks to the BAM index files
tissue=`echo $bam | rev | cut -c 5- | cut -d'/' -f1 | rev` ## USe the name of the file without .bam as trackLabel

## Add BAM in alignment mode (Alignments2)
echo '{
"label" : "'${tissue}'_alignment",
"key" : "'${tissue}'_alignment",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json

## Add BAM in coverage mode (SNPCoverage)
echo '{
"label" : "'${tissue}'_coverage",
"key" : "'${tissue}'_coverage",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "SNPCoverage"
}' | bin/add-track-json.pl data/trackList.json

done
</pre>

Make sure the BAM file can be read by a everybody if not use:
<pre>
chmod +r <BAM_file>
</pre>

Make sure that all directoryies in the full path of the BAMfile are executable:
<pre>
chmod +x <dir>
</pre>

===Variant tracks (VCF)===

To load a VCF file in JBrowse make sure the file is gzipped and indexed

<pre>
tabix -p vcf Gallus_gallus_incl_consequences.vcf.gz

echo ' {
"label" : "Gallus_gallus_incl_consequences",
"key" : "Gallus_gallus_incl_consequences",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../ensembl_data/VCF/Gallus_gallus_incl_consequences.vcf.gz",
"category" : "2. Variants",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
</pre>

===Wiggle/BigWig tracks (WIG)===

You can load single BigWig-files by following command:
<pre>
bin/add-bw-track --label <label> --bw_url <url>
</pre>

===Evidence tracks===

Evidence tracks can be loaded in bed, gff and gbk format using

<pre>
bin/flatfile-to-json.pl
</pre>

Examples are given above.

Get my bill

2023-06-16T09:01:06Z

Dawes001:

To estimate costs over a certain time-period the following script can be invoked:

<pre>
module load anunna
get_my_bill
</pre>

The script will, by default, report the cost over the current month, until present. The output will look similar to this:

!!! These results are advisory only !!!
User: user001
Currently run jobs this month: 1038
Total cost so far: 25.19 EUR
For account: 12345
Jobs: 0 Cost: 0.00 EUR
For account: project2
Jobs: 104 Cost: 4.61 EUR
For account: 56789
Jobs: 74 Cost: 6.59 EUR
For account: project 4
Jobs: 19 Cost: 2.09 EUR
For account: project5
Jobs: 80 Cost: 0.04 EUR
For account: project6
Jobs: 738 Cost: 11.86 EUR
For account: project7
Jobs: 1 Cost: 0.01 EUR
For account: project8
Jobs: 22 Cost: 0.00 EUR
Type Time Current Use Current Cost EUR
home 2015-02-13 23:54:53 7.131 GB 0.00
backup 2015-02-06 06:02:43 4.000 kB 0.00
nobackup 2015-02-10 10:45:29 5.348 TB 29.48
scratch 2015-02-08 13:48:40 0.233 TB 0.00
Total this month: 54.67 EUR

The script provides options in addition to the defaults, which can be invoked as follows:
Options:
-h - Show this help message
-g - Show results for your entire group
-d [disk|compute] - get extra detail on disk/compute usages
-t YYYY-MM - Show results for specific month

== See also ==
* [[Main_Page | Main page AgHPC Wiki]]
* [[Main_Page#Controlling_costs | Controlling costs @ AgHPC Wiki]]

Storage Systems Overview

2023-06-16T08:59:44Z

Dawes001:

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from nfs01. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of three major parts (and some minor):
* /lustre/backup - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/nobackup - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recivered.
* /lustre/scratch - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.
* /lustre/shared - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* /archive - an archive mount only accessible from the login nodes. Files here are sent to the Isilon for deeper storage. The cost of storing data here is much less than on the Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via snapshot, and there is a separated backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<source lang='bash'>
#SBATCH --tmp=<required space>
</source>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Environment Modules

2023-06-16T08:59:32Z

Dawes001:

== Preface ==
Environment modules [http://modules.sourceforge.net] are a smart way to provide interchangeable blocks of executables and reproducible environments for use in an HPC. It's also the only way to provide simultaneous versions of the same software without collisions, as each module is housed entirely in its own subfolder structure.

== Using modules ==
The module executable is automatically provided to you upon login. Most users have some modules automatically loaded as well; to see these, use

<nowiki>module list</nowiki>

You should be able to see which modules are loaded.

One of the most important modules to load is 'shared' - this is Anunna specific, as it will extend the MODULEPATH environment variable to use modules present in /shared as well as /cm/local/ . Without this, many modules will not be available to you.

== Loading modules ==
Availability of (different versions of) software can be checked by the following command:
<nowiki>module avail</nowiki>

For example, you should be able to find the basic module slurm. This provides the path to the sbatch, srun, etc. executables for job submission. To load this, simply:
<nowiki>module load slurm</nowiki>

And you will see that it will automatically load the latest version - no need to write out any further.

Many of the hand-installed programs have a path such as:
<nowiki>hdf5/gcc/64/1.8.14</nowiki>
Which translates into:
<nowiki>SOFTWARE/COMPILER/BITS/VERSION</nowiki>

You can elect to load this to various levels:

<nowiki>module load hdf5 # loads the latest version, not caring for compiler
module load hdf5/gcc # loads the latest gcc-compiled version, not caring for 32/64 bits (default 64)
module load hdf5/gcc/64 # loads the latest 64-bit gcc-compiled version
module load hdf5/gcc/64/1.8.14 # loads this specific version of hdf5</nowiki>

This allows your job scripts to either automatically be upgraded when the latest executables are installed, or elect to use only one specific version of a piece of code.

== Switching modules ==

If you want to remove a module, simply
<nowiki>module unload module/1</nowiki>
This will remove the executable path from your environment. It'll also follow the same logic as above, i.e. you can unload all loaded slurm modules independent of version by just unloading the base module name. You can then load up a new one. You can do this in one command, with:
<nowiki>module switch module/1 module/2</nowiki>

Some modules will not allow themselves to be loaded when another one is loaded, for instance, for sanity reasons it's not possible to load two java modules at the same time. Trying to do this will give:

<nowiki>Module 'module/2' conflicts with the currently loaded module(s) 'module/1'</nowiki>
If you're seeing this, you must unload or switch your modules rather than overloading them:
<nowiki>module switch module/2</nowiki>
This works if both modules have the same base path.

== External links ==
* http://modules.sourceforge.net
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Using environment modules

2023-06-16T08:59:02Z

Dawes001:

=== Environment Modules ===
[http://modules.sourceforge.net/ Environment modules] are a simple way to allow multiple potentially clashing programs to coexist on a large shared machine such as an HPC. It allows a user to specify exactly which programs are loaded, and even which version of each program, whilst simultaneously allowing the administrator the ability to automatically configure the appropriate environment variables for the system itself.

== Viewing Modules ==
Upon logging in to Anunna, you should find that when you do:
module list

You will see something like this:
<source lang='bash'>
-bash-4.1$ module list
Currently Loaded Modulefiles:
1) shared 2) slurm/2.5.7
</source>

This is a list of all loaded modules in your shell session. To get a list of all available modules, simply
module available

And this will show you the (very exhaustive) list of modules on Anunna:

<source lang='bash'>
-bash-4.1$ module avail

---------------------------- /shared/modulefiles ----------------------------
acml/gcc/64/5.3.1 netcdf/gcc/64/4.1.3
acml/gcc/fma4/5.3.1 netcdf/gcc/64/4.3.0
acml/gcc/mp/64/5.3.1 netcdf/gcc/64/4.3.2
acml/gcc/mp/fma4/5.3.1 netcdf/gcc/64/4.3.3
acml/gcc-int64/64/5.3.1 netcdf/gcc/64/4.3.3.1
acml/gcc-int64/fma4/5.3.1 netcdf/intel/64/4.1.3
...
</source>

Let's look at each of these module names. Each module is named for the application it provides, plus a subfolder of which compiler it was compiled with (if compiled), the number of address bits or options (if compiled), and the version.

If you want to see a list for a specific module, you can
module avail netcdf

And the complete list of versions will be shown.

== Loading Modules ==
To load a module, simply
module load foo

And the most recent version of module foo will automatically be loaded. If foo is compiled, it will automatically select the gcc version. If you want to specify a certain version, then
module load foo/gcc/64/1.0.0

Will load foo version 1, compiled with gcc. Be advised that this may not always work, as some modules are not compatible with each other, but a message will be shown if this is the case. Additionally, some modules will automatically load other modules with them for them to operate.

== Unloading Modules ==
If you want to remove a module that you've loaded, then
module unload foo

Will remove all module foo's loaded.

== Example ==
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

This script will not run at all in the default 2.4 version of Python on the cluster. In order for this script to run you must use Python3. To do this, first list all versions of Python:
<source lang='bash'>
-bash-4.1$ module avail python

---------------------------- /shared/modulefiles ----------------------------
python/2.7.6 python/3.3.3 python/3.4.2
</source>

Then you can load the specific version you need:
module load python/3.3.3

Now you have access to the executable python3.

== See also ==
* [[Environment_Modules | Environment Modules]]
* [[Control_R_environment_using_modules | Control R environment using modules]]
* [[Create_shortcut_log-in_command | Create a shortcut for the ssh log-in command]]
* [[Installing_R_packages_locally | Installing R packages locally]]

== External links ==
* http://modules.sourceforge.net
* https://modules.readthedocs.io/en/latest/ (documentation)
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Node usage graph

2023-06-16T08:55:36Z

Dawes001:

There is a graphing tool that uses elements directly from sacct to display information about the current cluster usage, node_usage_graph (located at in the anunna module ).

Example:
<pre>
[user@login0 ~]# module load anunna
[user@login0 ~]# usage_graph
node: |0% 100%|
fat001: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
fat002: CCCCCCCCC
MMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node001:

node002:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmm
node003:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MM
node004:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node005:CCCCCCCCCC

node006:CCCCCCCCCC

node007:CCCCCCCCCC

node008:CCCCCCCCCCccccc
MMMMMMMMMMMMMMMMMMMMM
node009:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
node010:

node011:

node012:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node013:

node014:

node015:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node016:CCCCCCCCCCCCCCCCCCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node017:

node018:

node019:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node020:

node021:

node022:

node023:

node024:CCCCCCCCCCCCCCC

node025:CCCCCCCCCCCCCCCCCCCCC

node026:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node027:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node028:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMM
node029:

node030:

node031:

node032:

node033:

node034:

node035:

node036:

node037:

node038:

node039:

node040:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node041:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCcccccc
MMMMMMMMMMMMMMMMMMMMMM
node042:RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
node049:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node050:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node051:

node052:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMMMMMmmmmmmmmmmmmmmm
node053:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node054:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
</pre>

This gives an overview of the current per-node resource usage. There are four types of letter:
* M: Memory reserved and in use
* m: Memory reserved and not in use
* C: CPU reserved and in use
* c: CPU reserved and not in use
* D: Drained node (not available for submission for some adminstrative reason
* R: Reserved node

It cannot however give you an indication of how much the queue is right now for any node. for that, squeue is a better resource.

Node usage graph

2023-06-16T08:55:20Z

Dawes001:

There is a graphing tool that uses elements directly from sacct to display information about the current cluster usage, node_usage_graph (located at /cm/shared/apps/accounting/node_usage_graph ).

Example:
<pre>
[user@login0 ~]# module load anunna
[user@login0 ~]# usage_graph
node: |0% 100%|
fat001: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
fat002: CCCCCCCCC
MMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node001:

node002:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmm
node003:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MM
node004:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node005:CCCCCCCCCC

node006:CCCCCCCCCC

node007:CCCCCCCCCC

node008:CCCCCCCCCCccccc
MMMMMMMMMMMMMMMMMMMMM
node009:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
node010:

node011:

node012:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node013:

node014:

node015:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node016:CCCCCCCCCCCCCCCCCCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node017:

node018:

node019:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node020:

node021:

node022:

node023:

node024:CCCCCCCCCCCCCCC

node025:CCCCCCCCCCCCCCCCCCCCC

node026:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node027:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node028:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMM
node029:

node030:

node031:

node032:

node033:

node034:

node035:

node036:

node037:

node038:

node039:

node040:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node041:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCcccccc
MMMMMMMMMMMMMMMMMMMMMM
node042:RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
node049:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node050:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node051:

node052:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMMMMMmmmmmmmmmmmmmmm
node053:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node054:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
</pre>

This gives an overview of the current per-node resource usage. There are four types of letter:
* M: Memory reserved and in use
* m: Memory reserved and not in use
* C: CPU reserved and in use
* c: CPU reserved and not in use
* D: Drained node (not available for submission for some adminstrative reason
* R: Reserved node

It cannot however give you an indication of how much the queue is right now for any node. for that, squeue is a better resource.

Main Page

2023-01-27T14:40:32Z

Dawes001:

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2022-06-02T13:44:51Z

Dawes001: /* Events */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =
* Upcoming courses on 23rd + 30th June!

* Linux Basic - 23rd June

* HPC Basic - 30th June

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Tariffs

2020-02-17T08:46:03Z

Dawes001:

== Computing: Calculations (cores)==
{| class="wikitable"
!Queue
!CPU core hour
!GB memory hour
|-
|Standard queue
|€ 0.0150
|€ 0.0015
|-
|High priority queue
|€ 0.0200
|€ 0.0020
|-
|Low priority queue
|€ 0.0100
|€ 0.0010
|}

== Computing: GPU Use==
{| class="wikitable"
!Tariff per device per hour (gpu/hour)
|-
|€ 0.3000
|}

== Storage ==
Tariffs per year per TB
{| class="wikitable"
!Lustre Backup
!Lustre Nobackup
!Lustre Scratch
!Home-dir
!Archive
|-
|€ 175
|€ 125
|€ 125
|€ 175
|€ 125
|}

== Reservations ==
{| class="wikitable"
!Tariff per node per day (node/day)
|-
|€ 30
|}

== Notes==

If you are a member of a group with a commitment, then these costs get deducted from that commitment. Typically we are fairly lax with enforcing limits - only once you get to around 150% of your commitment will we consider taking action (mainly coming to discuss things).

== Example ==

You are running a job that needs 4 cores, 32G of RAM and runs for 90 minutes in the std quality. To run this, you over-request resources slightly, and execute in a job that requests 4 CPUs, 40G of RAM and with a time limit of 3 hours. Your job terminates early. Thus, your costs are:

4 * 0.015 * 1.5 = 0.09 EUR for the CPU

40 * 0.0015 * 1.5 = 0.09 EUR for the memory

Total: 0.18 EUR

Setting up Python virtualenv

2019-12-11T15:32:51Z

Dawes001: /* Virtualenv kernels in Jupyter */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</source>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
pip install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"/home/myhome/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/cm/shared/apps/spark/my-spark-version"
},
"argv": [
"/home/myhome/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Setting up Python virtualenv

2019-12-11T15:32:23Z

Dawes001: /* Virtualenv kernels in Jupyter */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</source>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
pip install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"~/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/cm/shared/apps/spark/my-spark-version"
},
"argv": [
"/home/myhome/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Scheduler Overview (Slurm)

2019-10-17T15:19:26Z

Dawes001: /* Batch script */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --cpus-per-task=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --constraint=<gpu flavour e.g. K80, V100>
#SBATCH --partition=gpu
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-02T08:00:14Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --constraint=<gpu flavour e.g. K80, V100>
#SBATCH --partition=gpu
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-01T08:08:46Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --constraint=<gpu flavour e.g. K80, V100>
#SBATCH --partition=GPU
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-01T08:07:53Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --partition=GPU
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-01T08:06:55Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --partition=GPU
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Ssh without password

2019-07-26T16:29:57Z

Dawes001: /* Step 1: create a public key and copy to remote computer */

Secure shell (ssh) protocols can be configure to work without protocols. This is particularly helpful for machines that are used often.

== Configuring ssh without password from a POSIX-compliant terminal ==

=== Step 1: create a public key and copy to remote computer ===
* Log into a local Linux or MacOSX computer
* Type the following to generate the ssh key:
<source lang='bash'>
ssh-keygen
</source>
* Accept the default key location by pressing <code>Enter</code>.
* Secure permission of your authentication keys by closing permission to your home directory, .ssh directory, and authentication files
<source lang='bash'>
chmod go-wx $HOME
chmod 700 $HOME/.ssh
chmod 600 $HOME/.ssh/*
</source>
* Type the following to copy the key to the remote server (this will prompt for a password).
<source lang='bash'>
ssh-copy-id remote_username@remote_host
</source>

== Configuring ssh without password for Anunna ==

* Create a public key as in Step 1 of the previous section and copy it to Anunna. Note that a public/private key pair needs to be made only once per machine.
* Similar to step 2 of the previous section, add the public key to the <code>$HOME/.ssh/authorized_keys2</code> file. There is already a <code>$HOME/.ssh/authorized_keys</code> present. You may append the key to this file as an alternative, but take care not to remove content that is already there. The cluster is configured so that passwordless communication will all other nodes is default.

== Configuring ssh without password using PuTTY ==
Use pAGEaNT: http://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter9.html to generate local keys. You'll want have a copy of the pubkey in plaintext available.

Make sure to paste that plaintext string into ~/.ssh/authorized_keys in one single line. Chmod the file 600 (so it shows -rw------- in ls -l) and the directory .ssh to 700 (drwx------).

Now PuTTY will login passwordlessly whenever pAGEaNT is running.

Finally, get pAGEaNT to load on startup: http://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html

== See also ==
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External Links ==

Ssh without password

2019-07-26T16:29:12Z

Dawes001: /* Step 1: create a public key and copy to remote computer */

Secure shell (ssh) protocols can be configure to work without protocols. This is particularly helpful for machines that are used often.

== Configuring ssh without password from a POSIX-compliant terminal ==

=== Step 1: create a public key and copy to remote computer ===
* Log into a local Linux or MacOSX computer
* Type the following to generate the ssh key:
<source lang='bash'>
ssh-keygen
</source>
* Accept the default key location by pressing <code>Enter</code>.
* Secure permission of your authentication keys by closing permission to your home directory, .ssh directory, and authentication files
<source lang='bash'>
chmod go-w $HOME
chmod 700 $HOME/.ssh
chmod go-rwx $HOME/.ssh/*
</source>
* Type the following to copy the key to the remote server (this will prompt for a password).
<source lang='bash'>
ssh-copy-id remote_username@remote_host
</source>

== Configuring ssh without password for Anunna ==

* Create a public key as in Step 1 of the previous section and copy it to Anunna. Note that a public/private key pair needs to be made only once per machine.
* Similar to step 2 of the previous section, add the public key to the <code>$HOME/.ssh/authorized_keys2</code> file. There is already a <code>$HOME/.ssh/authorized_keys</code> present. You may append the key to this file as an alternative, but take care not to remove content that is already there. The cluster is configured so that passwordless communication will all other nodes is default.

== Configuring ssh without password using PuTTY ==
Use pAGEaNT: http://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter9.html to generate local keys. You'll want have a copy of the pubkey in plaintext available.

Make sure to paste that plaintext string into ~/.ssh/authorized_keys in one single line. Chmod the file 600 (so it shows -rw------- in ls -l) and the directory .ssh to 700 (drwx------).

Now PuTTY will login passwordlessly whenever pAGEaNT is running.

Finally, get pAGEaNT to load on startup: http://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html

== See also ==
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External Links ==

Ssh without password

2019-07-26T16:28:37Z

Dawes001:

Secure shell (ssh) protocols can be configure to work without protocols. This is particularly helpful for machines that are used often.

== Configuring ssh without password from a POSIX-compliant terminal ==

=== Step 1: create a public key and copy to remote computer ===
* Log into a local Linux or MacOSX computer
* Type the following to generate the ssh key:
<source lang='bash'>
ssh-keygen
</source>
* Accept the default key location by pressing <code>Enter</code>.
* Secure permission of your authentication keys by closing permission to your home directory, .ssh directory, and authentication files
<source lang='bash'>
chmod go-w $HOME
chmod 700 $HOME/.ssh
chmod go-rwx $HOME/.ssh/*
</source>
* Type the following to copy the key to the remote server (this will prompt for a password).
<source lang='bash'>
cd ~/.ssh
ssh-copy-id remote_username@remote_host
</source>

== Configuring ssh without password for Anunna ==

* Create a public key as in Step 1 of the previous section and copy it to Anunna. Note that a public/private key pair needs to be made only once per machine.
* Similar to step 2 of the previous section, add the public key to the <code>$HOME/.ssh/authorized_keys2</code> file. There is already a <code>$HOME/.ssh/authorized_keys</code> present. You may append the key to this file as an alternative, but take care not to remove content that is already there. The cluster is configured so that passwordless communication will all other nodes is default.

== Configuring ssh without password using PuTTY ==
Use pAGEaNT: http://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter9.html to generate local keys. You'll want have a copy of the pubkey in plaintext available.

Make sure to paste that plaintext string into ~/.ssh/authorized_keys in one single line. Chmod the file 600 (so it shows -rw------- in ls -l) and the directory .ssh to 700 (drwx------).

Now PuTTY will login passwordlessly whenever pAGEaNT is running.

Finally, get pAGEaNT to load on startup: http://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html

== See also ==
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External Links ==

Creating sbatch script

2019-07-15T15:05:38Z

Dawes001:

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
#SBATCH --qos=
#-----------------------------Required resources-----------------------
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
===partition for resource allocation===
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.

=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --comment=773320000
</source>
Charge resources used by this job to specified account. The comment is an arbitrary string. The comment may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting by feature===
<source lang='bash'>
#SBATCH --constraint=4gpercpu
</source>
The HPC nodes have features associated with them, such as Intel CPU's, or the amount of memory per node. If you know that your job requires a specific architecture or memory size, you can elect to constrain your job to only these features.

The example above will result in jobs being scheduled to the compute nodes with 4GB of memory per CPU. By using <code>12gpercpu</code> as option the job will specifically be scheduled to one of the larger nodes with 12GB per CPU.

All features can be seen using:
<source lang='bash'>
scontrol show nodes | grep ActiveFeatures | sort | uniq
</source>

===requesting specific resources===
<source lang='bash'>
#SBATCH --gres=gpu:1
</source>
In order to be able to use specific hardware resources, you need to request a Generic Resource. Once you do this, one of the resources will be allocated to your job when they are available. In the above example, one GPU is requested for use.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.

===adding a job name===
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.

===receiving mailed updates===
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=yourname001@wur.nl
</source>
Email address to use.

== See also ==
* [[Anunna | Anunna]]
* [[Using_Slurm#Batch_script | Submitting jobs to Slurm]]
* [[Array_jobs|Array job hints]]

Tariffs

2019-07-15T15:04:20Z

Dawes001:

== Computing: Calculations (cores)==
{| class="wikitable"
!Queue
!CPU core hour
!GB memory hour
|-
|Standard queue
|€ 0.0150
|€ 0.0015
|-
|High priority queue
|€ 0.0200
|€ 0.0020
|-
|Low priority queue
|€ 0.0100
|€ 0.0010
|}

== Computing: GPU Use==
{| class="wikitable"
!Tariff per device per hour (gpu/hour)
|-
|€ 0.3000
|}

== Storage ==
Tariffs per year per TB
{| class="wikitable"
!Lustre Nobackup
!Lustre Backup
!Home-dir
!Archive
|-
|€ 150
|€ 200
|€ 200
|€ 100
|}

== Reservations ==
{| class="wikitable"
!Tariff per node per day (node/day)
|-
|€ 30
|}

== Notes==

If you are a member of a group with a commitment, then these costs get deducted from that commitment. Typically we are fairly lax with enforcing limits - only once you get to around 150% of your commitment will we consider taking action (mainly coming to discuss things).

== Example ==

You are running a job that needs 4 cores, 32G of RAM and runs for 90 minutes in the std quality. To run this, you over-request resources slightly, and execute in a job that requests 4 CPUs, 40G of RAM and with a time limit of 3 hours. Your job terminates early. Thus, your costs are:

4 * 0.015 * 1.5 = 0.09 EUR for the CPU

40 * 0.0015 * 1.5 = 0.09 EUR for the memory

Total: 0.18 EUR

Performance Optimization/Multiple nodes (MPI)

2019-07-15T15:02:51Z

Dawes001:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<source lang='bash'>
module list
</source>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /cm/shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

Performance Optimization/Multiple nodes (MPI)

2019-07-15T15:02:32Z

Dawes001:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<source lang='bash'>
module list
</source>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /cm/shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --partition=ABGC --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

Array Jobs

2019-07-15T14:58:59Z

Dawes001:

SLURM can simplify your efforts if you are planning on submitting multiple independent jobs in parallel. Rather than having to use sbatch multiple times, you can instead use an array job to run your job.

Take the following example:

<source lang='bash'>
#SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt
#SBATCH --time=10
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=4000
#SBATCH --array=0-9%4

echo $SLURM_ARRAY_TASK_ID

</source>
Let's break this down step by step:

<source lang='bash'>
#SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt
</source>
This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.

<source lang='bash'>
#SBATCH --array=0-9%4
</source>
This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
<source lang='bash'>
#SBATCH --array=3,7-11
</source>
will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.

<source lang='bash'>
echo $SLURM_ARRAY_TASK_ID
</source>
This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.

So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n.

== Two dimensional arrays? ==
Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then

SLURM_ARRAY_TASK_ID=[ 0 1 2 3 4 5 6 7 8 9 ]

for each job. What if you need two variables to change instead of one?

Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:

<source lang='bash'>
A=$((93 / 10)) # A = 9
B=$((93 % 10)) # B = 3
</source>

As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:

<source lang='bash'>
#SBATCH --array=0-29 ## 5*6 entries, thus 30, including 0 this is 0-29
A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
mywork $A $B
</source>

Creating sbatch script

2019-07-15T14:58:38Z

Dawes001:

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
#SBATCH --qos=
#-----------------------------Required resources-----------------------
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
===partition for resource allocation===
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.

=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --comment=773320000
</source>
Charge resources used by this job to specified account. The comment is an arbitrary string. The comment may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting by feature===
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The HPC nodes have features associated with them, such as Intel CPU's, or the amount of memory per node. If you know that your job requires a specific architecture or memory size, you can elect to constrain your job to only these features.

The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

All features can be seen using:
<source lang='bash'>
scontrol show nodes | grep ActiveFeatures | sort | uniq
</source>

===requesting specific resources===
<source lang='bash'>
#SBATCH --gres=gpu:1
</source>
In order to be able to use specific hardware resources, you need to request a Generic Resource. Once you do this, one of the resources will be allocated to your job when they are available. In the above example, one GPU is requested for use.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.

===adding a job name===
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.

===receiving mailed updates===
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=yourname001@wur.nl
</source>
Email address to use.

== See also ==
* [[Anunna | Anunna]]
* [[Using_Slurm#Batch_script | Submitting jobs to Slurm]]
* [[Array_jobs|Array job hints]]

Scheduler Overview (Slurm)

2019-07-15T14:57:48Z

Dawes001:

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --reservation='GPU'
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

THIS IS PRONE TO CHANGE SHORTLY! [[User:Dawes001|Dawes001]] ([[User talk:Dawes001|talk]]) 14:57, 15 July 2019 (UTC)

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Training Materials

2019-06-27T08:29:50Z

Dawes001: /* HPC CUDA/AI Course - 2019-06-21 */

== Upcoming: HPC Basic Course - 2019-06-28==

A course for beginners will be organised on the 28th of June, aiming to help absolute beginners to begin to use the main job scheduler, SLURM. You can register [https://oneschool.wur.nl/Lists/Cursus/DispForm.aspx?ID=100 here].

== Upcoming Linux Basic Course - 2019-06-27 ==

A course for beginners will be organised on the 13th of June, to help beginner Linux users gain some skills in using Linux. You can register for this course [https://www.wur.nl/en/activity/Linux-basic-course-on-13-June-2019.htm here]

== HPC CUDA/AI Course - 2019-06-21 ==

A course for interested users for deep learning and neural networks, combined with some deep level manipulation of graphics cards was given by Dell on the 21st of July.

[[File:WUR_CUDA_210619.pdf|WUR CUDA Course]]
[[File:WUR_AI_101_210619.pdf|WUR AI Course 101]]
[[File:WUR_AI_201_210619.pdf|WUR AI Course 201]]

[[File:WUR_CUDA_2_210619.pdf|WUR CUDA Course]]
[[File:WUR_Deep_Learning_Frameworks_210619.pdf|WUR Deep Learning Frameworks Primer]]
[[File:WUR_Deep_Learning_Lab_210619.pdf|WUR Deep Learning Lab]]

== HPC Advanced Course - 2019-05-28 ==

A course for experienced users was organised on the 28th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20190506.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - 2019-05-07 ==

A course for beginners will be organised on the 7th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20190506.pdf|Basic Course]]

== Linux Basic Course - 2019-04-16 ==

A course for beginners was organised on the 16th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2018-10-16 ==

A course for experienced users was organised on the 16th of October, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_Slides_20181016.pdf|Advanced Course (Gwen)]]

[[File:HPC_advanced_course_20181008.pdf|Advanced Course (Jeremie)]]

== HPC Basic Course - 2018-10-11 ==

A course for beginners was organised on the 11th of October, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC_basic_course_20181008.pdf|Basic Course]]

== Basic Linux Course - 2018-10-02 ==

A course basic Linux usage was organised on the 2nd of October, to help beginner Linux users gain some skills in using Linux.

[https://etherpad.lug.wur.nl/p/UpkF2KXDVh]

== HPC Advanced Course - 2018-05-18 ==

A course for experienced users was organised on the 18th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_20180518-GD.pdf|Advanced Course (Gwen)]]

== HPC Basic Course - 2018-05-17 ==

A course for beginners was organised on the 17th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Basic Linux Course - 2018-04-19 ==

A course basic Linux usage was organised on the 19th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2017-11-09 ==

A course for experienced users was organised on the 9th of November, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_course_2017-11-08-JV.pdf|Advanced Course (Jeremie)]]

[[File:HPC_Advanced_course_2017-11-08-GD.pdf|Advanced Course (Gwen)]]

[[File:Checkpointing_2017-11-08.pdf|Checkpointing]]

== HPC Basic Course - 2017-10-30 ==

A course for beginners was organised on the 30th of October, aiming to help absolute beginners to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:HPC_basic_course_20171025.pdf | Basic introduction to Linux]]

== HPC Teaching - 2017-06-07 ==

A course for was organised on the 7th of June, aiming to help absolute beginners (and moderately experienced users) to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:Connecting_with_Secure_Shell_to_the_HPC_20170606.pdf | Basic introduction to Linux]]

[[File:Submitting_and_monitoring_jobs_on_the_HPC_20170602.pdf | Submitting and Monitoring Jobs]]

== Old Courses ==
* [http://www.basgen.nl/sdac/ Sequence Data Analysis Course (Dec. 2012)]

File:WUR Deep Learning Lab 210619.pdf

2019-06-27T08:28:39Z

Dawes001:

File:WUR Deep Learning Frameworks 210619.pdf

2019-06-27T08:28:23Z

Dawes001:

File:WUR CUDA 2 210619.pdf

2019-06-27T08:27:57Z

Dawes001:

Training Materials

2019-06-24T16:04:26Z

Dawes001:

== Upcoming: HPC Basic Course - 2019-06-28==

A course for beginners will be organised on the 28th of June, aiming to help absolute beginners to begin to use the main job scheduler, SLURM. You can register [https://oneschool.wur.nl/Lists/Cursus/DispForm.aspx?ID=100 here].

== Upcoming Linux Basic Course - 2019-06-27 ==

A course for beginners will be organised on the 13th of June, to help beginner Linux users gain some skills in using Linux. You can register for this course [https://www.wur.nl/en/activity/Linux-basic-course-on-13-June-2019.htm here]

== HPC CUDA/AI Course - 2019-06-21 ==

A course for interested users for deep learning and neural networks, combined with some deep level manipulation of graphics cards was given by Dell on the 21st of July.

[[File:WUR_CUDA_210619.pdf|WUR CUDA Course]]
[[File:WUR_AI_101_210619.pdf|WUR AI Course 101]]
[[File:WUR_AI_201_210619.pdf|WUR AI Course 201]]

== HPC Advanced Course - 2019-05-28 ==

A course for experienced users was organised on the 28th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20190506.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - 2019-05-07 ==

A course for beginners will be organised on the 7th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20190506.pdf|Basic Course]]

== Linux Basic Course - 2019-04-16 ==

A course for beginners was organised on the 16th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2018-10-16 ==

A course for experienced users was organised on the 16th of October, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_Slides_20181016.pdf|Advanced Course (Gwen)]]

[[File:HPC_advanced_course_20181008.pdf|Advanced Course (Jeremie)]]

== HPC Basic Course - 2018-10-11 ==

A course for beginners was organised on the 11th of October, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC_basic_course_20181008.pdf|Basic Course]]

== Basic Linux Course - 2018-10-02 ==

A course basic Linux usage was organised on the 2nd of October, to help beginner Linux users gain some skills in using Linux.

[https://etherpad.lug.wur.nl/p/UpkF2KXDVh]

== HPC Advanced Course - 2018-05-18 ==

A course for experienced users was organised on the 18th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_20180518-GD.pdf|Advanced Course (Gwen)]]

== HPC Basic Course - 2018-05-17 ==

A course for beginners was organised on the 17th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Basic Linux Course - 2018-04-19 ==

A course basic Linux usage was organised on the 19th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2017-11-09 ==

A course for experienced users was organised on the 9th of November, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_course_2017-11-08-JV.pdf|Advanced Course (Jeremie)]]

[[File:HPC_Advanced_course_2017-11-08-GD.pdf|Advanced Course (Gwen)]]

[[File:Checkpointing_2017-11-08.pdf|Checkpointing]]

== HPC Basic Course - 2017-10-30 ==

A course for beginners was organised on the 30th of October, aiming to help absolute beginners to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:HPC_basic_course_20171025.pdf | Basic introduction to Linux]]

== HPC Teaching - 2017-06-07 ==

A course for was organised on the 7th of June, aiming to help absolute beginners (and moderately experienced users) to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:Connecting_with_Secure_Shell_to_the_HPC_20170606.pdf | Basic introduction to Linux]]

[[File:Submitting_and_monitoring_jobs_on_the_HPC_20170602.pdf | Submitting and Monitoring Jobs]]

== Old Courses ==
* [http://www.basgen.nl/sdac/ Sequence Data Analysis Course (Dec. 2012)]

File:WUR CUDA 210619.pdf

2019-06-24T16:01:56Z

Dawes001:

File:WUR AI 101 210619.pdf

2019-06-24T16:01:22Z

Dawes001:

File:WUR AI 201 210619.pdf

2019-06-24T16:00:59Z

Dawes001:

File:Anunna Flyer 2019.svg

2019-05-06T08:48:54Z

Dawes001:

Tariffs

2019-04-23T08:30:08Z

Dawes001:

== Computing: Calculations (cores)==
{| class="wikitable"
!Queue
!CPU core hour
!GB memory hour
|-
|Standard queue
|€ 0.0150
|€ 0.0015
|-
|High priority queue
|€ 0.0200
|€ 0.0020
|-
|Low priority queue
|€ 0.0100
|€ 0.0010
|}

== Computing: GPU Use==
{| class="wikitable"
!Tariff per device per hour (gpu/hour)
|-
|€ 0.3000
|}

== Storage ==
Tariffs per year per TB
{| class="wikitable"
!Lustre Nobackup
!Lustre Backup
!Home-dir
!Archive
|-
|€ 150
|€ 200
|€ 200
|€ 100
|}

== Reservations ==
{| class="wikitable"
!Tariff per node per day (node/day)
|-
|€ 50
|}

== Notes==

If you are a member of a group with a commitment, then these costs get deducted from that commitment. Typically we are fairly lax with enforcing limits - only once you get to around 150% of your commitment will we consider taking action (mainly coming to discuss things).

== Example ==

You are running a job that needs 4 cores, 32G of RAM and runs for 90 minutes in the Std partition. To run this, you over-request resources slightly, and execute in a job that requests 4 CPUs, 40G of RAM and with a time limit of 3 hours. Your job terminates early. Thus, your costs are:

4 * 0.015 * 1.5 = 0.09 EUR for the CPU

40 * 0.0015 * 1.5 = 0.09 EUR for the memory

Total: 0.18 EUR

Tariffs

2019-04-23T08:25:47Z

Dawes001:

Scheduler Overview (Slurm)

2019-04-04T08:44:14Z

Dawes001:

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Queues ===
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue.<br>
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0)<br>
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are occupied by a Low queue jobs.
To find out which queues your account has been authorized for, type sinfo:
<source lang='bash'>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
ABGC_High up infinite 12 down* node[043-048,055-060]
ABGC_High up infinite 6 mix fat[001-002],node[002-005]
ABGC_High up infinite 44 idle node[001,006-042,049-054]
ABGC_Std up infinite 12 down* node[043-048,055-060]
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]
ABGC_Std up infinite 44 idle node[001,006-042,049-054]
ABGC_Low up infinite 12 down* node[043-048,055-060]
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]
ABGC_Low up infinite 44 idle node[001,006-042,049-054]
</source>

=== Defaults ===
There is no default queue, so you need to specify which queue to use when submitting a job.<br>
'''The default run time for a job is 1 hour!''' <br>
'''Default memory limit is 100MB per node!'''

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=ABGC_Std
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]
< text here >

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Storage Systems Overview

2019-04-04T08:43:24Z

Dawes001:

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from nfs01. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /cm/shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of three major parts (and some minor):
* /lustre/backup - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/nobackup - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recivered.
* /lustre/scratch - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.
* /lustre/shared - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* /archive - an archive mount only accessible from the login nodes. Files here are sent to the Isilon for deeper storage. The cost of storing data here is much less than on the Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via snapshot, and there is a separated backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<source lang='bash'>
#SBATCH --tmp=<required space>
</source>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Main Page

2019-04-04T08:42:41Z

Dawes001:

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]
* [[Tariffs | Costs associated with resource usage]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:lith010 | Jan van Lith (Wageningen UR,FB-IT, Infrastructure)]], [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:vaend001 | Catharina Vaendel(Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}