HPCwiki - User contributions [en]

Main Page

2022-06-02T13:44:51Z

Dawes001: /* Events */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =
* Upcoming courses on 23rd + 30th June!

* Linux Basic - 23rd June

* HPC Basic - 30th June

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Spark

2020-10-05T12:23:46Z

Dawes001: /* SPARK in Jupyter */

Apache Spark is a means of distributing compute resources across multiple worker machines. It is the successor to Hadoop, and allows for a wider distribution of code to be executed on the clustered resources. The only requirement for Spark to be able to operate is that each worker must be able to reach each other via TCP, thus it allows for compute to be executed on very simple resources, if the code itself can be translated into the MapReduce paradigm.

== SPARK on HPC ==
In order to create a personal SPARK cluster, you must first request resources on the HPC. Use this example submission script to initialise your cluster:

<nowiki>#!/bin/bash
#SBATCH --time=<length>
#SBATCH --mem-per-cpu=4000
#SBATCH --nodes=<number of nodes>
#SBATCH --tasks-per-node=<number of workers per node>
#SBATCH --job-name="my spark cluster"
#SBATCH --qos=QOS

module load spark/3.0.1-2.7
module load python/3.8.5

source $SPARK_HOME/wur/start-spark

tail -f /dev/null</nowiki>

This will spawn a new cluster of your desired dimensions once resources are available. This spark module has been written to output its logs to your home directory, at:

/home/WUR/yourid/.spark/<jobid>/

In this folder you will find the raw logs of the master and all worker threads. By default the master will consume 1Gb of memory from the first process, and so a single 4Gb 'cluster' will be provided with one 3Gb worker. You can adjust the CPU/memory use by adjusting the parameters in your batch script.

Within the log file you will find two unique files: master, and master-console. master will always contain the URI of the current spark cluster master access point, and master-console the URL of the console of it.

To access the web console, the easiest solution is to use links:

<nowiki>links http://myspark:8081</nowiki>

This will nicely render the page for you in the console. Ctrl-R reloads the page, q to quit.

There are several caveats to remember with this:

* The cluster exists (and consumes resources) until you cancel it with scancel <jobid>
* There is no security at all - any user of the HPC can access both these at any point if they know the port and host.

== Instant SPARK ==

You can also spin up clusters solely to execute scripts. Simply replace the last line from the example above:
<nowiki>tail -f /dev/null</nowiki>
with
<nowiki>spark-submit myscript.py</nowiki>

And after the script has executed, the cluster will automatically terminate.

== SPARK in Jupyter ==

There is a kernel available for using Spark from Jupyter. All this does is to set up the correct path to the python version and the spark binaries for you. In order to set up your Context, your first cell for each notebook should be:
<nowiki>import os, pyspark
with open(os.environ['HOME']+'/.spark/current/master') as f:
conf = (pyspark.SparkConf()
.setMaster(f.read().strip()))
.setAppName("MyName"))

sc = pyspark.SparkContext(conf=conf)</nowiki>

Using the cluster master name from the master file in your job output as above. Subsequent cells will then have sc defined. Run this cell only once - attempting to reconnect will throw an error. That application will run until the kernel is terminated and prevent other applications from being able to be executed - you may wish to manually terminate your kernel from the top bar in Jupyter to free resources.

As a teacher, you might want to put that master file somewhere else, such as /lustre/shared, so that students can connect to your cluster.

Spark

2020-10-05T12:21:30Z

Dawes001: /* SPARK on HPC */

Apache Spark is a means of distributing compute resources across multiple worker machines. It is the successor to Hadoop, and allows for a wider distribution of code to be executed on the clustered resources. The only requirement for Spark to be able to operate is that each worker must be able to reach each other via TCP, thus it allows for compute to be executed on very simple resources, if the code itself can be translated into the MapReduce paradigm.

== SPARK on HPC ==
In order to create a personal SPARK cluster, you must first request resources on the HPC. Use this example submission script to initialise your cluster:

<nowiki>#!/bin/bash
#SBATCH --time=<length>
#SBATCH --mem-per-cpu=4000
#SBATCH --nodes=<number of nodes>
#SBATCH --tasks-per-node=<number of workers per node>
#SBATCH --job-name="my spark cluster"
#SBATCH --qos=QOS

module load spark/3.0.1-2.7
module load python/3.8.5

source $SPARK_HOME/wur/start-spark

tail -f /dev/null</nowiki>

This will spawn a new cluster of your desired dimensions once resources are available. This spark module has been written to output its logs to your home directory, at:

/home/WUR/yourid/.spark/<jobid>/

In this folder you will find the raw logs of the master and all worker threads. By default the master will consume 1Gb of memory from the first process, and so a single 4Gb 'cluster' will be provided with one 3Gb worker. You can adjust the CPU/memory use by adjusting the parameters in your batch script.

Within the log file you will find two unique files: master, and master-console. master will always contain the URI of the current spark cluster master access point, and master-console the URL of the console of it.

To access the web console, the easiest solution is to use links:

<nowiki>links http://myspark:8081</nowiki>

This will nicely render the page for you in the console. Ctrl-R reloads the page, q to quit.

There are several caveats to remember with this:

* The cluster exists (and consumes resources) until you cancel it with scancel <jobid>
* There is no security at all - any user of the HPC can access both these at any point if they know the port and host.

== Instant SPARK ==

You can also spin up clusters solely to execute scripts. Simply replace the last line from the example above:
<nowiki>tail -f /dev/null</nowiki>
with
<nowiki>spark-submit myscript.py</nowiki>

And after the script has executed, the cluster will automatically terminate.

== SPARK in Jupyter ==

There is a kernel available for using Spark from Jupyter. All this does (for now) is to set up the correct path to the python version and the spark binaries for you. In order to set up your Context, your first cell for each notebook should be:
<nowiki>import pyspark
conf = (pyspark.SparkConf()
.setMaster("spark://mysparkcluster:7077")
.setAppName("MyName"))

sc = pyspark.SparkContext(conf=conf)</nowiki>

Using the cluster master name from the master file in your job output as above. Subsequent cells will then have sc defined. Run this cell only once - attempting to reconnect will throw an error. That application will run until the kernel is terminated and prevent other applications from being able to be executed - you may wish to manually terminate your kernel from the top bar in Jupyter to free resources.

Conda for teaching

2020-05-27T09:03:03Z

Dawes001:

You are going to give a teaching course, and you need a specific code environment.

== Setup ==
First - find a good location that everyone can read (and not write). I'd suggest somewhere under <code>/cm/shared/apps/SHARED/</code> as a starting point - this allows everyone to access this location. It's important not to put anything secret there - it's a public resource, so please bear that in mind.

Next - create a folder for your environment:

<source lang='bash'>
mkdir /cm/shared/apps/SHARED/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env
</source>
You may want to manipulate the permissions for this folder if someone is going to set this up with you. Consider the commands in [[Shared folders]].

Then, install Anaconda into it:

<source lang='bash'>
wget https://repo.anaconda.com/archive/Anaconda3-YEAR.MONTH-Linux-x86_64.sh
./Anaconda3-YEAR.MONTH-Linux-x86_64.sh -s -b -p /cm/shared/apps/SHARED/my_conda_env
</source>

Now you have a working conda environment in this folder. You can manipulate this here by running <code>/cm/shared/apps/SHARED/my_conda_env/bin/conda</code>, or, I would recommend creating a modulefile so that you can use it as default.

Create the following example modulefile in a matching <code>/cm/shared/modulefiles/SHARED/my_conda_env</code>:

<source lang='bash'>
#%Module -*- tcl -*-
##
## conda environment modulefile
##

set loadedmodules [split $::env(LOADEDMODULES) ":"]
set modulepath [split $ModulesCurrentModulefile "/"]
set envpath [lrange $modulepath 4 end]

set root /cm/shared/apps/[join $envpath "/"]

proc ModulesHelp { } {
global version

puts stderr "\tThis module provides the conda environment at $envpath"
}

if { [module-info mode] != "whatis" } {
puts stderr "[module-info mode] environent $envpath ."
}

module-whatis "Provides environment $envpath"

prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path LIBRARY_PATH $root/lib
prepend-path CPATH $root/include
prepend-path MANPATH $root/share/man
</source>

This will allow you to <code>module load SHARED/my_conda_env</code> and thus have <code>conda</code> pathed to the currently active environment.

== Jupyter Kernel ==

In order for students to be able to use this environment in jupyter, they will need a kernel definition.

Kernel definitions are usually a separate folder containing, in particular, a file called <code>kernel.json</code>, plus an icon that is displayed that represents this kernel, and other helper code.

The setup for this is that you should create the following folder for their access:

<source lang='bash'>
mkdir -p /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
</source>

This folder is something they will need to copy in place to their home directory, specifically <code>$HOME/.local/share/jupyter/kernels/</code>

Inside this folder, create the following <code>kernel.json</code> file. Watch out that the paths will need to match the current environment path if you're using a different location!

=== Python Kernel ===

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"PATH":

"/cm/shared/apps/SHARED/my_conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
},
"language": "python",
"argv": [
"/cm/shared/apps/SHARED/my_conda_env/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "my_conda_env"
}
</source>

For Python, that's it. Ipykernel is installed automatically on conda initialisation.

=== R Kernel ===

For an R kernel, you need to make sure that the IRkernel package is installed. This is the package that is used to communicate from Jupyter to your running R kernel.
<source lang='bash'>
#MISSING EXAMPLE CODE
#PROBABLY /cm/shared/apps/SHARED/my_conda_env/bin/conda install R_irkernel
</source>

You'll also need to create two files: <code>kernel.json</code> and <code>kernel.js</code>. the <code>kernel.js</code> is a helper script to allow jupyter to communicate to R effectively:

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"LD_LIBRARY_PATH":
"/cm/shared/apps/SHARED/my_conda_env/lib:/cm/shared/apps/SHARED/my_conda_env/lib64"
},
"argv": ["/cm/shared/apps/SHARED/my_conda_env/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
"display_name": "MAE50806-AdvMolEcol/Sandbox_R",
"language": "R"
}
</source>

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.js
</source>
<source lang='bash'>
const cmd_key = /Mac/.test(navigator.platform) ? 'Cmd' : 'Ctrl'

const edit_actions = [
{
name: 'R Assign',
shortcut: 'Alt--',
icon: 'fa-long-arrow-left',
help: 'R: Inserts the left-assign operator (<-)',
handler(cm) {
cm.replaceSelection(' <- ')
},
},
{
name: 'R Pipe',
shortcut: `Shift-${cmd_key}-M`,
icon: 'fa-angle-right',
help: 'R: Inserts the magrittr pipe operator (%>%)',
handler(cm) {
cm.replaceSelection(' %>% ')
},
},
{
name: 'R Help',
shortcut: 'F1',
icon: 'fa-book',
help: 'R: Shows the manpage for the item under the cursor',
handler(cm, cell) {
const {anchor, head} = cm.findWordAt(cm.getCursor())
const word = cm.getRange(anchor, head)

const callbacks = cell.get_callbacks()
const options = {silent: false, store_history: false, stop_on_error: true}
cell.last_msg_id = cell.notebook.kernel.execute(`help(\`${word}\`)`, callbacks, options)
},
},
]

const prefix = 'irkernel'

function add_edit_shortcut(notebook, actions, keyboard_manager, edit_action) {
const {name, shortcut, icon, help, handler} = edit_action

const action = {
icon, help,
help_index : 'zz',
handler: () => {
const cell = notebook.get_selected_cell()
handler(cell.code_mirror, cell)
},
}

const full_name = actions.register(action, name, prefix)

Jupyter.keyboard_manager.edit_shortcuts.add_shortcut(shortcut, full_name)
}

function render_math(pager, html) {
if (!html) return
const $container = pager.pager_element.find('#pager-container')
$container.find('p[style="text-align: center;"]').map((i, e) =>
e.outerHTML = `\\[${e.querySelector('i').innerHTML}\\]`)
$container.find('i').map((i, e) =>
e.outerHTML = `\$${e.innerHTML}\$`)
MathJax.Hub.Queue(['Typeset', MathJax.Hub, $container[0]])
}

define(['base/js/namespace'], ({
notebook,
actions,
keyboard_manager,
pager,
}) => ({
onload() {
edit_actions.forEach(a => add_edit_shortcut(notebook, actions, keyboard_manager, a))

pager.events.on('open_with_text.Pager', (event, {data: {'text/html': html}}) =>
render_math(pager, html))
},
}))
</source>

== Last Steps ==

In order to help your students get their kernel definitions into <code>$HOME/.local/share/jupyter/kernels/</code>, it's probably a good idea to write a small and simple notebook to do this when executed, or else instruct them to do:

<source lang='bash'>
cp -rv /cm/shared/apps/SHARED/my_conda_env/kernel/* .local/share/jupyter/kernels/
</source>

Conda for teaching

2020-05-27T09:01:02Z

Dawes001:

You are going to give a teaching course, and you need a specific code environment.

== Setup ==
First - find a good location that everyone can read (and not write). I'd suggest somewhere under /cm/shared/apps/SHARED/ as a starting point - this allows everyone to access this location. It's important not to put anything secret there - it's a public resource, so please bear that in mind.

Next - create a folder for your environment:

<source lang='bash'>
mkdir /cm/shared/apps/SHARED/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env
</source>
You may want to manipulate the permissions for this folder if someone is going to set this up with you. Consider the commands in [[Shared Folders]].

Then, install Anaconda into it:

<source lang='bash'>
wget https://repo.anaconda.com/archive/Anaconda3-YEAR.MONTH-Linux-x86_64.sh
./Anaconda3-YEAR.MONTH-Linux-x86_64.sh -s -b -p /cm/shared/apps/SHARED/my_conda_env
</source>

Now you have a working conda environment in this folder. You can manipulate this here by running /cm/shared/apps/SHARED/my_conda_env/bin/conda , or, I would recommend creating a modulefile so that you can use it as default.

Create the following example modulefile in a matching /cm/shared/modulefiles/SHARED/my_conda_env :

<source lang='bash'>
#%Module -*- tcl -*-
##
## conda environment modulefile
##

set loadedmodules [split $::env(LOADEDMODULES) ":"]
set modulepath [split $ModulesCurrentModulefile "/"]
set envpath [lrange $modulepath 4 end]

set root /cm/shared/apps/[join $envpath "/"]

proc ModulesHelp { } {
global version

puts stderr "\tThis module provides the conda environment at $envpath"
}

if { [module-info mode] != "whatis" } {
puts stderr "[module-info mode] environent $envpath ."
}

module-whatis "Provides environment $envpath"

prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path LIBRARY_PATH $root/lib
prepend-path CPATH $root/include
prepend-path MANPATH $root/share/man
</source>

This will allow you to <code>module load SHARED/my_conda_env</code> and thus have <code>conda</code> pathed to the currently active environment.

== Jupyter Kernel ==

In order for students to be able to use this environment in jupyter, they will need a kernel definition.

Kernel definitions are usually a separate folder containing, in particular, a file called <code>kernel.json</code>, plus an icon that is displayed that represents this kernel, and other helper code.

The setup for this is that you should create the following folder for their access:

<source lang='bash'>
mkdir -p /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
</source>

This folder is something they will need to copy in place to their home directory, specifically $HOME/.local/share/jupyter/kernels/

Inside this folder, create the following kernel.json file. Watch out that the paths will need to match the current environment path if you're using a different location!

=== Python Kernel ===

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"PATH":

"/cm/shared/apps/SHARED/my_conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
},
"language": "python",
"argv": [
"/cm/shared/apps/SHARED/my_conda_env/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "my_conda_env"
}
</source>

For Python, that's it. Ipykernel is installed automatically on conda initialisation.

=== R Kernel ===

For an R kernel, you need to make sure that the IRkernel package is installed. This is the package that is used to communicate from Jupyter to your running R kernel.<source lang='bash'>
#MISSING EXAMPLE CODE
#PROBABLY /cm/shared/apps/SHARED/my_conda_env/bin/conda install R_irkernel
</source>

You'll also need to create two files. the kernel.js is a helper script to allow jupyter to communicate to R effectively:

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"LD_LIBRARY_PATH":
"/cm/shared/apps/SHARED/my_conda_env/lib:/cm/shared/apps/SHARED/my_conda_env/lib64"
},
"argv": ["/cm/shared/apps/SHARED/my_conda_env/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
"display_name": "MAE50806-AdvMolEcol/Sandbox_R",
"language": "R"
}
</source>

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.js
</source>
<source lang='bash'>
const cmd_key = /Mac/.test(navigator.platform) ? 'Cmd' : 'Ctrl'

const edit_actions = [
{
name: 'R Assign',
shortcut: 'Alt--',
icon: 'fa-long-arrow-left',
help: 'R: Inserts the left-assign operator (<-)',
handler(cm) {
cm.replaceSelection(' <- ')
},
},
{
name: 'R Pipe',
shortcut: `Shift-${cmd_key}-M`,
icon: 'fa-angle-right',
help: 'R: Inserts the magrittr pipe operator (%>%)',
handler(cm) {
cm.replaceSelection(' %>% ')
},
},
{
name: 'R Help',
shortcut: 'F1',
icon: 'fa-book',
help: 'R: Shows the manpage for the item under the cursor',
handler(cm, cell) {
const {anchor, head} = cm.findWordAt(cm.getCursor())
const word = cm.getRange(anchor, head)

const callbacks = cell.get_callbacks()
const options = {silent: false, store_history: false, stop_on_error: true}
cell.last_msg_id = cell.notebook.kernel.execute(`help(\`${word}\`)`, callbacks, options)
},
},
]

const prefix = 'irkernel'

function add_edit_shortcut(notebook, actions, keyboard_manager, edit_action) {
const {name, shortcut, icon, help, handler} = edit_action

const action = {
icon, help,
help_index : 'zz',
handler: () => {
const cell = notebook.get_selected_cell()
handler(cell.code_mirror, cell)
},
}

const full_name = actions.register(action, name, prefix)

Jupyter.keyboard_manager.edit_shortcuts.add_shortcut(shortcut, full_name)
}

function render_math(pager, html) {
if (!html) return
const $container = pager.pager_element.find('#pager-container')
$container.find('p[style="text-align: center;"]').map((i, e) =>
e.outerHTML = `\\[${e.querySelector('i').innerHTML}\\]`)
$container.find('i').map((i, e) =>
e.outerHTML = `\$${e.innerHTML}\$`)
MathJax.Hub.Queue(['Typeset', MathJax.Hub, $container[0]])
}

define(['base/js/namespace'], ({
notebook,
actions,
keyboard_manager,
pager,
}) => ({
onload() {
edit_actions.forEach(a => add_edit_shortcut(notebook, actions, keyboard_manager, a))

pager.events.on('open_with_text.Pager', (event, {data: {'text/html': html}}) =>
render_math(pager, html))
},
}))
</source>

== Last Steps ==

In order to help your students get their kernel definitions into $HOME/.local/share/jupyter/kernels/, it's probably a good idea to write a small and simple notebook to do this when executed, or else instruct them to do:

<source lang='bash'>
cp -rv /cm/shared/apps/SHARED/my_conda_env/kernel/* .local/share/jupyter/kernels/
</source>

Conda for teaching

2020-05-27T08:59:13Z

Dawes001: /* Jupyter Kernel */

You are going to give a teaching course, and you need a specific code environment.

== Setup ==
First - find a good location that everyone can read (and not write). I'd suggest somewhere under /cm/shared/apps/SHARED/ as a starting point - this allows everyone to access this location. It's important not to put anything secret there - it's a public resource, so please bear that in mind.

Next - create a folder for your environment:

<source lang='bash'>
mkdir /cm/shared/apps/SHARED/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env
</source>
You may want to manipulate the permissions for this folder if someone is going to set this up with you. Consider the commands in [[Shared Folders]].

Then, install Anaconda into it:

<source lang='bash'>
wget https://repo.anaconda.com/archive/Anaconda3-YEAR.MONTH-Linux-x86_64.sh
./Anaconda3-YEAR.MONTH-Linux-x86_64.sh -s -b -p /cm/shared/apps/SHARED/my_conda_env
</source>

Now you have a working conda environment in this folder. You can manipulate this here by running /cm/shared/apps/SHARED/my_conda_env/bin/conda , or, I would recommend creating a modulefile so that you can use it as default.

Create the following example modulefile in a matching /cm/shared/modulefiles/SHARED/my_conda_env :

<source lang='bash'>
#%Module -*- tcl -*-
##
## conda environment modulefile
##

set loadedmodules [split $::env(LOADEDMODULES) ":"]
set modulepath [split $ModulesCurrentModulefile "/"]
set envpath [lrange $modulepath 4 end]

set root /cm/shared/apps/[join $envpath "/"]

proc ModulesHelp { } {
global version

puts stderr "\tThis module provides the conda environment at $envpath"
}

if { [module-info mode] != "whatis" } {
puts stderr "[module-info mode] environent $envpath ."
}

module-whatis "Provides environment $envpath"

prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path LIBRARY_PATH $root/lib
prepend-path CPATH $root/include
prepend-path MANPATH $root/share/man
</source>

This will allow you to `module load SHARED/my_conda_env` and thus have `conda` pathed to the currently active environment.

== Jupyter Kernel ==

In order for students to be able to use this environment in jupyter, they will need a kernel definition.

Kernel definitions are usually a separate folder containing, in particular, a file called `kernel.json`, plus an icon that is displayed that represents this kernel, and other helper code.

The setup for this is that you should create the following folder for their access:

<source lang='bash'>
mkdir -p /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
</source>

This folder is something they will need to copy in place to their home directory, specifically $HOME/.local/share/jupyter/kernels/

Inside this folder, create the following kernel.json file. Watch out that the paths will need to match the current environment path if you're using a different location!

=== Python Kernel ===

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"PATH":

"/cm/shared/apps/SHARED/my_conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
},
"language": "python",
"argv": [
"/cm/shared/apps/SHARED/my_conda_env/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "my_conda_env"
}
</source>

For Python, that's it. Ipykernel is installed automatically on conda initialisation.

=== R Kernel ===

For an R kernel, you need to make sure that the IRkernel package is installed. This is the package that is used to communicate from Jupyter to your running R kernel.<source lang='bash'>
#MISSING EXAMPLE CODE
#PROBABLY /cm/shared/apps/SHARED/my_conda_env/bin/conda install R_irkernel
</source>

You'll also need to create two files. the kernel.js is a helper script to allow jupyter to communicate to R effectively:

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"LD_LIBRARY_PATH":
"/cm/shared/apps/SHARED/my_conda_env/lib:/cm/shared/apps/SHARED/my_conda_env/lib64"
},
"argv": ["/cm/shared/apps/SHARED/my_conda_env/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
"display_name": "MAE50806-AdvMolEcol/Sandbox_R",
"language": "R"
}
</source>

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.js
</source>
<source lang='bash'>
const cmd_key = /Mac/.test(navigator.platform) ? 'Cmd' : 'Ctrl'

const edit_actions = [
{
name: 'R Assign',
shortcut: 'Alt--',
icon: 'fa-long-arrow-left',
help: 'R: Inserts the left-assign operator (<-)',
handler(cm) {
cm.replaceSelection(' <- ')
},
},
{
name: 'R Pipe',
shortcut: `Shift-${cmd_key}-M`,
icon: 'fa-angle-right',
help: 'R: Inserts the magrittr pipe operator (%>%)',
handler(cm) {
cm.replaceSelection(' %>% ')
},
},
{
name: 'R Help',
shortcut: 'F1',
icon: 'fa-book',
help: 'R: Shows the manpage for the item under the cursor',
handler(cm, cell) {
const {anchor, head} = cm.findWordAt(cm.getCursor())
const word = cm.getRange(anchor, head)

const callbacks = cell.get_callbacks()
const options = {silent: false, store_history: false, stop_on_error: true}
cell.last_msg_id = cell.notebook.kernel.execute(`help(\`${word}\`)`, callbacks, options)
},
},
]

const prefix = 'irkernel'

function add_edit_shortcut(notebook, actions, keyboard_manager, edit_action) {
const {name, shortcut, icon, help, handler} = edit_action

const action = {
icon, help,
help_index : 'zz',
handler: () => {
const cell = notebook.get_selected_cell()
handler(cell.code_mirror, cell)
},
}

const full_name = actions.register(action, name, prefix)

Jupyter.keyboard_manager.edit_shortcuts.add_shortcut(shortcut, full_name)
}

function render_math(pager, html) {
if (!html) return
const $container = pager.pager_element.find('#pager-container')
$container.find('p[style="text-align: center;"]').map((i, e) =>
e.outerHTML = `\\[${e.querySelector('i').innerHTML}\\]`)
$container.find('i').map((i, e) =>
e.outerHTML = `\$${e.innerHTML}\$`)
MathJax.Hub.Queue(['Typeset', MathJax.Hub, $container[0]])
}

define(['base/js/namespace'], ({
notebook,
actions,
keyboard_manager,
pager,
}) => ({
onload() {
edit_actions.forEach(a => add_edit_shortcut(notebook, actions, keyboard_manager, a))

pager.events.on('open_with_text.Pager', (event, {data: {'text/html': html}}) =>
render_math(pager, html))
},
}))
</source>

== Last Steps ==

In order to help your students get their kernel definitions into $HOME/.local/share/jupyter/kernels/, it's probably a good idea to write a small and simple notebook to do this when executed, or else instruct them to do:

<source lang='bash'>
cp -rv /cm/shared/apps/SHARED/my_conda_env/kernel/* .local/share/jupyter/kernels/
</source>

Conda for teaching

2020-05-27T08:45:06Z

Dawes001:

Conda for teaching

2020-05-27T08:40:47Z

Dawes001: /* Jupyter Kernel */

Conda for teaching

2020-05-27T08:38:31Z

Dawes001: Created page with "You are going to give a teaching course, and you need a specific code environment. == Setup == First - find a good location that everyone can read (and not write). I'd sugges..."

Tariffs

2020-02-17T08:46:03Z

Dawes001:

== Computing: Calculations (cores)==
{| class="wikitable"
!Queue
!CPU core hour
!GB memory hour
|-
|Standard queue
|€ 0.0150
|€ 0.0015
|-
|High priority queue
|€ 0.0200
|€ 0.0020
|-
|Low priority queue
|€ 0.0100
|€ 0.0010
|}

== Computing: GPU Use==
{| class="wikitable"
!Tariff per device per hour (gpu/hour)
|-
|€ 0.3000
|}

== Storage ==
Tariffs per year per TB
{| class="wikitable"
!Lustre Backup
!Lustre Nobackup
!Lustre Scratch
!Home-dir
!Archive
|-
|€ 175
|€ 125
|€ 125
|€ 175
|€ 125
|}

== Reservations ==
{| class="wikitable"
!Tariff per node per day (node/day)
|-
|€ 30
|}

== Notes==

If you are a member of a group with a commitment, then these costs get deducted from that commitment. Typically we are fairly lax with enforcing limits - only once you get to around 150% of your commitment will we consider taking action (mainly coming to discuss things).

== Example ==

You are running a job that needs 4 cores, 32G of RAM and runs for 90 minutes in the std quality. To run this, you over-request resources slightly, and execute in a job that requests 4 CPUs, 40G of RAM and with a time limit of 3 hours. Your job terminates early. Thus, your costs are:

4 * 0.015 * 1.5 = 0.09 EUR for the CPU

40 * 0.0015 * 1.5 = 0.09 EUR for the memory

Total: 0.18 EUR

Setting up Python virtualenv

2019-12-11T15:32:51Z

Dawes001: /* Virtualenv kernels in Jupyter */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</source>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
pip install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"/home/myhome/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/cm/shared/apps/spark/my-spark-version"
},
"argv": [
"/home/myhome/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Setting up Python virtualenv

2019-12-11T15:32:23Z

Dawes001: /* Virtualenv kernels in Jupyter */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</source>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
pip install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"~/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/cm/shared/apps/spark/my-spark-version"
},
"argv": [
"/home/myhome/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Scheduler Overview (Slurm)

2019-10-17T15:19:26Z

Dawes001: /* Batch script */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --cpus-per-task=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --constraint=<gpu flavour e.g. K80, V100>
#SBATCH --partition=gpu
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-02T08:00:14Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --constraint=<gpu flavour e.g. K80, V100>
#SBATCH --partition=gpu
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-01T08:08:46Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --constraint=<gpu flavour e.g. K80, V100>
#SBATCH --partition=GPU
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-01T08:07:53Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --gres=gpu:<num gpus>
#SBATCH --partition=GPU
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Scheduler Overview (Slurm)

2019-10-01T08:06:55Z

Dawes001: /* Using GPU */

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --partition=GPU
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Ssh without password

2019-07-26T16:29:57Z

Dawes001: /* Step 1: create a public key and copy to remote computer */

Ssh without password

2019-07-26T16:29:12Z

Dawes001: /* Step 1: create a public key and copy to remote computer */

Ssh without password

2019-07-26T16:28:37Z

Dawes001:

Creating sbatch script

2019-07-15T15:05:38Z

Dawes001:

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
#SBATCH --qos=
#-----------------------------Required resources-----------------------
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
===partition for resource allocation===
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.

=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --comment=773320000
</source>
Charge resources used by this job to specified account. The comment is an arbitrary string. The comment may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting by feature===
<source lang='bash'>
#SBATCH --constraint=4gpercpu
</source>
The HPC nodes have features associated with them, such as Intel CPU's, or the amount of memory per node. If you know that your job requires a specific architecture or memory size, you can elect to constrain your job to only these features.

The example above will result in jobs being scheduled to the compute nodes with 4GB of memory per CPU. By using <code>12gpercpu</code> as option the job will specifically be scheduled to one of the larger nodes with 12GB per CPU.

All features can be seen using:
<source lang='bash'>
scontrol show nodes | grep ActiveFeatures | sort | uniq
</source>

===requesting specific resources===
<source lang='bash'>
#SBATCH --gres=gpu:1
</source>
In order to be able to use specific hardware resources, you need to request a Generic Resource. Once you do this, one of the resources will be allocated to your job when they are available. In the above example, one GPU is requested for use.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.

===adding a job name===
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.

===receiving mailed updates===
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=yourname001@wur.nl
</source>
Email address to use.

== See also ==
* [[Anunna | Anunna]]
* [[Using_Slurm#Batch_script | Submitting jobs to Slurm]]
* [[Array_jobs|Array job hints]]

Tariffs

2019-07-15T15:04:20Z

Dawes001:

== Computing: Calculations (cores)==
{| class="wikitable"
!Queue
!CPU core hour
!GB memory hour
|-
|Standard queue
|€ 0.0150
|€ 0.0015
|-
|High priority queue
|€ 0.0200
|€ 0.0020
|-
|Low priority queue
|€ 0.0100
|€ 0.0010
|}

== Computing: GPU Use==
{| class="wikitable"
!Tariff per device per hour (gpu/hour)
|-
|€ 0.3000
|}

== Storage ==
Tariffs per year per TB
{| class="wikitable"
!Lustre Nobackup
!Lustre Backup
!Home-dir
!Archive
|-
|€ 150
|€ 200
|€ 200
|€ 100
|}

== Reservations ==
{| class="wikitable"
!Tariff per node per day (node/day)
|-
|€ 30
|}

== Notes==

If you are a member of a group with a commitment, then these costs get deducted from that commitment. Typically we are fairly lax with enforcing limits - only once you get to around 150% of your commitment will we consider taking action (mainly coming to discuss things).

== Example ==

You are running a job that needs 4 cores, 32G of RAM and runs for 90 minutes in the std quality. To run this, you over-request resources slightly, and execute in a job that requests 4 CPUs, 40G of RAM and with a time limit of 3 hours. Your job terminates early. Thus, your costs are:

4 * 0.015 * 1.5 = 0.09 EUR for the CPU

40 * 0.0015 * 1.5 = 0.09 EUR for the memory

Total: 0.18 EUR

Spark

2019-07-15T15:03:35Z

Dawes001: /* SPARK on HPC */

Apache Spark is a means of distributing compute resources across multiple worker machines. It is the successor to Hadoop, and allows for a wider distribution of code to be executed on the clustered resources. The only requirement for Spark to be able to operate is that each worker must be able to reach each other via TCP, thus it allows for compute to be executed on very simple resources, if the code itself can be translated into the MapReduce paradigm.

== SPARK on HPC ==
In order to create a personal SPARK cluster, you must first request resources on the HPC. Use this example submission script to initialise your cluster:

<nowiki>#!/bin/bash
#SBATCH --time=<length>
#SBATCH --mem-per-cpu=4000
#SBATCH --nodes=<number of nodes>
#SBATCH --tasks-per-node=<number of workers per node>
#SBATCH --job-name="my spark cluster"
#SBATCH --qos=QOS

module load python
module load spark

source $SPARK_HOME/wur/start-spark

tail -f /dev/null</nowiki>

This will spawn a new cluster of your desired dimensions once resources are available. This spark module has been written to output its logs to your home directory, at:

/home/WUR/yourid/.spark/<jobid>/

In this folder you will find the raw logs of the master and all worker threads. By default the master will consume 1Gb of memory from the first process, and so a single 4Gb 'cluster' will be provided with one 3Gb worker. You can adjust the CPU/memory use by adjusting the parameters in your batch script.

Within the log file you will find two unique files: master, and master-console. master will always contain the URI of the current spark cluster master access point, and master-console the URL of the console of it.

To access the web console, the easiest solution is to use links:

<nowiki>links http://myspark:8081</nowiki>

This will nicely render the page for you in the console. Ctrl-R reloads the page, q to quit.

There are several caveats to remember with this:

* The cluster exists (and consumes resources) until you cancel it with scancel <jobid>
* There is no security at all - any user of the HPC can access both these at any point if they know the port and host.

== Instant SPARK ==

You can also spin up clusters solely to execute scripts. Simply replace the last line from the example above:
<nowiki>tail -f /dev/null</nowiki>
with
<nowiki>spark-submit myscript.py</nowiki>

And after the script has executed, the cluster will automatically terminate.

== SPARK in Jupyter ==

There is a kernel available for using Spark from Jupyter. All this does (for now) is to set up the correct path to the python version and the spark binaries for you. In order to set up your Context, your first cell for each notebook should be:
<nowiki>import pyspark
conf = (pyspark.SparkConf()
.setMaster("spark://mysparkcluster:7077")
.setAppName("MyName"))

sc = pyspark.SparkContext(conf=conf)</nowiki>

Using the cluster master name from the master file in your job output as above. Subsequent cells will then have sc defined. Run this cell only once - attempting to reconnect will throw an error. That application will run until the kernel is terminated and prevent other applications from being able to be executed - you may wish to manually terminate your kernel from the top bar in Jupyter to free resources.

MPI on B4F cluster

2019-07-15T15:02:51Z

Dawes001:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<source lang='bash'>
module list
</source>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /cm/shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

MPI on B4F cluster

2019-07-15T15:02:32Z

Dawes001:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<source lang='bash'>
module list
</source>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /cm/shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --partition=ABGC --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

Array Jobs

2019-07-15T14:58:59Z

Dawes001:

SLURM can simplify your efforts if you are planning on submitting multiple independent jobs in parallel. Rather than having to use sbatch multiple times, you can instead use an array job to run your job.

Take the following example:

<source lang='bash'>
#SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt
#SBATCH --time=10
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=4000
#SBATCH --array=0-9%4

echo $SLURM_ARRAY_TASK_ID

</source>
Let's break this down step by step:

<source lang='bash'>
#SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt
</source>
This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.

<source lang='bash'>
#SBATCH --array=0-9%4
</source>
This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
<source lang='bash'>
#SBATCH --array=3,7-11
</source>
will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.

<source lang='bash'>
echo $SLURM_ARRAY_TASK_ID
</source>
This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.

So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n.

== Two dimensional arrays? ==
Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then

SLURM_ARRAY_TASK_ID=[ 0 1 2 3 4 5 6 7 8 9 ]

for each job. What if you need two variables to change instead of one?

Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:

<source lang='bash'>
A=$((93 / 10)) # A = 9
B=$((93 % 10)) # B = 3
</source>

As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:

<source lang='bash'>
#SBATCH --array=0-29 ## 5*6 entries, thus 30, including 0 this is 0-29
A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
mywork $A $B
</source>

Creating sbatch script

2019-07-15T14:58:38Z

Dawes001:

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
#SBATCH --qos=
#-----------------------------Required resources-----------------------
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
===partition for resource allocation===
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.

=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --comment=773320000
</source>
Charge resources used by this job to specified account. The comment is an arbitrary string. The comment may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting by feature===
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The HPC nodes have features associated with them, such as Intel CPU's, or the amount of memory per node. If you know that your job requires a specific architecture or memory size, you can elect to constrain your job to only these features.

The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

All features can be seen using:
<source lang='bash'>
scontrol show nodes | grep ActiveFeatures | sort | uniq
</source>

===requesting specific resources===
<source lang='bash'>
#SBATCH --gres=gpu:1
</source>
In order to be able to use specific hardware resources, you need to request a Generic Resource. Once you do this, one of the resources will be allocated to your job when they are available. In the above example, one GPU is requested for use.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.

===adding a job name===
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.

===receiving mailed updates===
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=yourname001@wur.nl
</source>
Email address to use.

== See also ==
* [[Anunna | Anunna]]
* [[Using_Slurm#Batch_script | Submitting jobs to Slurm]]
* [[Array_jobs|Array job hints]]

Scheduler Overview (Slurm)

2019-07-15T14:57:48Z

Dawes001:

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Quality of Service ===
When submitting a job, you may optionally assign a different Quality of Service to it. You can do this with:

<source lang='bash'>
#SBATCH --qos=std
</source>

By default, jobs will use std, the standard quality.

Optionally, you may elect to reduce the priority of your jobs to low. This comes with a limit of how long each job can be (8h) to prevent the cluster from being locked up entirely with low priority jobs.

The high quality provides a higher priority to jobs (20) than std (10), or low (1). It is naturally more expensive.

The highest priority goes to jobs in interactive quality (100), but you may not submit many jobs or many large jobs as this quality. This is exclusively for the use of immediate running jobs, ones that are going to have hands-on users behind them.

Jobs may be restarted and rescheduled if a job with higher priority needs cluster resources, but as of right now, this is not occurring.

=== Queues ===
The cluster consists of multiple partitions of nodes that you can submit to. The primary one is 'main'. There are other partitions as needed - current plans include 'gpu'.

You can see the partitions available with `sinfo`:

=== Defaults ===
The default partition is 'main'. This will work for most jobs.

The default qos is 'std'.

The default cpu count is 1.

The default run time for a job is '''1 hour'''.

The default memory limit is '''100MB per node'''.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

=== Using GPU ===
There are two GPU nodes, in order to run a job that uses GPU on one of these nodes, you can add

<source lang='bash'>
#SBATCH --reservation='GPU'
</source>
To your sbatch script. Without this parameter, your job won't run on one of these nodes.

THIS IS PRONE TO CHANGE SHORTLY! [[User:Dawes001|Dawes001]] ([[User talk:Dawes001|talk]]) 14:57, 15 July 2019 (UTC)

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Courses

2019-06-27T08:29:50Z

Dawes001: /* HPC CUDA/AI Course - 2019-06-21 */

== Upcoming: HPC Basic Course - 2019-06-28==

A course for beginners will be organised on the 28th of June, aiming to help absolute beginners to begin to use the main job scheduler, SLURM. You can register [https://oneschool.wur.nl/Lists/Cursus/DispForm.aspx?ID=100 here].

== Upcoming Linux Basic Course - 2019-06-27 ==

A course for beginners will be organised on the 13th of June, to help beginner Linux users gain some skills in using Linux. You can register for this course [https://www.wur.nl/en/activity/Linux-basic-course-on-13-June-2019.htm here]

== HPC CUDA/AI Course - 2019-06-21 ==

A course for interested users for deep learning and neural networks, combined with some deep level manipulation of graphics cards was given by Dell on the 21st of July.

[[File:WUR_CUDA_210619.pdf|WUR CUDA Course]]
[[File:WUR_AI_101_210619.pdf|WUR AI Course 101]]
[[File:WUR_AI_201_210619.pdf|WUR AI Course 201]]

[[File:WUR_CUDA_2_210619.pdf|WUR CUDA Course]]
[[File:WUR_Deep_Learning_Frameworks_210619.pdf|WUR Deep Learning Frameworks Primer]]
[[File:WUR_Deep_Learning_Lab_210619.pdf|WUR Deep Learning Lab]]

== HPC Advanced Course - 2019-05-28 ==

A course for experienced users was organised on the 28th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20190506.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - 2019-05-07 ==

A course for beginners will be organised on the 7th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20190506.pdf|Basic Course]]

== Linux Basic Course - 2019-04-16 ==

A course for beginners was organised on the 16th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2018-10-16 ==

A course for experienced users was organised on the 16th of October, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_Slides_20181016.pdf|Advanced Course (Gwen)]]

[[File:HPC_advanced_course_20181008.pdf|Advanced Course (Jeremie)]]

== HPC Basic Course - 2018-10-11 ==

A course for beginners was organised on the 11th of October, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC_basic_course_20181008.pdf|Basic Course]]

== Basic Linux Course - 2018-10-02 ==

A course basic Linux usage was organised on the 2nd of October, to help beginner Linux users gain some skills in using Linux.

[https://etherpad.lug.wur.nl/p/UpkF2KXDVh]

== HPC Advanced Course - 2018-05-18 ==

A course for experienced users was organised on the 18th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_20180518-GD.pdf|Advanced Course (Gwen)]]

== HPC Basic Course - 2018-05-17 ==

A course for beginners was organised on the 17th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Basic Linux Course - 2018-04-19 ==

A course basic Linux usage was organised on the 19th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2017-11-09 ==

A course for experienced users was organised on the 9th of November, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_course_2017-11-08-JV.pdf|Advanced Course (Jeremie)]]

[[File:HPC_Advanced_course_2017-11-08-GD.pdf|Advanced Course (Gwen)]]

[[File:Checkpointing_2017-11-08.pdf|Checkpointing]]

== HPC Basic Course - 2017-10-30 ==

A course for beginners was organised on the 30th of October, aiming to help absolute beginners to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:HPC_basic_course_20171025.pdf | Basic introduction to Linux]]

== HPC Teaching - 2017-06-07 ==

A course for was organised on the 7th of June, aiming to help absolute beginners (and moderately experienced users) to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:Connecting_with_Secure_Shell_to_the_HPC_20170606.pdf | Basic introduction to Linux]]

[[File:Submitting_and_monitoring_jobs_on_the_HPC_20170602.pdf | Submitting and Monitoring Jobs]]

== Old Courses ==
* [http://www.basgen.nl/sdac/ Sequence Data Analysis Course (Dec. 2012)]

File:WUR Deep Learning Lab 210619.pdf

2019-06-27T08:28:39Z

Dawes001:

File:WUR Deep Learning Frameworks 210619.pdf

2019-06-27T08:28:23Z

Dawes001:

File:WUR CUDA 2 210619.pdf

2019-06-27T08:27:57Z

Dawes001:

Courses

2019-06-24T16:04:26Z

Dawes001:

== Upcoming: HPC Basic Course - 2019-06-28==

A course for beginners will be organised on the 28th of June, aiming to help absolute beginners to begin to use the main job scheduler, SLURM. You can register [https://oneschool.wur.nl/Lists/Cursus/DispForm.aspx?ID=100 here].

== Upcoming Linux Basic Course - 2019-06-27 ==

A course for beginners will be organised on the 13th of June, to help beginner Linux users gain some skills in using Linux. You can register for this course [https://www.wur.nl/en/activity/Linux-basic-course-on-13-June-2019.htm here]

== HPC CUDA/AI Course - 2019-06-21 ==

A course for interested users for deep learning and neural networks, combined with some deep level manipulation of graphics cards was given by Dell on the 21st of July.

[[File:WUR_CUDA_210619.pdf|WUR CUDA Course]]
[[File:WUR_AI_101_210619.pdf|WUR AI Course 101]]
[[File:WUR_AI_201_210619.pdf|WUR AI Course 201]]

== HPC Advanced Course - 2019-05-28 ==

A course for experienced users was organised on the 28th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_advanced_course_20190506.pdf|Advanced Course 1]]
[[File:HPC_advanced_slides_20190528.pdf|Advanced Course 2]]

== HPC Basic Course - 2019-05-07 ==

A course for beginners will be organised on the 7th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC basic course 20190506.pdf|Basic Course]]

== Linux Basic Course - 2019-04-16 ==

A course for beginners was organised on the 16th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2018-10-16 ==

A course for experienced users was organised on the 16th of October, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_Slides_20181016.pdf|Advanced Course (Gwen)]]

[[File:HPC_advanced_course_20181008.pdf|Advanced Course (Jeremie)]]

== HPC Basic Course - 2018-10-11 ==

A course for beginners was organised on the 11th of October, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

[[File:HPC_basic_course_20181008.pdf|Basic Course]]

== Basic Linux Course - 2018-10-02 ==

A course basic Linux usage was organised on the 2nd of October, to help beginner Linux users gain some skills in using Linux.

[https://etherpad.lug.wur.nl/p/UpkF2KXDVh]

== HPC Advanced Course - 2018-05-18 ==

A course for experienced users was organised on the 18th of May, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_20180518-GD.pdf|Advanced Course (Gwen)]]

== HPC Basic Course - 2018-05-17 ==

A course for beginners was organised on the 17th of May, aiming to help absolute beginners to begin to use the main job scheduler, SLURM.

== Basic Linux Course - 2018-04-19 ==

A course basic Linux usage was organised on the 19th of April, to help beginner Linux users gain some skills in using Linux.

== HPC Advanced Course - 2017-11-09 ==

A course for experienced users was organised on the 9th of November, aiming to brush up users on techniques for submitting unusual jobs, and help provide some more helpful hints and techniques.

[[File:HPC_Advanced_course_2017-11-08-JV.pdf|Advanced Course (Jeremie)]]

[[File:HPC_Advanced_course_2017-11-08-GD.pdf|Advanced Course (Gwen)]]

[[File:Checkpointing_2017-11-08.pdf|Checkpointing]]

== HPC Basic Course - 2017-10-30 ==

A course for beginners was organised on the 30th of October, aiming to help absolute beginners to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:HPC_basic_course_20171025.pdf | Basic introduction to Linux]]

== HPC Teaching - 2017-06-07 ==

A course for was organised on the 7th of June, aiming to help absolute beginners (and moderately experienced users) to enhance their ability to use the main job scheduler, SLURM.

The slides for this course can be found here:

[[File:Connecting_with_Secure_Shell_to_the_HPC_20170606.pdf | Basic introduction to Linux]]

[[File:Submitting_and_monitoring_jobs_on_the_HPC_20170602.pdf | Submitting and Monitoring Jobs]]

== Old Courses ==
* [http://www.basgen.nl/sdac/ Sequence Data Analysis Course (Dec. 2012)]

File:WUR CUDA 210619.pdf

2019-06-24T16:01:56Z

Dawes001:

File:WUR AI 101 210619.pdf

2019-06-24T16:01:22Z

Dawes001:

File:WUR AI 201 210619.pdf

2019-06-24T16:00:59Z

Dawes001:

File:Anunna Flyer 2019.svg

2019-05-06T08:48:54Z

Dawes001:

Tariffs

2019-04-23T08:30:08Z

Dawes001:

== Computing: Calculations (cores)==
{| class="wikitable"
!Queue
!CPU core hour
!GB memory hour
|-
|Standard queue
|€ 0.0150
|€ 0.0015
|-
|High priority queue
|€ 0.0200
|€ 0.0020
|-
|Low priority queue
|€ 0.0100
|€ 0.0010
|}

== Computing: GPU Use==
{| class="wikitable"
!Tariff per device per hour (gpu/hour)
|-
|€ 0.3000
|}

== Storage ==
Tariffs per year per TB
{| class="wikitable"
!Lustre Nobackup
!Lustre Backup
!Home-dir
!Archive
|-
|€ 150
|€ 200
|€ 200
|€ 100
|}

== Reservations ==
{| class="wikitable"
!Tariff per node per day (node/day)
|-
|€ 50
|}

== Notes==

If you are a member of a group with a commitment, then these costs get deducted from that commitment. Typically we are fairly lax with enforcing limits - only once you get to around 150% of your commitment will we consider taking action (mainly coming to discuss things).

== Example ==

You are running a job that needs 4 cores, 32G of RAM and runs for 90 minutes in the Std partition. To run this, you over-request resources slightly, and execute in a job that requests 4 CPUs, 40G of RAM and with a time limit of 3 hours. Your job terminates early. Thus, your costs are:

4 * 0.015 * 1.5 = 0.09 EUR for the CPU

40 * 0.0015 * 1.5 = 0.09 EUR for the memory

Total: 0.18 EUR

Tariffs

2019-04-23T08:25:47Z

Dawes001:

Scheduler Overview (Slurm)

2019-04-04T08:44:14Z

Dawes001:

The resource allocation / scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Queues ===
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue.<br>
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0)<br>
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are occupied by a Low queue jobs.
To find out which queues your account has been authorized for, type sinfo:
<source lang='bash'>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
ABGC_High up infinite 12 down* node[043-048,055-060]
ABGC_High up infinite 6 mix fat[001-002],node[002-005]
ABGC_High up infinite 44 idle node[001,006-042,049-054]
ABGC_Std up infinite 12 down* node[043-048,055-060]
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]
ABGC_Std up infinite 44 idle node[001,006-042,049-054]
ABGC_Low up infinite 12 down* node[043-048,055-060]
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]
ABGC_Low up infinite 44 idle node[001,006-042,049-054]
</source>

=== Defaults ===
There is no default queue, so you need to specify which queue to use when submitting a job.<br>
'''The default run time for a job is 1 hour!''' <br>
'''Default memory limit is 100MB per node!'''

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --comment=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=ABGC_Std
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs (simple) ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Submitting multiple jobs (complex) ===
Lets's say you have three job scripts that depend on each other:

<source lang='bash'>job_1.sh #A simple initialisation script</source>
<source lang='bash'>job_2.sh #An array task</source>
<source lang='bash'>job_3.sh #Some finishing script, single run, after everything previous has finished</source>

You can create a script to simultaneously submit each job with a dependency on each other:

<source lang='bash'>#!/bin/bash
JOB1=$(sbatch job_1.sh| rev | cut -d ' ' -f 1 | rev) #Get me the last space-separated element

if ! [ "z$JOB1" == "z" ] ; then
echo "First job submitted as jobid $JOB1"
JOB2=$(sbatch --dependency=afterany:$JOB1 job_2.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB2" == "z" ] ; then
echo "Second job submitted as jobid $JOB2, following $JOB1"
JOB3=$(sbatch --dependency=afterany:$JOB2 job_3.sh| rev | cut -d ' ' -f 1 | rev)

if ! [ "z$JOB3" == "z" ] ; then
echo "Third job submitted as jobid $JOB3, following after every element of $JOB2"

fi
fi
fi
</source>

This will ensure that the subsequent jobs occur after any finishing of the former (even if they failed).

Please see [https://slurm.schedmd.com/sbatch.html#OPT_dependency the sbatch documentation] for other options available to you. Note that aftercorr makes a subsequent array jobs array elements start after the correspondingly numbered ones from the previous job.

=== Submitting array jobs ===
<source lang='bash'>
#SBATCH --array=0-10%4
</source>
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].

=== Using /tmp ===
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.

In order to be sure that you're able to use space in /tmp, you can add
<source lang='bash'>
#SBATCH --tmp=<required size>
</source>
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
login ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=login0:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@login jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@login jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@login jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sinteractive ==
sinteractive is a tiny wrapper on srun to create interactive jobs quickly and easily. It allows you to get a shell on one of the nodes, with similar limits as you would do for a normal job. To use it, simply run:
<source lang='bash'>
sinteractive -c <num_cpus> --mem <amount_mem> --time <minutes> -p <partition>
</source>
You will then be presented with a new shell prompt on one of the compute nodes (run 'hostname' to see which!). From here, you can test out code in an interactive fashion as needs be.

Be advised though - not filling in the above fields will get you a shell with 1 CPU and 100Mb of RAM for 1 hour. This is useful for quick testing, however.

=== sinteractive source ===
<source lang='bash'>
#!/bin/bash
srun "$@" -I60 -N 1 -n 1 --pty bash -i
</source>

=== interactive Slurm - using salloc ===
If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on the login node, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,comment,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Comment Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on Anunna ==

[[MPI_on_B4F_cluster | Main article: MPI on Anunna]]
< text here >

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[Tariffs | Costs associated with resource usage]]
* [[B4F_cluster | Anunna]]
* [[BCM_on_B4F_cluster | BCM on Anunna]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Filesystems

2019-04-04T08:43:24Z

Dawes001:

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from nfs01. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /cm/shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of three major parts (and some minor):
* /lustre/backup - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/nobackup - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recivered.
* /lustre/scratch - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.
* /lustre/shared - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* /archive - an archive mount only accessible from the login nodes. Files here are sent to the Isilon for deeper storage. The cost of storing data here is much less than on the Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via snapshot, and there is a separated backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<source lang='bash'>
#SBATCH --tmp=<required space>
</source>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Main Page

2019-04-04T08:42:41Z

Dawes001:

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]
* [[Tariffs | Costs associated with resource usage]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:lith010 | Jan van Lith (Wageningen UR,FB-IT, Infrastructure)]], [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:vaend001 | Catharina Vaendel(Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Tariffs

2019-04-04T08:41:47Z

Dawes001: Created page with "== Computing: Calculations (cores)== {| class="wikitable" !Queue !CPU core hour !GB memory hour |- |Standard queue |€ 0.0150 |€ 0.0015 |- |High priority queue |€ 0.0200..."

File:Anunna.svg.png

2019-03-19T14:11:52Z

Dawes001:

B4F cluster

2019-03-19T14:03:41Z

Dawes001:

= Historical =
This page contains historical information about the WUR HPC Cluster as of 2013.

== B4F Cluster ==
The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS].

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.

{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.

{{-}}

== Architecture of the cluster ==

[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast internet connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}
=== Nodes ===
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is RHEL6. The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').

Information from the Cluster Management Portal, as it appeared on November 23, 2013:
<code>DEVICE INFORMATION
Hostname State Memory Cores CPU Speed GPU NICs IB Category
master1, master2 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1
node001..node042, node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 1200 MHz 3 1 default
node043..node048, node055..node060 DOWN N/A N/A N/A N/A N/A N/A N/A default
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 5 1 mds
storage01 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss
storage02..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 oss
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 7 1 login
fat001, fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2299 MHz 5 1 fat </code>

Main cluster node configuration:
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration
* The NFS server is a PowerEdge R720XD, which will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution. The NFS node will also act as a login node, where users log in and compile applications and submit jobs.
* 50 compute nodes
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each
Hyperthreading is disabled in compute nodes.

=== Filesystems ===

[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]
The B4F Cluster has two primary file systems, each with different properties and purposes.
==== Parallel File System: Lustre ====
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.

The hardware components of the PFS:
* 2x Dell PowerEdge R720
* 1x Dell PowerVault MD3220
* 6x Dell PowerEdge R620
* 6x Dell PowerVault MD3260

==== Network File System (NFS): $HOME dirs ====
Each user will have his/her own home directory. The path of the home directory will be:

/home/[name partner]/[username]

/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using:

<source lang='bash'>
quota
</source>

The NFS is supported through the NFS server that also serves as access point to the cluster.

Hardware components of the NFS:
* 1x Dell PowerEdge R720XD
* 1x Dell PowerVault MD3220

=== Network ===
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The B4F Cluster is housed at the main server centre of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-ICT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==

=== Project Leader ===
* Stephen Janssen (Wageningen UR, FB-IT, Service Management)

=== Daily Project Management ===
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]
* Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)
[[Maintenance_and_Management | Maintenance and Management]]

=== Steering Group ===
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication.
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)
* Petra Caessens (CAT-AgroFood)
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)
* Edda Neuteboom (CAT_AgroFood, secretariat)
* Martien Groenen (Wageningen UR, chair, replaced Johan van Arendonk).

=== IT Workgroup ===
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.
* Stephen Janssen (Wageningen UR, FB-IT, Service Management)
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]
* Gwen Dawes (Wageningen UR, FB-IT, Infrastructure, replaced Andre ten Böhmer)
* Wes Barris (Cobb)
* Ton Dings (Hendrix Genetics)
* Henk van Dongen (Topigs)
* Harry Dijkstra (CRV)
* Mario Calus (ABGC-WLR)
* Hendrik-Jan Megens (ABGC-ABG)
{{-}}

=== User Group ===
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.

== Access Policy ==
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

=== Contact Persons ===
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):

==== Cobb-Vantress ====
* Wes Barris
* Jun Chen

==== ABGC ====
===== Animal Breeding and Genetics =====
* [[User:Hulze001 |Alex Hulzebosch]]
* [[User:Hjmegens | Hendrik-Jan Megens]]

===== Wageningen Livestock Research =====
* Mario Calus
* Ina Hulsegge
==== CRV ====
* Frido Hamoen
* Chris Schrooten
==== Hendrix Genetics ====
* Ton Dings
* Abe Huisman
* Addie Vereijken
==== Topigs ====
* Henk van Dongen
* Egiel Hanenbarg
* Naomi Duijvensteijn

== Using the B4F Cluster ==
=== Gaining access to the B4F Cluster ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"

| width="45%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]
| width="45%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
|}

Create shortcut log-in command

2019-03-19T14:00:32Z

Dawes001:

It can be very tedious to have to type full command to log into a remote machine using the secure shell (ssh) when a remote machine is used often.

== Add a custom command to the .bashrc ==
A simple solution is to add a custom command to the .basrc. Open the .bashrc (which is in the root of home directory) using any plain text editor. E.g.:
<source lang='bash'>
vi $HOME/.bashrc
</source>

Then, using the <code>alias</code> function, define a new function derived from the <code>ssh</code> command and your personal credentials:
<source lang='bash'>
alias lx7="ssh username@scomp1095.wurnet.nl"
alias hpc="ssh username@login.anunna.wur.nl"
</source>

The new function will be available from next login, therefore log out of the active terminal and log back in. From the freshly opened terminal, issuing the following command:
<source lang='bash'>
lx7
</source>
should give you direct access to <code>scomp1095</code>, for instance. If you do this in combination with a [[ssh_without_password | passwordless setup]], you will not even be prompted for a password.

== Using a ssh config file ==
Instead of using the alias function, it is possible to use a ssh config file. The ssh config file can be found at $HOME/.ssh/config .
If the ssh config file does not exist, you can create it with, e.g.,:
<source lang='bash'>
touch $HOME/.ssh.config
</source>
If the ssh config file exists, you can add the different options you use to log in of the different servers/clusters. For example, for Anunna, you can add:

<source lang='bash'>
Host hpc
Hostname login.anunna.wur.nl
User username
IdentityFile PATH_TO_PRIVATE_KEY (if applicable)
ForwardX11 yes (if applicable)
</source>

After the addition to the ssh congig file, you can log in as follows:
<source lang='bash'>
ssh hpc
</source>

You may add to the ssh config file several configurations for the same or several clusters. For example, for the two aliases in the section [[Add a custom command to the .bashrc | Add a custom command to the .bashrc ]], you can add the following lines to your ssh config file:
<source lang='bash'>
Host scomp
HostName scomp1095.wurnet.nl
User username
Host hpc
Hostname login.anunna.wur.nl
User username
</source>

For more information on the possible different options for the ssh config file, you may consult the [http://linux.die.net/man/5/ssh_config man page].

== See also ==
* [[ssh_without_password | ssh without password]]
* [[Lx6_and_Lx7_compute_nodes | Lx6 and Lx7 compute nodes]]
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External links ==

Create shortcut log-in command

2019-03-19T13:59:47Z

Dawes001:

It can be very tedious to have to type full command to log into a remote machine using the secure shell (ssh) when a remote machine is used often.

== Add a custom command to the .bashrc ==
A simple solution is to add a custom command to the .basrc. Open the .bashrc (which is in the root of home directory) using any plain text editor. E.g.:
<source lang='bash'>
vi $HOME/.bashrc
</source>

Then, using the <code>alias</code> function, define a new function derived from the <code>ssh</code> command and your personal credentials:
<source lang='bash'>
alias lx7="ssh username@scomp1095.wurnet.nl"
alias hpc="ssh username@nfs01.hpcagrogenomics.wur.nl"
</source>

The new function will be available from next login, therefore log out of the active terminal and log back in. From the freshly opened terminal, issuing the following command:
<source lang='bash'>
lx7
</source>
should give you direct access to <code>scomp1095</code>, for instance. If you do this in combination with a [[ssh_without_password | passwordless setup]], you will not even be prompted for a password.

== Using a ssh config file ==
Instead of using the alias function, it is possible to use a ssh config file. The ssh config file can be found at $HOME/.ssh/config .
If the ssh config file does not exist, you can create it with, e.g.,:
<source lang='bash'>
touch $HOME/.ssh.config
</source>
If the ssh config file exists, you can add the different options you use to log in of the different servers/clusters. For example, for Anunna, you can add:

<source lang='bash'>
Host hpc
Hostname nfs01.hpcagrogenomics.wur.nl
User username
IdentityFile PATH_TO_PRIVATE_KEY (if applicable)
ForwardX11 yes (if applicable)
</source>

After the addition to the ssh congig file, you can log in as follows:
<source lang='bash'>
ssh hpc
</source>

You may add to the ssh config file several configurations for the same or several clusters. For example, for the two aliases in the section [[Add a custom command to the .bashrc | Add a custom command to the .bashrc ]], you can add the following lines to your ssh config file:
<source lang='bash'>
Host scomp
HostName scomp1095.wurnet.nl
User username
Host hpc
Hostname nfs01.hpcagrogenomics.wur.nl
User username
</source>

For more information on the possible different options for the ssh config file, you may consult the [http://linux.die.net/man/5/ssh_config man page].

== See also ==
* [[ssh_without_password | ssh without password]]
* [[Lx6_and_Lx7_compute_nodes | Lx6 and Lx7 compute nodes]]
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External links ==

Setting up Python virtualenv

2019-03-19T13:58:57Z

Dawes001:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</source>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
pip install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"~/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/cm/shared/apps/spark/my-spark-version"
},
"argv": [
"~/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Ssh without password

2019-03-19T13:58:23Z

Dawes001:

Secure shell (ssh) protocols can be configure to work without protocols. This is particularly helpful for machines that are used often.

== Configuring ssh without password from a POSIX-compliant terminal ==

=== Step 1: create a public key and copy to remote computer ===
* Log into a local Linux or MacOSX computer
* Type the following to generate the ssh key:
<source lang='bash'>
ssh-keygen -t dsa
</source>
* Accept the default key location by pressing <code>Enter</code>.
* Secure permission of your authentication keys by closing permission to your home directory, .ssh directory, and authentication files
<source lang='bash'>
chmod go-w $HOME
chmod 700 $HOME/.ssh
chmod go-rwx $HOME/.ssh/*
</source>
* Type the following to copy the key to the remote server (this will prompt for a password).
<source lang='bash'>
cd ~/.ssh
scp id_dsa.pub remote_username@remote_host:/tmp
</source>
* Type the following to add the ssh key to the remote user's authorization keys (this will prompt for a password).
ssh remote_username@remote_host 'cat /tmp/id_dsa.pub >>/home/

=== Step 2: configure the public key from the local computer on the remote computer ===
* log in to the remote computer using ssh (this will prompt for a password)
<source lang='bash'>
ssh remote_username@remote_host
</source>
* copy the public key generated on the local computer, that was copied to <code>/tmp</code> on the remote computer, is now appended to <code>.ssh/authorized_keys2</code>. Note that a file called <code>.ssh/authorized_keys</code> may already be present.
<source lang='bash'>
cat /tmp/id_dsa.pub >>$HOME/.ssh/authorized_keys2
</source>
* similar to your local computer, make sure the permission on your remote home folder, .ssh folder, and authentication files are properly set (NOTE: this may already be the case).
<source lang='bash'>
chmod go-w $HOME
chmod 700 $HOME/.ssh
chmod go-rwx $HOME/.ssh/*
</source>
* Lastly, remove the public key from the temporary folder.
<source lang='bash'>
rm /tmp/id_dsa.pub
</source>
* If you log out of the remote computer and back in again, from the local computer from which you copied the public key, you will notice you will no longer be prompted for a password.

== Configuring ssh without password for Anunna ==

* Create a public key as in Step 1 of the previous section and copy it to Anunna. Note that a public/private key pair needs to be made only once per machine.
* Similar to step 2 of the previous section, add the public key to the <code>$HOME/.ssh/authorized_keys2</code> file. There is already a <code>$HOME/.ssh/authorized_keys</code> present. You may append the key to this file as an alternative, but take care not to remove content that is already there. The cluster is configured so that passwordless communication will all other nodes is default.

== Configuring ssh without password using PuTTY ==
Use pAGEaNT: http://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter9.html to generate local keys. You'll want have a copy of the pubkey in plaintext available.

Make sure to paste that plaintext string into ~/.ssh/authorized_keys in one single line. Chmod the file 600 (so it shows -rw------- in ls -l)

Now PuTTY will login passwordlessly whenever pAGEaNT is running.

Finally, get pAGEaNT to load on startup: http://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html

== See also ==
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External Links ==

Ssh without password

2019-03-19T13:57:46Z

Dawes001:

Secure shell (ssh) protocols can be configure to work without protocols. This is particularly helpful for machines that are used often.

== Configuring ssh without password from a POSIX-compliant terminal ==

=== Step 1: create a public key and copy to remote computer ===
* Log into a local Linux or MacOSX computer
* Type the following to generate the ssh key:
<source lang='bash'>
ssh-keygen -t dsa
</source>
* Accept the default key location by pressing <code>Enter</code>.
* Secure permission of your authentication keys by closing permission to your home directory, .ssh directory, and authentication files
<source lang='bash'>
chmod go-w $HOME
chmod 700 $HOME/.ssh
chmod go-rwx $HOME/.ssh/*
</source>
* Type the following to copy the key to the remote server (this will prompt for a password).
<source lang='bash'>
cd ~/.ssh
scp id_dsa.pub remote_username@remote_host:/tmp
</source>
* Type the following to add the ssh key to the remote user's authorization keys (this will prompt for a password).
ssh remote_username@remote_host 'cat /tmp/id_dsa.pub >>/home/

=== Step 2: configure the public key from the local computer on the remote computer ===
* log in to the remote computer using ssh (this will prompt for a password)
<source lang='bash'>
ssh remote_username@remote_host
</source>
* copy the public key generated on the local computer, that was copied to <code>/tmp</code> on the remote computer, is now appended to <code>.ssh/authorized_keys2</code>. Note that a file called <code>.ssh/authorized_keys</code> may already be present.
<source lang='bash'>
cat /tmp/id_dsa.pub >>$HOME/.ssh/authorized_keys2
</source>
* similar to your local computer, make sure the permission on your remote home folder, .ssh folder, and authentication files are properly set (NOTE: this may already be the case).
<source lang='bash'>
chmod go-w $HOME
chmod 700 $HOME/.ssh
chmod go-rwx $HOME/.ssh/*
</source>
* Lastly, remove the public key from the temporary folder.
<source lang='bash'>
rm /tmp/id_dsa.pub
</source>
* If you log out of the remote computer and back in again, from the local computer from which you copied the public key, you will notice you will no longer be prompted for a password.

== Configuring ssh without password for Anunna ==

* Create a public key as in Step 1 of the previous section and copy it to the B4F Cluster. Note that a public/private key pair needs to be made only once per machine.
* Similar to step 2 of the previous section, add the public key to the <code>$HOME/.ssh/authorized_keys2</code> file. There is already a <code>$HOME/.ssh/authorized_keys</code> present. You may append the key to this file as an alternative, but take care not to remove content that is already there. The cluster is configured so that passwordless communication will all other nodes is default.

== Configuring ssh without password using PuTTY ==
Use pAGEaNT: http://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter9.html to generate local keys. You'll want have a copy of the pubkey in plaintext available.

Make sure to paste that plaintext string into ~/.ssh/authorized_keys in one single line. Chmod the file 600 (so it shows -rw------- in ls -l)

Now PuTTY will login passwordlessly whenever pAGEaNT is running.

Finally, get pAGEaNT to load on startup: http://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html

== See also ==
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External Links ==