HPCwiki - User contributions [en]

Anunna

2013-12-02T17:54:24Z

Hjmegens: /* Filesystems */

The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS].

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.

{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.

{{-}}

== Architecture of the cluster ==

[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast internet connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}
=== Nodes ===
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is RHEL6. The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').

Information from the Cluster Management Portal, as it appeared on November 23, 2013:
<code>DEVICE INFORMATION
Hostname State Memory Cores CPU Speed GPU NICs IB Category
master1, master2 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1
node001..node042, node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 1200 MHz 3 1 default
node043..node048, node055..node060 DOWN N/A N/A N/A N/A N/A N/A N/A default
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 5 1 mds
storage01 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss
storage02..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 oss
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 7 1 login
fat001, fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2299 MHz 5 1 fat </code>

Main cluster node configuration:
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration
* The NFS server is a PowerEdge R720XD, which will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution. The NFS node will also act as a login node, where users log in and compile applications and submit jobs.
* 50 compute nodes
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each
Hyperthreading is disabled in compute nodes.

=== Filesystems ===

[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]
The B4F Cluster has two primary file systems, each with different properties and purposes.
==== Parallel File System: Lustre ====
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.

The hardware components of the PFS:
* 2x Dell PowerEdge R720
* 1x Dell PowerVault MD3220
* 6x Dell PowerEdge R620
* 6x Dell PowerVault MD3260

==== Network File System (NFS): $HOME dirs ====
Each user will have his/her own home directory. The path of the home directory will be:

/home/[name partner]/[username]

/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using:

<source lang='bash'>
quota
</source>

The NFS is supported through the NFS server that also serves as access point to the cluster.

Hardware components of the NFS:
* 1x Dell PowerEdge R720XD
* 1x Dell PowerVault MD3220

=== Network ===
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The B4F Cluster is housed at the main server centre of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-ICT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==

=== Project Leader ===
* Stephen Janssen (WUR-FB-ICT)

=== Daily Project Management ===
* [[User:Kmpollmann | Koen Pollmann (WUR-FB-ICT)]]
* Andre ten Böhmer (WUR-FB-ICT)

=== Steering Group ===
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication.
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)
* Petra Caessens (CAT-AgroFood)
* Wojtek Sablik (WUR-FB-ICT)
* Edda Neuteboom (CAT_AgroFood, secretariat)
* Johan van Arendonk (Wageningen UR, chair).

=== IT Workgroup ===
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.
* Stephen Janssen (WUR-FB-ICT)
* Koen Pollmann (WUR-FB-ICT)
* Andre ten Böhmer (WUR-FB-ICT)
* Wes Barris (Cobb)
* Ton Dings (Hendrix Genetics)
* Henk van Dongen (Topigs)
* Harry Dijkstra (CRV)
* Mario Calus (ABGC-WLR)
* Hendrik-Jan Megens (ABGC-ABG)
{{-}}

=== User Group ===
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.

== Access Policy ==
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

=== Contact Persons ===
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):

==== Cobb-Vantress ====
* Wes Barris
* Jun Chen

==== ABGC ====
===== Animal Breeding and Genetics =====
* [[User:Hulze001 |Alex Hulzebosch]]
* [[User:Hjmegens | Hendrik-Jan Megens]]

===== Wageningen Livestock Research =====
* Mario Calus
* Ina Hulsegge
==== CRV ====
* Frido Hamoen
* Chris Schrooten
==== Hendrix Genetics ====
* Ton Dings
* Abe Huisman
* Addie Vereijken
==== Topigs ====
* Henk van Dongen
* Egiel Hanenbarg
* Naomi Duijvensteijn

== Using the B4F Cluster ==
=== Gaining access to the B4F Cluster ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== See also ==
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"

| width="45%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]
| width="45%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
|}

Log in to Anunna

2013-12-02T15:11:32Z

Hjmegens: /* Log on using ssh */

== Log on using ssh ==
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:
nfs01.hpcagrogenomics.wur.nl

To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work.'''
'''Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@nfs01.hpcagrogenomics.wur.nl
</source>

=== PuTTY on Windows ===
<need a volunteer to write some text here>

== Log on to worker nodes ==

Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== winSCP ===
<need a windows user as a volunteer to write some text....>

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.

<is this going to be enabled on the Cluster?.

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Log in to Anunna

2013-12-02T15:10:57Z

Hjmegens: /* Log on using ssh */

== Log on using ssh ==
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:
nfs01.hpcagrogenomics.wur.nl

To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work. Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@nfs01.hpcagrogenomics.wur.nl
</source>

=== PuTTY on Windows ===
<need a volunteer to write some text here>

== Log on to worker nodes ==

Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== winSCP ===
<need a windows user as a volunteer to write some text....>

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.

<is this going to be enabled on the Cluster?.

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Log in to Anunna

2013-12-02T14:38:57Z

Hjmegens:

== Log on using ssh ==
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:
nfs01.hpcagrogenomics.wur.nl

To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work. Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@nfs01.hpcagrogenomics.wur.nl
</source>

=== PuTTY on Windows ===
<need a volunteer to write some text here>

== Log on to worker nodes ==

Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== winSCP ===
<need a windows user as a volunteer to write some text....>

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.

<is this going to be enabled on the Cluster?.

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Log in to Anunna

2013-12-02T14:36:41Z

Hjmegens: /* Log on using ssh */

== Log on using ssh ==
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:
nfs01.hpcagrogenomics.wur.nl

To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work. Anything requiring even moderate resources should be scheduled using SLURM!'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@nfs01.hpcagrogenomics.wur.nl
</source>

=== PuTTY on Windows ===
<need a volunteer to write some text here>

== Log on to worker nodes ==

Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== winSCP ===
<need a windows user as a volunteer to write some text....>

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.

<is this going to be enabled on the Cluster?.

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Control R environment using modules

2013-11-29T22:59:12Z

Hjmegens:

== Setting up a semi-public repository ==

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

== Create an R module ==

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>
Note: need to do this only once per version; make sure that the file does not already exist.

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile modified for WUR/ABGC use
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"
}

module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root

</source>

== Load Module ==

<source lang='bash'>
module avail
</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

<source lang='bash'>
module load R/3.0.2_wur
</source>

== Install R package and check install path ==

<source lang='rsplus'>
install.packages('car')
</source>

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

== See also ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==

Control R environment using modules

2013-11-29T22:57:56Z

Hjmegens:

Control R environment using modules

2013-11-29T22:56:22Z

Hjmegens:

== Setting up a semi-public repository ==

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

== Create an R module ==

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>
Note: need to do this only once per version; make sure that the file does not already exist.

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile modified for WUR/ABGC use
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"
}

module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root

</source>

== Load Module ==

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

<source lang='rsplus'>
install.packages('car')
</source>

== Install R package and check install path ==

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

== See also ==

== External links ==

Control R environment using modules

2013-11-29T22:54:12Z

Hjmegens:

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>
Note: need to do this only once per version; make sure that the file does not already exist.

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile modified for WUR/ABGC use
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"
}

module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root

</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

<source lang='rsplus'>
install.packages('car')
</source>

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

Control R environment using modules

2013-11-29T22:53:39Z

Hjmegens:

<source lang='bash'>
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>
Note: need to do this only once per version; make sure that the folder does not already exist.

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>
Note: need to do this only once per version; make sure that the file does not already exist.

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile modified for WUR/ABGC use
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"
}

module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root

</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

<source lang='rsplus'>
install.packages('car')
</source>

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

Control R environment using modules

2013-11-29T22:50:11Z

Hjmegens:

<source lang='bash'>
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/
</source>

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile modified for WUR/ABGC use
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"
}

module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root

</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

<source lang='rsplus'>
install.packages('car')
</source>

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

Control R environment using modules

2013-11-29T22:48:04Z

Hjmegens:

<source lang='bash'>
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/
</source>

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment"
}

module-whatis "Adds R v3.0.2 to your environment"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root
</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

<source lang='rsplus'>
install.packages('car')
</source>

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

Control R environment using modules

2013-11-29T22:47:21Z

Hjmegens:

<source lang='bash'>
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/
</source>

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment"
}

module-whatis "Adds R v3.0.2 to your environment"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root
</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

<source lang='r'>
install.packages('car')
</source>

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

Control R environment using modules

2013-11-29T22:46:02Z

Hjmegens:

<source lang='bash'>
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/
</source>

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment"
}

module-whatis "Adds R v3.0.2 to your environment"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root
</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

> installed.packages()
Package LibPath
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"

Control R environment using modules

2013-11-29T22:43:54Z

Hjmegens:

<source lang='bash'>
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/
</source>

<source lang='bash'>
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/
</source>

<source lang='bash'>
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur
</source>

<source lang='tcl'>
#%Module1.0#######################################################################
## R3 modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment"
}

module-whatis "Adds R v3.0.2 to your environment"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root
</source>

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur

Control R environment using modules

2013-11-29T22:42:57Z

Hjmegens:

Control R environment using modules

2013-11-29T22:41:57Z

Hjmegens: Created page with " /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/ mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/ cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur ..."

/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/

cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur

<source lang='tcl'>

#%Module1.0#######################################################################
## R3 modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds R v3.0.2 to your environment"
}

module-whatis "Adds R v3.0.2 to your environment"

set r3_root /cm/shared/apps/R3/
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/

prepend-path PATH $r3_root/bin
prepend-path MANPATH $r3_root/share/man
prepend-path LD_LIBRARY_PATH $r3_root/lib64
prepend-path R_LIBS_SITE $r3_wur_root
</source>

Installing R packages locally

2013-11-29T22:38:33Z

Hjmegens: /* See also */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Start up R:

<source lang='bash'>
R # Invoke R
</source>
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.
<source lang='rsplus'>
install.packages("name-of-your-package",lib="~/R/library")
</source>

== See also ==
* [[Control_R_environment_using_modules | Control R environment on the B4F Cluster using modules]]

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]

[http://cran-mirror.cs.uu.nl CRAN homepage]

[http://en.wikipedia.org/wiki/R_(programming_language) R article from Wikipedia]

Environment Modules

2013-11-29T20:47:42Z

Hjmegens: /* Loading modules */

== Loading modules ==
Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* http://modules.sourceforge.net
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Environment Modules

2013-11-29T20:47:02Z

Hjmegens: /* External links */

== Loading modules ==
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3
== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* http://modules.sourceforge.net
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Environment Modules

2013-11-29T20:46:40Z

Hjmegens:

== Loading modules ==
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3
== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
http://modules.sourceforge.net
http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Environment Modules

2013-11-29T20:45:19Z

Hjmegens: Created page with " == See also == * B4F Cluster == External links == http://modules.sourceforge.net http://www.admin-magazine.com/HPC/Articles/Environment-Modules"

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
http://modules.sourceforge.net
http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Setting local variables

2013-11-29T19:17:25Z

Hjmegens:

== Environment Variables ==

== Modify <code>.bashrc</code> and <code>.bash_profile</code> ==

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[Lx6_and_Lx7_compute_nodes | Lx6 and Lx7 compute nodes]]
* [[Create_shortcut_log-in_command | Create a shortcut for the ssh log-in command]]
* [[Installing_R_packages_locally | Installing R packages locally]]

== External links ==

Setting local variables

2013-11-29T19:15:13Z

Hjmegens:

== Environment Variables ==

== Modify <code>.bashrc</code> and <code>.bash_profile</code> ==

== See also ==

== External links ==

Setting local variables

2013-11-29T19:14:41Z

Hjmegens:

== Environment Variables ==

== Modify <code>.bashrc</code> and <code>.bash_profile</code> ==

Setting local variables

2013-11-29T19:14:17Z

Hjmegens: Created page with "== Environment Variables == == Modify <code>.bashrc</code> and <code>.basch_profile</code> =="

== Environment Variables ==

== Modify <code>.bashrc</code> and <code>.basch_profile</code> ==

Anunna

2013-11-29T18:00:51Z

Hjmegens: /* Installing software by users */

The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS].

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.

{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.

{{-}}

== Architecture of the cluster ==

[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast internet connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}
=== Nodes ===
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is RHEL6. The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').

Information from the Cluster Management Portal, as it appeared on November 23, 2013:
<code>DEVICE INFORMATION
Hostname State Memory Cores CPU Speed GPU NICs IB Category
master1, master2 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1
node001..node042, node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 1200 MHz 3 1 default
node043..node048, node055..node060 DOWN N/A N/A N/A N/A N/A N/A N/A default
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 5 1 mds
storage01 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss
storage02..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 oss
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 7 1 login
fat001, fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2299 MHz 5 1 fat </code>

Main cluster node configuration:
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration
* The NFS server is a PowerEdge R720XD, which will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution. The NFS node will also act as a login node, where users log in and compile applications and submit jobs.
* 50 compute nodes
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each
Hyperthreading is disabled in compute nodes.

=== Filesystems ===

[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]
The B4F Cluster has two primary file systems, each with different properties and purposes.
==== Parallel File System: Lustre ====
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.

The hardware components of the PFS:
* 2x Dell PowerEdge R720
* 1x Dell PowerVault MD3220
* 6x Dell PowerEdge R620
* 6x Dell PowerVault MD3260

==== Network File System (NFS): $HOME dirs ====
Each user will have his/her own home directory. The path of the home directory will be:

/home/[name partner]/[username]

/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily.

The NFS is supported through the NFS server that also serves as access point to the cluster.

Hardware components of the NFS:
* 1x Dell PowerEdge R720XD
* 1x Dell PowerVault MD3220

=== Network ===
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The B4F Cluster is housed at the main server centre of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-ICT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==

=== Project Leader ===
* Stephen Janssen (WUR-FB-ICT)

=== Daily Project Management ===
* [[User:Kmpollmann | Koen Pollmann (WUR-FB-ICT)]]
* Andre ten Böhmer (WUR-FB-ICT)

=== Steering Group ===
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication.
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)
* Petra Caessens (CAT-AgroFood)
* Wojtek Sablik (WUR-FB-ICT)
* Edda Neuteboom (CAT_AgroFood, secretariat)
* Johan van Arendonk (Wageningen UR, chair).

=== IT Workgroup ===
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.
* Stephen Janssen (WUR-FB-ICT)
* Koen Pollmann (WUR-FB-ICT)
* Andre ten Böhmer (WUR-FB-ICT)
* Wes Barris (Cobb)
* Ton Dings (Hendrix Genetics)
* Henk van Dongen (Topigs)
* Harry Dijkstra (CRV)
* Mario Calus (ABGC-WLR)
* Hendrik-Jan Megens (ABGC-ABG)
{{-}}

=== User Group ===
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.

== Access Policy ==
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

=== Contact Persons ===
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):

==== Cobb-Vantress ====
* Wes Barris
* Jun Chen

==== ABGC ====
===== Animal Breeding and Genetics =====
* [[User:Hulze001 |Alex Hulzebosch]]
* [[User:Hjmegens | Hendrik-Jan Megens]]

===== Wageningen Livestock Research =====
* Mario Calus
* Ina Hulsegge
==== CRV ====
* Frido Hamoen
* Chris Schrooten
==== Hendrix Genetics ====
* Ton Dings
* Abe Huisman
* Addie Vereijken
==== Topigs ====
* Henk van Dongen
* Egiel Hanenbarg
* Naomi Duijvensteijn

== Using the B4F Cluster ==
=== Gaining access to the B4F Cluster ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== See also ==
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"

| width="45%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]
| width="45%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
|}

Anunna

2013-11-29T18:00:06Z

Hjmegens: /* Using the B4F Cluster */

The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS].

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.

{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.

{{-}}

== Architecture of the cluster ==

[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast internet connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}
=== Nodes ===
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is RHEL6. The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').

Information from the Cluster Management Portal, as it appeared on November 23, 2013:
<code>DEVICE INFORMATION
Hostname State Memory Cores CPU Speed GPU NICs IB Category
master1, master2 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1
node001..node042, node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 1200 MHz 3 1 default
node043..node048, node055..node060 DOWN N/A N/A N/A N/A N/A N/A N/A default
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 5 1 mds
storage01 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss
storage02..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 oss
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 7 1 login
fat001, fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2299 MHz 5 1 fat </code>

Main cluster node configuration:
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration
* The NFS server is a PowerEdge R720XD, which will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution. The NFS node will also act as a login node, where users log in and compile applications and submit jobs.
* 50 compute nodes
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each
Hyperthreading is disabled in compute nodes.

=== Filesystems ===

[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]
The B4F Cluster has two primary file systems, each with different properties and purposes.
==== Parallel File System: Lustre ====
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.

The hardware components of the PFS:
* 2x Dell PowerEdge R720
* 1x Dell PowerVault MD3220
* 6x Dell PowerEdge R620
* 6x Dell PowerVault MD3260

==== Network File System (NFS): $HOME dirs ====
Each user will have his/her own home directory. The path of the home directory will be:

/home/[name partner]/[username]

/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily.

The NFS is supported through the NFS server that also serves as access point to the cluster.

Hardware components of the NFS:
* 1x Dell PowerEdge R720XD
* 1x Dell PowerVault MD3220

=== Network ===
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The B4F Cluster is housed at the main server centre of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-ICT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==

=== Project Leader ===
* Stephen Janssen (WUR-FB-ICT)

=== Daily Project Management ===
* [[User:Kmpollmann | Koen Pollmann (WUR-FB-ICT)]]
* Andre ten Böhmer (WUR-FB-ICT)

=== Steering Group ===
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication.
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)
* Petra Caessens (CAT-AgroFood)
* Wojtek Sablik (WUR-FB-ICT)
* Edda Neuteboom (CAT_AgroFood, secretariat)
* Johan van Arendonk (Wageningen UR, chair).

=== IT Workgroup ===
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.
* Stephen Janssen (WUR-FB-ICT)
* Koen Pollmann (WUR-FB-ICT)
* Andre ten Böhmer (WUR-FB-ICT)
* Wes Barris (Cobb)
* Ton Dings (Hendrix Genetics)
* Henk van Dongen (Topigs)
* Harry Dijkstra (CRV)
* Mario Calus (ABGC-WLR)
* Hendrik-Jan Megens (ABGC-ABG)
{{-}}

=== User Group ===
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.

== Access Policy ==
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

=== Contact Persons ===
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):

==== Cobb-Vantress ====
* Wes Barris
* Jun Chen

==== ABGC ====
===== Animal Breeding and Genetics =====
* [[User:Hulze001 |Alex Hulzebosch]]
* [[User:Hjmegens | Hendrik-Jan Megens]]

===== Wageningen Livestock Research =====
* Mario Calus
* Ina Hulsegge
==== CRV ====
* Frido Hamoen
* Chris Schrooten
==== Hendrix Genetics ====
* Ton Dings
* Abe Huisman
* Addie Vereijken
==== Topigs ====
* Henk van Dongen
* Egiel Hanenbarg
* Naomi Duijvensteijn

== Using the B4F Cluster ==
=== Gaining access to the B4F Cluster ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]

=== Installing software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== See also ==
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"

| width="45%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]
| width="45%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
|}

Using Slurm

2013-11-29T14:57:23Z

Hjmegens: /* Running MPI jobs on B4F cluster */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sallocate ==
< text here>

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

== Running MPI jobs on B4F cluster ==
< text here >

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:57:06Z

Hjmegens: /* allocating resources interactively: sallocate */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sallocate ==
< text here>

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

== Running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:56:48Z

Hjmegens: /* removing jobs from a list: scancel */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

== Running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:56:34Z

Hjmegens: /* running MPI jobs on B4F cluster */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

== Running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:56:19Z

Hjmegens: /* Get overview of past and current jobs: sacct */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:52:46Z

Hjmegens:

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== Get overview of past and current jobs: sacct ==
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:48:54Z

Hjmegens: /* Monitoring time limit set for a specific job */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:46:22Z

Hjmegens: /* Monitoring time limit set for a specific job */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
<source lang='bash'>
squeue -l -j 3532
</source>

Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-29T14:45:45Z

Hjmegens: /* monitoring submitted jobs: squeue */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
<source lang='bash'>
squeue -l -j 3532
<source>

Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Setting up Python virtualenv

2013-11-27T22:35:00Z

Hjmegens: /* creating a new virtual environment */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
virtualenv newenv
</source>

Should virtualenv not be installed, the virtualenv script can be downloaded and accessed directly:
<source lang='bash'>
curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz
tar -xzvf virtualenv-1.9.tar.gz
python3 virtualenv-1.9/virtualenv.py testenv
</source>

When the new environment is created, one will see a message similar to this:
<code>
New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.
</code>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(newenv)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>easy_install</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
easy_install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>
== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<code>
File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^
</code>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Using Slurm

2013-11-27T20:30:07Z

Hjmegens: /* Submitting jobs: sbatch */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the 'squeue' command:
squeue

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-27T20:10:08Z

Hjmegens: /* Batch script */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --time=1200
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:

<source lang='bash'>
sbatch run_calc_pi.sh
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the 'squeue' command:
squeue

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-27T20:09:27Z

Hjmegens: /* removing jobs from a list: scancel */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
# #SBATCH --time=100
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:

<source lang='bash'>
sbatch run_calc_pi.sh
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the 'squeue' command:
squeue

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== allocating resources interactively: sallocate ==

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2013-11-27T20:08:56Z

Hjmegens: /* monitoring submitted jobs: squeue */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
# #SBATCH --time=100
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=research

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:

<source lang='bash'>
sbatch run_calc_pi.sh
</source>

== monitoring submitted jobs: squeue ==
Once a job is submitted, the status can be monitored using the 'squeue' command:
squeue

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

== removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. The For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3347
</source>

== allocating resources interactively: sallocate ==

== running MPI jobs on B4F cluster ==

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Anunna

2013-11-27T19:56:30Z

Hjmegens: /* Nodes */

The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS].

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.

{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.

{{-}}

== Architecture of the cluster ==

[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast internet connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}
=== Nodes ===
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is RHEL6. The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').

Information from the Cluster Management Portal, as it appeared on November 23, 2013:
<code>DEVICE INFORMATION
Hostname State Memory Cores CPU Speed GPU NICs IB Category
master1, master2 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1
node001..node042, node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 1200 MHz 3 1 default
node043..node048, node055..node060 DOWN N/A N/A N/A N/A N/A N/A N/A default
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 5 1 mds
storage01 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss
storage02..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 oss
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 7 1 login
fat001, fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2299 MHz 5 1 fat </code>

Main cluster node configuration:
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration
* The NFS server is a PowerEdge R720XD, which will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution. The NFS node will also act as a login node, where users log in and compile applications and submit jobs.
* 50 compute nodes
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each
Hyperthreading is disabled in compute nodes.

=== Filesystems ===

[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]
The B4F Cluster has two primary file systems, each with different properties and purposes.
==== Parallel File System: Lustre ====
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.

The hardware components of the PFS:
* 2x Dell PowerEdge R720
* 1x Dell PowerVault MD3220
* 6x Dell PowerEdge R620
* 6x Dell PowerVault MD3260

==== Network File System (NFS): $HOME dirs ====
Each user will have his/her own home directory. The path of the home directory will be:

/home/[name partner]/[username]

/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily.

The NFS is supported through the NFS server that also serves as access point to the cluster.

Hardware components of the NFS:
* 1x Dell PowerEdge R720XD
* 1x Dell PowerVault MD3220

=== Network ===
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The B4F Cluster is housed at the main server centre of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-ICT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==

=== Project Leader ===
* Stephen Janssen (WUR-FB-ICT)

=== Daily Project Management ===
* [[User:Kmpollmann | Koen Pollmann (WUR-FB-ICT)]]
* Andre ten Böhmer (WUR-FB-ICT)

=== Steering Group ===
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication.
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)
* Petra Caessens (CAT-AgroFood)
* Wojtek Sablik (WUR-FB-ICT)
* Edda Neuteboom (CAT_AgroFood, secretariat)
* Johan van Arendonk (Wageningen UR, chair).

=== IT Workgroup ===
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.
* Stephen Janssen (WUR-FB-ICT)
* Koen Pollmann (WUR-FB-ICT)
* Andre ten Böhmer (WUR-FB-ICT)
* Wes Barris (Cobb)
* Ton Dings (Hendrix Genetics)
* Henk van Dongen (Topigs)
* Harry Dijkstra (CRV)
* Mario Calus (ABGC-WLR)
* Hendrik-Jan Megens (ABGC-ABG)
{{-}}

=== User Group ===
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.

== Access Policy ==
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

=== Contact Persons ===
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):

==== Cobb-Vantress ====
* Wes Barris
* Jun Chen

==== ABGC ====
===== Animal Breeding and Genetics =====
* [[User:Hulze001 |Alex Hulzebosch]]
* [[User:Hjmegens | Hendrik-Jan Megens]]

===== Wageningen Livestock Research =====
* Mario Calus
* Ina Hulsegge
==== CRV ====
* Frido Hamoen
* Chris Schrooten
==== Hendrix Genetics ====
* Ton Dings
* Abe Huisman
* Addie Vereijken
==== Topigs ====
* Henk van Dongen
* Egiel Hanenbarg
* Naomi Duijvensteijn

== Using the B4F Cluster ==
=== Gaining access to the B4F Cluster ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]

== See also ==
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"

| width="45%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]
| width="45%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
|}

Installing R packages locally

2013-11-26T20:30:01Z

Hjmegens: /* External links */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Start up R:

<source lang='bash'>
R # Invoke R
</source>
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.
<source lang='rsplus'>
install.packages("name-of-your-package",lib="~/R/library")
</source>

== See also ==

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]

[http://cran-mirror.cs.uu.nl CRAN homepage]

[http://en.wikipedia.org/wiki/R_(programming_language) R article from Wikipedia]

Installing R packages locally

2013-11-26T20:29:15Z

Hjmegens: /* External links */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Start up R:

<source lang='bash'>
R # Invoke R
</source>
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.
<source lang='rsplus'>
install.packages("name-of-your-package",lib="~/R/library")
</source>

== See also ==

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]
[http://cran-mirror.cs.uu.nl CRAN homepage]
[http://en.wikipedia.org/wiki/R_(programming_language) R article from Wikipedia]

Installing R packages locally

2013-11-26T20:28:55Z

Hjmegens: /* External links */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Start up R:

<source lang='bash'>
R # Invoke R
</source>
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.
<source lang='rsplus'>
install.packages("name-of-your-package",lib="~/R/library")
</source>

== See also ==

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]
[http://cran-mirror.cs.uu.nl CRAN homepage]
[http://en.wikipedia.org/wiki/R_(programming_language) | R article from Wikipedia]

Installing R packages locally

2013-11-26T20:27:30Z

Hjmegens: /* Installing to a local library search location */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Start up R:

<source lang='bash'>
R # Invoke R
</source>
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.
<source lang='rsplus'>
install.packages("name-of-your-package",lib="~/R/library")
</source>

== See also ==

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]

Installing R packages locally

2013-11-26T20:25:53Z

Hjmegens: /* Installing to a local library search location */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Installation is dead easy. Start up R and tell R to fetch your package from CRAN, compile whatever needs compiling and set everything else up.

Beware - each package will only work for the platform (i.e. Linux or Solaris) where you installed it. If you want a package on both Linux and Solaris, you'll need to install it in different directories for each system type.

<source lang='bash'>
R # Invoke R
</source>
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice:
<source lang='rsplus'>
install.packages("name-of-your-package",lib="~/R/library")
</source>

== See also ==

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]

Installing R packages locally

2013-11-26T20:23:54Z

Hjmegens: /* Installing to a local library search location */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Installation is dead easy. Start up R and tell R to fetch your package from CRAN, compile whatever needs compiling and set everything else up.

Beware - each package will only work for the platform (i.e. Linux or Solaris) where you installed it. If you want a package on both Linux and Solaris, you'll need to install it in different directories for each system type.

<source lang='bash'>
R # Invoke R
</source>
<source lang='rsplus'>
install.packages("name-of-your-package",lib="~/R/library")
</source>
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>

== See also ==

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]

Installing R packages locally

2013-11-26T20:22:47Z

Hjmegens: /* Installing to a local library search location */

== Specifying a local library search location ==

Specify a local library search location.

You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.

If necessary, create a place for your R libraries
<source lang='bash'>
mkdir ~/R ~/R/library # Only need do this once
</source>
Set your R library path
<source lang='bash'>
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron
</source>
== Installing to a local library search location ==

Installation is dead easy. Start up R and tell R to fetch your package from CRAN, compile whatever needs compiling and set everything else up.

Beware - each package will only work for the platform (i.e. Linux or Solaris) where you installed it. If you want a package on both Linux and Solaris, you'll need to install it in different directories for each system type.

<source lang='bash'>
R # Invoke R
</source>
<source lang='R'>
install.packages("name-of-your-package",lib="~/R/library")
</source>
<source lang='bash'>
R_LIBS_USER="~/R/library"
</source>

== See also ==

== External links ==
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]