https://wiki.anunna.wur.nl/api.php?action=feedcontributions&feedformat=atom&user=Duque004HPCwiki - User contributions [en]2026-04-17T22:38:17ZUser contributionsMediaWiki 1.43.1https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1922Shared folders2019-01-30T11:19:30Z<p>Duque004: </p>
<hr />
<div><br />
== Working with shared folders in the Lustre file system ==<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
== Identifying a suitable user group ==<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
<code>groups <username></code><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
<code>groups duque004 | grep isric</code><br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
== Creating a shared folder with correct permissions ==<br />
<br />
The Lustre file system is accessible in the <code>/lustre</code> folder and then divided into the <code>/backup</code> and <code>/nobackup</code> sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named <code>SHARED</code> in which users are to create their own assets.<br />
<br />
You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:<br />
<br />
<code>mkdir /lustre/nobackup/SHARED/myTeamWorkspace</code> <br />
<br />
Or in alternative:<br />
<br />
<code>cd /lustre/nobackup/SHARED<br />
<br />
mkdir myTeamWorkspace</code> <br />
<br />
== Setting permissions ==<br />
<br />
Three basic steps are involved in stepping permissions correctly:<br />
<br />
1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:<br />
<br />
<code>chgrp -R my-team-group myTeamWorkspace</code><br />
<br />
2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the <code>w</code> character from the <code>+rw</code> bit:<br />
<br />
<code>chmod -R g+rw myTeamWorkspace</code><br />
<br />
3. Set default ownership within the group. This guarantees that any new files or folders created within the shared folder are owned by default owned by your team group:<br />
<br />
<code>chmod -R g+s myTeamWorkspace</code><br />
<br />
In case the contents of the shared are sensitive or private, and should be accessed by your team, you can block access from any other users with the following command:<br />
<br />
<code>chmod -R o-rw myTeamWorkspace</code><br />
<br />
== Further reading ==<br />
<br />
[https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-permissions An Introduction to Linux Permissions]<br />
<br />
[https://www.linode.com/docs/tools-reference/linux-users-and-groups/ Linux Users and Groups]<br />
<br />
[https://www.digitalocean.com/community/tutorials/linux-permissions-basics-and-how-to-use-umask-on-a-vps#types-of-permissions Linux Permissions Basics and How to Use Umask on a VPS]<br />
<br />
[http://www.yolinux.com/TUTORIALS/LinuxTutorialManagingGroups.html Linux Tutorial - Managing Group Access on Linux and UNIX]</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1921Shared folders2019-01-30T11:18:02Z<p>Duque004: </p>
<hr />
<div><br />
= Working with shared folders in the Lustre file system =<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
== Identifying a suitable user group ==<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
<code>groups <username></code><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
<code>groups duque004 | grep isric</code><br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
== Creating a shared folder with correct permissions ==<br />
<br />
The Lustre file system is accessible in the <code>/lustre</code> folder and then divided into the <code>/backup</code> and <code>/nobackup</code> sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named <code>SHARED</code> in which users are to create their own assets.<br />
<br />
You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:<br />
<br />
<code>mkdir /lustre/nobackup/SHARED/myTeamWorkspace</code> <br />
<br />
Or in alternative:<br />
<br />
<code>cd /lustre/nobackup/SHARED<br />
<br />
mkdir myTeamWorkspace</code> <br />
<br />
== Setting permissions ==<br />
<br />
Three basic steps are involved in stepping permissions correctly:<br />
<br />
1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:<br />
<br />
<code>chgrp -R my-team-group myTeamWorkspace</code><br />
<br />
2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the <code>w</code> character from the <code>+rw</code> bit:<br />
<br />
<code>chmod -R g+rw myTeamWorkspace</code><br />
<br />
3. Set default ownership within the group. This guarantees that any new files or folders created within the shared folder are owned by default owned by your team group:<br />
<br />
<code>chmod -R g+s myTeamWorkspace</code><br />
<br />
In case the contents of the shared are sensitive or private, and should be accessed by your team, you can block access from any other users with the following command:<br />
<br />
<code>chmod -R o-rw myTeamWorkspace</code><br />
<br />
== Further reading ==<br />
<br />
[https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-permissions An Introduction to Linux Permissions]<br />
<br />
[https://www.linode.com/docs/tools-reference/linux-users-and-groups/ Linux Users and Groups]<br />
<br />
[https://www.digitalocean.com/community/tutorials/linux-permissions-basics-and-how-to-use-umask-on-a-vps#types-of-permissions Linux Permissions Basics and How to Use Umask on a VPS]<br />
<br />
[http://www.yolinux.com/TUTORIALS/LinuxTutorialManagingGroups.html Linux Tutorial - Managing Group Access on Linux and UNIX]</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1920Shared folders2019-01-30T11:16:28Z<p>Duque004: /* Further reading */</p>
<hr />
<div><br />
== Working with shared folders in the Lustre file system ==<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
=== Identifying a suitable user group ===<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
<code>groups <username></code><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
<code>groups duque004 | grep isric</code><br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
=== Creating a shared folder with correct permissions ===<br />
<br />
The Lustre file system is accessible in the <code>/lustre</code> folder and then divided into the <code>/backup</code> and <code>/nobackup</code> sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named <code>SHARED</code> in which users are to create their own assets.<br />
<br />
You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:<br />
<br />
<code>mkdir /lustre/nobackup/SHARED/myTeamWorkspace</code> <br />
<br />
Or in alternative:<br />
<br />
<code>cd /lustre/nobackup/SHARED<br />
<br />
mkdir myTeamWorkspace</code> <br />
<br />
=== Setting permissions ===<br />
<br />
Three basic steps are involved in stepping permissions correctly:<br />
<br />
1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:<br />
<br />
<code>chgrp -R my-team-group myTeamWorkspace</code><br />
<br />
2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the <code>w</code> character from the <code>+rw</code> bit:<br />
<br />
<code>chmod -R g+rw myTeamWorkspace</code><br />
<br />
3. Set default ownership within the group. This guarantees that any new files or folders created within the shared folder are owned by default owned by your team group:<br />
<br />
<code>chmod -R g+s myTeamWorkspace</code><br />
<br />
In case the contents of the shared are sensitive or private, and should be accessed by your team, you can block access from any other users with the following command:<br />
<br />
<code>chmod -R o-rw myTeamWorkspace</code><br />
<br />
=== Further reading ===<br />
<br />
[https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-permissions An Introduction to Linux Permissions]<br />
<br />
[https://www.linode.com/docs/tools-reference/linux-users-and-groups/ Linux Users and Groups]<br />
<br />
[https://www.digitalocean.com/community/tutorials/linux-permissions-basics-and-how-to-use-umask-on-a-vps#types-of-permissions Linux Permissions Basics and How to Use Umask on a VPS]<br />
<br />
[http://www.yolinux.com/TUTORIALS/LinuxTutorialManagingGroups.html Linux Tutorial - Managing Group Access on Linux and UNIX]</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1919Shared folders2019-01-30T11:05:35Z<p>Duque004: /* Setting permissions */</p>
<hr />
<div><br />
== Working with shared folders in the Lustre file system ==<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
=== Identifying a suitable user group ===<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
<code>groups <username></code><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
<code>groups duque004 | grep isric</code><br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
=== Creating a shared folder with correct permissions ===<br />
<br />
The Lustre file system is accessible in the <code>/lustre</code> folder and then divided into the <code>/backup</code> and <code>/nobackup</code> sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named <code>SHARED</code> in which users are to create their own assets.<br />
<br />
You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:<br />
<br />
<code>mkdir /lustre/nobackup/SHARED/myTeamWorkspace</code> <br />
<br />
Or in alternative:<br />
<br />
<code>cd /lustre/nobackup/SHARED<br />
<br />
mkdir myTeamWorkspace</code> <br />
<br />
=== Setting permissions ===<br />
<br />
Three basic steps are involved in stepping permissions correctly:<br />
<br />
1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:<br />
<br />
<code>chgrp -R my-team-group myTeamWorkspace</code><br />
<br />
2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the <code>w</code> character from the <code>+rw</code> bit:<br />
<br />
<code>chmod -R g+rw myTeamWorkspace</code><br />
<br />
3. Set default ownership within the group. This guarantees that any new files or folders created within the shared folder are owned by default owned by your team group:<br />
<br />
<code>chmod -R g+s myTeamWorkspace</code><br />
<br />
In case the contents of the shared are sensitive or private, and should be accessed by your team, you can block access from any other users with the following command:<br />
<br />
<code>chmod -R o-rw myTeamWorkspace</code><br />
<br />
=== Further reading ===</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1918Shared folders2019-01-30T10:53:57Z<p>Duque004: /* Creating a shared folder with correct permissions */</p>
<hr />
<div><br />
== Working with shared folders in the Lustre file system ==<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
=== Identifying a suitable user group ===<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
<code>groups <username></code><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
<code>groups duque004 | grep isric</code><br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
=== Creating a shared folder with correct permissions ===<br />
<br />
The Lustre file system is accessible in the <code>/lustre</code> folder and then divided into the <code>/backup</code> and <code>/nobackup</code> sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named <code>SHARED</code> in which users are to create their own assets.<br />
<br />
You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:<br />
<br />
<code>mkdir /lustre/nobackup/SHARED/myTeamWorkspace</code> <br />
<br />
Or in alternative:<br />
<br />
<code>cd /lustre/nobackup/SHARED<br />
<br />
mkdir myTeamWorkspace</code> <br />
<br />
=== Setting permissions ===<br />
<br />
Three basic steps are involved in stepping permissions correctly:<br />
<br />
1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:<br />
<br />
<code>chgrp -R my-team-group myTeamWorkspace</code><br />
<br />
2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the <code>w</code> character from the <code>+rw</code> bit:<br />
<br />
<code>chmod -R g+rw myTeamWorkspace</code><br />
<br />
3. Set default ownership within the group. This guarantees that any new files or folders created within the folder are owned by your team group by default:<br />
<br />
<code>chmod -R g+s myTeamWorkspace</code><br />
<br />
=== Further reading ===</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1917Shared folders2019-01-30T10:53:04Z<p>Duque004: /* Creating a shared folder in Lustre */</p>
<hr />
<div><br />
== Working with shared folders in the Lustre file system ==<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
=== Identifying a suitable user group ===<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
<code>groups <username></code><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
<code>groups duque004 | grep isric</code><br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
=== Creating a shared folder with correct permissions ===<br />
<br />
The Lustre file system is accessible in the <code>/lustre</code> folder and then divided into the <code>/backup</code> and <code>/nobackup</code> sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named <code>SHARED</code> in which users are to create their own assets.<br />
<br />
You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:<br />
<br />
<code>mkdir /lustre/nobackup/SHARED/myTeamWorkspace</code> <br />
<br />
Or in alternative:<br />
<br />
<code>cd /lustre/nobackup/SHARED<br />
<br />
mkdir myTeamWorkspace</code> <br />
<br />
Now come the permissions. Three basic steps are involved:<br />
<br />
1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:<br />
<br />
<code>chgrp -R my-team-group myTeamWorkspace</code><br />
<br />
2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the <code>w</code> character from the <code>+rw</code> bit:<br />
<br />
<code>chmod -R g+rw myTeamWorkspace</code><br />
<br />
3. Set default ownership within the group. This guarantees that any new files or folders created within the folder are owned by your team group by default:<br />
<br />
<code>chmod -R g+s myTeamWorkspace</code><br />
<br />
=== Further reading ===</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1916Shared folders2019-01-30T10:37:06Z<p>Duque004: /* Identifying a suitable user group */</p>
<hr />
<div><br />
== Working with shared folders in the Lustre file system ==<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
=== Identifying a suitable user group ===<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
<code>groups <username></code><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
<code>groups duque004 | grep isric</code><br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
=== Creating a shared folder in Lustre ===<br />
<br />
The</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Shared_folders&diff=1915Shared folders2019-01-30T10:36:21Z<p>Duque004: Created page with " == Working with shared folders in the Lustre file system == If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User ca..."</p>
<hr />
<div><br />
== Working with shared folders in the Lustre file system ==<br />
<br />
If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.<br />
<br />
=== Identifying a suitable user group ===<br />
<br />
Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:<br />
<br />
$ groups <username><br />
<br />
This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.<br />
<br />
For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:<br />
<br />
$ groups duque004 | grep isric<br />
<br />
In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.<br />
<br />
=== Creating a shared folder in Lustre ===<br />
<br />
The</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1914Main Page2019-01-30T10:16:43Z<p>Duque004: </p>
<hr />
<div>The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR. <br />
<br />
The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan). <br />
<br />
= Using the HPC-Ag =<br />
== Gaining access to the HPC-Ag ==<br />
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
* [[Services | Alternative access methods, and extra features and services on the HPC]]<br />
* [[Filesystems | Accessible storage methods on the HPC]]<br />
<br />
== Access Policy ==<br />
[[Access_Policy | Main Article: Access Policy]]<br />
<br />
Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of the HPC-Ag is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.<br />
<br />
= Events =<br />
* [[Courses]] that have happened and are happening<br />
* [[Downtime]] that will affect all users<br />
* [[Meetings]] that may affect the policies of the HPC<br />
<br />
= Other Software =<br />
<br />
== Cluster Management Software and Scheduler ==<br />
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
== Installation of software by users ==<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]<br />
* [[Installing WRF and WPS]]<br />
<br />
== Installed software ==<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
= Useful Notes = <br />
<br />
== Being in control of Environment parameters ==<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== Controlling costs ==<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]<br />
<br />
== Management ==<br />
[[HPC_management | Main Article: HPC management]]<br />
<br />
Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:lith010 | Jan van Lith (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].<br />
<br />
= Miscellaneous =<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[History_of_the_Cluster | Historical information on the startup of the HPC]]<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]<br />
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]<br />
* [[Manual GitLab | GitLab: Create projects and add scripts]]<br />
* [[Monitoring_executions | Monitoring job execution]]<br />
* [[Shared_folders | Working with shared folders in the Lustre file system]]<br />
<br />
= See also =<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[BCData | BCData]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
= External links =<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1903Monitoring job execution2018-08-27T13:26:26Z<p>Duque004: /* Output stream redirection */</p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the ''stdout'' and ''stderr'' streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the <code>tail</code> command is particularly useful. To continuously follow the output to a text file use:<br />
<br />
<code>tail -f output_987654.txt</code><br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
<code>tail -n X output_987654.txt</code><br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands <code>cat</code> and <code>less</code>:<br />
<br />
<code>cat output_987654.txt | less</code><br />
<br />
Use the Q key to exit <code>less</code>.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it start by changing your SLURM script so that your programme is launched with the <code>srun</code> command (this should be the last line in the script):<br />
<br />
<code>srun python3 calc_pi.py</code><br />
<br />
Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:<br />
<br />
<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code><br />
<br />
<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way is to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name): <br />
<br />
<code>srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py</code><br />
<br />
This will log the output of <code>top</code> to the sdtout stream file specific in the SLURM script every 3 seconds. Using the <code>tail</code> command you will be able to see logs like:<br />
<br />
<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</pre></div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1902Monitoring job execution2018-08-27T13:25:24Z<p>Duque004: /* Using top */</p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the <code>tail</code> command is particularly useful. To continuously follow the output to a text file use:<br />
<br />
<code>tail -f output_987654.txt</code><br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
<code>tail -n X output_987654.txt</code><br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
<code>cat output_987654.txt | less</code><br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it start by changing your SLURM script so that your programme is launched with the <code>srun</code> command (this should be the last line in the script):<br />
<br />
<code>srun python3 calc_pi.py</code><br />
<br />
Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:<br />
<br />
<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code><br />
<br />
<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way is to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name): <br />
<br />
<code>srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py</code><br />
<br />
This will log the output of <code>top</code> to the sdtout stream file specific in the SLURM script every 3 seconds. Using the <code>tail</code> command you will be able to see logs like:<br />
<br />
<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</pre></div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1901Monitoring job execution2018-08-27T13:24:15Z<p>Duque004: /* Using sstat */</p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the <code>tail</code> command is particularly useful. To continuously follow the output to a text file use:<br />
<br />
<code>tail -f output_987654.txt</code><br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
<code>tail -n X output_987654.txt</code><br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
<code>cat output_987654.txt | less</code><br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it start by changing your SLURM script so that your programme is launched with the <code>srun</code> command (this should be the last line in the script):<br />
<br />
<code>srun python3 calc_pi.py</code><br />
<br />
Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:<br />
<br />
<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code><br />
<br />
<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name): <br />
<br />
<code>srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py</code><br />
<br />
This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:<br />
<br />
<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</pre></div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1900Monitoring job execution2018-08-27T13:22:52Z<p>Duque004: /* Output stream redirection */</p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the <code>tail</code> command is particularly useful. To continuously follow the output to a text file use:<br />
<br />
<code>tail -f output_987654.txt</code><br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
<code>tail -n X output_987654.txt</code><br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
<code>cat output_987654.txt | less</code><br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the <code>srun</code> command (this should be the last line in the script):<br />
<br />
<code>srun python3 calc_pi.py</code><br />
<br />
Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:<br />
<br />
<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code><br />
<br />
<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name): <br />
<br />
<code>srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py</code><br />
<br />
This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:<br />
<br />
<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</pre></div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_executions&diff=1899Monitoring executions2018-08-23T09:38:24Z<p>Duque004: Duque004 moved page Monitoring executions to Monitoring job execution</p>
<hr />
<div>#REDIRECT [[Monitoring job execution]]</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1898Monitoring job execution2018-08-23T09:38:24Z<p>Duque004: Duque004 moved page Monitoring executions to Monitoring job execution</p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the <code>tail</code> command is particularly useful. To continuous follow the output to a text file use:<br />
<br />
<code>tail -f output_987654.txt</code><br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
<code>tail -n X output_987654.txt</code><br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
<code>cat output_987654.txt | less</code><br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the <code>srun</code> command (this should be the last line in the script):<br />
<br />
<code>srun python3 calc_pi.py</code><br />
<br />
Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:<br />
<br />
<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code><br />
<br />
<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name): <br />
<br />
<code>srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py</code><br />
<br />
This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:<br />
<br />
<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</pre></div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1897Monitoring job execution2018-08-23T09:37:17Z<p>Duque004: </p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the <code>tail</code> command is particularly useful. To continuous follow the output to a text file use:<br />
<br />
<code>tail -f output_987654.txt</code><br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
<code>tail -n X output_987654.txt</code><br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
<code>cat output_987654.txt | less</code><br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the <code>srun</code> command (this should be the last line in the script):<br />
<br />
<code>srun python3 calc_pi.py</code><br />
<br />
Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:<br />
<br />
<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code><br />
<br />
<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name): <br />
<br />
<code>srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py</code><br />
<br />
This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:<br />
<br />
<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</pre></div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1896Monitoring job execution2018-08-23T09:36:34Z<p>Duque004: </p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the <code>tail command is particularly useful. To continuous follow the output to a text file use:<br />
<br />
<code>tail -f output_987654.txt</code><br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
<code>tail -n X output_987654.txt</code><br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
<code>cat output_987654.txt | less</code><br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the <code>srun</code> command (this should be the last line in the script):<br />
<br />
<code>srun python3 calc_pi.py</code><br />
<br />
Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:<br />
<br />
<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code><br />
<br />
<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name): <br />
<br />
<code>srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py</code><br />
<br />
This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:<br />
<br />
<code>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</code></div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1895Monitoring job execution2018-08-23T09:32:26Z<p>Duque004: </p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].<br />
<br />
For this purpose the `tail`command is particularly useful. To continuous follow the output to a text file use:<br />
<br />
tail -f output_987654.txt<br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
tail -n X output_987654.txt<br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
cat output_987654.txt | less<br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
sstat is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the srun command (this should be the last line in the script):<br />
<br />
srun python3 calc_pi.py<br />
<br />
Note down the job number. During execution you can then use sstat, passing the job number with the -j flag:<br />
<br />
sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654<br />
<br />
sstat can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].<br />
<br />
=== Using top ===<br />
<br />
Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html top] command. This requires adding an extra command to your SLURM script, again using srun (where "user001" should be replaced by your user name): <br />
<br />
srun --overcommit --ntasks=1 top -b -u user001 &<br />
python3 calc_pi.py<br />
<br />
This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:<br />
<br />
top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53<br />
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie<br />
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st<br />
KiB Mem : 10439453+total, 18650486+free, 43690281+used, 42053763+buff/cache<br />
KiB Swap: 26214400+total, 26214393+free, 52 used. 54182425+avail Mem <br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R<br />
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top<br />
28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+<br />
29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun<br />
29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1894Monitoring job execution2018-08-23T09:13:02Z<p>Duque004: </p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [[https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file | text files specified in the SLURM script]].<br />
<br />
For this purpose the `tail`command is particularly useful. To continuous follow the output to a text file use:<br />
<br />
tail -f output_987654.txt<br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
tail -n X output_987654.txt<br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
cat output_987654.txt | less<br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
sstat is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the srun command (this should be the last line in the script):<br />
<br />
srun python3 calc_pi.py<br />
<br />
Note down the job number. During execution you can then use sstat, passing the job number with the -j flag:<br />
<br />
sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654<br />
<br />
sstat can provide information on many different variables, for more details check [[https://slurm.schedmd.com/sstat.html | the manual]].<br />
<br />
=== Logging the output of top ===</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Monitoring_job_execution&diff=1893Monitoring job execution2018-08-23T09:07:19Z<p>Duque004: Created page with " == Output stream redirection == The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to https://wiki.hpcagrogenomics.wur..."</p>
<hr />
<div><br />
== Output stream redirection ==<br />
<br />
The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [[https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file | text files specified in the SLURM script]].<br />
<br />
For this purpose the `tail`command is particularly useful. To continuous follow the output to a text file use:<br />
<br />
tail -f output_987654.txt<br />
<br />
To obtain the last X lines of a text file use:<br />
<br />
tail -n X output_987654.txt<br />
<br />
Replacing X by the desired number of lines.<br />
<br />
If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:<br />
<br />
cat output_987654.txt | less<br />
<br />
Use the Q key to exit less.<br />
<br />
== Monitoring resource usage ==<br />
<br />
While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.<br />
<br />
=== Using sstat ===<br />
<br />
sstat is a SLURM tool that can be used to obtain instantaneous <br />
<br />
srun python3 calc_pi.py<br />
<br />
sstat --format=AveCPU,AveRSS,MaxRSS -P -j 7466208<br />
<br />
=== Logging the output of top ===</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1892Main Page2018-08-23T08:40:03Z<p>Duque004: Added link to monitor execution page</p>
<hr />
<div>The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR. <br />
<br />
The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan). <br />
<br />
= Using the HPC-Ag =<br />
== Gaining access to the HPC-Ag ==<br />
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
* [[Services | Alternative access methods, and extra features and services on the HPC]]<br />
* [[Filesystems | Accessible storage methods on the HPC]]<br />
<br />
== Access Policy ==<br />
[[Access_Policy | Main Article: Access Policy]]<br />
<br />
Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of the HPC-Ag is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.<br />
<br />
= Events =<br />
* [[Courses]] that have happened and are happening<br />
* [[Downtime]] that will affect all users<br />
* [[Meetings]] that may affect the policies of the HPC<br />
<br />
= Other Software =<br />
<br />
== Cluster Management Software and Scheduler ==<br />
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
== Installation of software by users ==<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]<br />
* [[Installing WRF and WPS]]<br />
<br />
== Installed software ==<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
= Useful Notes = <br />
<br />
== Being in control of Environment parameters ==<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== Controlling costs ==<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]<br />
<br />
== Management ==<br />
[[HPC_management | Main Article: HPC management]]<br />
<br />
Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:lith010 | Jan van Lith (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].<br />
<br />
= Miscellaneous =<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[History_of_the_Cluster | Historical information on the startup of the HPC]]<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]<br />
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]<br />
* [[Manual GitLab | GitLab: Create projects and add scripts]]<br />
* [[Monitoring_executions | Monitoring job execution]]<br />
<br />
= See also =<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[BCData | BCData]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
= External links =<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Duque004https://wiki.anunna.wur.nl/index.php?title=SLURM_Compare&diff=1886SLURM Compare2018-08-01T13:45:35Z<p>Duque004: /* Rosetta Stone of Workload Managers */</p>
<hr />
<div>=== Rosetta Stone of Workload Managers ===<br />
<br />
PBS/Torque, Slurm, LSF, SGE and LoadLeveler [http://slurm.schedmd.com/rosetta.html Rosetta Stone]<br />
<br />
This table lists the most common command, environment variables, and job specification options used by the major workload management systems: PBS/Torque, Slurm, LSF, SGE and LoadLeveler. Each of these workload managers has unique features, but the most commonly used functionality is available in all of these environments as listed in the table. This should be considered a work in progress and contributions to improve the document are welcome.<br />
{| class="wikitable"<br />
!User Commands!!PBS/Torque!!Slurm!!LSF!!SGE!!LoadLeveler<br />
|-<br />
||Job submission|| qsub [script_file]|| sbatch [script_file]|| bsub [script_file]|| qsub [script_file]|| llsubmit [script_file] <br />
|-<br />
||Job deletion ||qdel [job_id]|| scancel [job_id]|| bkill [job_id]|| qdel [job_id]|| llcancel [job_id] <br />
|-<br />
||Job status (by job)|| qstat [job_id]|| squeue --job [job_id]|| bjobs [job_id]|| qstat -u \* [-j job_id]|| llq -u [username] <br />
|-<br />
||Job status (by user)|| qstat -u [user_name]|| squeue -u [user_name]|| bjobs -u [user_name]|| qstat [-u user_name]|| llq -u [user_name] <br />
|-<br />
||Job hold ||qhold [job_id]|| scontrol hold [job_id]|| bstop [job_id]|| qhold [job_id]|| llhold -r [job_id] <br />
|-<br />
||Job release|| qrls [job_id]|| scontrol release [job_id]|| bresume [job_id]|| qrls [job_id]|| llhold -r [job_id] <br />
|-<br />
||Queue list|| qstat -Q|| squeue ||bqueues ||qconf -sql|| llclass <br />
|-<br />
||Node list ||pbsnodes -l|| sinfo -N OR scontrol show nodes|| bhosts|| qhost|| llstatus -L machine <br />
|-<br />
||Cluster status|| qstat -a|| sinfo|| bqueues|| qhost -q|| llstatus -L cluster <br />
|-<br />
||GUI|| xpbsmon|| sview|| xlsf OR xlsbatch|| qmon|| xload <br />
|-<br />
|-<br />
||'''Environment'''||'''PBS/Torque'''||'''Slurm'''||'''LSF'''||'''SGE'''||'''LoadLeveler''' <br />
|-<br />
||Job ID|| $PBS_JOBID|| $SLURM_JOBID|| $LSB_JOBID|| $JOB_ID|| $LOAD_STEP_ID <br />
|-<br />
||Submit Directory|| $PBS_O_WORKDIR|| $SLURM_SUBMIT_DIR|| $LSB_SUBCWD|| $SGE_O_WORKDIR|| $LOADL_STEP_INITDIR <br />
|-<br />
||Submit Host|| $PBS_O_HOST|| $SLURM_SUBMIT_HOST|| $LSB_SUB_HOST|| $SGE_O_HOST ||<br />
|-<br />
||Node List|| $PBS_NODEFILE|| $SLURM_JOB_NODELIST|| $LSB_HOSTS/LSB_MCPU_HOST|| $PE_HOSTFILE|| $LOADL_PROCESSOR_LIST <br />
|-<br />
||Job Array Index|| $PBS_ARRAYID|| $SLURM_ARRAY_TASK_ID|| $LSB_JOBINDEX|| $SGE_TASK_ID ||<br />
|-<br />
|-<br />
|-<br />
|}</div>Duque004https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1861Using Slurm2018-04-19T13:05:31Z<p>Duque004: /* Batch script */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue.<br><br />
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0)<br><br />
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are occupied by a Low queue jobs.<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_High up infinite 12 down* node[043-048,055-060]<br />
ABGC_High up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_High up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Std up infinite 12 down* node[043-048,055-060]<br />
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Std up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Low up infinite 12 down* node[043-048,055-060]<br />
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Low up infinite 44 idle node[001,006-042,049-054]<br />
</source><br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 100MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
[[Creating_sbatch_script | Main Article: Creating a sbatch script]]<br />
<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC_Std<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs (simple) ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Submitting array jobs ===<br />
<source lang='bash'><br />
#SBATCH --array=0-10%4<br />
</source><br />
SLURM allows you to submit multiple jobs using the same template. Further information about this can be found [[Array_jobs|here]].<br />
<br />
=== Using /tmp ===<br />
There is a local disk of ~300G that can be used to temporarily stage some of your workload attached to each node. This is free to use, but please remember to clean up your data after usage.<br />
<br />
In order to be sure that you're able to use space in /tmp, you can add<br />
<source lang='bash'><br />
#SBATCH --tmp=<required size><br />
</source><br />
To your sbatch script. This will prevent your job from being run on nodes where there is no free space, or it's aimed to be used by another job at the same time.<br />
<br />
== Monitoring submitted jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
=== Check on a pending job ===<br />
A submitted job could result in a pending state when there are not enough resources available to this job.<br />
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.<br />
<source lang='bash'><br />
[@nfs01 jobs]$ sbatch hpl_student.job<br />
Submitted batch job 740338<br />
<br />
[@nfs01 jobs]$ squeue -l -j 740338<br />
Fri Feb 21 15:32:31 2014<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)<br />
<br />
[@nfs01 jobs]$ squeue --start -j 740338<br />
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)<br />
</source><br />
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: salloc ==<br />
It's possible to set up an interactive session using salloc. Run salloc as follows:<br />
<source lang='bash'><br />
salloc -p <partition, say, ABGC_Low><br />
</source><br />
And because of the magic of SallocDefaultCommand, you will immediately be transported to a new prompt.<br />
<br />
Here, run 'hostname' to see which node your shell has been transported to.<br />
<br />
If you don't want your shell to be transported but want a new remote shell, do:<br />
<source lang='bash'><br />
salloc -p ABGC_Low $SHELL<br />
</source><br />
Now your shell will stay on nfs01, but you can do:<br />
<source lang='bash'><br />
srun <command> &<br />
</source><br />
To submit tasks to this new shell!<br />
<br />
Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html<br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Duque004