https://wiki.anunna.wur.nl/api.php?action=feedcontributions&feedformat=atom&user=HjmegensHPCwiki - User contributions [en]2026-08-09T13:13:05ZUser contributionsMediaWiki 1.45.3https://wiki.anunna.wur.nl/index.php?title=SSH_Access&diff=807SSH Access2013-12-02T15:11:32Z<p>Hjmegens: /* Log on using ssh */</p>
<hr />
<div>== Log on using ssh ==<br />
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:<br />
nfs01.hpcagrogenomics.wur.nl<br />
<br />
To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].<br />
<br />
Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.<br />
<br />
The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.<br />
<br />
'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work.''' <br />
'''Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''<br />
<br />
=== CLI from a Linux/MacOSX terminal ===<br />
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.<br />
<br />
<source lang='bash'><br />
ssh [user name]@nfs01.hpcagrogenomics.wur.nl<br />
</source><br />
<br />
=== PuTTY on Windows ===<br />
<need a volunteer to write some text here><br />
<br />
== Log on to worker nodes ==<br />
<br />
Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].<br />
<br />
<source lang='bash'><br />
ssh [user name]@[node name]<br />
</source><br />
<br />
For instance:<br />
<br />
<source lang='bash'><br />
ssh dummy001@node049<br />
</source><br />
<br />
== File transfer using ssh-based file transfer protocols ==<br />
=== Copying files to/from the cluster: scp ===<br />
<br />
From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:<br />
<br />
Syntax of the scp command requires from-to order:<br />
<br />
<source lang='bash'><br />
scp <source> <destination><br />
</source><br />
<br />
<source lang='bash'><br />
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
<br />
This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== rsync ===<br />
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.<br />
<source lang='bash'><br />
rsync <source> <destination><br />
</source><br />
e.g.:<br />
<source lang='bash'><br />
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== winSCP ===<br />
<need a windows user as a volunteer to write some text....><br />
<br />
=== FileZilla ===<br />
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].<br />
<br />
=== Samba/CIFS based protocols ===<br />
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.<br />
<br />
<is this going to be enabled on the Cluster?.<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[ssh_without_password | ssh without password]]<br />
<br />
== External links ==<br />
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]<br />
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]<br />
* [http://winscp.net/eng/index.php WinSCP homepage]<br />
* [https://filezilla-project.org FileZilla homepage]<br />
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=SSH_Access&diff=806SSH Access2013-12-02T15:10:57Z<p>Hjmegens: /* Log on using ssh */</p>
<hr />
<div>== Log on using ssh ==<br />
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:<br />
nfs01.hpcagrogenomics.wur.nl<br />
<br />
To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].<br />
<br />
Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.<br />
<br />
The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.<br />
<br />
'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work. Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''<br />
<br />
=== CLI from a Linux/MacOSX terminal ===<br />
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.<br />
<br />
<source lang='bash'><br />
ssh [user name]@nfs01.hpcagrogenomics.wur.nl<br />
</source><br />
<br />
=== PuTTY on Windows ===<br />
<need a volunteer to write some text here><br />
<br />
== Log on to worker nodes ==<br />
<br />
Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].<br />
<br />
<source lang='bash'><br />
ssh [user name]@[node name]<br />
</source><br />
<br />
For instance:<br />
<br />
<source lang='bash'><br />
ssh dummy001@node049<br />
</source><br />
<br />
== File transfer using ssh-based file transfer protocols ==<br />
=== Copying files to/from the cluster: scp ===<br />
<br />
From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:<br />
<br />
Syntax of the scp command requires from-to order:<br />
<br />
<source lang='bash'><br />
scp <source> <destination><br />
</source><br />
<br />
<source lang='bash'><br />
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
<br />
This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== rsync ===<br />
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.<br />
<source lang='bash'><br />
rsync <source> <destination><br />
</source><br />
e.g.:<br />
<source lang='bash'><br />
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== winSCP ===<br />
<need a windows user as a volunteer to write some text....><br />
<br />
=== FileZilla ===<br />
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].<br />
<br />
=== Samba/CIFS based protocols ===<br />
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.<br />
<br />
<is this going to be enabled on the Cluster?.<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[ssh_without_password | ssh without password]]<br />
<br />
== External links ==<br />
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]<br />
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]<br />
* [http://winscp.net/eng/index.php WinSCP homepage]<br />
* [https://filezilla-project.org FileZilla homepage]<br />
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=SSH_Access&diff=805SSH Access2013-12-02T14:38:57Z<p>Hjmegens: </p>
<hr />
<div>== Log on using ssh ==<br />
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:<br />
nfs01.hpcagrogenomics.wur.nl<br />
<br />
To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].<br />
<br />
Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.<br />
<br />
The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.<br />
<br />
'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work. Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''<br />
<br />
=== CLI from a Linux/MacOSX terminal ===<br />
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.<br />
<br />
<source lang='bash'><br />
ssh [user name]@nfs01.hpcagrogenomics.wur.nl<br />
</source><br />
<br />
=== PuTTY on Windows ===<br />
<need a volunteer to write some text here><br />
<br />
== Log on to worker nodes ==<br />
<br />
Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].<br />
<br />
<source lang='bash'><br />
ssh [user name]@[node name]<br />
</source><br />
<br />
For instance:<br />
<br />
<source lang='bash'><br />
ssh dummy001@node049<br />
</source><br />
<br />
== File transfer using ssh-based file transfer protocols ==<br />
=== Copying files to/from the cluster: scp ===<br />
<br />
From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:<br />
<br />
Syntax of the scp command requires from-to order:<br />
<br />
<source lang='bash'><br />
scp <source> <destination><br />
</source><br />
<br />
<source lang='bash'><br />
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
<br />
This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== rsync ===<br />
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.<br />
<source lang='bash'><br />
rsync <source> <destination><br />
</source><br />
e.g.:<br />
<source lang='bash'><br />
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== winSCP ===<br />
<need a windows user as a volunteer to write some text....><br />
<br />
=== FileZilla ===<br />
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].<br />
<br />
=== Samba/CIFS based protocols ===<br />
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.<br />
<br />
<is this going to be enabled on the Cluster?.<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[ssh_without_password | ssh without password]]<br />
<br />
== External links ==<br />
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]<br />
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]<br />
* [http://winscp.net/eng/index.php WinSCP homepage]<br />
* [https://filezilla-project.org FileZilla homepage]<br />
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=SSH_Access&diff=804SSH Access2013-12-02T14:36:41Z<p>Hjmegens: /* Log on using ssh */</p>
<hr />
<div>== Log on using ssh ==<br />
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh. The address of the nfs server is:<br />
nfs01.hpcagrogenomics.wur.nl<br />
<br />
To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].<br />
<br />
Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.<br />
<br />
The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.<br />
<br />
'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work. Anything requiring even moderate resources should be scheduled using SLURM!'''<br />
<br />
=== CLI from a Linux/MacOSX terminal ===<br />
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.<br />
<br />
<source lang='bash'><br />
ssh [user name]@nfs01.hpcagrogenomics.wur.nl<br />
</source><br />
<br />
=== PuTTY on Windows ===<br />
<need a volunteer to write some text here><br />
<br />
== Log on to worker nodes ==<br />
<br />
Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].<br />
<br />
<source lang='bash'><br />
ssh [user name]@[node name]<br />
</source><br />
<br />
For instance:<br />
<br />
<source lang='bash'><br />
ssh dummy001@node049<br />
</source><br />
<br />
== File transfer using ssh-based file transfer protocols ==<br />
=== Copying files to/from the cluster: scp ===<br />
<br />
From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:<br />
<br />
Syntax of the scp command requires from-to order:<br />
<br />
<source lang='bash'><br />
scp <source> <destination><br />
</source><br />
<br />
<source lang='bash'><br />
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
<br />
This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== rsync ===<br />
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.<br />
<source lang='bash'><br />
rsync <source> <destination><br />
</source><br />
e.g.:<br />
<source lang='bash'><br />
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/<br />
</source><br />
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].<br />
<source lang='bash'><br />
man scp<br />
</source><br />
<br />
=== winSCP ===<br />
<need a windows user as a volunteer to write some text....><br />
<br />
=== FileZilla ===<br />
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].<br />
<br />
=== Samba/CIFS based protocols ===<br />
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing.<br />
<br />
<is this going to be enabled on the Cluster?.<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[ssh_without_password | ssh without password]]<br />
<br />
== External links ==<br />
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]<br />
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]<br />
* [http://winscp.net/eng/index.php WinSCP homepage]<br />
* [https://filezilla-project.org FileZilla homepage]<br />
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=785Control R environment using modules2013-11-29T22:59:12Z<p>Hjmegens: </p>
<hr />
<div>== Setting up a semi-public repository ==<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
== Create an R module ==<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
Note: need to do this only once per version; make sure that the file does not already exist.<br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile modified for WUR/ABGC use<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
<br />
</source><br />
<br />
== Load Module ==<br />
<br />
<source lang='bash'><br />
module avail<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='bash'><br />
module load R/3.0.2_wur<br />
</source><br />
<br />
== Install R package and check install path ==<br />
<br />
<source lang='rsplus'><br />
install.packages('car')<br />
</source><br />
<br />
<br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"<br />
<br />
== See also ==<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=784Control R environment using modules2013-11-29T22:57:56Z<p>Hjmegens: </p>
<hr />
<div>== Setting up a semi-public repository ==<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
== Create an R module ==<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
Note: need to do this only once per version; make sure that the file does not already exist.<br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile modified for WUR/ABGC use<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
<br />
</source><br />
<br />
== Load Module ==<br />
<br />
<source lang='bash'><br />
module avail<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='bash'><br />
module load R/3.0.2_wur<br />
</source><br />
<br />
== Install R package and check install path ==<br />
<br />
<source lang='rsplus'><br />
install.packages('car')<br />
</source><br />
<br />
<br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"<br />
<br />
== See also ==<br />
<br />
== External links ==</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=783Control R environment using modules2013-11-29T22:56:22Z<p>Hjmegens: </p>
<hr />
<div>== Setting up a semi-public repository ==<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
== Create an R module ==<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
Note: need to do this only once per version; make sure that the file does not already exist.<br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile modified for WUR/ABGC use<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
<br />
</source><br />
<br />
== Load Module ==<br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='rsplus'><br />
install.packages('car')<br />
</source><br />
<br />
== Install R package and check install path ==<br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"<br />
<br />
== See also ==<br />
<br />
== External links ==</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=782Control R environment using modules2013-11-29T22:54:12Z<p>Hjmegens: </p>
<hr />
<div><br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
Note: need to do this only once per version; make sure that the file does not already exist.<br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile modified for WUR/ABGC use<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='rsplus'><br />
install.packages('car')<br />
</source><br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=781Control R environment using modules2013-11-29T22:53:39Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
Note: need to do this only once per version; make sure that the folder does not already exist.<br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
Note: need to do this only once per version; make sure that the file does not already exist.<br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile modified for WUR/ABGC use<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='rsplus'><br />
install.packages('car')<br />
</source><br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=780Control R environment using modules2013-11-29T22:50:11Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
</source><br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile modified for WUR/ABGC use<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment and sets WUR/ABGC libraries"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment and set WUR/ABGC libraries"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='rsplus'><br />
install.packages('car')<br />
</source><br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=779Control R environment using modules2013-11-29T22:48:04Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
</source><br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='rsplus'><br />
install.packages('car')<br />
</source><br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=778Control R environment using modules2013-11-29T22:47:21Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
</source><br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
<source lang='r'><br />
install.packages('car')<br />
</source><br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=777Control R environment using modules2013-11-29T22:46:02Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
</source><br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur<br />
<br />
> installed.packages()<br />
Package LibPath <br />
car "car" "/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2"</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=776Control R environment using modules2013-11-29T22:43:54Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
</source><br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
</source><br />
<br />
----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------<br />
bwa/0.5.9 bwa/0.7.5a R/3.0.2_wur</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=775Control R environment using modules2013-11-29T22:42:57Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
</source><br />
<br />
<source lang='bash'><br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
</source><br />
<br />
<source lang='bash'><br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
</source><br />
<br />
<source lang='tcl'><br />
#%Module1.0#######################################################################<br />
## R3 modulefile<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
</source></div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Control_R_environment_using_modules&diff=774Control R environment using modules2013-11-29T22:41:57Z<p>Hjmegens: Created page with " /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/ mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/ cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur ..."</p>
<hr />
<div><br />
/cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
mkdir /cm/shared/apps/WUR/ABGC/modulefiles/R/<br />
<br />
cp /cm/shared/modulefiles/R/3.0.2 /cm/shared/apps/WUR/ABGC/modulefiles/R/3.0.2_wur<br />
<br />
<source lang='tcl'><br />
<br />
#%Module1.0#######################################################################<br />
## R3 modulefile<br />
##<br />
proc ModulesHelp { } {<br />
<br />
puts stderr "\tAdds R v3.0.2 to your environment"<br />
}<br />
<br />
module-whatis "Adds R v3.0.2 to your environment"<br />
<br />
set r3_root /cm/shared/apps/R3/<br />
set r3_wur_root /cm/shared/apps/WUR/ABGC/R/R_libraries/3.0.2/<br />
<br />
prepend-path PATH $r3_root/bin<br />
prepend-path MANPATH $r3_root/share/man<br />
prepend-path LD_LIBRARY_PATH $r3_root/lib64<br />
prepend-path R_LIBS_SITE $r3_wur_root<br />
</source></div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=773Installing R packages locally2013-11-29T22:38:33Z<p>Hjmegens: /* See also */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Start up R:<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.<br />
<source lang='rsplus'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<br />
== See also ==<br />
* [[Control_R_environment_using_modules | Control R environment on the B4F Cluster using modules]]<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]<br />
<br />
[http://cran-mirror.cs.uu.nl CRAN homepage]<br />
<br />
[http://en.wikipedia.org/wiki/R_(programming_language) R article from Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Environment_Modules&diff=766Environment Modules2013-11-29T20:47:42Z<p>Hjmegens: /* Loading modules */</p>
<hr />
<div><br />
== Loading modules ==<br />
Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
<br />
== External links ==<br />
* http://modules.sourceforge.net<br />
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Environment_Modules&diff=765Environment Modules2013-11-29T20:47:02Z<p>Hjmegens: /* External links */</p>
<hr />
<div><br />
== Loading modules ==<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
<br />
== External links ==<br />
* http://modules.sourceforge.net<br />
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Environment_Modules&diff=764Environment Modules2013-11-29T20:46:40Z<p>Hjmegens: </p>
<hr />
<div><br />
== Loading modules ==<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
<br />
== External links ==<br />
http://modules.sourceforge.net<br />
http://www.admin-magazine.com/HPC/Articles/Environment-Modules</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Environment_Modules&diff=763Environment Modules2013-11-29T20:45:19Z<p>Hjmegens: Created page with " == See also == * B4F Cluster == External links == http://modules.sourceforge.net http://www.admin-magazine.com/HPC/Articles/Environment-Modules"</p>
<hr />
<div><br />
<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
<br />
== External links ==<br />
http://modules.sourceforge.net<br />
http://www.admin-magazine.com/HPC/Articles/Environment-Modules</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Setting_local_variables&diff=758Setting local variables2013-11-29T19:17:25Z<p>Hjmegens: </p>
<hr />
<div>== Environment Variables ==<br />
<br />
== Modify <code>.bashrc</code> and <code>.bash_profile</code> ==<br />
<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[Lx6_and_Lx7_compute_nodes | Lx6 and Lx7 compute nodes]]<br />
* [[Create_shortcut_log-in_command | Create a shortcut for the ssh log-in command]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
<br />
== External links ==</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Setting_local_variables&diff=757Setting local variables2013-11-29T19:15:13Z<p>Hjmegens: </p>
<hr />
<div>== Environment Variables ==<br />
<br />
== Modify <code>.bashrc</code> and <code>.bash_profile</code> ==<br />
<br />
<br />
== See also ==<br />
<br />
== External links ==</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Setting_local_variables&diff=756Setting local variables2013-11-29T19:14:41Z<p>Hjmegens: </p>
<hr />
<div>== Environment Variables ==<br />
<br />
== Modify <code>.bashrc</code> and <code>.bash_profile</code> ==</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Setting_local_variables&diff=755Setting local variables2013-11-29T19:14:17Z<p>Hjmegens: Created page with "== Environment Variables == == Modify <code>.bashrc</code> and <code>.basch_profile</code> =="</p>
<hr />
<div>== Environment Variables ==<br />
<br />
== Modify <code>.bashrc</code> and <code>.basch_profile</code> ==</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=752Scheduler Overview (Slurm)2013-11-29T14:57:23Z<p>Hjmegens: /* Running MPI jobs on B4F cluster */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=751Scheduler Overview (Slurm)2013-11-29T14:57:06Z<p>Hjmegens: /* allocating resources interactively: sallocate */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=750Scheduler Overview (Slurm)2013-11-29T14:56:48Z<p>Hjmegens: /* removing jobs from a list: scancel */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=749Scheduler Overview (Slurm)2013-11-29T14:56:34Z<p>Hjmegens: /* running MPI jobs on B4F cluster */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=748Scheduler Overview (Slurm)2013-11-29T14:56:19Z<p>Hjmegens: /* Get overview of past and current jobs: sacct */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=747Scheduler Overview (Slurm)2013-11-29T14:52:46Z<p>Hjmegens: </p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== Get overview of past and current jobs: sacct ==<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0 <br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=746Scheduler Overview (Slurm)2013-11-29T14:48:54Z<p>Hjmegens: /* Monitoring time limit set for a specific job */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=745Scheduler Overview (Slurm)2013-11-29T14:46:22Z<p>Hjmegens: /* Monitoring time limit set for a specific job */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=744Scheduler Overview (Slurm)2013-11-29T14:45:45Z<p>Hjmegens: /* monitoring submitted jobs: squeue */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
<source><br />
<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Setting_up_Python_virtualenv&diff=743Setting up Python virtualenv2013-11-27T22:35:00Z<p>Hjmegens: /* creating a new virtual environment */</p>
<hr />
<div>With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary. <br />
<br />
== creating a new virtual environment ==<br />
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:<br />
<source lang='bash'><br />
virtualenv newenv<br />
</source><br />
<br />
Should virtualenv not be installed, the virtualenv script can be downloaded and accessed directly:<br />
<source lang='bash'><br />
curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz<br />
tar -xzvf virtualenv-1.9.tar.gz <br />
python3 virtualenv-1.9/virtualenv.py testenv<br />
</source><br />
<br />
When the new environment is created, one will see a message similar to this:<br />
<code><br />
New python executable in newenv/bin/python3<br />
Also creating executable in newenv/bin/python<br />
Installing Setuptools.........................................................................done.<br />
Installing Pip................................................................................done.<br />
</code><br />
<br />
== activating a virtual environment ==<br />
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:<br />
<source lang='bash'><br />
source newenv/bin/activate<br />
</source><br />
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.<br />
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.<br />
<code><br />
(newenv)user@host:~$<br />
</code><br />
<br />
== installing modules on the virtual environment ==<br />
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>easy_install</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.<br />
<source lang='bash'><br />
easy_install numpy<br />
</source><br />
Similarly, installing packages from source works exactly the same as usual.<br />
<source lang='bash'><br />
python setup.py install<br />
</source><br />
<br />
== deactivating a virtual environment ==<br />
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.<br />
<source lang='bash'><br />
deactivate<br />
</source><br />
== Make IPython work under virtualenv ==<br />
IPython may not work initially under a virtual environment. It may produce an error message like below:<br />
<br />
<code><br />
File "/usr/bin/ipython", line 11<br />
print "Could not start qtconsole. Please install ipython-qtconsole"<br />
^<br />
</code><br />
<br />
This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.<br />
<source lang='bash'><br />
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython<br />
</source><br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
<br />
== External links ==<br />
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]<br />
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=742Scheduler Overview (Slurm)2013-11-27T20:30:07Z<p>Hjmegens: /* Submitting jobs: sbatch */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the 'squeue' command:<br />
squeue<br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=741Scheduler Overview (Slurm)2013-11-27T20:10:08Z<p>Hjmegens: /* Batch script */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the 'squeue' command:<br />
squeue<br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=740Scheduler Overview (Slurm)2013-11-27T20:09:27Z<p>Hjmegens: /* removing jobs from a list: scancel */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
# #SBATCH --time=100<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the 'squeue' command:<br />
squeue<br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Scheduler_Overview_(Slurm)&diff=739Scheduler Overview (Slurm)2013-11-27T20:08:56Z<p>Hjmegens: /* monitoring submitted jobs: squeue */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
# #SBATCH --time=100<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=research<br />
<br />
time python3 calc_pi.py<br />
</source><br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the 'squeue' command:<br />
squeue<br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
== removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. The For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3347<br />
</source><br />
<br />
== allocating resources interactively: sallocate ==<br />
<br />
== running MPI jobs on B4F cluster ==<br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=727Installing R packages locally2013-11-26T20:30:01Z<p>Hjmegens: /* External links */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Start up R:<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.<br />
<source lang='rsplus'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]<br />
<br />
[http://cran-mirror.cs.uu.nl CRAN homepage]<br />
<br />
[http://en.wikipedia.org/wiki/R_(programming_language) R article from Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=726Installing R packages locally2013-11-26T20:29:15Z<p>Hjmegens: /* External links */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Start up R:<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.<br />
<source lang='rsplus'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]<br />
[http://cran-mirror.cs.uu.nl CRAN homepage]<br />
[http://en.wikipedia.org/wiki/R_(programming_language) R article from Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=725Installing R packages locally2013-11-26T20:28:55Z<p>Hjmegens: /* External links */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Start up R:<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.<br />
<source lang='rsplus'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]<br />
[http://cran-mirror.cs.uu.nl CRAN homepage]<br />
[http://en.wikipedia.org/wiki/R_(programming_language) | R article from Wikipedia]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=724Installing R packages locally2013-11-26T20:27:30Z<p>Hjmegens: /* Installing to a local library search location */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Start up R:<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice. This example will install from CRAN.<br />
<source lang='rsplus'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=723Installing R packages locally2013-11-26T20:25:53Z<p>Hjmegens: /* Installing to a local library search location */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Installation is dead easy. Start up R and tell R to fetch your package from CRAN, compile whatever needs compiling and set everything else up.<br />
<br />
Beware - each package will only work for the platform (i.e. Linux or Solaris) where you installed it. If you want a package on both Linux and Solaris, you'll need to install it in different directories for each system type.<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
Then, from the R environment, install the packages you require while pointing at the root R-package directory of choice:<br />
<source lang='rsplus'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=722Installing R packages locally2013-11-26T20:23:54Z<p>Hjmegens: /* Installing to a local library search location */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Installation is dead easy. Start up R and tell R to fetch your package from CRAN, compile whatever needs compiling and set everything else up.<br />
<br />
Beware - each package will only work for the platform (i.e. Linux or Solaris) where you installed it. If you want a package on both Linux and Solaris, you'll need to install it in different directories for each system type.<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
<source lang='rsplus'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=721Installing R packages locally2013-11-26T20:22:47Z<p>Hjmegens: /* Installing to a local library search location */</p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Installation is dead easy. Start up R and tell R to fetch your package from CRAN, compile whatever needs compiling and set everything else up.<br />
<br />
Beware - each package will only work for the platform (i.e. Linux or Solaris) where you installed it. If you want a package on both Linux and Solaris, you'll need to install it in different directories for each system type.<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
</source><br />
<source lang='R'><br />
install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=720Installing R packages locally2013-11-26T20:21:59Z<p>Hjmegens: </p>
<hr />
<div><br />
<br />
== Specifying a local library search location ==<br />
<br />
Specify a local library search location.<br />
<br />
You can use several library trees of add-on packages. The easiest way to tell R to use these via a 'dotfile' by creating the following file '$HOME/.Renviron' (watch the quotes and ~ character):<br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
This specifies a keyword (<code>R_LIBS_USER</code>) which points to a colon-separated list of directories at which R library trees are rooted. You do not have to specify the default tree for R packages.<br />
<br />
If necessary, create a place for your R libraries<br />
<source lang='bash'><br />
mkdir ~/R ~/R/library # Only need do this once<br />
</source><br />
Set your R library path<br />
<source lang='bash'><br />
echo 'R_LIBS_USER="~/R/library"' > $HOME/.Renviron<br />
</source><br />
== Installing to a local library search location ==<br />
<br />
Installation is dead easy. Start up R and tell R to fetch your package from CRAN, compile whatever needs compiling and set everything else up.<br />
<br />
Beware - each package will only work for the platform (i.e. Linux or Solaris) where you installed it. If you want a package on both Linux and Solaris, you'll need to install it in different directories for each system type.<br />
<br />
<source lang='bash'><br />
R # Invoke R<br />
<source lang='bash'><br />
> install.packages("name-of-your-package",lib="~/R/library")<br />
</source><br />
<source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=719Installing R packages locally2013-11-26T20:18:42Z<p>Hjmegens: /* External links */</p>
<hr />
<div><source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html source for material for this aricle]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=718Installing R packages locally2013-11-26T20:18:31Z<p>Hjmegens: /* External links */</p>
<hr />
<div><source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
[http://csg.sph.umich.edu/docs/R/localpackages.html | source for material for this aricle]</div>Hjmegenshttps://wiki.anunna.wur.nl/index.php?title=Installing_R_packages_locally&diff=717Installing R packages locally2013-11-26T15:00:57Z<p>Hjmegens: </p>
<hr />
<div><source lang='bash'><br />
R_LIBS_USER="~/R/library"<br />
</source><br />
<br />
== See also ==<br />
<br />
== External links ==<br />
http://csg.sph.umich.edu/docs/R/localpackages.html</div>Hjmegens