HPCwiki - User contributions [en]

Main Page

2020-03-02T16:51:32Z

Megen002: /* Using Anunna */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:bexke002 | Stefan Bexkens (Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2020-03-02T16:50:52Z

Megen002: /* Using Anunna */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.

== testing ==
* [[New_page | my shiny new test page]]

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:bexke002 | Stefan Bexkens (Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

New page

2020-03-02T16:49:20Z

Megen002: Created page with "Now here is some stuff"

Now here is some stuff

Creating sbatch script

2017-06-07T14:26:35Z

Megen002: /* receiving mailed updates */

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
##SBATCH --account=
#-----------------------------Required resources-----------------------
#SBATCH --partition=
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting the normal or fat nodes===
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.

===adding a job name===
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.

===partition for resource allocation===
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.

===receiving mailed updates===
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=yourname001@wur.nl
</source>
Email address to use.

== See also ==
* [[B4F_cluster | HPC Agrogenomics Cluster]]
* [[SLURM_on_B4F_cluster#Batch_script | Redirect to SLURM on B4F cluster]]

Creating sbatch script

2017-06-07T14:25:52Z

Megen002: /* time limit */

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
##SBATCH --account=
#-----------------------------Required resources-----------------------
#SBATCH --partition=
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting the normal or fat nodes===
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.

===adding a job name===
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.

===partition for resource allocation===
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.

===receiving mailed updates===
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

== See also ==
* [[B4F_cluster | HPC Agrogenomics Cluster]]
* [[SLURM_on_B4F_cluster#Batch_script | Redirect to SLURM on B4F cluster]]

Creating sbatch script

2017-06-07T14:24:34Z

Megen002: /* output (stderr,stdout) directed to file */

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
##SBATCH --account=
#-----------------------------Required resources-----------------------
#SBATCH --partition=
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting the normal or fat nodes===
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.

===adding a job name===
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.

===partition for resource allocation===
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.

===receiving mailed updates===
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

== See also ==
* [[B4F_cluster | HPC Agrogenomics Cluster]]
* [[SLURM_on_B4F_cluster#Batch_script | Redirect to SLURM on B4F cluster]]

Creating sbatch script

2017-06-07T14:22:39Z

Megen002: /* Explanation of used SBATCH parameters */

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
##SBATCH --account=
#-----------------------------Required resources-----------------------
#SBATCH --partition=
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.

===time limit===
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

===memory limit===
<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

===number of tasks===
<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.

===constraints: selecting the normal or fat nodes===
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

===output (stderr,stdout) directed to file===
<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>

===adding a job name===
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.
<source lang='bash'>

===partition for resource allocation===
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.
<source lang='bash'>

===receiving mailed updates===
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

== See also ==
* [[B4F_cluster | HPC Agrogenomics Cluster]]
* [[SLURM_on_B4F_cluster#Batch_script | Redirect to SLURM on B4F cluster]]

Creating sbatch script

2017-06-07T14:17:44Z

Megen002:

== A skeleton Slurm script ==
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
##SBATCH --account=
#-----------------------------Required resources-----------------------
#SBATCH --partition=
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

==Explanation of used SBATCH parameters==
=== Adding accounting information or project number ===
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

----

<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:
<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

----

<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

== See also ==
* [[B4F_cluster | HPC Agrogenomics Cluster]]
* [[SLURM_on_B4F_cluster#Batch_script | Redirect to SLURM on B4F cluster]]

Using Slurm

2017-06-07T14:12:21Z

Megen002: /* Batch script */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Queues ===
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue. 
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0) 
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are occupied by a Low queue jobs.
To find out which queues your account has been authorized for, type sinfo:
<source lang='bash'>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
ABGC_High up infinite 12 down* node[043-048,055-060]
ABGC_High up infinite 6 mix fat[001-002],node[002-005]
ABGC_High up infinite 44 idle node[001,006-042,049-054]
ABGC_Std up infinite 12 down* node[043-048,055-060]
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]
ABGC_Std up infinite 44 idle node[001,006-042,049-054]
ABGC_Low up infinite 12 down* node[043-048,055-060]
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]
ABGC_Low up infinite 44 idle node[001,006-042,049-054]
</source>

=== Defaults ===
There is no default queue, so you need to specify which queue to use when submitting a job. 
'''The default run time for a job is 1 hour!''' 
'''Default memory limit is 100MB per node!'''

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a batch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --account=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=ABGC_Std
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
nfs01 ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=nfs01:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@nfs01 jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@nfs01 jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@nfs01 jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: salloc ==
It's possible to set up an interactive session using salloc. Run salloc as follows:
<source lang='bash'>
salloc -p <partition, say, ABGC_Low>
</source>
And because of the magic of SallocDefaultCommand, you will immediately be transported to a new prompt.

Here, run 'hostname' to see which node your shell has been transported to.

If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on nfs01, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on B4F cluster ==

[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]
< text here >

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Creating sbatch script

2017-06-07T14:12:05Z

Megen002:

A skeleton Slurm script:
<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
##SBATCH --account=
#-----------------------------Required resources-----------------------
#SBATCH --partition=
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

Explanation of used SBATCH parameters:
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

----

<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:
<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

----

<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

Creating sbatch script

2017-06-07T13:06:27Z

Megen002:

Creating sbatch script

2017-06-07T13:06:08Z

Megen002: Created page with "A skeletion Slurm script: <source lang='bash'> #-----------------------------Mail address----------------------------- #SBATCH --mail-user= #SBATCH --mail-type=ALL #-------..."

A skeletion Slurm script:

<source lang='bash'>

#-----------------------------Mail address-----------------------------
#SBATCH --mail-user=
#SBATCH --mail-type=ALL
#-----------------------------Output files-----------------------------
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#-----------------------------Other information------------------------
#SBATCH --comment=
##SBATCH --account=
#-----------------------------Required resources-----------------------
#SBATCH --partition=
#SBATCH --time=0-0:0:0
#SBATCH --ntasks=
#SBATCH --cpus-per-task=
#SBATCH --mem-per-cpu=

#-----------------------------Environment, Operations and Job steps----
#load modules

#export variables

#your job

</source>

Using Slurm

2017-06-07T13:01:18Z

Megen002: /* Batch script */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Queues ===
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue. 
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0) 
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are occupied by a Low queue jobs.
To find out which queues your account has been authorized for, type sinfo:
<source lang='bash'>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
ABGC_High up infinite 12 down* node[043-048,055-060]
ABGC_High up infinite 6 mix fat[001-002],node[002-005]
ABGC_High up infinite 44 idle node[001,006-042,049-054]
ABGC_Std up infinite 12 down* node[043-048,055-060]
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]
ABGC_Std up infinite 44 idle node[001,006-042,049-054]
ABGC_Low up infinite 12 down* node[043-048,055-060]
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]
ABGC_Low up infinite 44 idle node[001,006-042,049-054]
</source>

=== Defaults ===
There is no default queue, so you need to specify which queue to use when submitting a job. 
'''The default run time for a job is 1 hour!''' 
'''Default memory limit is 100MB per node!'''

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
[[Creating_sbatch_script | Main Article: Creating a batch script]]

The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --account=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=ABGC_Std
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>
Explanation of used SBATCH parameters:
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

----

<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:
<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

----

<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
nfs01 ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=nfs01:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@nfs01 jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@nfs01 jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@nfs01 jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: salloc ==
It's possible to set up an interactive session using salloc. Run salloc as follows:
<source lang='bash'>
salloc -p <partition, say, ABGC_Low>
</source>
And because of the magic of SallocDefaultCommand, you will immediately be transported to a new prompt.

Here, run 'hostname' to see which node your shell has been transported to.

If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on nfs01, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on B4F cluster ==

[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]
< text here >

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Using Slurm

2017-06-07T12:53:07Z

Megen002: /* Submitting jobs: sbatch */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Queues ===
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue. 
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0) 
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are occupied by a Low queue jobs.
To find out which queues your account has been authorized for, type sinfo:
<source lang='bash'>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
ABGC_High up infinite 12 down* node[043-048,055-060]
ABGC_High up infinite 6 mix fat[001-002],node[002-005]
ABGC_High up infinite 44 idle node[001,006-042,049-054]
ABGC_Std up infinite 12 down* node[043-048,055-060]
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]
ABGC_Std up infinite 44 idle node[001,006-042,049-054]
ABGC_Low up infinite 12 down* node[043-048,055-060]
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]
ABGC_Low up infinite 44 idle node[001,006-042,049-054]
</source>

=== Defaults ===
There is no default queue, so you need to specify which queue to use when submitting a job. 
'''The default run time for a job is 1 hour!''' 
'''Default memory limit is 100MB per node!'''

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --account=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=ABGC_Std
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>
Explanation of used SBATCH parameters:
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the <tt>scontrol</tt> command. For WUR users a projectnumber or KTP number would be advisable.
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

----

<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:
<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

----

<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
nfs01 ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=nfs01:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@nfs01 jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@nfs01 jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@nfs01 jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: salloc ==
It's possible to set up an interactive session using salloc. Run salloc as follows:
<source lang='bash'>
salloc -p <partition, say, ABGC_Low>
</source>
And because of the magic of SallocDefaultCommand, you will immediately be transported to a new prompt.

Here, run 'hostname' to see which node your shell has been transported to.

If you don't want your shell to be transported but want a new remote shell, do:
<source lang='bash'>
salloc -p ABGC_Low $SHELL
</source>
Now your shell will stay on nfs01, but you can do:
<source lang='bash'>
srun <command> &
</source>
To submit tasks to this new shell!

Be aware that the time limit of salloc is default 1 hour. If you intend to run jobs for longer times than this, you need to edit the settings for it. See: https://computing.llnl.gov/linux/slurm/salloc.html

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on B4F cluster ==

[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]
< text here >

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

List of users

2017-06-07T12:45:34Z

Megen002: /* Alumni */

== Active users ==

List of users. Alphabetical order (last/family name). Provide some background information by adding information to the 'User:username' page.

* [[User:Barris01 | Wes Barris (Cobb)]]
* [[User:Basti015 | John Bastiaansen (WUR/ABGC)]]
* [[User:Binsb003 | Rianne van Binsbergen (WUR/ABGC)]]
* [[User:Bosse014 | Mirte Bosse (WUR/ABGC)]]
* [[User:Bouwm024 | Aniek Bouwman (WUR/ABGC)]]
* [[User:Brasc001 | Pim Brascamp (WUR/ABGC)]]
* [[User:Calus001 | Mario Calus (WUR/ABGC)]]
* [[User:dongen01 | Henk van Dongen (TOPIGS)]]
* [[User:Frans004 | Wietse Franssen (WUR/ESG)]]
* [[User:Haars001 | Jan van Haarst (WUR/PRI)]]
* [[User:Hulse002 | Ina Hulsegge (WUR/ABGC)]]
* [[User:Hulze001 | Alex Hulzebosch (WUR/ABGC)]]
* [[User:Lopes01 | Marcos Soares Lopes (TOPIGS)]]
* [[User:Madse001 | Ole Madsen (WUR/ABGC)]]
* [[User:Megen002 | Hendrik-Jan Megens (WUR/ABGC)]]
* [[User:Nijve002 | Harm Nijveen (WUR/Bioinformatics)]]
* [[User:Schroo01 | Chris Schrooten (CRV)]]
* [[User:Schur010 | Anouk Schurink (WUR/ABGC)]]
* [[User:Smit089 | Sandra Smit (WUR/Bioinformatics)]]
* [[User:Vande018 | Jeremie Vandenplas (WUR/ABGC)]]
* [[User:Veerk001 | Roel Veerkamp (WUR/ABGC)]]
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]
* [[User:derks047 | Martijn Derks (WUR/ABGC)]]

== FB-ICT Management of the HPC ==

* [[User:Dawes001 | Gwen Dawes (WUR, FB-IT) - HPC System Adiministrator]]
* [[User:Janss115 | Stephen Janssen (WUR, FB-IT, Service Management)]]
* [[User:pollm001 | Koen Pollmann (WUR, FB-IT, Infrastructure)]]

== Alumni ==
* [[User:Bohme001 | Andre ten Böhmer (WUR, FB-IT, Infrastructure)]]
* [[User:Frant001 | Laurent Frantz (WUR/ABGC)]]
* [[User:Herrer01 | Juanma Herrero (WUR/ABGC)]]
* [[User:paude004 | Yogesh Paudel (WUR/ABGC)]]

List of users

2017-06-07T12:45:16Z

Megen002: /* Active users */

== Active users ==

List of users. Alphabetical order (last/family name). Provide some background information by adding information to the 'User:username' page.

* [[User:Barris01 | Wes Barris (Cobb)]]
* [[User:Basti015 | John Bastiaansen (WUR/ABGC)]]
* [[User:Binsb003 | Rianne van Binsbergen (WUR/ABGC)]]
* [[User:Bosse014 | Mirte Bosse (WUR/ABGC)]]
* [[User:Bouwm024 | Aniek Bouwman (WUR/ABGC)]]
* [[User:Brasc001 | Pim Brascamp (WUR/ABGC)]]
* [[User:Calus001 | Mario Calus (WUR/ABGC)]]
* [[User:dongen01 | Henk van Dongen (TOPIGS)]]
* [[User:Frans004 | Wietse Franssen (WUR/ESG)]]
* [[User:Haars001 | Jan van Haarst (WUR/PRI)]]
* [[User:Hulse002 | Ina Hulsegge (WUR/ABGC)]]
* [[User:Hulze001 | Alex Hulzebosch (WUR/ABGC)]]
* [[User:Lopes01 | Marcos Soares Lopes (TOPIGS)]]
* [[User:Madse001 | Ole Madsen (WUR/ABGC)]]
* [[User:Megen002 | Hendrik-Jan Megens (WUR/ABGC)]]
* [[User:Nijve002 | Harm Nijveen (WUR/Bioinformatics)]]
* [[User:Schroo01 | Chris Schrooten (CRV)]]
* [[User:Schur010 | Anouk Schurink (WUR/ABGC)]]
* [[User:Smit089 | Sandra Smit (WUR/Bioinformatics)]]
* [[User:Vande018 | Jeremie Vandenplas (WUR/ABGC)]]
* [[User:Veerk001 | Roel Veerkamp (WUR/ABGC)]]
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]
* [[User:derks047 | Martijn Derks (WUR/ABGC)]]

== FB-ICT Management of the HPC ==

* [[User:Dawes001 | Gwen Dawes (WUR, FB-IT) - HPC System Adiministrator]]
* [[User:Janss115 | Stephen Janssen (WUR, FB-IT, Service Management)]]
* [[User:pollm001 | Koen Pollmann (WUR, FB-IT, Infrastructure)]]

== Alumni ==
* [[User:Bohme001 | Andre ten Böhmer (WUR, FB-IT, Infrastructure)]]
* [[User:Herrer01 | Juanma Herrero (WUR/ABGC)]]
* [[User:paude004 | Yogesh Paudel (WUR/ABGC)]]

User:Megen002

2017-06-07T12:42:20Z

Megen002:

== Hendrik-Jan Megens ==

[[File:HJM.jpg|right]]
* Profile on [https://www.vcard.wur.nl/Views/Profile/View.aspx?id=4846 We@WUR]
* Profile on [http://www.linkedin.com/pub/hendrik-jan-megens/24/536/2b8 LinkedIn]
* Profile on [http://scholar.google.nl/citations?user=kGUIXOYAAAAJ Google Scholar]

Assistant Professor at Wageningen University, Animal Breeding and Genomics Centre.

'''Scientific interests:'''

Coming from a wet-lab background I discovered I had more talent for programming than for pipetting. I have moved into applied bioinformatics since 2004, while retaining focus on my research interests:
* evolutionary genomics (generation and maintenance of, and selection on, structural and single nucleotide variation; speciation and outbreeding depression; inbreeding depression and heterosis)
* population genetics (genetic consequences of population management, domestication, selection)

'''Main current projects'''
* Genome sequencing of pig and turkey.
* Re-sequencing projects on various livestock species
* Functional aspects of genome variation
* De novo assembly and annotation of genomes

We are currently sequencing >300 pigs, wild boar, and outgroup species. The project aims to elucidate major patterns in biogeography and domestication of the pig, resulting from selection and demography.

'''Procedures'''
* Main sequencing platform is Illumina (We started in 2008 on Solexa GA, to currently Illumina HiSeq).
* Depending on research questions various short-read mapping programs are used (Mosaik, BWA, BWA/Stampy, MrsFAST)
* Variant calling (Samtools, GATK).
* Functional analysis of variants (Annovar, customly scripted tools)
* Various population- and phylogenomic approaches to tackle specific questions (RAxML, Beagle, coalhmm, etc., and customly scripted tools)

'''Main programming languages:'''
* Python
* R
* Linux shell scripting
* Perl (once upon a time....)
* SQL

Favorite distros: Ubuntu, Debian, Fedora, CentOS, Raspbian

== See also ==
[[List_of_users | List of users of the HPC Agrogenomics]]

JBrowse

2016-12-16T13:03:21Z

Megen002: /* Install JBrowse */

=== Typical commands used to set up a JBrowse ===

Author: Martijn Derks

* JBrowse is available for multiple species:
** https://jbrowse.hpcagrogenomics.wur.nl/pig/
** https://jbrowse.hpcagrogenomics.wur.nl/chicken/
** https://jbrowse.hpcagrogenomics.wur.nl/cattle/
** https://jbrowse.hpcagrogenomics.wur.nl/turkey/
** https://jbrowse.hpcagrogenomics.wur.nl/Cyprinus_carpio/
* Users are free to add usefull commands to this tutorial

=== Install JBrowse ===

Download the latest JBrowse here: http://jbrowse.org/

Make a directory in <code>/cm/shared/apps/jbrowse/</code> for your species of interested (e.g. <code>mkdir Cyprinus_carpio</code>). Move the downloaded JBrowse source files there. All further procedures detailed in this Wiki page assume working from that directory (NOTE: if your species of interest is already there, contact the maintainer of that JBrowse instance).
Run the setup script to install perl dependencies and required modules

<source lang='bash'>
unzip JBrowse-1.12.0.zip
mv JBrowse-1.12.0/* $PWD
./setup.sh
</source>

=== Add reference sequence ===

Example code for chicken genome

<source lang='bash'>
bin/prepare-refseqs.pl --fasta /lustre/nobackup/WUR/ABGC/shared/public_data_store/genomes/chicken/Ensembl74/Gallus_gallus.Galgal4.74.dna.toplevel.fa
</source>

To remove tracks use following command:

<source lang='bash'>
bin/remove-track.pl -D --trackLabel 'trackname'
</source>

===Add annotation files (GFF/BED)===

Data can be downloaded from the Ensembl FTP site: http://www.ensembl.org/info/data/ftp/index.html

Add gene features:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Gene spans" --className feature5 --type gene --noSubfeatures --config '{ "category": "GalGal4.83 Annotation" }' --trackLabel Genes --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

Add corresponding transcripts:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Transcripts" --className transcript --subfeatureClasses '{"exon": "exon", "CDS": "CDS", "five_prime_UTR": "five_prime_UTR", "three_prime_UTR": "three_prime_UTR"}' --config '{ "category": "GalGal4.83 Annotation" }' --type transcript --trackLabel Transcripts --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

===Alignment tracks (BAM)===

You can load single BAM-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bam in /<dir>*.bam; do
ln -s $bam track_symlinks/ ## Make symlinks from the BAM files
ln -s $bam.bai track_symlinks/ ## Make symlinks to the BAM index files
tissue=`echo $bam | rev | cut -c 5- | cut -d'/' -f1 | rev` ## USe the name of the file without .bam as trackLabel

## Add BAM in alignment mode (Alignments2)
echo '{
"label" : "'${tissue}'_alignment",
"key" : "'${tissue}'_alignment",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json

## Add BAM in coverage mode (SNPCoverage)
echo '{
"label" : "'${tissue}'_coverage",
"key" : "'${tissue}'_coverage",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "SNPCoverage"
}' | bin/add-track-json.pl data/trackList.json

done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Variant tracks (VCF)===

To load a VCF file in JBrowse make sure the file is gzipped and indexed

<source lang='bash'>
tabix -p vcf Gallus_gallus_incl_consequences.vcf.gz

echo ' {
"label" : "Gallus_gallus_incl_consequences",
"key" : "Gallus_gallus_incl_consequences",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../ensembl_data/VCF/Gallus_gallus_incl_consequences.vcf.gz",
"category" : "2. Variants",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
</source>

===Wiggle/BigWig tracks (WIG)===

You can load single BigWig-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bw in /lustre/nobackup/WUR/ABGC/shared/Chicken/Ensembl_Rna_Seq/*.bw; do
ln -s $bw track_symlinks/
tissue=`echo $bw | rev | cut -c 8- | cut -d'/' -f1 | rev`
echo '{
"label" : "'${tissue}'_BWcoverage",
"key" : "'${tissue}'_BWcoverage",
"storeClass" : "BigWig",
"urlTemplate" : "../track_symlinks/'${tissue}.bam.bw'",
"category" : "3. RNA-seq alignments",
"type" : "JBrowse/View/Track/Wiggle/XYPlot",
"variance_band" : true
}' | bin/add-track-json.pl data/trackList.json
done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Evidence tracks===

Evidence tracks can be loaded in bed, gff and gbk format using

<source lang='bash'>
bin/flatfile-to-json.pl
</source>

Examples are given above.

JBrowse

2016-12-16T12:56:54Z

Megen002: /* Typical commands used to set up a JBrowse */

=== Typical commands used to set up a JBrowse ===

Author: Martijn Derks

* JBrowse is available for multiple species:
** https://jbrowse.hpcagrogenomics.wur.nl/pig/
** https://jbrowse.hpcagrogenomics.wur.nl/chicken/
** https://jbrowse.hpcagrogenomics.wur.nl/cattle/
** https://jbrowse.hpcagrogenomics.wur.nl/turkey/
** https://jbrowse.hpcagrogenomics.wur.nl/Cyprinus_carpio/
* Users are free to add usefull commands to this tutorial

=== Install JBrowse ===

Download the latest JBrowse here: http://jbrowse.org/

Make a directory in <code>/cm/shared/apps/jbrowse/</code> for your species of interested (e.g. <code>mkdir Cyprinus_carpio</code>). Move the downloaded JBrowse source files there. All further procedures detailed in this Wiki page assume working from that directory (NOTE: if your species of interest is already there, contact the maintainer of that JBrowse instance).
Run the setup script to install perl dependencies and required modules

<source lang='bash'>
unzip JBrowse-1.12.0.zip
cd JBrowse-1.12.0
./setup.sh
</source>

=== Add reference sequence ===

Example code for chicken genome

<source lang='bash'>
bin/prepare-refseqs.pl --fasta /lustre/nobackup/WUR/ABGC/shared/public_data_store/genomes/chicken/Ensembl74/Gallus_gallus.Galgal4.74.dna.toplevel.fa
</source>

To remove tracks use following command:

<source lang='bash'>
bin/remove-track.pl -D --trackLabel 'trackname'
</source>

===Add annotation files (GFF/BED)===

Data can be downloaded from the Ensembl FTP site: http://www.ensembl.org/info/data/ftp/index.html

Add gene features:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Gene spans" --className feature5 --type gene --noSubfeatures --config '{ "category": "GalGal4.83 Annotation" }' --trackLabel Genes --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

Add corresponding transcripts:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Transcripts" --className transcript --subfeatureClasses '{"exon": "exon", "CDS": "CDS", "five_prime_UTR": "five_prime_UTR", "three_prime_UTR": "three_prime_UTR"}' --config '{ "category": "GalGal4.83 Annotation" }' --type transcript --trackLabel Transcripts --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

===Alignment tracks (BAM)===

You can load single BAM-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bam in /<dir>*.bam; do
ln -s $bam track_symlinks/ ## Make symlinks from the BAM files
ln -s $bam.bai track_symlinks/ ## Make symlinks to the BAM index files
tissue=`echo $bam | rev | cut -c 5- | cut -d'/' -f1 | rev` ## USe the name of the file without .bam as trackLabel

## Add BAM in alignment mode (Alignments2)
echo '{
"label" : "'${tissue}'_alignment",
"key" : "'${tissue}'_alignment",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json

## Add BAM in coverage mode (SNPCoverage)
echo '{
"label" : "'${tissue}'_coverage",
"key" : "'${tissue}'_coverage",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "SNPCoverage"
}' | bin/add-track-json.pl data/trackList.json

done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Variant tracks (VCF)===

To load a VCF file in JBrowse make sure the file is gzipped and indexed

<source lang='bash'>
tabix -p vcf Gallus_gallus_incl_consequences.vcf.gz

echo ' {
"label" : "Gallus_gallus_incl_consequences",
"key" : "Gallus_gallus_incl_consequences",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../ensembl_data/VCF/Gallus_gallus_incl_consequences.vcf.gz",
"category" : "2. Variants",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
</source>

===Wiggle/BigWig tracks (WIG)===

You can load single BigWig-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bw in /lustre/nobackup/WUR/ABGC/shared/Chicken/Ensembl_Rna_Seq/*.bw; do
ln -s $bw track_symlinks/
tissue=`echo $bw | rev | cut -c 8- | cut -d'/' -f1 | rev`
echo '{
"label" : "'${tissue}'_BWcoverage",
"key" : "'${tissue}'_BWcoverage",
"storeClass" : "BigWig",
"urlTemplate" : "../track_symlinks/'${tissue}.bam.bw'",
"category" : "3. RNA-seq alignments",
"type" : "JBrowse/View/Track/Wiggle/XYPlot",
"variance_band" : true
}' | bin/add-track-json.pl data/trackList.json
done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Evidence tracks===

Evidence tracks can be loaded in bed, gff and gbk format using

<source lang='bash'>
bin/flatfile-to-json.pl
</source>

Examples are given above.

JBrowse

2016-12-16T12:13:40Z

Megen002: /* Install JBrowse */

=== Typical commands used to set up a JBrowse ===

Author: Martijn Derks

* JBrowse is available for multiple species:
** https://jbrowse.hpcagrogenomics.wur.nl/pig/
** https://jbrowse.hpcagrogenomics.wur.nl/chicken/
** https://jbrowse.hpcagrogenomics.wur.nl/cattle/
** https://jbrowse.hpcagrogenomics.wur.nl/turkey/
* Users are free to add usefull commands to this tutorial

=== Install JBrowse ===

Download the latest JBrowse here: http://jbrowse.org/

Make a directory in <code>/cm/shared/apps/jbrowse/</code> for your species of interested (e.g. <code>mkdir Cyprinus_carpio</code>). Move the downloaded JBrowse source files there. All further procedures detailed in this Wiki page assume working from that directory (NOTE: if your species of interest is already there, contact the maintainer of that JBrowse instance).
Run the setup script to install perl dependencies and required modules

<source lang='bash'>
unzip JBrowse-1.12.0.zip
cd JBrowse-1.12.0
./setup.sh
</source>

=== Add reference sequence ===

Example code for chicken genome

<source lang='bash'>
bin/prepare-refseqs.pl --fasta /lustre/nobackup/WUR/ABGC/shared/public_data_store/genomes/chicken/Ensembl74/Gallus_gallus.Galgal4.74.dna.toplevel.fa
</source>

To remove tracks use following command:

<source lang='bash'>
bin/remove-track.pl -D --trackLabel 'trackname'
</source>

===Add annotation files (GFF/BED)===

Data can be downloaded from the Ensembl FTP site: http://www.ensembl.org/info/data/ftp/index.html

Add gene features:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Gene spans" --className feature5 --type gene --noSubfeatures --config '{ "category": "GalGal4.83 Annotation" }' --trackLabel Genes --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

Add corresponding transcripts:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Transcripts" --className transcript --subfeatureClasses '{"exon": "exon", "CDS": "CDS", "five_prime_UTR": "five_prime_UTR", "three_prime_UTR": "three_prime_UTR"}' --config '{ "category": "GalGal4.83 Annotation" }' --type transcript --trackLabel Transcripts --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

===Alignment tracks (BAM)===

You can load single BAM-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bam in /<dir>*.bam; do
ln -s $bam track_symlinks/ ## Make symlinks from the BAM files
ln -s $bam.bai track_symlinks/ ## Make symlinks to the BAM index files
tissue=`echo $bam | rev | cut -c 5- | cut -d'/' -f1 | rev` ## USe the name of the file without .bam as trackLabel

## Add BAM in alignment mode (Alignments2)
echo '{
"label" : "'${tissue}'_alignment",
"key" : "'${tissue}'_alignment",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json

## Add BAM in coverage mode (SNPCoverage)
echo '{
"label" : "'${tissue}'_coverage",
"key" : "'${tissue}'_coverage",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "SNPCoverage"
}' | bin/add-track-json.pl data/trackList.json

done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Variant tracks (VCF)===

To load a VCF file in JBrowse make sure the file is gzipped and indexed

<source lang='bash'>
tabix -p vcf Gallus_gallus_incl_consequences.vcf.gz

echo ' {
"label" : "Gallus_gallus_incl_consequences",
"key" : "Gallus_gallus_incl_consequences",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../ensembl_data/VCF/Gallus_gallus_incl_consequences.vcf.gz",
"category" : "2. Variants",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
</source>

===Wiggle/BigWig tracks (WIG)===

You can load single BigWig-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bw in /lustre/nobackup/WUR/ABGC/shared/Chicken/Ensembl_Rna_Seq/*.bw; do
ln -s $bw track_symlinks/
tissue=`echo $bw | rev | cut -c 8- | cut -d'/' -f1 | rev`
echo '{
"label" : "'${tissue}'_BWcoverage",
"key" : "'${tissue}'_BWcoverage",
"storeClass" : "BigWig",
"urlTemplate" : "../track_symlinks/'${tissue}.bam.bw'",
"category" : "3. RNA-seq alignments",
"type" : "JBrowse/View/Track/Wiggle/XYPlot",
"variance_band" : true
}' | bin/add-track-json.pl data/trackList.json
done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Evidence tracks===

Evidence tracks can be loaded in bed, gff and gbk format using

<source lang='bash'>
bin/flatfile-to-json.pl
</source>

Examples are given above.

JBrowse

2016-12-16T12:13:04Z

Megen002: /* Install JBrowse */

=== Typical commands used to set up a JBrowse ===

Author: Martijn Derks

* JBrowse is available for multiple species:
** https://jbrowse.hpcagrogenomics.wur.nl/pig/
** https://jbrowse.hpcagrogenomics.wur.nl/chicken/
** https://jbrowse.hpcagrogenomics.wur.nl/cattle/
** https://jbrowse.hpcagrogenomics.wur.nl/turkey/
* Users are free to add usefull commands to this tutorial

=== Install JBrowse ===

Download the latest JBrowse here: http://jbrowse.org/

Make a directory in <code>/cm/shared/apps/jbrowse/</code> for your species of interested (e.g. Cyprinus_carpio). Move the downloaded JBrowse source files there. All further procedures detailed in this Wiki page assume working from that directory (NOTE: if your species of interest is already there, contact the maintainer of that JBrowse instance).
Run the setup script to install perl dependencies and required modules

<source lang='bash'>
unzip JBrowse-1.12.0.zip
cd JBrowse-1.12.0
./setup.sh
</source>

=== Add reference sequence ===

Example code for chicken genome

<source lang='bash'>
bin/prepare-refseqs.pl --fasta /lustre/nobackup/WUR/ABGC/shared/public_data_store/genomes/chicken/Ensembl74/Gallus_gallus.Galgal4.74.dna.toplevel.fa
</source>

To remove tracks use following command:

<source lang='bash'>
bin/remove-track.pl -D --trackLabel 'trackname'
</source>

===Add annotation files (GFF/BED)===

Data can be downloaded from the Ensembl FTP site: http://www.ensembl.org/info/data/ftp/index.html

Add gene features:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Gene spans" --className feature5 --type gene --noSubfeatures --config '{ "category": "GalGal4.83 Annotation" }' --trackLabel Genes --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

Add corresponding transcripts:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Transcripts" --className transcript --subfeatureClasses '{"exon": "exon", "CDS": "CDS", "five_prime_UTR": "five_prime_UTR", "three_prime_UTR": "three_prime_UTR"}' --config '{ "category": "GalGal4.83 Annotation" }' --type transcript --trackLabel Transcripts --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

===Alignment tracks (BAM)===

You can load single BAM-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bam in /<dir>*.bam; do
ln -s $bam track_symlinks/ ## Make symlinks from the BAM files
ln -s $bam.bai track_symlinks/ ## Make symlinks to the BAM index files
tissue=`echo $bam | rev | cut -c 5- | cut -d'/' -f1 | rev` ## USe the name of the file without .bam as trackLabel

## Add BAM in alignment mode (Alignments2)
echo '{
"label" : "'${tissue}'_alignment",
"key" : "'${tissue}'_alignment",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json

## Add BAM in coverage mode (SNPCoverage)
echo '{
"label" : "'${tissue}'_coverage",
"key" : "'${tissue}'_coverage",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "SNPCoverage"
}' | bin/add-track-json.pl data/trackList.json

done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Variant tracks (VCF)===

To load a VCF file in JBrowse make sure the file is gzipped and indexed

<source lang='bash'>
tabix -p vcf Gallus_gallus_incl_consequences.vcf.gz

echo ' {
"label" : "Gallus_gallus_incl_consequences",
"key" : "Gallus_gallus_incl_consequences",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../ensembl_data/VCF/Gallus_gallus_incl_consequences.vcf.gz",
"category" : "2. Variants",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
</source>

===Wiggle/BigWig tracks (WIG)===

You can load single BigWig-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bw in /lustre/nobackup/WUR/ABGC/shared/Chicken/Ensembl_Rna_Seq/*.bw; do
ln -s $bw track_symlinks/
tissue=`echo $bw | rev | cut -c 8- | cut -d'/' -f1 | rev`
echo '{
"label" : "'${tissue}'_BWcoverage",
"key" : "'${tissue}'_BWcoverage",
"storeClass" : "BigWig",
"urlTemplate" : "../track_symlinks/'${tissue}.bam.bw'",
"category" : "3. RNA-seq alignments",
"type" : "JBrowse/View/Track/Wiggle/XYPlot",
"variance_band" : true
}' | bin/add-track-json.pl data/trackList.json
done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Evidence tracks===

Evidence tracks can be loaded in bed, gff and gbk format using

<source lang='bash'>
bin/flatfile-to-json.pl
</source>

Examples are given above.

JBrowse

2016-12-16T12:08:54Z

Megen002: /* Install JBrowse */

=== Typical commands used to set up a JBrowse ===

Author: Martijn Derks

* JBrowse is available for multiple species:
** https://jbrowse.hpcagrogenomics.wur.nl/pig/
** https://jbrowse.hpcagrogenomics.wur.nl/chicken/
** https://jbrowse.hpcagrogenomics.wur.nl/cattle/
** https://jbrowse.hpcagrogenomics.wur.nl/turkey/
* Users are free to add usefull commands to this tutorial

=== Install JBrowse ===

Download the latest JBrowse here: http://jbrowse.org/

Run the setup script to install perl dependencies and required modules

<source lang='bash'>
unzip JBrowse-1.12.0.zip
cd JBrowse-1.12.0
./setup.sh
</source>

=== Add reference sequence ===

Example code for chicken genome

<source lang='bash'>
bin/prepare-refseqs.pl --fasta /lustre/nobackup/WUR/ABGC/shared/public_data_store/genomes/chicken/Ensembl74/Gallus_gallus.Galgal4.74.dna.toplevel.fa
</source>

To remove tracks use following command:

<source lang='bash'>
bin/remove-track.pl -D --trackLabel 'trackname'
</source>

===Add annotation files (GFF/BED)===

Data can be downloaded from the Ensembl FTP site: http://www.ensembl.org/info/data/ftp/index.html

Add gene features:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Gene spans" --className feature5 --type gene --noSubfeatures --config '{ "category": "GalGal4.83 Annotation" }' --trackLabel Genes --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

Add corresponding transcripts:

<source lang='bash'>
bin/flatfile-to-json.pl --key "Transcripts" --className transcript --subfeatureClasses '{"exon": "exon", "CDS": "CDS", "five_prime_UTR": "five_prime_UTR", "three_prime_UTR": "three_prime_UTR"}' --config '{ "category": "GalGal4.83 Annotation" }' --type transcript --trackLabel Transcripts --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</source>

===Alignment tracks (BAM)===

You can load single BAM-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bam in /<dir>*.bam; do
ln -s $bam track_symlinks/ ## Make symlinks from the BAM files
ln -s $bam.bai track_symlinks/ ## Make symlinks to the BAM index files
tissue=`echo $bam | rev | cut -c 5- | cut -d'/' -f1 | rev` ## USe the name of the file without .bam as trackLabel

## Add BAM in alignment mode (Alignments2)
echo '{
"label" : "'${tissue}'_alignment",
"key" : "'${tissue}'_alignment",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json

## Add BAM in coverage mode (SNPCoverage)
echo '{
"label" : "'${tissue}'_coverage",
"key" : "'${tissue}'_coverage",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "SNPCoverage"
}' | bin/add-track-json.pl data/trackList.json

done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Variant tracks (VCF)===

To load a VCF file in JBrowse make sure the file is gzipped and indexed

<source lang='bash'>
tabix -p vcf Gallus_gallus_incl_consequences.vcf.gz

echo ' {
"label" : "Gallus_gallus_incl_consequences",
"key" : "Gallus_gallus_incl_consequences",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../ensembl_data/VCF/Gallus_gallus_incl_consequences.vcf.gz",
"category" : "2. Variants",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
</source>

===Wiggle/BigWig tracks (WIG)===

You can load single BigWig-files by following command:
<source lang='bash'>
bin/add-bam-track --label <label> --bam_url <url>
</source>

To load multiple BAM files present in a certain directory use:
<source lang='bash'>

for bw in /lustre/nobackup/WUR/ABGC/shared/Chicken/Ensembl_Rna_Seq/*.bw; do
ln -s $bw track_symlinks/
tissue=`echo $bw | rev | cut -c 8- | cut -d'/' -f1 | rev`
echo '{
"label" : "'${tissue}'_BWcoverage",
"key" : "'${tissue}'_BWcoverage",
"storeClass" : "BigWig",
"urlTemplate" : "../track_symlinks/'${tissue}.bam.bw'",
"category" : "3. RNA-seq alignments",
"type" : "JBrowse/View/Track/Wiggle/XYPlot",
"variance_band" : true
}' | bin/add-track-json.pl data/trackList.json
done
</source>

Make sure the BAM file can be read by a everybody if not use:
<source lang='bash'>
chmod +r <BAM_file>
</source>

Make sure that all directoryies in the full path of the BAMfile are executable:
<source lang='bash'>
chmod +x <dir>
</source>

===Evidence tracks===

Evidence tracks can be loaded in bed, gff and gbk format using

<source lang='bash'>
bin/flatfile-to-json.pl
</source>

Examples are given above.

Virtual environment Python 3.4 or higher

2016-01-22T08:58:28Z

Megen002:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your <code>$HOME</code> dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>p35_myproj</code>.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

Note that like with any command you can make an alias in your <code>~/.bashrc</code>. Just add something like this line to your <code>.bashrc</code>:

<source lang='bash'>
alias p35myproj='source ~/my_envs/p35_myproj/bin/activate'
</source>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using <code>pip</code>.

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>).
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Make IPython work under virtualenv ==
IPython can simply be installed through pip.

<source lang='bash'>
pip install ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://docs.python.org/3/library/venv.html#module-venv Python docs on virtenv]

Virtual environment Python 3.4 or higher

2016-01-22T08:57:44Z

Megen002: /* activating a virtual environment */

Virtual environment Python 3.4 or higher

2016-01-22T08:56:28Z

Megen002: /* activating a virtual environment */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your <code>$HOME</code> dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>p35_myproj</code>.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

Note that like with any command you can make an alias in your ~/.bashrc. Just add something like this line to your .bashrc:

<source lang='bash'>
alias p35myproj='source ~/my_envs/p35_myproj/bin/activate'
</source>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using <code>pip</code>.

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>).
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Make IPython work under virtualenv ==
IPython can simply be installed through pip.

<source lang='bash'>
pip install ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://docs.python.org/3/library/venv.html#module-venv Python docs on virtenv]

Main Page

2016-01-22T08:51:42Z

Megen002: /* Installation of software by users */

The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan).

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The Agrogenomics Cluster was originally conceived as being the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the HPC-Ag acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know-how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.
{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The Agrogenomics cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.
{{-}}

== Architecture of the cluster ==
[[Architecture_of_the_HPC | Main Article: Architecture of the Agrogenomics HPC]]
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new Agrogenomics HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The Agrogenomics Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the Agrogenomics Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==
[[HPC_management | Main Article: HPC management]]

Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of the HPC-Ag is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.

== Users ==

* [[List_of_users | List of users (alphabetical order)]]
* [[Mailinglist | Electronic mail discussion lists]]

== Using the HPC-Ag ==
=== Gaining access to the HPC-Ag ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]

=== Installed software ===

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

=== Being in control of Environment parameters ===

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Controlling costs ===

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Miscellaneous ==
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Virtual environment Python 3.4 or higher

2016-01-22T08:50:25Z

Megen002: /* External links */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your <code>$HOME</code> dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>p35_myproj</code>.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using <code>pip</code>.

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>).
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Make IPython work under virtualenv ==
IPython can simply be installed through pip.

<source lang='bash'>
pip install ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://docs.python.org/3/library/venv.html#module-venv Python docs on virtenv]

Virtual environment Python 3.4 or higher

2016-01-22T08:50:13Z

Megen002: /* External links */

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your <code>$HOME</code> dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>p35_myproj</code>.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using <code>pip</code>.

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>).
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Make IPython work under virtualenv ==
IPython can simply be installed through pip.

<source lang='bash'>
pip install ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://docs.python.org/3/library/venv.html#module-venv | Python docs on virtenv]

Virtual environment Python 3.4 or higher

2016-01-22T08:49:47Z

Megen002:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your <code>$HOME</code> dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>p35_myproj</code>.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using <code>pip</code>.

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>).
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Make IPython work under virtualenv ==
IPython can simply be installed through pip.

<source lang='bash'>
pip install ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://docs.python.org/3/library/venv.html#module-venv]

Virtual environment Python 3.4 or higher

2016-01-22T08:39:56Z

Megen002:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your <code>)$HOME</code>) dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>)p35_myproj</code>).

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using <code>)pip</code>).

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>).
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>
== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<code>
File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^
</code>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Virtual environment Python 3.4 or higher

2016-01-22T08:37:16Z

Megen002:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your $HOME dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example p35_myproj.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using pip.

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>
== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<code>
File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^
</code>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Virtual environment Python 3.4 or higher

2016-01-22T08:36:20Z

Megen002:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your $HOME dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example p35_myproj.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(p35_myproj)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. An easy way of installing modules is using pip.

Before you start installing modules, first update pip itself:
<source lang='bash'>
pip install --upgrade pip
</source>

you can then install other modules as you like, for instance numpy:

<source lang='bash'>
pip install numpy
</source>

<code>
(p35_myproj) [user@nfs01 ~]$ pip install numpy
Collecting numpy
Using cached numpy-1.10.4.tar.gz
Installing collected packages: numpy
Running setup.py install for numpy ... done
Successfully installed numpy-1.10.4
</code>

Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>
== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<code>
File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^
</code>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Virtual environment Python 3.4 or higher

2016-01-22T08:29:58Z

Megen002:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your $HOME dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example p35_myproj.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source ~/my_envs/p35_myproj/bin/activate
</source>

This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(newenv)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>easy_install</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
easy_install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>
== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<code>
File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^
</code>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Virtual environment Python 3.4 or higher

2016-01-22T08:07:28Z

Megen002: Created page with "With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is..."

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

== creating a new virtual environment ==

If you do not already have a directory in your $HOME dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).

<source lang='bash'>
mkdir ~/my_envs
</source>

Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):

<source lang='bash'>
module load python/3.5.0
</source>
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example p35_myproj.

<source lang='bash'>
pyvenv install ~/my_envs/p35_myproj
</source>

Should virtualenv not be installed, the virtualenv script can be downloaded and accessed directly:
<source lang='bash'>
curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz
tar -xzvf virtualenv-1.9.tar.gz
python3 virtualenv-1.9/virtualenv.py testenv
</source>

When the new environment is created, one will see a message similar to this:
<code>
New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.
</code>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(newenv)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>easy_install</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
easy_install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>
== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<code>
File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^
</code>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Setting up Python virtualenv

2016-01-22T07:57:44Z

Megen002:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
virtualenv newenv
</source>

Should virtualenv not be installed, the virtualenv script can be downloaded and accessed directly:
<source lang='bash'>
curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz
tar -xzvf virtualenv-1.9.tar.gz
python3 virtualenv-1.9/virtualenv.py testenv
</source>

When the new environment is created, one will see a message similar to this:
<code>
New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.
</code>

== activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<code>
(newenv)user@host:~$
</code>

== installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>easy_install</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
easy_install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>
== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<code>
File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^
</code>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== See also ==
* [[B4F_cluster | B4F Cluster]]

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

List of users

2015-12-10T13:42:18Z

Megen002: /* Alumni */

== Active users ==

List of users. Alphabetical order (last/family name). Provide some background information by adding information to the 'User:username' page.

* [[User:Barris01 | Wes Barris (Cobb)]]
* [[User:Basti015 | John Bastiaansen (WUR/ABGC)]]
* [[User:Binsb003 | Rianne van Binsbergen (WUR/ABGC)]]
* [[User:Bosse014 | Mirte Bosse (WUR/ABGC)]]
* [[User:Bouwm024 | Aniek Bouwman (WUR/ABGC)]]
* [[User:Brasc001 | Pim Brascamp (WUR/ABGC)]]
* [[User:Calus001 | Mario Calus (WUR/ABGC)]]
* [[User:dongen01 | Henk van Dongen (TOPIGS)]]
* [[User:Frant001 | Laurent Frantz (WUR/ABGC)]]
* [[User:Frans004 | Wietse Franssen (WUR/ESG)]]
* [[User:Haars001 | Jan van Haarst (WUR/PRI)]]
* [[User:Hulse002 | Ina Hulsegge (WUR/ABGC)]]
* [[User:Hulze001 | Alex Hulzebosch (WUR/ABGC)]]
* [[User:Lopes01 | Marcos Soares Lopes (TOPIGS)]]
* [[User:Madse001 | Ole Madsen (WUR/ABGC)]]
* [[User:Megen002 | Hendrik-Jan Megens (WUR/ABGC)]]
* [[User:Nijve002 | Harm Nijveen (WUR/Bioinformatics)]]
* [[User:Schroo01 | Chris Schrooten (CRV)]]
* [[User:Schur010 | Anouk Schurink (WUR/ABGC)]]
* [[User:Smit089 | Sandra Smit (WUR/Bioinformatics)]]
* [[User:Vande018 | Jeremie Vandenplas (WUR/ABGC)]]
* [[User:Veerk001 | Roel Veerkamp (WUR/ABGC)]]
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]

== FB-ICT Management of the HPC ==

* [[User:Dawes001 | Gwen Dawes (WUR, FB-IT) - HPC System Adiministrator]]
* [[User:Janss115 | Stephen Janssen (WUR, FB-IT, Service Management)]]
* [[User:pollm001 | Koen Pollmann (WUR, FB-IT, Infrastructure)]]

== Alumni ==
* [[User:Bohme001 | Andre ten Böhmer (WUR, FB-IT, Infrastructure)]]
* [[User:Herrer01 | Juanma Herrero (WUR/ABGC)]]
* [[User:paude004 | Yogesh Paudel (WUR/ABGC)]]

List of users

2015-12-10T13:42:01Z

Megen002: /* Active users */

== Active users ==

List of users. Alphabetical order (last/family name). Provide some background information by adding information to the 'User:username' page.

* [[User:Barris01 | Wes Barris (Cobb)]]
* [[User:Basti015 | John Bastiaansen (WUR/ABGC)]]
* [[User:Binsb003 | Rianne van Binsbergen (WUR/ABGC)]]
* [[User:Bosse014 | Mirte Bosse (WUR/ABGC)]]
* [[User:Bouwm024 | Aniek Bouwman (WUR/ABGC)]]
* [[User:Brasc001 | Pim Brascamp (WUR/ABGC)]]
* [[User:Calus001 | Mario Calus (WUR/ABGC)]]
* [[User:dongen01 | Henk van Dongen (TOPIGS)]]
* [[User:Frant001 | Laurent Frantz (WUR/ABGC)]]
* [[User:Frans004 | Wietse Franssen (WUR/ESG)]]
* [[User:Haars001 | Jan van Haarst (WUR/PRI)]]
* [[User:Hulse002 | Ina Hulsegge (WUR/ABGC)]]
* [[User:Hulze001 | Alex Hulzebosch (WUR/ABGC)]]
* [[User:Lopes01 | Marcos Soares Lopes (TOPIGS)]]
* [[User:Madse001 | Ole Madsen (WUR/ABGC)]]
* [[User:Megen002 | Hendrik-Jan Megens (WUR/ABGC)]]
* [[User:Nijve002 | Harm Nijveen (WUR/Bioinformatics)]]
* [[User:Schroo01 | Chris Schrooten (CRV)]]
* [[User:Schur010 | Anouk Schurink (WUR/ABGC)]]
* [[User:Smit089 | Sandra Smit (WUR/Bioinformatics)]]
* [[User:Vande018 | Jeremie Vandenplas (WUR/ABGC)]]
* [[User:Veerk001 | Roel Veerkamp (WUR/ABGC)]]
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]

== FB-ICT Management of the HPC ==

* [[User:Dawes001 | Gwen Dawes (WUR, FB-IT) - HPC System Adiministrator]]
* [[User:Janss115 | Stephen Janssen (WUR, FB-IT, Service Management)]]
* [[User:pollm001 | Koen Pollmann (WUR, FB-IT, Infrastructure)]]

== Alumni ==
* [[User:Bohme001 | Andre ten Böhmer (WUR, FB-IT, Infrastructure)]]
* [[User:paude004 | Yogesh Paudel (WUR/ABGC)]]

Main Page

2015-03-26T22:33:46Z

Megen002: /* Access Policy */

The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan).

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The Agrogenomics Cluster was originally conceived as being the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the HPC-Ag acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know-how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.
{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The Agrogenomics cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.
{{-}}

== Architecture of the cluster ==
[[Architecture_of_the_HPC | Main Article: Architecture of the Agrogenomics HPC]]
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new Agrogenomics HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The Agrogenomics Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the Agrogenomics Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==
[[HPC_management | Main Article: HPC management]]

Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of the HPC-Ag is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.

== Users ==

* [[List_of_users | List of users (alphabetical order)]]
* [[Mailinglist | Electronic mail discussion lists]]

== Using the HPC-Ag ==
=== Gaining access to the HPC-Ag ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Installed software ===

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

=== Being in control of Environment parameters ===

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Controlling costs ===

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Miscellaneous ==
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2015-03-26T22:26:53Z

Megen002: /* Access Policy */

The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan).

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The Agrogenomics Cluster was originally conceived as being the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the HPC-Ag acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know-how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.
{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The Agrogenomics cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.
{{-}}

== Architecture of the cluster ==
[[Architecture_of_the_HPC | Main Article: Architecture of the Agrogenomics HPC]]
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new Agrogenomics HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The Agrogenomics Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the Agrogenomics Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==
[[HPC_management | Main Article: HPC management]]

Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of the AgHPC is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from CAT-AGRO or FB-ICT.

== Users ==

* [[List_of_users | List of users (alphabetical order)]]
* [[Mailinglist | Electronic mail discussion lists]]

== Using the HPC-Ag ==
=== Gaining access to the HPC-Ag ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Installed software ===

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

=== Being in control of Environment parameters ===

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Controlling costs ===

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Miscellaneous ==
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2015-03-26T22:24:25Z

Megen002: /* Access Policy */

The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan).

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The Agrogenomics Cluster was originally conceived as being the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the HPC-Ag acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know-how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.
{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The Agrogenomics cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.
{{-}}

== Architecture of the cluster ==
[[Architecture_of_the_HPC | Main Article: Architecture of the Agrogenomics HPC]]
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new Agrogenomics HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The Agrogenomics Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the Agrogenomics Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==
[[HPC_management | Main Article: HPC management]]

Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

== Users ==

* [[List_of_users | List of users (alphabetical order)]]
* [[Mailinglist | Electronic mail discussion lists]]

== Using the HPC-Ag ==
=== Gaining access to the HPC-Ag ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Installed software ===

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

=== Being in control of Environment parameters ===

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Controlling costs ===

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Miscellaneous ==
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Manual GitLab

2015-03-26T22:08:45Z

Megen002: /* Example of local commands to execute */

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine, e.g.:
mkdir ~/Git_Stuff

# 2. configuration step 1:
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2: make private-public rsa key pair
ssh-keygen -t rsa
cd ~/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# IMPORTANT: make sure you use the *public* key, not the private part of the key pair!
# Go back to the terminal:
cd ~/Git_Stuff
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in ~/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cp myscript.sh ~/Git_Stuff/Turkey_Association
cd ~/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-03-26T22:03:56Z

Megen002:

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# IMPORTANT: make sure you use the *public* key, not the private part of the key pair!
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Main Page

2015-03-26T20:42:21Z

Megen002: /* Miscellaneous */

The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan).

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The Agrogenomics Cluster was originally conceived as being the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the HPC-Ag acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know-how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.
{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The Agrogenomics cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.
{{-}}

== Architecture of the cluster ==
[[Architecture_of_the_HPC | Main Article: Architecture of the Agrogenomics HPC]]
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new Agrogenomics HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The Agrogenomics Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the Agrogenomics Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==
[[HPC_management | Main Article: HPC management]]

Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-IT reagdring non WUR accounts). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

== Users ==

* [[List_of_users | List of users (alphabetical order)]]
* [[Mailinglist | Electronic mail discussion lists]]

== Using the HPC-Ag ==
=== Gaining access to the HPC-Ag ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Installed software ===

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

=== Being in control of Environment parameters ===

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Controlling costs ===

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Miscellaneous ==
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Manual GitLab

2015-03-26T20:27:53Z

Megen002:

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-03-26T20:27:27Z

Megen002: /* External links */

Manual GitLab: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-03-26T20:27:08Z

Megen002: /* External links */

Manual GitLab: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Convert between MediaWiki and other formats

2015-03-26T20:24:17Z

Megen002:

There are various programs that can convert to and from MediaWiki format, as used in this Wiki, and other formats. This page lists some of them and provides explanation of how to use them.

== Pandoc ==

[http://johnmacfarlane.net/pandoc/demos.html Pandoc] is an open source tool that can convert between different markup styles and languages.

There is a [http://johnmacfarlane.net/pandoc/try/ web version of Pandoc] but that seems limited to small documents only. There's also a CLI version, and if you are running a Linux desktop machine chances are you can install it from the repository. E.g. on Ubuntu 14.04:

<source lang=bash>
sudo apt-get install pandoc
</source>

The following example translates from MediaWiki format to Markdown format:

<source lang=bash>
pandoc -f mediawiki -t markdown -s myfile.mw -o myfile.md
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
[http://en.wikipedia.org/wiki/Markup_language Wikipedia entry on markup languages]

Manual GitLab

2015-03-26T20:23:02Z

Megen002:

Manual GitLab: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]

Manual GitLab

2015-03-26T20:21:40Z

Megen002:

Manual GitLab: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]

Manual GitLab

2015-03-26T20:20:14Z

Megen002:

Manual GitLab: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]