HPCwiki - User contributions [en]

Manual GitLab

2015-04-09T13:01:57Z

Herre011: /* Example of local commands to execute */

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, install git in your local machine and sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine, e.g.:
mkdir ~/Git_Stuff

# 2. configuration step 1:
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2: make private-public rsa key pair
ssh-keygen -t rsa
cd ~/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# IMPORTANT: make sure you use the *public* key, not the private part of the key pair!
# The configuration is finished.

## Create a project:

# 1. Log in https://git.wageningenur.nl. Then create a project (name of the project should not have spaces or dash '-'). In the example, the project is called Turkey_Association
# 2. Go back to the terminal:
cd ~/Git_Stuff
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git

# Once I have made this connection, my project will appear as a folder in ~/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cp myscript.sh ~/Git_Stuff/Turkey_Association
cd ~/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master

## Remove a directory:
git rm -r one-of-the-directories
git commit -m "Remove duplicated directory"
git push origin master

</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-04-09T12:25:06Z

Herre011: /* Example of local commands to execute */

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, install git in your local machine and sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine, e.g.:
mkdir ~/Git_Stuff

# 2. configuration step 1:
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2: make private-public rsa key pair
ssh-keygen -t rsa
cd ~/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# IMPORTANT: make sure you use the *public* key, not the private part of the key pair!
# The configuration is finished.

## Create a project:

# 1. Log in https://git.wageningenur.nl. Then create a project (name of the project should not have spaces or dash '-'). In the example, the project is called Turkey_Association
# 2. Go back to the terminal:
cd ~/Git_Stuff
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git

# Once I have made this connection, my project will appear as a folder in ~/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cp myscript.sh ~/Git_Stuff/Turkey_Association
cd ~/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-04-09T12:24:49Z

Herre011: /* Example of local commands to execute */

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, install git in your local machine and sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine, e.g.:
mkdir ~/Git_Stuff

# 2. configuration step 1:
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2: make private-public rsa key pair
ssh-keygen -t rsa
cd ~/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# IMPORTANT: make sure you use the *public* key, not the private part of the key pair!
# The configuration is finished.

## Create a project:

# 1. Log in https://git.wageningenur.nl. Then create a project (name of the project should not have spaces or dash '-'. In the example, the project is called Turkey_Association
# 2. Go back to the terminal:
cd ~/Git_Stuff
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git

# Once I have made this connection, my project will appear as a folder in ~/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cp myscript.sh ~/Git_Stuff/Turkey_Association
cd ~/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-04-09T11:11:41Z

Herre011: /* Signing up */

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, install git in your local machine and sign up at GitLab@WUR:
https://git.wageningenur.nl

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine, e.g.:
mkdir ~/Git_Stuff

# 2. configuration step 1:
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2: make private-public rsa key pair
ssh-keygen -t rsa
cd ~/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# IMPORTANT: make sure you use the *public* key, not the private part of the key pair!
# Go back to the terminal:
cd ~/Git_Stuff
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in ~/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cp myscript.sh ~/Git_Stuff/Turkey_Association
cd ~/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-04-09T11:11:12Z

Herre011: /* Signing up */

Manual GitLab@WUR: Create projects and add files

== Signing up ==
If you haven't done so already, first sign up at GitLab@WUR:
https://git.wageningenur.nl

and install git in your local machine

== Example of local commands to execute ==
<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine, e.g.:
mkdir ~/Git_Stuff

# 2. configuration step 1:
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2: make private-public rsa key pair
ssh-keygen -t rsa
cd ~/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# IMPORTANT: make sure you use the *public* key, not the private part of the key pair!
# Go back to the terminal:
cd ~/Git_Stuff
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in ~/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cp myscript.sh ~/Git_Stuff/Turkey_Association
cd ~/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master
</source>

== See also ==
[[Bioinformatics_protocols_ABG_Chairgroup | Bioinformatics tips, tricks, workflows at ABGC]]

== External links ==
*[http://en.wikipedia.org/wiki/Git_(software) Wikipedia entry on Git]
*[https://github.com GitHub, a public Git repository (not to be confused with GitLab@WUR, but uses the same Git versioning software)]

Manual GitLab

2015-03-25T12:16:34Z

Herre011:

Manual GitLab: Create projects and add files

<source lang='bash'>

## This shows an example, step by step, to create a project and add files into that project.

## Configuration:

# 1. Create a folder in your machine:
/home/juanma/Git_Stuff

# 2. configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ # and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

# 3. configuration step 2:
cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub
# Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
# Go back to the terminal:
git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git
# Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

## Add files:
# Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master

Main Page

2015-03-25T12:14:14Z

Herre011: /* Miscellaneous */

The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan).

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The Agrogenomics Cluster was originally conceived as being the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the HPC-Ag acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know-how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.
{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The Agrogenomics cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.
{{-}}

== Architecture of the cluster ==
[[Architecture_of_the_HPC | Main Article: Architecture of the Agrogenomics HPC]]
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new Agrogenomics HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The Agrogenomics Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the Agrogenomics Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==
[[HPC_management | Main Article: HPC management]]

Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-IT reagdring non WUR accounts). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

== Users ==

* [[List_of_users | List of users (alphabetical order)]]
* [[Mailinglist | Electronic mail discussion lists]]

== Using the HPC-Ag ==
=== Gaining access to the HPC-Ag ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Installed software ===

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

=== Being in control of Environment parameters ===

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Controlling costs ===

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Miscellaneous ==
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scrits]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Manual GitLab

2015-03-25T12:07:42Z

Herre011: Created page with "Manual GitLab: Create projects and add files ## This shows an example, step by step, to create a project and add files into that project. 1. Configuration: Create a folder i..."

Manual GitLab: Create projects and add files ##
This shows an example, step by step, to create a project and add files into that project.

1. Configuration:

Create a folder in your machine:
/home/juanma/Git_Stuff

configuration step 1: Copy into that folder the commands from the git-web: https://git.wageningenur.nl/ and then ABGC_Genomics/

cd /home/juanma/Git_Stuff
git config --global user.name "Herrero Medrano, Juan"
git config --global user.email "juan.herreromedrano@wur.nl"

configuration step 2:

cd /home/juanma/Git_Stuff
ssh-keygen -t rsa
cd /home/juanma/.ssh/
cat id_rsa.pub

Copy the code and go to the git-web: Profile settings -> SSH Keys. Paste the code and add key.
configuration step 3: Go back to the terminal:

git clone git@git.wageningenur.nl:ABGC_Genomics/Turkey_Association.git

Once I have made this connection, my project will appear as a folder in /home/juanma/Git_Stuff

2. Add files:
Add scripts to the project "Turkey_Association" :
cd /home/juanma/Git_Stuff/Turkey_Association
cp myscript.sh /home/juanma/Git_Stuff/Turkey_Association
git add myscript.sh
git commit -m "myfirst_commit"
git push origin master

Main Page

2015-03-25T11:59:37Z

Herre011: /* Miscellaneous */

The Agrogenomics cluster is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

The Agrogenomics HPC was an initiative of the [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) consortium, consisting of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. Currently, in addition to the original partners, the HPC (HPC-Ag) is used by other groups from Wageningen UR (Bioinformatics, Centre for Crop Systems Analysis, Environmental Sciences Group, and Plant Research International) and plant breeding industry (Rijk Zwaan).

== Rationale and Requirements for a new cluster ==
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]
The Agrogenomics Cluster was originally conceived as being the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the HPC-Ag acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know-how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.
{{-}}

== Process of acquisition and financing ==

[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]
The Agrogenomics cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.
{{-}}

== Architecture of the cluster ==
[[Architecture_of_the_HPC | Main Article: Architecture of the Agrogenomics HPC]]
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]
The new Agrogenomics HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.
{{-}}

== Housing at Theia ==
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]
The Agrogenomics Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the Agrogenomics Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.
{{-}}
{| width="90%"
|- valign="top"
| width="10%" |

| width="30%" |
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]
| width="70%" |
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]
|}
{{-}}

== Management ==
[[HPC_management | Main Article: HPC management]]

Project Leader of the HPC is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]] and [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]].

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-IT reagdring non WUR accounts). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

== Users ==

* [[List_of_users | List of users (alphabetical order)]]
* [[Mailinglist | Electronic mail discussion lists]]

== Using the HPC-Ag ==
=== Gaining access to the HPC-Ag ===
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]

=== Cluster Management Software and Scheduler ===
The HPC-Ag uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

=== Installation of software by users ===

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Installed software ===

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

=== Being in control of Environment parameters ===

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

=== Controlling costs ===

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Miscellaneous ==
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | Create projects and add scrits]]

== See also ==
* [[Maintenance_and_Management | Maintenance and Management]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

== External links ==
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

User:Herrer01

2015-02-19T10:16:41Z

Herre011: /* Juan Manuel Herrero Medrano */

== Juan Manuel Herrero Medrano ==

Postdoctoral position at Wageningen University, Animal Breeding and Genomics Centre.

'''Scientific interests:'''

* Bioinformatics
* Next Generation Sequence data and High-density SNP arrays analysis
* Functional genomics
* Population genomics

'''Main current projects'''
* Storage, mapping variants calling analysis on re-sequence data on various livestock species
* Functional aspects of turkey genome variation
* De novo assembly and annotation of genomes

'''Procedures'''
* Main sequencing platform is Illumina (We started in 2008 on Solexa GA, to currently Illumina HiSeq).
* Short-read mapping (Mosaik, BWA)
* Variant calling (Samtools, GATK, ANGSD).
* Functional analysis of variants (Annovar, Variant Effect Predictor)
* Various population- and phylogenomic approaches

'''Main programming languages:'''
* Linux shell scripting
* Python
* R

User:Herrer01

2015-02-19T10:14:55Z

Herre011: Created page with "== Juan Manuel Herrero Medrano == Postdoctoral position at Wageningen University, Animal Breeding and Genomics Centre. '''Scientific interests:''' * Bioinformatics * Next..."

== Juan Manuel Herrero Medrano ==

Postdoctoral position at Wageningen University, Animal Breeding and Genomics Centre.

'''Scientific interests:'''

* Bioinformatics
* Next Generation Sequence data and High-density SNP arrays analysis
* Functional genomics
* Population genomics

'''Main current projects'''
* Storage, mapping variants calling analysis on re-sequence data on various livestock species
* Functional aspects of genome variation in Turkey
* De novo assembly and annotation of genomes

'''Procedures'''
* Main sequencing platform is Illumina (We started in 2008 on Solexa GA, to currently Illumina HiSeq).
* Depending on research questions various short-read mapping programs are used (Mosaik, BWA, BWA/Stampy, MrsFAST)
* Variant calling (Samtools, GATK).
* Functional analysis of variants (Annovar, Variant Effect Predictor)
* Various population- and phylogenomic approaches

'''Main programming languages:'''
* Linux shell scripting
* Python
* R

List of users

2015-02-19T10:07:16Z

Herre011: /* Active users */

== Active users ==

List of users. Alphabetical order (last/family name). Provide some background information by adding information to the 'User:username' page.

* [[User:Barris01 | Wes Barris (Cobb)]]
* [[User:Basti015 | John Bastiaansen (ABGC)]]
* [[User:Binsb003 | Rianne van Binsbergen (ABGC)]]
* [[User:Bosse014 | Mirte Bosse (ABGC)]]
* [[User:Bouwm024 | Aniek Bouwman (ABGC)]]
* [[User:Brasc001 | Pim Brascamp (ABGC)]]
* [[User:Calus001 | Mario Calus (ABGC)]]
* [[User:dongen01 | Henk van Dongen (TOPIGS)]]
* [[User:Frant001 | Laurent Frantz (ABGC)]]
* [[User:Haars001 | Jan van Haarst (PRI)]]
* [[User:Herrer01 | Juanma Herrero (ABGC)]]
* [[User:Hulse002 | Ina Hulsegge (ABGC)]]
* [[User:Hulze001 | Alex Hulzebosch (ABGC)]]
* [[User:Lopes01 | Marcos Soares Lopes (TOPIGS)]]
* [[User:Madse001 | Ole Madsen (ABGC)]]
* [[User:Megen002 | Hendrik-Jan Megens (ABGC)]]
* [[User:Nijve002 | Harm Nijveen (Bioinformatica)]]
* [[User:paude004 | Yogesh Paudel (ABGC)]]
* [[User:Schroo01 | Chris Schrooten (CRV)]]
* [[User:Smit089 | Sandra Smit (Bioinformatica)]]
* [[User:Vande018 | Jeremie Vandenplas (ABGC)]]
* [[User:Veerk001 | Roel Veerkamp (ABGC)]]
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]

== FB-ICT Management of the HPC ==

* [[User:Bohme001 | Andre ten Böhmer (WUR, FB-IT, Infrastructure)]]
* [[User:Janss115 | Stephen Janssen (WUR, FB-IT, Service Management)]]
* [[User:pollm001 | Koen Pollmann (WUR, FB-IT, Infrastructure)]]

== Alumni ==

Plink 1.9

2015-02-05T15:06:43Z

Herre011: Created page with "[https://www.cog-genomics.org/plink2/ PLINK] is a free, open-source whole genome association analysis toolset, designed to perform a range of basic, large-scale analyses in a..."

[https://www.cog-genomics.org/plink2/ PLINK] is a free, open-source whole genome association analysis toolset, designed to perform a range of basic, large-scale analyses in a computationally efficient manner. Development of the baseline PLINK toolset has resumed and beta testing for the next (1.90) release has now begun.The new release, by Chris Chang and colleagues, is a complete rewrite of the original code and represents a very significant improvement in overall speed and functionality. Moving forward, these changes should enable PLINK to meet the demands of ever-larger genetic datasets.

== Module file ==
The module file can be found in this location:
/cm/shared/modulefiles/SHARED/

<source lang='tcl'>
#%Module1.0#######################################################################
## plink 1.9 modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds plink v1.9 to your environment"
}

module-whatis "Adds plink v1.9 to your environment"

set plink_v19_root /cm/shared/apps/SHARED/plink/1.9

prepend-path PATH $plink_v19_root

</source>

== Installation details ==
Installed from downloaded 64bit binary, see external links.

== See also ==
* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

== External links ==
* [http://pngu.mgh.harvard.edu/~purcell/plink2/]

ANGSD 0.614

2015-02-05T14:49:21Z

Herre011: Created page with "[http://popgen.dk/wiki/index.php/ANGSD ANGSD] ANGSD is a software for analyzing next generation sequencing data. The software can handle a number of different input types from..."

[http://popgen.dk/wiki/index.php/ANGSD ANGSD] ANGSD is a software for analyzing next generation sequencing data. The software can handle a number of different input types from mapped reads to imputed genotype probabilities. Most methods take genotype uncertainty into account instead of basing the analysis on called genotypes. This is especially useful for low and medium depth data. The software is written in C++ and has been used on large sample sizes.

== Module file ==
The module file can be found in this location:
/cm/shared/modulefiles/SHARED/

<source lang='tcl'>
#%Module1.0#######################################################################
## ANGSD 0.614 modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds ANGSD v0.614 to your environment"
}

module-whatis "Adds ANGSD v0.614 to your environment"

set angsd_v0614_root /cm/shared/apps/WUR/ABGC/ANGSD/angsd0.614

prepend-path PATH $angsd_v0614_root

</source>

== Installation details ==
Installed from downloaded 64bit binary, see external links.

== See also ==
* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

== External links ==
* [http://popgen.dk/wiki/index.php/ANGSD]
* [https://github.com/ANGSD/angsd]

Globally installed software

2015-02-05T14:42:42Z

Herre011: /* Available as global SHARED modules */

== Available as modules ==
* gcc/4.8.1
* python/2.7.6
* python/3.3.3
* R/3.0.2

== Globally installed on all nodes ==

* Perl5.10
* pigz
* Python2.6
* BioPerl v1.61
* [http://samtools.sourceforge.net/tabix.shtml bgzip]
* [http://samtools.sourceforge.net/tabix.shtml tabix v0.2.5]

== Available as global SHARED modules ==
Software can be deposited in:
/cm/shared/apps/SHARED/

Modules can be found in:
/cm/shared/modulefiles/SHARED/

{| width="90%"
|- valign="top"
| width="30%" |
* [[allpathslg_48961 | ALLPATHS-LG/48961]]
* [[allpathslg 51910 | ALLPATHS-LG/51910]]
* [[ANGSD_0.614 | angsd/0.614]]
* [[augustus_2.7 | augustus/2.7]]
* [[bedtools2.18 | bedtools/2.18.0]]
* [[BLAST | BLAST+/2.2.28]]
* [[blat_v35 |blat/v35]]
* [[bowtie2_v2.2.1 | bowtie/2-2.2.1]]
* [[bowtie1_v1.0.0 | bowtie/1-1.0.0]]
* [[bwa_5.9 | bwa/0.5.9]]
* [[bwa_7.5 | bwa/0.7.5a]]
* [[cegma_2.4 | cegma/2.4]]
* [[Cufflinks | cufflinks/2.1.1]]

| width="30%" |
* [[exonerate_2.2.0 | exonerate/2.2.0-x86_64]]
* [[geneid_1.4.4 | geneid/1.4.4]]
* [[genewise_2.2.3 | genewise/2.2.3-rc7]]
* [[gmap_2014-01-21 | gmap/2014-01-21]]
* [[hmmer_3.1 | hmmer/3.1b1]]
* [[jellyfish_2.1.1 | jellyfish/2.1.1]]
* [[MAFFT_7.130 | MAFFT/7.130]]
* [[maker_2.2.8 | maker/2.28]]
* [[Muscle_3.8.31 | muscle/3.8.31]]
* [[Plink_1.07 | Plink/1.07]]
* [[Plink_1.9 | Plink/1.9]]
* [[Provean_1.1.3 | provean/1.1.3]]
* [[RepeatMasker_4.0.3 | RepeatMasker/4.0.3]]

| width="30%" |
* [[RepeatModeler_1.0.7 | RepeatModeler/1.0.7]]
* [[RAxML8.0.0 | RAxML/8.0.0]]
* [[samtools v0.1.12a | samtools/0.1.12a]]
* [[samtools v0.1.19 | samtools/0.1.19]]
* [[snap | snap/2013-11-29]]
* [[soapdenovo2_r240 | SOAPdenovo2/r240]]
* [[sra_toolkit_2.3.4 | sra-toolkit/2.3.4]]
* [[TopHat_2.0.11 | tophat/2.0.11]]
* [[Trinity_r20131110 | Trinity/r20131110]]
* [[wgs_assembler_8.1 | wgs-assembler/8.1]]
|}

== Adding a module to your current session ==
Use <code>module apropos</code> to find the module you wish to use, then <code>module load</code> to enable it.

== Adding a custom module directory to your environment ==
To allow the <code>module</code> program to find the custom module directory, the location of that directory has to be added to <code>MODULEPATH</code> variable.

<source lang='bash'>
export MODULEPATH=$MODULEPATH:/cm/shared/apps/WUR/ABGC/modulefiles
</source>
This can be made permanent by adding this line of code to the <code>.bash_profile</code> file in the root of your home folder. To then load the modified <code>MODULEPATH</code> variable you have to load <code>.bash_profile</code> again:
<source lang='bash'>
source .bash_profile
</source>
This needs to be done only for terminals that are already open. Next time you login, <code>.bash_profile</code> will be loaded automatically.

You can check if the modules are found.
<source lang='bash'>
module avail
</source>
This should give output that includes something similar to this:

---------------------------------------- /cm/shared/modulefiles/ -----------------------------------------
ALLPATHS-LG/48961 bwa/0.7.5a jellyfish/2.1.1 RepeatMasker/4.0.3
augustus/2.7 cegma/2.4 MAFFT/7.130 RepeatModeler/1.0.7
bedtools/2.18.0 cufflinks/2.1.1 maker/2.28 samtools/0.1.12a
BLAST+/2.2.28 exonerate/2.2.0-x86_64 muscle/3.8.31 samtools/0.1.19
blat/v35 geneid/1.4.4 plink/1.07 snap/2013-11-29
bowtie/2-2.2.1 genewise/2.2.3-rc7 provean/1.1.3 SOAPdenovo2/r240
bwa/0.5.9 hmmer/3.1b1 RAxML/8.0.0 tophat/2.0.11

== See also ==
* [[Main_Page#Using_the_B4F_Cluster | Using the B4F Cluster]]
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[ABGC_modules | modules specific for ABGC ]]

== External links ==