Haars001: /* Gaining access to Anunna */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:bexke002 | Stefan Bexkens (Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Ssh without password

2021-01-08T10:22:53Z

Haars001:

Secure shell (ssh) protocols can be configure to work without protocols. This is particularly helpful for machines that are used often.

== Configuring ssh without password from a POSIX-compliant terminal ==

=== Step 1: create a public key and copy to remote computer ===
* Log into a local Linux or MacOSX computer
* Type the following to generate the ssh key:
<source lang='bash'>
ssh-keygen -t ed25519 -a 200 -C $USER@$(hostname)
</source>
* Accept the default key location by pressing <code>Enter</code>.
* Please use a different password/passphrase for your SSH key than your WUR password.
* Secure permission of your authentication keys by closing permission to your home directory, .ssh directory, and authentication files
<source lang='bash'>
chmod go-wx $HOME
chmod 700 $HOME/.ssh
chmod 600 $HOME/.ssh/*
</source>
* Type the following to copy the key to the remote server (this will prompt for a password).
<source lang='bash'>
ssh-copy-id remote_username@remote_host
</source>

== Configuring ssh without password for Anunna ==

* Create a public key as in Step 1 of the previous section and copy it to Anunna. Note that a public/private key pair needs to be made only once per machine.
* Similar to step 2 of the previous section, add the public key to the <code>$HOME/.ssh/authorized_keys2</code> file. There is already a <code>$HOME/.ssh/authorized_keys</code> present. You may append the key to this file as an alternative, but take care not to remove content that is already there. The cluster is configured so that passwordless communication will all other nodes is default.

== Configuring ssh without password using PuTTY ==
Use pAGEaNT: http://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter9.html to generate local keys. You'll want have a copy of the pubkey in plaintext available.

Make sure to paste that plaintext string into ~/.ssh/authorized_keys in one single line. Chmod the file 600 (so it shows -rw------- in ls -l) and the directory .ssh to 700 (drwx------).

Now PuTTY will login passwordlessly whenever pAGEaNT is running.

Finally, get pAGEaNT to load on startup: http://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html

== Configuring ssh without password on a Mac ==
* Create a public key as in Step 1 of the first section and copy it to Anunna.
* Add the passphrase that you entered above to the keychain on your mac:
ssh-add -K /path/to/private/key/file

== Selecting which settings to use ==

To have your SSH client to use certain settings, one can use a config file, at ~/.ssh/config

For example :
<source lang='bash'>

Host *.wurnet.nl *.wur.nl
User haars001
Compression no
RequestTTY force

Host *
Compression yes
Protocol 2
ServerAliveInterval 120
ServerAliveCountMax 50
TCPKeepAlive no
ConnectTimeout 60
IdentityFile ~/.ssh/id_ed25519
AddKeysToAgent yes

</source>

As the config file is used top to bottom, the connection wur(net).nl servers will be using no compression, but the rest of the servers you might access will.
More options and settings can be found by using `man ssh_config`

== See also ==
* [[log_in_to_Anunna | Logging into cluster using ssh and file transfer]]

== External Links ==

Ssh without password

2021-01-08T10:06:17Z

Haars001: Make ed25519 the default

Running scripts on a fixed timeschedule (cron)

2020-10-20T13:24:09Z

Haars001:

== Using crontab ==

With crontab you can run jobs on a fixed time schedule.

This means that you can e.g. download some data every day.

To start an edit use '''crontab -l''' , the info on how that file should look can be found by using '''man 5 crontab'''.

Be aware of the following :

The scripts will run on the login node, so do not use a lot of resources.
The crontab entry will be wiped upon reboot !

Running scripts on a fixed timeschedule (cron)

2020-10-20T13:21:49Z

Haars001:

== Using crontab ==

With crontab you can run jobs on a fixed time schedule.

This means that you can e.g. download some data every day.

To start an edit with '''crontab -l''' , the info on that file can be found by using '''man 5 crontab'''.

Be aware of the following :

The scripts will run on the login node, so do not use a lot of resources
The crontab entry will be wiped upon reboot, so if you want to make sure that it is there, it wise to set that up.

Running scripts on a fixed timeschedule (cron)

2020-10-20T13:21:20Z

Haars001: Created page with " == Using crontab == With crontab you can run jobs on a fixed timeschedule. This means that you can e.g. download some data every day. To start an edit with '''crontab -l'..."

== Using crontab ==

With crontab you can run jobs on a fixed timeschedule.

This means that you can e.g. download some data every day.

To start an edit with '''crontab -l''' , the info on that file can be found by using '''man 5 crontab'''.

Be aware of the following :

The scripts will run on the login node, so do not use a lot of resources
The crontab entry will be wiped upon reboot, so if you want to make sure that it is there, it wise to set that up.

Main Page

2020-10-20T12:50:35Z

Haars001: /* Installation of software by users */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:bexke002 | Stefan Bexkens (Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2020-10-20T08:51:04Z

Haars001: /* External links */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:bexke002 | Stefan Bexkens (Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2020-10-20T08:48:40Z

Haars001: /* Access Policy */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Accessible storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =
* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Project Leader of Anunna is Stephen Janssen (Wageningen UR,FB-IT, Service Management). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, Infrastructure)]] and [[User:bexke002 | Stefan Bexkens (Wageningen UR,FB-IT, Infrastructure)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]
* [http://www.cobb-vantress.com Cobb-Vantress homepage]

| width="30%" |
* [https://www.crv4all.nl CRV homepage]
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]
* [http://www.topigs.com TOPIGS homepage]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Tariffs

2020-10-20T08:41:54Z

Haars001: /* Notes */

== Computing: Calculations (cores)==
{| class="wikitable"
!Queue
!CPU core hour
!GB memory hour
|-
|Standard queue
|€ 0.0150
|€ 0.0015
|-
|High priority queue
|€ 0.0200
|€ 0.0020
|-
|Low priority queue
|€ 0.0100
|€ 0.0010
|}

== Computing: GPU Use==
{| class="wikitable"
!Tariff per device per hour (gpu/hour)
|-
|€ 0.3000
|}

== Storage ==
Tariffs per year per TB
{| class="wikitable"
!Lustre Backup
!Lustre Nobackup
!Lustre Scratch
!Home-dir
!Archive
|-
|€ 175
|€ 125
|€ 125
|€ 175
|€ 125
|}

== Reservations ==
{| class="wikitable"
!Tariff per node per day (node/day)
|-
|€ 30
|}

== Notes==

If you are a member of a group with a commitment, then these costs get deducted from that commitment.
Once you get to around 125% of your commitment we will take action to fix things.

== Example ==

You are running a job that needs 4 cores, 32G of RAM and runs for 90 minutes in the std quality. To run this, you over-request resources slightly, and execute in a job that requests 4 CPUs, 40G of RAM and with a time limit of 3 hours. Your job terminates early. Thus, your costs are:

4 * 0.015 * 1.5 = 0.09 EUR for the CPU

40 * 0.0015 * 1.5 = 0.09 EUR for the memory

Total: 0.18 EUR

Shared folders

2020-09-04T07:38:29Z

Haars001: /* ACL shared directories */

== Working with shared folders in the Lustre file system ==

If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports Anunna.

There are two main methods available to you: Access Control List (ACL) access (that you can administer yourself) or group access (that is centrally administered).

== ACL shared directories ==
You may create a folder that can be accessed by yourself and someone else in the following manner:

<source lang='bash'>
cd /lustre/shared
mkdir shared_folder
chmod 700 shared_folder
setfacl -R -m u:my_id:rwx shared_folder
setfacl -R -d -m u:my_id:rwx shared_folder
</source>

Then, for each person who you want to have access to this:
<source lang='bash'>
setfacl -R -m u:my_friend:rwx shared_folder
setfacl -R -d -m u:my_friend:rwx shared_folder
</source>

Adding users later can be done using the same method, but it might be hard.
You may have trouble updating ACLs on files that aren't yours, and you cannot change ownership of files to yourself.
Each user with files in the folder will need to update their ACLs appropriately themselves, or you can contact your sysadmins to assist.

== Group shared directories ==

Users access Anunna cluster with their WUR-wide account. This means that all the membership information is also available on Anunna. To check of which groups is your user a member of, use the following command:

<code>groups <username></code>

This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.

For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:

<code>groups duque004 | grep isric</code>

In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.

=== Creating a shared folder with correct permissions ===

The Lustre file system is accessible in the <code>/lustre</code> folder and then divided into the <code>/backup</code> and <code>/nobackup</code> sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named <code>SHARED</code> in which users are to create their own assets.

You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:

<code>mkdir /lustre/nobackup/SHARED/myTeamWorkspace</code>

Or in alternative:

<code>cd /lustre/nobackup/SHARED

mkdir myTeamWorkspace</code>

=== Setting permissions ===

Three basic steps are involved in stepping permissions correctly:

1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:

<code>chgrp -R my-team-group myTeamWorkspace</code>

2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the <code>w</code> character from the <code>+rw</code> bit:

<code>chmod -R g+rw myTeamWorkspace</code>

3. Set default ownership within the group. This guarantees that any new files or folders created within the shared folder are owned by default owned by your team group:

<code>chmod -R g+s myTeamWorkspace</code>

In case the contents of the shared are sensitive or private, and should be accessed by your team, you can block access from any other users with the following command:

<code>chmod -R o-rw myTeamWorkspace</code>

== Further reading ==

[https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-permissions An Introduction to Linux Permissions]

[https://www.linode.com/docs/tools-reference/linux-users-and-groups/ Linux Users and Groups]

[https://www.digitalocean.com/community/tutorials/linux-permissions-basics-and-how-to-use-umask-on-a-vps#types-of-permissions Linux Permissions Basics and How to Use Umask on a VPS]

[http://www.yolinux.com/TUTORIALS/LinuxTutorialManagingGroups.html Linux Tutorial - Managing Group Access on Linux and UNIX]

Using conda to install a new kernel into your notebook

2020-07-10T14:42:51Z

Haars001: Created page with "As you can read here, you can add a new kernel to your Jupyter manually. If you use conda, the following steps..."

As you can read [[Setting_up_Python_virtualenv#Virtualenv_kernels_in_Jupyter|here]], you can add a new kernel to your Jupyter manually.

If you use conda, the following steps can be used to create the necessary files, and share your environment :

Install:

<nowiki>
conda create -y -n kernel_test python=3 ipykernel && conda activate kernel_test
python -m ipykernel install --user --name kernel_test</nowiki>

Cleanup (make sure you have activated the right enviroment):

<nowiki>
jupyter kernelspec uninstall kernel_test
conda deactivate && conda remove -y -n kernel_test</nowiki>

Setting up Python virtualenv

2020-07-10T14:36:54Z

Haars001: Add link to page describing conda alternative

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<source lang='bash'>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</source>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<source lang='bash'>
source newenv/bin/activate
</source>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<source lang='bash'>
pip install numpy
</source>
Similarly, installing packages from source works exactly the same as usual.
<source lang='bash'>
python setup.py install
</source>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<source lang='bash'>
deactivate
</source>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

(an alternative way to the manual way (using conda) is described [[Using conda to install a new kernel into your notebook|here ]])

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"/home/myhome/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/cm/shared/apps/spark/my-spark-version"
},
"argv": [
"/home/myhome/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<source lang='bash'>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</source>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Using environment modules

2020-07-10T14:18:09Z

Haars001:

=== Environment Modules ===
[http://modules.sourceforge.net/ Environment modules] are a simple way to allow multiple potentially clashing programs to coexist on a large shared machine such as an HPC. It allows a user to specify exactly which programs are loaded, and even which version of each program, whilst simultaneously allowing the administrator the ability to automatically configure the appropriate environment variables for the system itself.

== Viewing Modules ==
Upon logging in to Anunna, you should find that when you do:
module list

You will see something like this:
<source lang='bash'>
-bash-4.1$ module list
Currently Loaded Modulefiles:
1) shared 2) slurm/2.5.7
</source>

This is a list of all loaded modules in your shell session. To get a list of all available modules, simply
module available

And this will show you the (very exhaustive) list of modules on Anunna:

<source lang='bash'>
-bash-4.1$ module avail

---------------------------- /cm/shared/modulefiles ----------------------------
acml/gcc/64/5.3.1 netcdf/gcc/64/4.1.3
acml/gcc/fma4/5.3.1 netcdf/gcc/64/4.3.0
acml/gcc/mp/64/5.3.1 netcdf/gcc/64/4.3.2
acml/gcc/mp/fma4/5.3.1 netcdf/gcc/64/4.3.3
acml/gcc-int64/64/5.3.1 netcdf/gcc/64/4.3.3.1
acml/gcc-int64/fma4/5.3.1 netcdf/intel/64/4.1.3
...
</source>

Let's look at each of these module names. Each module is named for the application it provides, plus a subfolder of which compiler it was compiled with (if compiled), the number of address bits or options (if compiled), and the version.

If you want to see a list for a specific module, you can
module avail netcdf

And the complete list of versions will be shown.

== Loading Modules ==
To load a module, simply
module load foo

And the most recent version of module foo will automatically be loaded. If foo is compiled, it will automatically select the gcc version. If you want to specify a certain version, then
module load foo/gcc/64/1.0.0

Will load foo version 1, compiled with gcc. Be advised that this may not always work, as some modules are not compatible with each other, but a message will be shown if this is the case. Additionally, some modules will automatically load other modules with them for them to operate.

== Unloading Modules ==
If you want to remove a module that you've loaded, then
module unload foo

Will remove all module foo's loaded.

== Example ==
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

This script will not run at all in the default 2.4 version of Python on the cluster. In order for this script to run you must use Python3. To do this, first list all versions of Python:
<source lang='bash'>
-bash-4.1$ module avail python

---------------------------- /cm/shared/modulefiles ----------------------------
python/2.7.6 python/3.3.3 python/3.4.2
</source>

Then you can load the specific version you need:
module load python/3.3.3

Now you have access to the executable python3.

== See also ==
* [[Environment_Modules | Environment Modules]]
* [[Control_R_environment_using_modules | Control R environment using modules]]
* [[Create_shortcut_log-in_command | Create a shortcut for the ssh log-in command]]
* [[Installing_R_packages_locally | Installing R packages locally]]

== External links ==
* http://modules.sourceforge.net
* https://modules.readthedocs.io/en/latest/ (documentation)
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Access Policy

2020-06-30T12:46:55Z

Haars001:

Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to Anunna. Access needs to be granted actively (by creation of an account on the cluster by FB-IT reagdring non WUR accounts). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated.

== Contact Persons ==
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):

=== WUR ===
==== ESG ====
* Ronald Hutjes
* Reinder Ronda

==== Bioinformatics ====
* Dick de Ridder

==== PRI ====
* Sara Diaz Trivino

==== ABGC ====
===== Animal Breeding and Genetics =====
* [[User:Hulze001 |Alex Hulzebosch]]
* [[User:Megen002 | Hendrik-Jan Megens]]
===== Wageningen Livestock Research =====
* Mario Calus
* Ina Hulsegge

=== Cobb-Vantress ===
* Wes Barris
* Jun Chen
=== CRV ===
* Frido Hamoen
* Chris Schrooten
=== Hendrix Genetics ===
* Ton Dings
* Abe Huisman
* Addie Vereijken
=== Topigs ===
* [[User:dongen01 | Henk van Dongen]]
* Egiel Hanenbarg
* Naomi Duijvensteijn

== See also ==
[[Main_Page | Main page]]

Log in to Anunna

2020-06-08T09:21:37Z

Haars001: /* Samba/CIFS based protocols */

== Log on using ssh ==
One can log in to [[Anunna | Anunna]] using ssh (default port tcp 22). The address of the login server is:
login.anunna.wur.nl

You will be automatically redirected to the currently valid login server. To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Windows an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the Login server can only act as access point and is not to be used for any serious CPU or RAM intensive work.'''
'''Anything requiring even moderate resources should be [[Using_Slurm |scheduled using SLURM!]]'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@login.anunna.wur.nl
</source>

=== PuTTY on Windows ===
Putty is a free, powerful, and widely used SSH client that runs on Windows.
It is extremely useful for those people who have a computer running Windows
on their desk but must remotely connect to a computer running UNIX/Linux.
Putty is one of a set of utilities that all work together to provide
convenient connectivity between Windows and UNIX/Linux environments.
Some of these utilities include:

* Putty -- the SSH client
* Pageant -- the authentication agent used with Putty
* Puttygen -- the RSA key generation utility
* Pscp -- the SCP secure file copy utility

Depending on your tasks, the above utilities are probably your minimum
set of tools to make convenient connections and file transfers between a
computer running Windows and a computer running UNIX/Linux.

==== Putty Configuration ====

Putty is able to store the configuration or connection profiles for a
number of remote UNIX/Linix clients. Each of profile can be created
and later edited by Right-clicking on a putty window header and choosing
"New Session...". The minimum set of items that need to be configured for
a given connection are:

* Session
** Host Name [login.anunna.wur.nl]
** Saved Session name [your name for this connection]
* Terminal
** Keyboard
*** Backspace key -> Control-H
* Connection
** Data
*** Auto-login username [your remote username]
** SSH
*** Auth
**** Private key file for authentication [pathname to your .ppk file]

Obviously, there are many other useful things that can be configured and
customized in Putty but the above list should be considered a minimum.
Please note that after making any change to a putty session you must
explicitly save your changes.

==== Creating an SSH Key Pair ====

Puttygen is the utility used for creating both a .ppk file (private
key) and the public authorized key information. Briefly, here are
the steps needed to create a key pair:

* Run (double-click) the Puttygen application
* Click on "Generate"
* Replace the comment with something meaningful -- maybe your name
* Type in your passphrase (password) twice
* Save the .ppk file in a secure location on your Windows computer
* Use your mouse to copy the public key string then paste it into the ~/.ssh/authorized_keys file on the remote computer

Note: The full pathname of this .ppk file is used in the last step of Putty
configuration as described above.

==== Using Pageant as an Interface for Putty ====

Pageant is a Putty helper program that is used for two main purposes:

* Pageant is used to hold the passphrase to your key pair
* Pageant is used as a convenience application to run a Putty session from any of your saved profiles

There is no configuration needed in Pageant. You simply need to
run this program at login. Any easy way to do this is to create a
shortcut in your startup folder that points to the Pageant executable.
Once this has been done, every time you log in you will see a little
icon of a computer with a hat in your taskbar. The first step in using
this is to right-click on it and select "Add Key". Navigate to your
.ppk file and select "Open". It will prompt you for your passphrase.
At this point your passphrase has been conveniently stored for you so
that when you use Putty to connect to your various remote computers,
you won't have to type in your passphrase over and over again.
The next step is to right-click on the Pageant icon again and select
one of your saved sessions. If you have done everything correctly
you will be logged right in so that you no longer have to type your
passphrase.

== Log on to worker nodes ==

In a complete emergency, it is then possible to log on to any of the worker nodes via the login node. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. This is not normally allowed - be aware that running tasks outside of SLURM is prohibited, but so far there has not been any serious abuse of this. This is provided to allow you to get a little more insight in what your job is doing.

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

Again, it is not permitted to run jobs outside the scheduling software (slurm). So logging on to a worker node is for analyses of running jobs only.

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== WinSCP ===
[http://en.wikipedia.org/wiki/WinSCP WinSCP] is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, password and server type (Unix, see Site Manager;Advanced), files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on Anunna. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet.

There are two mount points that are available
# your home folder ( '''\\cifs.anunna.wur.nl\[username]''' )
# the Lustre mount ( '''\\cifs.anunna.wur.nl\lustre''' )

You can enter these in the location bar of File Explorer.

== See also ==
* [[Using _Slurm | Submit jobs with Slurm]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Log in to Anunna

2020-06-08T09:19:08Z

Haars001: /* Samba/CIFS based protocols */

== Log on using ssh ==
One can log in to [[Anunna | Anunna]] using ssh (default port tcp 22). The address of the login server is:
login.anunna.wur.nl

You will be automatically redirected to the currently valid login server. To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Windows an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the Login server can only act as access point and is not to be used for any serious CPU or RAM intensive work.'''
'''Anything requiring even moderate resources should be [[Using_Slurm |scheduled using SLURM!]]'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@login.anunna.wur.nl
</source>

=== PuTTY on Windows ===
Putty is a free, powerful, and widely used SSH client that runs on Windows.
It is extremely useful for those people who have a computer running Windows
on their desk but must remotely connect to a computer running UNIX/Linux.
Putty is one of a set of utilities that all work together to provide
convenient connectivity between Windows and UNIX/Linux environments.
Some of these utilities include:

* Putty -- the SSH client
* Pageant -- the authentication agent used with Putty
* Puttygen -- the RSA key generation utility
* Pscp -- the SCP secure file copy utility

Depending on your tasks, the above utilities are probably your minimum
set of tools to make convenient connections and file transfers between a
computer running Windows and a computer running UNIX/Linux.

==== Putty Configuration ====

Putty is able to store the configuration or connection profiles for a
number of remote UNIX/Linix clients. Each of profile can be created
and later edited by Right-clicking on a putty window header and choosing
"New Session...". The minimum set of items that need to be configured for
a given connection are:

* Session
** Host Name [login.anunna.wur.nl]
** Saved Session name [your name for this connection]
* Terminal
** Keyboard
*** Backspace key -> Control-H
* Connection
** Data
*** Auto-login username [your remote username]
** SSH
*** Auth
**** Private key file for authentication [pathname to your .ppk file]

Obviously, there are many other useful things that can be configured and
customized in Putty but the above list should be considered a minimum.
Please note that after making any change to a putty session you must
explicitly save your changes.

==== Creating an SSH Key Pair ====

Puttygen is the utility used for creating both a .ppk file (private
key) and the public authorized key information. Briefly, here are
the steps needed to create a key pair:

* Run (double-click) the Puttygen application
* Click on "Generate"
* Replace the comment with something meaningful -- maybe your name
* Type in your passphrase (password) twice
* Save the .ppk file in a secure location on your Windows computer
* Use your mouse to copy the public key string then paste it into the ~/.ssh/authorized_keys file on the remote computer

Note: The full pathname of this .ppk file is used in the last step of Putty
configuration as described above.

==== Using Pageant as an Interface for Putty ====

Pageant is a Putty helper program that is used for two main purposes:

* Pageant is used to hold the passphrase to your key pair
* Pageant is used as a convenience application to run a Putty session from any of your saved profiles

There is no configuration needed in Pageant. You simply need to
run this program at login. Any easy way to do this is to create a
shortcut in your startup folder that points to the Pageant executable.
Once this has been done, every time you log in you will see a little
icon of a computer with a hat in your taskbar. The first step in using
this is to right-click on it and select "Add Key". Navigate to your
.ppk file and select "Open". It will prompt you for your passphrase.
At this point your passphrase has been conveniently stored for you so
that when you use Putty to connect to your various remote computers,
you won't have to type in your passphrase over and over again.
The next step is to right-click on the Pageant icon again and select
one of your saved sessions. If you have done everything correctly
you will be logged right in so that you no longer have to type your
passphrase.

== Log on to worker nodes ==

In a complete emergency, it is then possible to log on to any of the worker nodes via the login node. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. This is not normally allowed - be aware that running tasks outside of SLURM is prohibited, but so far there has not been any serious abuse of this. This is provided to allow you to get a little more insight in what your job is doing.

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

Again, it is not permitted to run jobs outside the scheduling software (slurm). So logging on to a worker node is for analyses of running jobs only.

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== WinSCP ===
[http://en.wikipedia.org/wiki/WinSCP WinSCP] is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, password and server type (Unix, see Site Manager;Advanced), files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on Anunna. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet.

There are two mount points that are available
# your home folder ( '''\\cifs.anunna.wur.nl\[username]''' )
# the Lustre mount ( '''\\cifs.anunna.wur.nl\lustre''' )

== See also ==
* [[Using _Slurm | Submit jobs with Slurm]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Node usage graph

2017-02-09T13:37:04Z

Haars001:

There is a graphing tool that uses elements directly from sacct to display information about the current cluster usage, node_usage_graph (located at /cm/shared/apps/accounting/node_usage_graph ).

Example:
<source lang="text">
[user@nfs01 ~]# /cm/shared/apps/accounting/node_usage_graph
node: |0% 100%|
fat001: ############################################
fat002: ######################################################
node001:#############################################################################################################
node002:#############################################################################################################
node003:########################################################################################
node004:#############################################################################################################
node005:#############################################################################################################
node006:#############################################################
node007:######################################################################################################
node008:###########################
node009:###########################
node010:########################################
node011:##########################################################################
node012:######################################################################################################
node013:########################################################################################
node014:######
node015:#############################################################################################################
node016:###############################################################################################
node017:########################################
node018:#############################################################################################################
node019:###########################
node020:###############################################################################################
node021:###########################
node022:###############################################
node023:########################################################################################
node024:########################################
node025:###############################################
node026:####################################################################
node027:######################################################################################################
node028:########################################################################################
node029:#############################################################################################################
node030:###########################
node031:######################################################
node032:#############
node033:#############################################################################################################
node034:#############
node035:###########################
node036:
node037:
node038:###########################
node039:###########################
node040:######################################################################################################
node041:###########################
node042:###########################
node049:#############################################################
node050:######################################################
node051:######################################################################################################
node052:#############################################################################################################
node053:###############################################
node054:#############################################################################################################
</source>

This gives an overview of the current per-node resource usage. It cannot however give you an indication of how much the queue is right now for any node.

User:Haars001

2013-12-19T19:49:33Z

Haars001: /* Assistant Research at Wageningen UR, Plant Research International, BU Bioscience, Applied Bioinformatics */

== Jan van Haarst ==

[[File:Haarst,_Jan_van_-_S_-_Portrait_-_1120x1654px_-_GA--20131010-ND7_2171.jpg|200px|right]]
=== Assistant Research at Wageningen UR, Plant Research International, BU Bioscience, Applied Bioinformatics ===
* Profile on [https://www.vcard.wur.nl/Views/Profile/View.aspx?id=3014 We@WUR]
* Profile on [https://www.linkedin.com/in/jvhaarst LinkedIn]
* Profile on [http://scholar.google.com/citations?user=eoB6EPcAAAAJ Google Scholar]

My main line of work revolves around de novo assembly projects. Mainly plants, but every now and then something else comes along.
Besides that, I need to tinker on our own infrastructure, and help colleagues from our BU Bioscience with their bioinformatics analyses.
These range from degradome analysis up to BiSulfite Sequencing analysis, and all those short things that pop up every week.

File:Haarst, Jan van - S - Portrait - 1120x1654px - GA--20131010-ND7 2171.jpg

2013-12-19T19:45:53Z

Haars001:

User:Haars001

2013-12-19T19:45:01Z

Haars001: /* Jan van Haarst */

== Jan van Haarst ==

=== Assistant Research at Wageningen UR, Plant Research International, BU Bioscience, Applied Bioinformatics ===

* Profile on [https://www.vcard.wur.nl/Views/Profile/View.aspx?id=3014 We@WUR]
* Profile on [https://www.linkedin.com/in/jvhaarst LinkedIn]
* Profile on [http://scholar.google.com/citations?user=eoB6EPcAAAAJ Google Scholar]

My main line of work revolves around de novo assembly projects. Mainly plants, but every now and then something else comes along.
Besides that, I need to tinker on our own infrastructure, and help colleagues from our BU Bioscience with their bioinformatics analyses.
These range from degradome analysis up to BiSulfite Sequencing analysis, and all those short things that pop up every week.

User:Haars001

2013-12-19T19:44:16Z

Haars001: Created page with "== Jan van Haarst == * Profile on [https://www.vcard.wur.nl/Views/Profile/View.aspx?id=3014 We@WUR] * Profile on [https://www.linkedin.com/in/jvhaarst LinkedIn] * Profile on ..."

== Jan van Haarst ==

* Profile on [https://www.vcard.wur.nl/Views/Profile/View.aspx?id=3014 We@WUR]
* Profile on [https://www.linkedin.com/in/jvhaarst LinkedIn]
* Profile on [http://scholar.google.com/citations?user=eoB6EPcAAAAJ Google Scholar]

Senior Researcher and lecturer at Wageningen University, Animal Breeding and Genomics Centre.
Assistant Research at Wageningen UR, Plant Research International, BU Bioscience, Applied Bioinformatics

My main line of work revolves around de nov assembly projects. Mainly plants, but every now and then something else comes along.
Besides that, I need to tinker on our own infrastructure, and help colleagues from our BU Bioscience with their bioinformatics analyses.
These range from degradome analysis up to BiSulfite Sequencing analysis, and all those short things that pop up every week.