HPCwiki - User contributions [en]

ABGC modules

2015-01-23T15:20:12Z

Hulze001: /* Modules available for ABGC */

All ABGC modules can be found in:

/cm/shared/apps/WUR/ABGC/modulefiles

== Modules available for ABGC ==

* [[asreml_3.0 | asreml/3.0fl-64]]
* [[asreml_4.0 | asreml/4.0kr]]
* [[asreml_4.1 | asreml/4.1.0]]
* [[Perl5.10_WUR_module | Perl/5.10.1_wur]]
* [[R3.0.2_WUR_module | R/3.0.2_wur]]

== Adding a custom module directory to your environment ==
To allow the <code>module</code> program to find the custom module directory, the location of that directory has to be added to <code>MODULEPATH</code> variable.

<source lang='bash'>
export MODULEPATH=$MODULEPATH:/cm/shared/apps/WUR/ABGC/modulefiles
</source>
This can be made permanent by adding this line of code to the <code>.bash_profile</code> file in the root of your home folder. To then load the modified <code>MODULEPATH</code> variable you have to load <code>.bash_profile</code> again:
<source lang='bash'>
source .bash_profile
</source>
This needs to be done only for terminals that are already open. Next time you login, <code>.bash_profile</code> will be loaded automatically.

You can check if the modules are found.
<source lang='bash'>
module avail
</source>
This should give output that includes something similar to this:

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a

== See also ==
* [[Main_Page#Using_the_B4F_Cluster | Using the B4F Cluster]]
* [[Globally_installed_software | Globally installed software]]
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==

Using Slurm

2014-09-19T09:37:27Z

Hulze001: /* Queues */

The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.

== Queues and defaults ==

=== Queues ===
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue.<br>
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0)<br>
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are occupied by a Low queue jobs.
To find out which queues your account has been authorized for, type sinfo:
<source lang='bash'>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
ABGC_High up infinite 12 down* node[043-048,055-060]
ABGC_High up infinite 6 mix fat[001-002],node[002-005]
ABGC_High up infinite 44 idle node[001,006-042,049-054]
ABGC_Std up infinite 12 down* node[043-048,055-060]
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]
ABGC_Std up infinite 44 idle node[001,006-042,049-054]
ABGC_Low up infinite 12 down* node[043-048,055-060]
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]
ABGC_Low up infinite 44 idle node[001,006-042,049-054]
</source>

=== Defaults ===
There is no default queue, so you need to specify which queue to use when submitting a job.<br>
'''The default run time for a job is 1 hour!''' <br>
'''Default memory limit is 100MB per node!'''

== Submitting jobs: sbatch ==

=== Example ===
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

=== Loading modules ===
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:
module avail

In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:
module load python/3.3.3

=== Batch script ===
The following shell/slurm script can then be used to schedule the job using the sbatch command:
<source lang='bash'>
#!/bin/bash
#SBATCH --account=773320000
#SBATCH --time=1200
#SBATCH --mem=2048
#SBATCH --ntasks=1
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=calc_pi.py
#SBATCH --partition=ABGC_Std
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@org.nl

time python3 calc_pi.py
</source>
Explanation of used SBATCH parameters:
<source lang='bash'>
#SBATCH --account=773320000
</source>
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.
<source lang='bash'>
#SBATCH --time=1200
</source>
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.

----

<source lang='bash'>
#SBATCH --mem=2048
</source>
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission:
<source lang='bash'>
#SBATCH --mem X
</source>

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:
<source lang='bash'>
$ sacct -o MaxRSS -j JOBID
</source>
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.

----

<source lang='bash'>
#SBATCH --ntasks=1
</source>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:
<source lang='bash'>
#SBATCH --nodes=1
</source>
This should force your job to be scheduled to a single node.

Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.
<source lang='bash'>
#SBATCH --constraint=normalmem
</source>
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes.

<source lang='bash'>
#SBATCH --output=output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --error=error_output_%j.txt
</source>
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.
<source lang='bash'>
#SBATCH --job-name=calc_pi.py
</source>
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.
<source lang='bash'>
#SBATCH --partition=ABGC_Std
</source>
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.
<source lang='bash'>
#SBATCH --mail-type=ALL
</source>
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
<source lang='bash'>
#SBATCH --mail-user=email@org.nl
</source>
Email address to use.

=== Submitting ===
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:
<source lang='bash'>
sbatch run_calc_pi.sh
</source>

=== Submitting multiple jobs ===
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done
</source>

=== Interactive X11/GUI jobs ===
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:
<source lang='bash'>module load hpl/2.1
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl
</source>

== Monitoring submitted jobs ==
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.

=== Generic monitoring of all running jobs ===
<source lang='bash'>
squeue
</source>

You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009
3385 research BOV-WUR- megen002 R 44:38 1 node049
3386 research BOV-WUR- megen002 R 44:38 1 node050
3387 research BOV-WUR- megen002 R 44:38 1 node051
3388 research BOV-WUR- megen002 R 44:38 1 node052
3389 research BOV-WUR- megen002 R 44:38 1 node053
3390 research BOV-WUR- megen002 R 44:38 1 node054
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]
3393 research BOV-WUR- megen002 R 44:38 1 node001
3394 research BOV-WUR- megen002 R 44:38 1 node002
3395 research BOV-WUR- megen002 R 44:38 1 node003

=== Monitoring time limit set for a specific job ===
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.
<source lang='bash'>
squeue -l -j 3532
</source>
Information similar to the following should appear:
Fri Nov 29 15:41:00 2013
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054

=== Query a specific active job: scontrol ===
Show all the details of a currently active job, so not a completed job.
<source lang='bash'>
nfs01 ~]$ scontrol show jobid 4241
JobId=4241 Name=WB20F06
UserId=megen002(16795409) GroupId=domain users(16777729)
Priority=1 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=research AllocNode:Sid=nfs01:21799
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node023
BatchHost=node023
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/lustre/scratch/WUR/ABGC/...
WorkDir=/lustre/scratch/WUR/ABGC/...
</source>

=== Check on a pending job ===
A submitted job could result in a pending state when there are not enough resources available to this job.
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.
<source lang='bash'>
[@nfs01 jobs]$ sbatch hpl_student.job
Submitted batch job 740338

[@nfs01 jobs]$ squeue -l -j 740338
Fri Feb 21 15:32:31 2014
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)

[@nfs01 jobs]$ squeue --start -j 740338
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)
</source>
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.

== Removing jobs from a list: scancel ==
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:
<source lang='bash'>
scancel 3401
</source>

== Allocating resources interactively: sallocate ==
< text here>

== Get overview of past and current jobs: sacct ==
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:
<source lang='bash'>
sacct
</source>
This should provide information similar to the following:

JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
3385 BOV-WUR-58 research 12 COMPLETED 0:0
3385.batch batch 1 COMPLETED 0:0
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0
3386.batch batch 1 CANCELLED 0:15
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0

Or in more detail for a specific job:
<source lang='bash'>
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220
</source>
This should provide information about job id 4220:

JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- --------
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0
4220.batch batch 1 1 00:30:52 COMPLETED 0:0

'''Job Status Codes'''

Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.

{| class="wikitable"
|-
!Code!!State!!Description
|-
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
|-
|CD|| COMPLETED|| Job has terminated all processes on all nodes.
|-
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
|-
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.
|-
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.
|-
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.
|-
|PD|| PENDING|| Job is awaiting resource allocation.
|-
|R|| RUNNING|| Job currently has an allocation.
|-
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.
|-
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.
|-
|-
|}

== Running MPI jobs on B4F cluster ==

[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]
< text here >

== Understanding which resources are available to you: sinfo ==
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.

<source lang='bash'>
sinfo
</source>

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
student* up infinite 12 down* node[043-048,055-060]
student* up infinite 50 idle fat[001-002],node[001-042,049-054]
research up infinite 12 down* node[043-048,055-060]
research up infinite 50 idle fat[001-002],node[001-042,049-054]
ABGC up infinite 12 down* node[043-048,055-060]
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]
* [[SLURM_Compare | SLURM compared to other common schedulers]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==
* [http://slurm.schedmd.com Slurm official documentation]
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]

Log in to Anunna

2014-09-19T09:14:38Z

Hulze001: /* winSCP */

== Log on using ssh ==
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh (default port tcp 22). The address of the nfs server is:
nfs01.hpcagrogenomics.wur.nl

To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work.'''
'''Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@nfs01.hpcagrogenomics.wur.nl
</source>

=== PuTTY on Windows ===
Putty is a free, powerful, and widely used SSH client that runs on Windows.
It is extremely useful for those people who have a computer running Windows
on their desk but must remotely connect to a computer running UNIX/Linux.
Putty is one of a set of utilities that all work together to provide
convenient connectivity between Windows and UNIX/Linux environments.
Some of these utilities include:

* Putty -- the SSH client
* Pageant -- the authentication agent used with Putty
* Puttygen -- the RSA key generation utility
* Pscp -- the SCP secure file copy utility

Depending on your tasks, the above utilities are probably your minimum
set of tools to make convenient connections and file transfers between a
computer running Windows and a computer running UNIX/Linux.

==== Putty Configuration ====

Putty is able to store the configuration or connection profiles for a
number of remote UNIX/Linix clients. Each of profile can be created
and later edited by Right-clicking on a putty window header and choosing
"New Session...". The minimum set of items that need to be configured for
a given connection are:

* Session
** Host Name [nfs01.hpcagrogenomics.wur.nl]
** Saved Session name [your name for this connection]
* Terminal
** Keyboard
*** Backspace key -> Control-H
* Connection
** Data
*** Auto-login username [your remote username]
** SSH
*** Auth
**** Private key file for authentication [pathname to your .ppk file]

Obviously, there are many other useful things that can be configured and
customized in Putty but the above list should be considered a minimum.
Please note that after making any change to a putty session you must
explicitly save your changes.

==== Creating an SSH Key Pair ====

Puttygen is the utility used for creating both a .ppk file (private
key) and the public authorized key information. Briefly, here are
the steps needed to create a key pair:

* Run (double-click) the Puttygen application
* Click on "Generate"
* Replace the comment with something meaningful -- maybe your name
* Type in your passphrase (password) twice
* Save the .ppk file in a secure location on your Windows computer
* Use your mouse to copy the public key string then paste it into the ~/.ssh/authorized_keys file on the remote computer

Note: The full pathname of this .ppk file is used in the last step of Putty
configuration as described above.

==== Using Pageant as an Interface for Putty ====

Pageant is a Putty helper program that is used for two main purposes:

* Pageant is used to hold the passphrase to your key pair
* Pageant is used as a convenience application to run a Putty session from any of your saved profiles

There is no configuration needed in Pageant. You simply need to
run this program at login. Any easy way to do this is to create a
shortcut in your startup folder that points to the Pageant executable.
Once this has been done, every time you log in you will see a little
icon of a computer with a hat in your taskbar. The first step in using
this is to right-click on it and select "Add Key". Navigate to your
.ppk file and select "Open". It will prompt you for your passphrase.
At this point your passphrase has been conveniently stored for you so
that when you use Putty to connect to your various remote computers,
you won't have to type in your passphrase over and over again.
The next step is to right-click on the Pageant icon again and select
one of your saved sessions. If you have done everything correctly
you will be logged right in so that you no longer have to type your
passphrase.

== Log on to worker nodes ==

Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

It is not permitted to run jobs outside the scheduling software (slurm). So logging on to a worker node is for analyses of running jobs only.

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== WinSCP ===
[http://en.wikipedia.org/wiki/WinSCP WinSCP] is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (nfs01.hpcagronomics.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet. If you enter the following UNC path \\nfs01.hpcagrogenomics.wur.nl\[username] in your Windows client, it will list the available (authenticated) shares (your home directory).

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Log in to Anunna

2014-09-19T09:01:13Z

Hulze001: /* Samba/CIFS based protocols */

== Log on using ssh ==
One can log into the [[B4F_cluster | B4F Cluster]] (more specifically the nfs server) using ssh (default port tcp 22). The address of the nfs server is:
nfs01.hpcagrogenomics.wur.nl

To log on one has to use an ssh ([http://en.wikipedia.org/wiki/Secure_Shell secure shell]) client. Such client systems are always available from Linux or MacOS systems. For Window an ssh-client may need to be installed. The most popular ssh-client for Windows is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY].

Note that current access may be restricted to certain IP-ranges. Furthermore, ssh-protocols may be prohibited on systems where port 22 is unavailable due to firewall.

The ssh-connection can also be configured to work [[ssh_without_password | without password]], which means that no password needs to be provided at each log-in or secure copy attempt.

'''IMPORTANT: the NFS server can only act as access point and is not to be used for any serious CPU or RAM intensive work.'''
'''Anything requiring even moderate resources should be [[SLURM_on_B4F_cluster |scheduled using SLURM!]]'''

=== CLI from a Linux/MacOSX terminal ===
A Command Line Interface ([http://en.wikipedia.org/wiki/Command-line_interface CLI]) ssh client is available from any Linux or MacOSX terminal. Secure shell (ssh) protocols require port 22 to be open. Should a connection be refused, the firewall settings of the system should be checked. Alternatively, local ICT regulations may prohibit the use of port 22. Wageningen UR FB-ICT for instance does not allow traffic through port 22 over WiFi to certain systems.

<source lang='bash'>
ssh [user name]@nfs01.hpcagrogenomics.wur.nl
</source>

=== PuTTY on Windows ===
Putty is a free, powerful, and widely used SSH client that runs on Windows.
It is extremely useful for those people who have a computer running Windows
on their desk but must remotely connect to a computer running UNIX/Linux.
Putty is one of a set of utilities that all work together to provide
convenient connectivity between Windows and UNIX/Linux environments.
Some of these utilities include:

* Putty -- the SSH client
* Pageant -- the authentication agent used with Putty
* Puttygen -- the RSA key generation utility
* Pscp -- the SCP secure file copy utility

Depending on your tasks, the above utilities are probably your minimum
set of tools to make convenient connections and file transfers between a
computer running Windows and a computer running UNIX/Linux.

==== Putty Configuration ====

Putty is able to store the configuration or connection profiles for a
number of remote UNIX/Linix clients. Each of profile can be created
and later edited by Right-clicking on a putty window header and choosing
"New Session...". The minimum set of items that need to be configured for
a given connection are:

* Session
** Host Name [nfs01.hpcagrogenomics.wur.nl]
** Saved Session name [your name for this connection]
* Terminal
** Keyboard
*** Backspace key -> Control-H
* Connection
** Data
*** Auto-login username [your remote username]
** SSH
*** Auth
**** Private key file for authentication [pathname to your .ppk file]

Obviously, there are many other useful things that can be configured and
customized in Putty but the above list should be considered a minimum.
Please note that after making any change to a putty session you must
explicitly save your changes.

==== Creating an SSH Key Pair ====

Puttygen is the utility used for creating both a .ppk file (private
key) and the public authorized key information. Briefly, here are
the steps needed to create a key pair:

* Run (double-click) the Puttygen application
* Click on "Generate"
* Replace the comment with something meaningful -- maybe your name
* Type in your passphrase (password) twice
* Save the .ppk file in a secure location on your Windows computer
* Use your mouse to copy the public key string then paste it into the ~/.ssh/authorized_keys file on the remote computer

Note: The full pathname of this .ppk file is used in the last step of Putty
configuration as described above.

==== Using Pageant as an Interface for Putty ====

Pageant is a Putty helper program that is used for two main purposes:

* Pageant is used to hold the passphrase to your key pair
* Pageant is used as a convenience application to run a Putty session from any of your saved profiles

There is no configuration needed in Pageant. You simply need to
run this program at login. Any easy way to do this is to create a
shortcut in your startup folder that points to the Pageant executable.
Once this has been done, every time you log in you will see a little
icon of a computer with a hat in your taskbar. The first step in using
this is to right-click on it and select "Add Key". Navigate to your
.ppk file and select "Open". It will prompt you for your passphrase.
At this point your passphrase has been conveniently stored for you so
that when you use Putty to connect to your various remote computers,
you won't have to type in your passphrase over and over again.
The next step is to right-click on the Pageant icon again and select
one of your saved sessions. If you have done everything correctly
you will be logged right in so that you no longer have to type your
passphrase.

== Log on to worker nodes ==

Once logged into the nfs server, it is then possible to log on to any of the worker nodes. Logging on to the worker nodes does not require password authentication, you should therefore not be prompted to provide a password. Before logging onto a node it should be checked whether that node is busy. Status of nodes can be ascertained through the [[ BCM_on_B4F_cluster|BCM Portal]].

<source lang='bash'>
ssh [user name]@[node name]
</source>

For instance:

<source lang='bash'>
ssh dummy001@node049
</source>

It is not permitted to run jobs outside the scheduling software (slurm). So logging on to a worker node is for analyses of running jobs only.

== File transfer using ssh-based file transfer protocols ==
=== Copying files to/from the cluster: scp ===

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'>
scp <source> <destination>
</source>

<source lang='bash'>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>

This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].

<source lang='bash'>
man scp
</source>

=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
<source lang='bash'>
rsync <source> <destination>
</source>
e.g.:
<source lang='bash'>
rsync -av /home/WUR/[username]/folder_to_transfer [username]@nfs01.hpcagrogenomics.wur.nl:/lustre/scratch/WUR/ABGC/
</source>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
<source lang='bash'>
man scp
</source>

=== winSCP ===
<need a windows user as a volunteer to write some text....>

=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, and password, files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on the Cluster. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].

=== Samba/CIFS based protocols ===
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet. If you enter the following UNC path \\nfs01.hpcagrogenomics.wur.nl\[username] in your Windows client, it will list the available (authenticated) shares (your home directory).

== See also ==
* [[B4F_cluster | B4F Cluster]]
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]
* [[ssh_without_password | ssh without password]]

== External links ==
* [http://en.wikipedia.org/wiki/Secure_Shell secure shell on Wikipedia]
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]

Globally installed software

2014-09-03T08:54:48Z

Hulze001: /* Adding a custom module directory to your environment */

== Available as modules ==
* gcc/4.8.1
* python/2.7.6
* python/3.3.3
* R/3.0.2

== Globally installed on all nodes ==

* Perl5.10
* pigz
* Python2.6
* BioPerl v1.61
* [http://samtools.sourceforge.net/tabix.shtml bgzip]
* [http://samtools.sourceforge.net/tabix.shtml tabix v0.2.5]

== Available as global SHARED modules ==
Software can be deposited in:
/cm/shared/apps/SHARED/

Modules can be found in:
/cm/shared/modulefiles/SHARED/

{| width="90%"
|- valign="top"
| width="30%" |
* [[allpathslg_48961 | ALLPATHS-LG/48961]]
* [[augustus_2.7 | augustus/2.7]]
* [[bedtools2.18 | bedtools/2.18.0]]
* [[BLAST | BLAST+/2.2.28]]
* [[blat_v35 |blat/v35]]
* [[bowtie2_v2.2.1 | bowtie/2-2.2.1]]
* [[bowtie1_v1.0.0 | bowtie/1-1.0.0]]
* [[bwa_5.9 | bwa/0.5.9]]
* [[bwa_7.5 | bwa/0.7.5a]]
* [[cegma_2.4 | cegma/2.4]]
* [[Cufflinks | cufflinks/2.1.1]]

| width="30%" |
* [[exonerate_2.2.0 | exonerate/2.2.0-x86_64]]
* [[geneid_1.4.4 | geneid/1.4.4]]
* [[genewise_2.2.3 | genewise/2.2.3-rc7]]
* [[gmap_2014-01-21 | gmap/2014-01-21]]
* [[hmmer_3.1 | hmmer/3.1b1]]
* [[jellyfish_2.1.1 | jellyfish/2.1.1]]
* [[MAFFT_7.130 | MAFFT/7.130]]
* [[maker_2.2.8 | maker/2.28]]
* [[Muscle_3.8.31 | muscle/3.8.31]]
* [[Plink_1.07 | Plink/1.07]]
* [[Provean_1.1.3 | provean/1.1.3]]

| width="30%" |
* [[RepeatMasker_4.0.3 | RepeatMasker/4.0.3]]
* [[RepeatModeler_1.0.7 | RepeatModeler/1.0.7]]
* [[RAxML8.0.0 | RAxML/8.0.0]]
* [[samtools v0.1.12a | samtools/0.1.12a]]
* [[samtools v0.1.19 | samtools/0.1.19]]
* [[snap | snap/2013-11-29]]
* [[soapdenovo2_r240 | SOAPdenovo2/r240]]
* [[sra_toolkit_2.3.4 | sra-toolkit/2.3.4]]
* [[TopHat_2.0.11 | tophat/2.0.11]]
* [[Trinity_r20131110 | Trinity/r20131110]]
* [[wgs_assembler_8.1 | wgs-assembler/8.1]]
|}

== Adding a custom module directory to your environment ==
To allow the <code>module</code> program to find the custom module directory, the location of that directory has to be added to <code>MODULEPATH</code> variable.

<source lang='bash'>
export MODULEPATH=$MODULEPATH:/cm/shared/apps/WUR/ABGC/modulefiles
</source>
This can be made permanent by adding this line of code to the <code>.bash_profile</code> file in the root of your home folder. To then load the modified <code>MODULEPATH</code> variable you have to load <code>.bash_profile</code> again:
<source lang='bash'>
source .bash_profile
</source>
This needs to be done only for terminals that are already open. Next time you login, <code>.bash_profile</code> will be loaded automatically.

You can check if the modules are found.
<source lang='bash'>
module avail
</source>
This should give output that includes something similar to this:

---------------------------------------- /cm/shared/modulefiles/ -----------------------------------------
ALLPATHS-LG/48961 bwa/0.7.5a jellyfish/2.1.1 RepeatMasker/4.0.3
augustus/2.7 cegma/2.4 MAFFT/7.130 RepeatModeler/1.0.7
bedtools/2.18.0 cufflinks/2.1.1 maker/2.28 samtools/0.1.12a
BLAST+/2.2.28 exonerate/2.2.0-x86_64 muscle/3.8.31 samtools/0.1.19
blat/v35 geneid/1.4.4 plink/1.07 snap/2013-11-29
bowtie/2-2.2.1 genewise/2.2.3-rc7 provean/1.1.3 SOAPdenovo2/r240
bwa/0.5.9 hmmer/3.1b1 RAxML/8.0.0 tophat/2.0.11

== See also ==
* [[Main_Page#Using_the_B4F_Cluster | Using the B4F Cluster]]
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[ABGC_modules | modules specific for ABGC ]]

== External links ==

ABGC modules

2014-07-11T14:03:34Z

Hulze001: /* Modules available for ABGC */

All ABGC modules can be found in:

/cm/shared/apps/WUR/ABGC/modulefiles

== Modules available for ABGC ==

* [[asreml_3.0 | asreml/3.0fl-64]]
* [[asreml_4.0 | asreml/4.0kr]]
* [[Perl5.10_WUR_module | Perl/5.10.1_wur]]
* [[R3.0.2_WUR_module | R/3.0.2_wur]]

== Adding a custom module directory to your environment ==
To allow the <code>module</code> program to find the custom module directory, the location of that directory has to be added to <code>MODULEPATH</code> variable.

<source lang='bash'>
export MODULEPATH=$MODULEPATH:/cm/shared/apps/WUR/ABGC/modulefiles
</source>
This can be made permanent by adding this line of code to the <code>.bash_profile</code> file in the root of your home folder. To then load the modified <code>MODULEPATH</code> variable you have to load <code>.bash_profile</code> again:
<source lang='bash'>
source .bash_profile
</source>
This needs to be done only for terminals that are already open. Next time you login, <code>.bash_profile</code> will be loaded automatically.

You can check if the modules are found.
<source lang='bash'>
module avail
</source>
This should give output that includes something similar to this:

----------------------------------- /cm/shared/apps/WUR/ABGC/modulefiles -----------------------------------
bwa/0.5.9 bwa/0.7.5a

== See also ==
* [[Main_Page#Using_the_B4F_Cluster | Using the B4F Cluster]]
* [[Globally_installed_software | Globally installed software]]
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== External links ==