HPCwiki - User contributions [en]

Old binaries

2023-06-19T09:27:57Z

Dawes0011:

On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed.

This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications.

However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no longer be automatically available. This will make older installs fail to execute, giving, for instance, errors related to glibc, or missing symbols.

Thankfully, the entire library from previous OS builds is made available on the shared filesystem, granting older binaries the ability to access them whilst also running on a younger OS.

There is a special environment variable: <code>LD_LIBRARY_PATH</code> which tells the linker (ld) where extra to go find libraries. We use this extensively in the underlying module system on Anunna to control which libraries can be used. One such module, <code>sl7_libs</code> automatically adds the older libraries in to older binaries, and has been added as a prerequisite for most older installs.

If you've compiled something yourself that isn't a module, you may find yourself needing this module. Simply:

<code>
module load sl7-libs/main
</code>

To add this path.

== Jupyter ==

This may also extend to custom-built kernels on Jupyter. Here, however, you can't add this module, as instantiating a kernel is done though <code>kernel.json</code> instead of bash.

To add the needed libraries, you need to modify your notebook to look like the following:
<code>
{
"language": "python",
"argv": [
"/path/to/my/venv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "myvenv",
"env" : {"LD_LIBRARY_PATH": "/usr/lib:/usr/lib64:/usr/lib/x86_64-linux-gnu:/shared/legacyapps/sl7-libs/lib:/shared/legacyapps/sl7-libs/lib64" }
}
</code>

When editing JSON, please be mindful that all elements in an object must be separated by commas, and that only doublequotes may be used.

To test your JSON, try the following:

<code>
cat ~/.local.share/jupyter/kernels/mykernel/kernel.json | jq
</code>

If <code>jq</code> repeats your JSON, then it can parse it and you are good to go.

Old binaries

2023-06-19T09:27:26Z

Dawes0011:

On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed.

This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications.

However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no longer be automatically available. This will make older installs fail to execute, giving, for instance, errors related to glibc, or missing symbols.

Thankfully, the entire library from previous OS builds is made available on the shared filesystem, granting older binaries the ability to access them whilst also running on a younger OS.

There is a special environment variable: <code>LD_LIBRARY_PATH</code> which tells the linker (ld) where extra to go find libraries. We use this extensively in the underlying module system on Anunna to control which libraries can be used. One such module, <code>sl7_libs</code> automatically adds the older libraries in to older binaries, and has been added as a prerequisite for most older installs.

If you've compiled something yourself that isn't a module, you may find yourself needing this module. Simply:
<code>
module load sl7-libs/main
</code>

To add this path.

== Jupyter ==

This may also extend to custom-built kernels on Jupyter. Here, however, you can't add this module, as instantiating a kernel is done though <code>kernel.json</code> instead of bash.

To add the needed libraries, you need to modify your notebook to look like the following:
<code>
{
"language": "python",
"argv": [
"/path/to/my/venv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "myvenv",
"env" : {"LD_LIBRARY_PATH": "/usr/lib:/usr/lib64:/usr/lib/x86_64-linux-gnu:/shared/legacyapps/sl7-libs/lib:/shared/legacyapps/sl7-libs/lib64" }
}
</code>

When editing JSON, please be mindful that all elements in an object must be separated by commas, and that only doublequotes may be used.

To test your JSON, try the following:

<code>
cat ~/.local.share/jupyter/kernels/mykernel/kernel.json | jq
</code>

If <code>jq</code> repeats your JSON, then it can parse it and you are good to go.

Old binaries

2023-06-19T09:27:09Z

Dawes0011: Created page with "On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed. This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications. However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no l..."

On Linux, most binaries rely on shared libraries that are loaded by the OS before the binary is executed.

This allows them to share common runtimes, and avoid having to redo the same work multiple times. This is useful for optimisation and security, as debugging only has to be done in one place for multiple applications.

However, a significant number of these libraries are automatically inserted by the OS, and as the cluster upgrades and changes some of these will no longer be automatically available. This will make older installs fail to execute, giving, for instance, errors related to glibc, or missing symbols.

Thankfully, the entire library from previous OS builds is made available on the shared filesystem, granting older binaries the ability to access them whilst also running on a younger OS.

There is a special environment variable: <code>LD_LIBRARY_PATH</code> which tells the linker (ld) where extra to go find libraries. We use this extensively in the underlying module system on Anunna to control which libraries can be used. One such module, <code>sl7_libs</code> automatically adds the older libraries in to older binaries, and has been added as a prerequisite for most older installs.

If you've compiled something yourself that isn't a module, you may find yourself needing this module. Simply:
<code>
module load sl7-libs/main
</code>

To add this path.

== Jupyter ==

This may also extend to custom-built kernels on Jupyter. Here, however, you can't add this module, as instantiating a kernel is done though <code>kernel.json</code> instead of bash.

To add the needed libraries, you need to modify your notebook to look like the following:
<code>
{
"language": "python",
"argv": [
"/path/to/my/venv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "myvenv",
"env" : {"LD_LIBRARY_PATH": "/usr/lib:/usr/lib64:/usr/lib/x86_64-linux-gnu:/shared/legacyapps/sl7-libs/lib:/shared/legacyapps/sl7-libs/lib64" }
}
</code>

When editing JSON, please be mindful that all elements in an object must be separated by commas, and that only doublequotes may be used.

To test your JSON, try the following:
<code>
cat ~/.local.share/jupyter/kernels/mykernel/kernel.json | jq
</code>

If <code>jq</code> repeats your JSON, then it can parse it and you are good to go.

Main Page

2023-06-19T09:11:41Z

Dawes0011: /* Miscellaneous */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]
* [[Old_binaries | Running older binaries on the updated OS]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Main Page

2023-06-19T09:11:27Z

Dawes0011: /* Miscellaneous */

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]
* [[Old_binaries | running older binaries on the updated OS]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}

Filesystems

2023-06-16T09:04:11Z

Dawes0011:

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from nfs01. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of two major parts (and some minor):
* /lustre/backup - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/nobackup - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recivered.
* /lustre/shared - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

And additionally:
* /lustre/scratch - A separated, low resilience filesystem. Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* /archive - an archive mount only accessible from the login nodes. Files here are sent to the Isilon for deeper storage. The cost of storing data here is much less than on the Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via snapshot, and there is a separated backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<pre>
#SBATCH --tmp=<required space>
</pre>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Setting up Python virtualenv

2023-06-16T09:02:11Z

Dawes0011:

With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.

NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an [[virtual_environment_Python_3.4_or_higher | alternative set-up of your virtual environment if using Python 3.4 or higher]].

== Creating a new virtual environment ==
It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
<pre>
module load python/my-favourite-version (e.g. 2.7.12)
virtualenv newenv
OR
pyvenv newenv (For versions >3.4)
</pre>

When the new environment is created, one will see a message similar to this:
<nowiki> New python executable in newenv/bin/python3
Also creating executable in newenv/bin/python
Installing Setuptools.........................................................................done.
Installing Pip................................................................................done.</nowiki>

== Activating a virtual environment ==
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
<pre>
source newenv/bin/activate
</pre>
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
<nowiki> (newenv)user@host:~$</nowiki>

== Installing modules on the virtual environment ==
Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
<pre>
pip install numpy
</pre>
Similarly, installing packages from source works exactly the same as usual.
<pre>
python setup.py install
</pre>

== deactivating a virtual environment ==
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
<pre>
deactivate
</pre>

== Virtualenv kernels in Jupyter ==
Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

(an alternative way to the manual way (using conda) is described [[Using conda to install a new kernel into your notebook|here ]])

* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>
* Create the following directory path in your homedir if it doesn't already exist:
<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>
* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>
* Add this file to this folder:
<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json
{
"language": "python",
"argv": [
"/home/myhome/path/to/my/virtualenv/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "testing"
}</nowiki>
* Reload Jupyterhub page. testing should now exist in your kernels list.

You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
{
"language": "python",
"env": {
"SPARK_HOME":
"/shared/apps/spark/my-spark-version"
},
"argv": [
"/home/myhome/my/spark/venv/bin/python",
"-m",
"ipykernel",
"-c", "import findspark; findspark.init()",
"-f",
"{connection_file}"
],
"display_name": "My Spark kernel"
}</nowiki>
(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)

== Make IPython work under virtualenv ==
IPython may not work initially under a virtual environment. It may produce an error message like below:

<nowiki> File "/usr/bin/ipython", line 11
print "Could not start qtconsole. Please install ipython-qtconsole"
^</nowiki>

This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
<pre>
ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
</pre>

== External links ==
* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

MPI on B4F cluster

2023-06-16T09:01:59Z

Dawes0011:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<pre>
module list
</pre>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

MPI on B4F cluster

2023-06-16T09:01:42Z

Dawes0011:

== A simple 'Hello World' example ==
Consider the following simple MPI version, in C, of the 'Hello World' example:

<source lang='cpp'>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv) {
int size,rank,namelen;
char processor_name[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);
MPI_Get_processor_name(processor_name, &namelen);
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);
MPI_Finalize();
}
</source>

Before compiling, make sure that the compilers that are required available.
<source lang='bash'>
module list
</source>

To avoid conflicts between libraries, the safest way is purging all modules:
<source lang='bash'>
module purge
</source>

The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.
<source lang='bash'>
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7
</source>

Compile the <code>hello_mpi.c</code> code.
<source lang='bash'>
mpicc hello_mpi.c -o test_hello_world
</source>

If desired, a list of libraries compiled into the executable can be viewed:
<source lang='bash'>
ldd test_hello_world
</source>

linux-vdso.so.1 => (0x00002aaaaaacb000)
libmpi.so.1 => /shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Running the executable on two nodes, with four tasks per node, can be done like this:
<source lang='bash'>
srun --nodes=2 --ntasks-per-node=4 --mpi=openmpi ./test_hello_world
</source>

This will result in the following output:
Hello MPI! Process 4 of 8 on node011
Hello MPI! Process 1 of 8 on node010
Hello MPI! Process 7 of 8 on node011
Hello MPI! Process 6 of 8 on node011
Hello MPI! Process 5 of 8 on node011
Hello MPI! Process 2 of 8 on node010
Hello MPI! Process 0 of 8 on node010
Hello MPI! Process 3 of 8 on node010

== A mvapich2 sbatch example ==
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.
<source lang='bash'>
$ module load mvapich2/gcc
$ vim batch.sh
#!/bin/sh
#SBATCH --comment=projectx
#SBATCH --time=30-0
#SBATCH -n 32
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=MPItest
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@wur.nl

echo "Starting at `date`"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0

mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10

echo "Program finished with exit code $? at: `date`"

$ sbatch batch.sh

</source>

JBrowse

2023-06-16T09:01:30Z

Dawes0011:

=== Typical commands used to set up a JBrowse ===

Author: Martijn Derks

* JBrowse is available for multiple species:
** https://jbrowse.hpcagrogenomics.wur.nl/pig/
** https://jbrowse.hpcagrogenomics.wur.nl/chicken/
** https://jbrowse.hpcagrogenomics.wur.nl/cattle/
** https://jbrowse.hpcagrogenomics.wur.nl/turkey/
** https://jbrowse.hpcagrogenomics.wur.nl/Cyprinus_carpio/
* Users are free to add usefull commands to this tutorial

=== Install JBrowse ===

Download the latest JBrowse here: http://jbrowse.org/

Make a directory in <code>/shared/apps/jbrowse/</code> for your species of interested (e.g. <code>mkdir Cyprinus_carpio</code>). Move the downloaded JBrowse source files there. All further procedures detailed in this Wiki page assume working from that directory (NOTE: if your species of interest is already there, contact the maintainer of that JBrowse instance).
Run the setup script to install perl dependencies and required modules

<pre>
unzip JBrowse-1.12.0.zip
mv JBrowse-1.12.0/* $PWD
./setup.sh
</pre>

=== Add reference sequence ===

Example code for chicken genome

<pre>
bin/prepare-refseqs.pl --fasta /lustre/nobackup/WUR/ABGC/shared/public_data_store/genomes/chicken/Ensembl74/Gallus_gallus.Galgal4.74.dna.toplevel.fa
</pre>

To remove tracks use following command:

<pre>
bin/remove-track.pl -D --trackLabel 'trackname'
</pre>

===Add annotation files (GFF/BED)===

Data can be downloaded from the Ensembl FTP site: http://www.ensembl.org/info/data/ftp/index.html

Add gene features:

<pre>
bin/flatfile-to-json.pl --key "Genes" --type gene --config '{ "category": "GalGal4.83 Annotation" }' --trackLabel Genes --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</pre>

Add corresponding transcripts:

<pre>
bin/flatfile-to-json.pl --key "Transcripts" --className transcript --subfeatureClasses '{"exon": "exon", "CDS": "CDS", "five_prime_UTR": "five_prime_UTR", "three_prime_UTR": "three_prime_UTR"}' --config '{ "category": "GalGal4.83 Annotation" }' --type transcript --trackLabel Transcripts --gff ../ensembl_data/Gallus_gallus.Galgal4.83.gff3
</pre>

===Alignment tracks (BAM)===

You can load single BAM-files by following command:
<pre>
bin/add-bam-track --label <label> --bam_url <url>
</pre>

To load multiple BAM files present in a certain directory use:
<pre>

for bam in /<dir>*.bam; do
ln -s $bam track_symlinks/ ## Make symlinks from the BAM files
ln -s $bam.bai track_symlinks/ ## Make symlinks to the BAM index files
tissue=`echo $bam | rev | cut -c 5- | cut -d'/' -f1 | rev` ## USe the name of the file without .bam as trackLabel

## Add BAM in alignment mode (Alignments2)
echo '{
"label" : "'${tissue}'_alignment",
"key" : "'${tissue}'_alignment",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "Alignments2"
}' | bin/add-track-json.pl data/trackList.json

## Add BAM in coverage mode (SNPCoverage)
echo '{
"label" : "'${tissue}'_coverage",
"key" : "'${tissue}'_coverage",
"storeClass" : "JBrowse/Store/SeqFeature/BAM",
"urlTemplate" : "../track_symlinks/'${tissue}'",
"category" : "3. RNA-seq alignments",
"type" : "SNPCoverage"
}' | bin/add-track-json.pl data/trackList.json

done
</pre>

Make sure the BAM file can be read by a everybody if not use:
<pre>
chmod +r <BAM_file>
</pre>

Make sure that all directoryies in the full path of the BAMfile are executable:
<pre>
chmod +x <dir>
</pre>

===Variant tracks (VCF)===

To load a VCF file in JBrowse make sure the file is gzipped and indexed

<pre>
tabix -p vcf Gallus_gallus_incl_consequences.vcf.gz

echo ' {
"label" : "Gallus_gallus_incl_consequences",
"key" : "Gallus_gallus_incl_consequences",
"storeClass" : "JBrowse/Store/SeqFeature/VCFTabix",
"urlTemplate" : "../../ensembl_data/VCF/Gallus_gallus_incl_consequences.vcf.gz",
"category" : "2. Variants",
"type" : "HTMLVariants"
} ' | bin/add-track-json.pl data/trackList.json
</pre>

===Wiggle/BigWig tracks (WIG)===

You can load single BigWig-files by following command:
<pre>
bin/add-bw-track --label <label> --bw_url <url>
</pre>

===Evidence tracks===

Evidence tracks can be loaded in bed, gff and gbk format using

<pre>
bin/flatfile-to-json.pl
</pre>

Examples are given above.

Whole genome alignment pipeline

2023-06-16T09:01:16Z

Dawes0011:

Author: Carolina Pita Barros 
Contact: carolina.pitabarros@wur.nl 
ABG 

For up-to-date documentation see [https://github.com/CarolinaPB/whole-genome-alignment here]

= Whole genome alignment pipeline =

Path to pipeline: /lustre/nobackup/WUR/ABGC/shared/PIPELINES/whole-genome-alignment


== First follow the instructions here: ==

[https://carolinapb.github.io/2021-06-23-how-to-run-my-pipelines/ Step by step guide on how to use my pipelines] 
Click [https://github.com/CarolinaPB/snakemake-template/blob/master/Short%20introduction%20to%20Snakemake.pdf here] for an introduction to Snakemake


== ABOUT ==

This pipeline aligns one or more genomes to a specified genome and plots the alignment.


==== Tools used: ====

* [https://github.com/lh3/minimap2 minimap2]
* R
** [https://github.com/tpoorten/dotPlotly/blob/master/pafCoordsDotPlotly.R pafCoordsDotPlotly.R]

{|
!align="center"| [[File:whole-genome-alignment-workflow.png]]
|-
|align="center"| ''Pipeline workflow''
|}


=== Edit config.yaml with the paths to your files ===

<syntaxhighlight lang="yaml"># genome alignment parameters:
GENOME: /path/to/genome #genome fasta to be compared
COMPARISON_GENOME:
<genome1>: path/to/genome1.fasta
<genome2>: path/to/genome2.fasta
<genome3>: path/to/genome3.fasta

# filter alignments less than cutoff X bp
MIN_ALIGNMENT_LENGTH: 10000
MIN_QUERY_LENGTH: 50000

PREFIX: <prefix>

OUTDIR: /path/to/outdir</syntaxhighlight>
* GENOME: path to the genome fasta file (can be compressed). This is the genome that you want to be compared to all the others
* COMPARISON_GENOME: genome fasta (can be compressed) for whole genome comparison. Add your species name and the path to the fasta file. ex: chicken: /path/to/chicken.fna.gz. You can add several genomes, one on each line.
* MIN_ALIGNMENT_LENGTH and MIN_QUERY_LENGTH - parameters for plotting. If your plot is coming out blank or if there’s an error with the plotting step, try lowering these thresholds. This happens because the alignments are not large enough.
* PREFIX: name of your species (ex: turkey)
* OUTDIR: directory where snakemake will run and where the results will be written to

If you want the results to be written to this directory (not to a new directory), comment out or remove

<pre>OUTDIR: /path/to/outdir</pre>

== ADDITIONAL SET UP ==


=== Installing R packages ===

First load R: <code>module load R/4.0.2</code>

Enter the R environment by writing <code>R</code> and clicking enter. Install the packages:

<pre>list.of.packages <- c("optparse", "data.table", "ggplot2")
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages)</pre>
If you get an error like this:

<pre>Warning in install.packages(new.packages) :
'lib = "/shared/apps/R/3.6.2/lib64/R/library"' is not writable</pre>
Follow the instructions on how to install R packages locally [https://wiki.anunna.wur.nl/index.php/Installing_R_packages_locally here] and try to install the packages again.


== RESULTS ==

The most important files and directories are:

* '''<run_date>_files.txt''' dated file with an overview of the files used to run the pipeline (for documentation purposes)
* '''genome_alignment/{prefix}''vs''{species}.paf''' paf format file w

Get my bill

2023-06-16T09:01:06Z

Dawes0011:

To estimate costs over a certain time-period the following script can be invoked:

<pre>
module load anunna
get_my_bill
</pre>

The script will, by default, report the cost over the current month, until present. The output will look similar to this:

!!! These results are advisory only !!!
User: user001
Currently run jobs this month: 1038
Total cost so far: 25.19 EUR
For account: 12345
Jobs: 0 Cost: 0.00 EUR
For account: project2
Jobs: 104 Cost: 4.61 EUR
For account: 56789
Jobs: 74 Cost: 6.59 EUR
For account: project 4
Jobs: 19 Cost: 2.09 EUR
For account: project5
Jobs: 80 Cost: 0.04 EUR
For account: project6
Jobs: 738 Cost: 11.86 EUR
For account: project7
Jobs: 1 Cost: 0.01 EUR
For account: project8
Jobs: 22 Cost: 0.00 EUR
Type Time Current Use Current Cost EUR
home 2015-02-13 23:54:53 7.131 GB 0.00
backup 2015-02-06 06:02:43 4.000 kB 0.00
nobackup 2015-02-10 10:45:29 5.348 TB 29.48
scratch 2015-02-08 13:48:40 0.233 TB 0.00
Total this month: 54.67 EUR

The script provides options in addition to the defaults, which can be invoked as follows:
Options:
-h - Show this help message
-g - Show results for your entire group
-d [disk|compute] - get extra detail on disk/compute usages
-t YYYY-MM - Show results for specific month

== See also ==
* [[Main_Page | Main page AgHPC Wiki]]
* [[Main_Page#Controlling_costs | Controlling costs @ AgHPC Wiki]]

Hadoop

2023-06-16T09:00:40Z

Dawes0011:

Hadoop is another kind of architecture to promote writing code that can be executed in a parallel fashion. It's become more of an environment than a piece of software these days, as it typically contains:

* HBase, flat table storage
* Hive, a form of data warehousing
* MapReduce, a style of coding to promote largely non-interacting workflows in order to leverage slow interconnects between workers
* HDFS, a distributed filesystem between workers
And many other components.

This install of Hadoop has been optimised for executing a standalone HDFS filesystem, primarily for teaching purposes.

== WARNING ==
HDFS is incredibly insecure. As soon as anyone knows the location of your nameserver, they may spoof into it to access your data, as any user. This can be prevented by using Kerberos tickets, however that is currently beyond our level of complexity. Do not store anything you care about here.

=== Usage ===
load up the module, and java:
<source lang="bash">
module load hadoop
module load java
</source>

This will set your environment variables correctly.

There is an example submission script at:
<nowiki>/shared/apps/hadoop/current/wur/example_sbatch.sh</nowiki>
Copy this to your preferred location and submit it.

Edit this script and adjust the settings accordingly. You can override the location that HDFS will store its blocks by altering the environment variable HADOOP_TMP_DIR, which will default to /lustre/scratch.

Once it's run, the output log will tell you the location. I'd recommend setting a static location in your job script so that it's reproducibly located, but the HADOOP_CONF_DIR will point to the correct fs.defaultFS once you've started it.

=== Mechanism details ===
The example script will run an executable called start-hdfs.sh. This simply starts by creating ~/.hadoop/$SLURM_JOB_ID, then assigning that as HADOOP_LOG_DIR. Next, it creates a subfolder conf, assigns that to HADOOP_CONF_DIR, and writes the core-site.xml and hdfs-site.xml files into it. Last, it creates a symlink from ~/.hadoop/current to point to your currently running HDFS instance. On the client side, all HADOOP_CONF_DIR's point to ~/.hadoop/current/conf so that whichever job is submitted, this symlink will point to the currently running config for you.

If you're trying to share your HDFS instance with other users, you should:
* Copy the ~/.hadoop/current/conf/* to somewhere shared
* Tell them to set the environment variable HADOOP_CONF_DIR to that location

Population structural variant calling pipeline

2023-06-16T09:00:29Z

Dawes0011:

Author: Carolina Pita Barros 
Contact: carolina.pitabarros@wur.nl 
ABG 

[https://github.com/CarolinaPB/population-structural-var-calling-smoove/tree/single_run Link to the repository]

== First follow the instructions here: ==

[https://carolinapb.github.io/2021-06-23-how-to-run-my-pipelines/ Step by step guide on how to use my pipelines] 
Click [https://github.com/CarolinaPB/snakemake-template/blob/master/Short%20introduction%20to%20Snakemake.pdf here] for an introduction to Snakemake

== ABOUT ==

This is a pipeline to perform structural variant calling in a population using Smoove. It also runs VEP and performs PCA. In addition to the VCF with the SVs, you also get a .tsv file with some summarized information on the SVs: it includes allele frequency per population, as well as VEP annotation and depth fold change as described in [https://github.com/brentp/duphold duphold]: 
> DHBFC: fold-change for the variant depth relative to bins in the genome with similar GC-content. 
> DHFFC: fold-change for the variant depth relative to Flanking regions.

==== Tools used: ====

* Smoove - SV calling
* VEP - determines the effect of the variants
* Plink - perform PCA
* R - plot PCA
* SURVIVOR - basic SV stats
* Python
** PyVcf - add depth to vcf and create final table
** bamgroupreads.py + samblaster - create bam files with split and discordant reads

{|
!align="center"| [[File:Pop-sv-calling-workflow.png]]
|-
|align="center"| ''Pipeline workflow''
|}

=== Edit config.yaml with the paths to your files ===

<pre>OUTDIR: /path/to/output
READS_DIR: /path/to/reads/ # don't add the reads files, just the directory where they are
SAMPLE_LIST: /path/to/file
REFERENCE: /path/to/assembly
CONTIGS_IGNORE: /path/to/file
SPECIES: <species_name>
PREFIX: <output name>
NUM_CHRS: <number of chromosomes>
BWA_MEM_M: <Y/N></pre>

* OUTDIR - directory where snakemake will run and where the results will be written to
* READS_DIR - path to the directory that contains the reads
* SAMPLE_LIST - three column csv with the sample name, name of the bam files to use in the second column and the name of the corresponding population on the third column. These bams should all be in the same directory (READS_DIR)
* Example:
<blockquote>sample1,sample1.bam,Pop1 
sample2,sample2.bam,Pop1 
sample3,sample3.bam,Pop2 
sample4,sample4.bam,Pop2 </blockquote>
Tip: use the name of the bam file without the .bam extension as the sample name. Ex: from sample1.bam to sample1
* REFERENCE - path to the assembly file
* CONTIGS_IGNORE - contigs to be excluded from SV calling (usually the small contigs)
** If you don't want to exclude contigs you'll need to edit the Snakefile to remove this line <code>--excludechroms {params.contigs} \</code>
* SPECIES - species name to be used for VEP
* PREFIX - prefix for the created files
* NUM_CHRS - number of chromosomes for your species (necessary for plink). ex: 38
* BWA_MEM_M - if you mapped your reads with `bwa mem` using the `-M` parameter and you want split read support in your VCF you need to run an extra step. For this write `Y`.
For a more detailed explanation see here [https://carolinapb.github.io/2021-10-28-smoove-SR-support/ Smoove SR support]

If you want the results to be written to this directory (not to a new directory), comment out or remove

<pre>OUTDIR: /path/to/outdir</pre>
== ADDITIONAL SET UP ==
=== Configuring VEP ===
This pipeline uses VEP in offline mode, which increases performance. In order to use it in this mode, the cache for the species used needs to be installed:
Check if the cache file for your species already exist in <code>/lustre/nobackup/SHARED/cache/</code>. If it doesn’t, create it with

<pre>/usr/bin/perl /shared/apps/SHARED/ensembl-vep/INSTALL.pl --CACHEDIR /lustre/nobackup/SHARED/cache/ --AUTO c -n --SPECIES <species></pre>
When multiple assemblies are found you need to run it again with <code>--ASSEMBLY <assembly name></code>, where “assembly name” is the name of the assembly you want to use.

==== Other option: ====

You can install VEP with

<pre>conda install -c bioconda ensembl-vep</pre>
and install the cache with

<pre>vep_install --CACHEDIR <where/to/install/cache> --AUTO c -n --SPECIES <species></pre>
When multiple assemblies are found you need to run it again with <code>--ASSEMBLY <assembly name></code>, where “assembly name” is the name of the assembly you want to use.

In the Snakefile, in rule <code>run_vep</code>, replace <code>/shared/apps/SHARED/ensembl-vep/vep</code> with <code>vep</code>

=== Installing R packages ===
First load R:
<pre>module load R/3.6.2</pre>
Enter the R environment by writing <pre>R</pre> and clicking enter.
Install the packages:
<pre>list.of.packages <- c("optparse", "data.table", "ggplot2")
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages)</pre>

If you get an error like this:
<pre>Warning in install.packages(new.packages) :
'lib = "/shared/apps/R/3.6.2/lib64/R/library"' is not writable</pre>
Follow the instructions on how to install R packages locally [https://wiki.anunna.wur.nl/index.php/Installing_R_packages_locally here] and try to install the packages again.

== RESULTS ==

* '''<run_date>_files.txt''' Dated file with an overview of the files used to run the pipeline (for documentation purposes)
* '''2_merged'''
** {prefix}.smoove-counts.html - shows a summary of the number of reads before and after filtering
* '''5_postprocessing''' directory that contains the final VCF file containing the structural variants found. This file has been annotated with VEP
** {prefix}.smoove.square.vep.vcf.gz - Final VCF - with VEP annotation, not filtered for quality
** {prefix}.smoove.square.vep.vcf.gz_summary.html - statistics from VEP
** {prefix}.nosex, {prefix}.log, {prefix}.eigenvec, {prefix}.eigenval - output files from the PCA
** {prefix}_DUP_DEL_INV_table.tsv - table with the most important information extracted from the VCF. Contains information about the SV, allele frequency for each population, VEP annotation and depth information. The variants have been filtered with Minimum Quality score = 30
** {prefix}_DUP_DEL_INV.vcf - vcf file with annotated duplications, deletions and inversions. It has been filtered with Minimum Quality score = 30 and the DEPTH* field was added
** {prefix}_BND.vcf - vcf file with variants annotated with BND
* '''6_metrics''' directory that contains general stats about the number of SVs found
* '''FIGURES''' directory that contains the PCA plot

What you do with the results from this structural variant calling pipeline depends on your research question: a possible next step would be to explore the '''{prefix}_DUP_DEL_INV_table.tsv''' file and look at the largest SVs found (sort by ''SVLEN'') or at a specific effect in the ANNOTATION column, such as “frameshift_variant”.

See [https://m.ensembl.org/info/genome/variation/prediction/predicted_data.html VEP effect descriptions] for a short description of the effects annotated by VEP.

-----

*The '''DEPTH''' field in the vcf has six fields, corresponding to the average depth across all samples.

<pre>DEPTH=(DHBFC_1/1, DHBFC_0/1, DHBFC_0/0, DHFFC_1/1, DHFFC_0/1, DHFFC_0/0)</pre>
Depth fold change as described in [https://github.com/brentp/duphold duphold]:
<blockquote>DHBFC: fold-change for the variant depth relative to bins in the genome with similar GC-content. 
DHFFC: fold-change for the variant depth relative to Flanking regions.</blockquote>

These fields are also in the <code>{prefix}_DUP_DEL_INV_table.tsv</code> file

Maker protocols Pmajor

2023-06-16T09:00:11Z

Dawes0011:

This page describes the various rounds of [http://www.yandell-lab.org/software/maker.html Maker]-based annotations for the [http://en.wikipedia.org/wiki/Parus_major ''Parus major'' (Great Tit)] genome.

== Round 1 ==
=== Rationale ===
For this round no P. major-based ESTs were available. Zebrafinch (T. guttata) is the closest relative for which a reasonably complete gene-model set is available. As a first pass, it was decided to let gene predictions be driven by ab-inititio predictions rather than by Zebrafinch EST.

=== Invoking maker script ===
Do not forget to load the <code>maker</code> module:
<source lang='bash'>
module load maker/2.28
</source>

script submitted by SLURM (<code>sbatch</code> command):
<source lang='bash'>
#!/bin/bash
#SBATCH --time=48000
#SBATCH --nodes=1
#SBATCH --ntasks=16
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=test_maker
#SBATCH --mail-type=ALL
#SBATCH --mail-user=hendrik-jan.megens@wur.nl
maker
</source>

=== Maker settings ===
==== content of <code>maker_opts.ctl</code> ====
#-----Genome (these are always required)
genome=Pam.fa #genome sequence (fasta file or fasta embeded in GFF3 file)
organism_type=eukaryotic #eukaryotic or prokaryotic. Default is eukaryotic

#-----Re-annotation Using MAKER Derived GFF3
maker_gff= #MAKER derived GFF3 file
est_pass=0 #use ESTs in maker_gff: 1 = yes, 0 = no
altest_pass=0 #use alternate organism ESTs in maker_gff: 1 = yes, 0 = no
protein_pass=0 #use protein alignments in maker_gff: 1 = yes, 0 = no
rm_pass=0 #use repeats in maker_gff: 1 = yes, 0 = no
model_pass=0 #use gene models in maker_gff: 1 = yes, 0 = no
pred_pass=0 #use ab-initio predictions in maker_gff: 1 = yes, 0 = no
other_pass=0 #passthrough anyything else in maker_gff: 1 = yes, 0 = no

#-----EST Evidence (for best results provide a file for at least one)
est= #set of ESTs or assembled mRNA-seq in fasta format
altest= Taeniopygia_guttata.taeGut3.2.4.74.cdna.all.fa #EST/cDNA sequence file in fasta format from an alternate organism
est_gff= #aligned ESTs or mRNA-seq from an external GFF3 file
altest_gff= #aligned ESTs from a closly relate species in GFF3 format

#-----Protein Homology Evidence (for best results provide a file for at least one)
protein= Taeniopygia_guttata.taeGut3.2.4.74.pep.all.fa #protein sequence file in fasta format (i.e. from mutiple oransisms)
protein_gff= #aligned protein homology evidence from an external GFF3 file

#-----Repeat Masking (leave values blank to skip repeat masking)
model_org=Metazoa #select a model organism for RepBase masking in RepeatMasker
rmlib= #provide an organism specific repeat library in fasta format for RepeatMasker
repeat_protein= #provide a fasta file of transposable element proteins for RepeatRunner
rm_gff= #pre-identified repeat elements from an external GFF3 file
prok_rm=0 #forces MAKER to repeatmask prokaryotes (no reason to change this), 1 = yes, 0 = no
softmask=1 #use soft-masking rather than hard-masking in BLAST (i.e. seg and dust filtering)

#-----Gene Prediction
snaphmm= /shared/apps/WUR/ABGC/snap/snap-2013-11-29/HMM/mam54.hmm #SNAP HMM file
gmhmm= #GeneMark HMM file
augustus_species= chicken #Augustus gene prediction species model
fgenesh_par_file= #FGENESH parameter file
pred_gff= #ab-initio predictions from an external GFF3 file
model_gff= #annotated gene models from an external GFF3 file (annotation pass-through)
est2genome=0 #infer gene predictions directly from ESTs, 1 = yes, 0 = no
protein2genome=0 #infer predictions from protein homology, 1 = yes, 0 = no
unmask=0 #also run ab-initio prediction programs on unmasked sequence, 1 = yes, 0 = no

#-----Other Annotation Feature Types (features MAKER doesn't recognize)
other_gff= #extra features to pass-through to final MAKER generated GFF3 file

#-----External Application Behavior Options
alt_peptide=C #amino acid used to replace non-standard amino acids in BLAST databases
cpus=16 #max number of cpus to use in BLAST and RepeatMasker (not for MPI, leave 1 when using MPI)

#-----MAKER Behavior Options
max_dna_len=100000 #length for dividing up contigs into chunks (increases/decreases memory usage)
min_contig=1 #skip genome contigs below this length (under 10kb are often useless)

pred_flank=200 #flank for extending evidence clusters sent to gene predictors
pred_stats=0 #report AED and QI statistics for all predictions as well as models
AED_threshold=1 #Maximum Annotation Edit Distance allowed (bound by 0 and 1)
min_protein=0 #require at least this many amino acids in predicted proteins
alt_splice=0 #Take extra steps to try and find alternative splicing, 1 = yes, 0 = no
always_complete=0 #extra steps to force start and stop codons, 1 = yes, 0 = no
map_forward=0 #map names and attributes forward from old GFF3 genes, 1 = yes, 0 = no
keep_preds=0 #Concordance threshold to add unsupported gene prediction (bound by 0 and 1)

split_hit=10000 #length for the splitting of hits (expected max intron size for evidence alignments)
single_exon=0 #consider single exon EST evidence when generating annotations, 1 = yes, 0 = no
single_length=250 #min length required for single exon ESTs if 'single_exon is enabled'
correct_est_fusion=0 #limits use of ESTs in annotation to avoid fusion genes

tries=2 #number of times to try a contig if there is a failure for some reason
clean_try=1 #remove all data from previous run before retrying, 1 = yes, 0 = no
clean_up=1 #removes theVoid directory with individual analysis files, 1 = yes, 0 = no
TMP= #specify a directory other than the system default temporary directory for temporary files

==== content of <code>maker_exe.ctl</code> ====
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/shared/apps/WUR/ABGC/blast/ncbi-blast-2.2.28+/bin/makeblastdb #location of NCBI+ makeblastdb executable
blastn=/shared/apps/WUR/ABGC/blast/ncbi-blast-2.2.28+/bin/blastn #location of NCBI+ blastn executable
blastx=/shared/apps/WUR/ABGC/blast/ncbi-blast-2.2.28+/bin/blastx #location of NCBI+ blastx executable
tblastx=/shared/apps/WUR/ABGC/blast/ncbi-blast-2.2.28+/bin/tblastx #location of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/shared/apps/WUR/ABGC/RepeatMasker/RepeatMasker-4-0-3/RepeatMasker #location of RepeatMasker executable
exonerate=/shared/apps/WUR/ABGC/exonerate/exonerate-2.2.0-x86_64/bin/exonerate #location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/shared/apps/WUR/ABGC/snap/snap-2013-11-29/snap #location of snap executable
gmhmme3= #location of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/shared/apps/WUR/ABGC/augustus/augustus.2.7/src/augustus #location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild= #location of probuild executable (required for genemark)

==== contents of <code>maker_bopts.ctl</code>====
#-----BLAST and Exonerate Statistics Thresholds
blast_type=ncbi+ #set to 'ncbi+', 'ncbi' or 'wublast'

pcov_blastn=0.8 #Blastn Percent Coverage Threhold EST-Genome Alignments
pid_blastn=0.85 #Blastn Percent Identity Threshold EST-Genome Aligments
eval_blastn=1e-10 #Blastn eval cutoff
bit_blastn=40 #Blastn bit cutoff
depth_blastn=0 #Blastn depth cutoff (0 to disable cutoff)

pcov_blastx=0.5 #Blastx Percent Coverage Threhold Protein-Genome Alignments
pid_blastx=0.4 #Blastx Percent Identity Threshold Protein-Genome Aligments
eval_blastx=1e-06 #Blastx eval cutoff
bit_blastx=30 #Blastx bit cutoff
depth_blastx=0 #Blastx depth cutoff (0 to disable cutoff)

pcov_tblastx=0.8 #tBlastx Percent Coverage Threhold alt-EST-Genome Alignments
pid_tblastx=0.85 #tBlastx Percent Identity Threshold alt-EST-Genome Aligments
eval_tblastx=1e-10 #tBlastx eval cutoff
bit_tblastx=40 #tBlastx bit cutoff
depth_tblastx=0 #tBlastx depth cutoff (0 to disable cutoff)

pcov_rm_blastx=0.5 #Blastx Percent Coverage Threhold For Transposable Element Masking
pid_rm_blastx=0.4 #Blastx Percent Identity Threshold For Transposbale Element Masking
eval_rm_blastx=1e-06 #Blastx eval cutoff for transposable element masking
bit_rm_blastx=30 #Blastx bit cutoff for transposable element masking

ep_score_limit=20 #Exonerate protein percent of maximal score threshold
en_score_limit=20 #Exonerate nucleotide percent of maximal score threshold

== See also ==
[[Maker_2.2.8 | Maker pipeline as installed on Anunna]]

== External links ==
* [http://www.yandell-lab.org/software/maker.html Maker homepage]
* [http://gmod.org/wiki/MAKER_Tutorial_2013 Maker tutorial]

Filesystems

2023-06-16T08:59:44Z

Dawes0011:

Anunna currently has multiple filesystem mounts that are available cluster-wide:

== Global ==
* /home - This mount uses NFS to mount the home directories directly from nfs01. Each user has a 200G quota for this filesystem, as it is regularly backed up to tape, and can reliably be restored from up to a week's history.

* /shared - This mount provides a consistent set of binaries for the entire cluster.

* /lustre - This large mount uses the Lustre filesystem to provide files from multiple redundant servers. Access is provided per group, thus:
/lustre/[level]/[partner]/[unit]
e.g.
/lustre/backup/WUR/ABGC/
It comprises of three major parts (and some minor):
* /lustre/backup - In case of disaster, this data is stored a second time on a separate machine. Whilst this backup is purely in case of complete tragedy (such as some immense filesystem error, or multiple component failure), it can potentially be used to revert mistakes if you are very fast about reporting them. There is however no guarantee of this service.
* /lustre/nobackup - This is the 'normal' filesystem for Lustre - no backups, just stored on the filesystem. Without having a backup needed, the cost of data here is not as much as under /lustre/backup, but in case of disaster cannot be recivered.
* /lustre/scratch - Files here may be removed after some time if the filesystem gets too full (Typically 30 days). You should tidy up this data yourself once work is complete.
* /lustre/shared - Same as /lustre/backup, except publicly available. This is where truly shared data lives that isn't assigned to a specific group.

=== Private shared directories ===
If you are working with a group of users on a similar project, you might consider making a [[Shared_folders|Shared directory]] to coordinate. Information on how to do so is in the linked article.

== Local ==
Specific to certain machines are some other filesystems that are available to you:
* /archive - an archive mount only accessible from the login nodes. Files here are sent to the Isilon for deeper storage. The cost of storing data here is much less than on the Lustre, but it cannot be used for compute work. This location is only available to WUR users. Files are able to be reverted via snapshot, and there is a separated backup, however this only comes in fortnightly (14 day) intervals.

* /tmp - On each worker node there is a /tmp mount that can be used for temporary local caching. Be advised that you should clean this up, lest your files become a hindrance to other users. You can request a node with free space in your sbatch script like so:
<source lang='bash'>
#SBATCH --tmp=<required space>
</source>

* /dev/shm - On each worker you may also create a virtual filesystem directly into memory, for extremely fast data access. Be advised that this will count against the memory used for your job, but it is also the fastest available filesystem if needed.

== See also ==
* [[Tariffs | Costs associated with resource usage]]

== External links ==
* [http://wiki.lustre.org/index.php/Main_Page Lustre website]

Environment Modules

2023-06-16T08:59:32Z

Dawes0011:

== Preface ==
Environment modules [http://modules.sourceforge.net] are a smart way to provide interchangeable blocks of executables and reproducible environments for use in an HPC. It's also the only way to provide simultaneous versions of the same software without collisions, as each module is housed entirely in its own subfolder structure.

== Using modules ==
The module executable is automatically provided to you upon login. Most users have some modules automatically loaded as well; to see these, use

<nowiki>module list</nowiki>

You should be able to see which modules are loaded.

One of the most important modules to load is 'shared' - this is Anunna specific, as it will extend the MODULEPATH environment variable to use modules present in /shared as well as /cm/local/ . Without this, many modules will not be available to you.

== Loading modules ==
Availability of (different versions of) software can be checked by the following command:
<nowiki>module avail</nowiki>

For example, you should be able to find the basic module slurm. This provides the path to the sbatch, srun, etc. executables for job submission. To load this, simply:
<nowiki>module load slurm</nowiki>

And you will see that it will automatically load the latest version - no need to write out any further.

Many of the hand-installed programs have a path such as:
<nowiki>hdf5/gcc/64/1.8.14</nowiki>
Which translates into:
<nowiki>SOFTWARE/COMPILER/BITS/VERSION</nowiki>

You can elect to load this to various levels:

<nowiki>module load hdf5 # loads the latest version, not caring for compiler
module load hdf5/gcc # loads the latest gcc-compiled version, not caring for 32/64 bits (default 64)
module load hdf5/gcc/64 # loads the latest 64-bit gcc-compiled version
module load hdf5/gcc/64/1.8.14 # loads this specific version of hdf5</nowiki>

This allows your job scripts to either automatically be upgraded when the latest executables are installed, or elect to use only one specific version of a piece of code.

== Switching modules ==

If you want to remove a module, simply
<nowiki>module unload module/1</nowiki>
This will remove the executable path from your environment. It'll also follow the same logic as above, i.e. you can unload all loaded slurm modules independent of version by just unloading the base module name. You can then load up a new one. You can do this in one command, with:
<nowiki>module switch module/1 module/2</nowiki>

Some modules will not allow themselves to be loaded when another one is loaded, for instance, for sanity reasons it's not possible to load two java modules at the same time. Trying to do this will give:

<nowiki>Module 'module/2' conflicts with the currently loaded module(s) 'module/1'</nowiki>
If you're seeing this, you must unload or switch your modules rather than overloading them:
<nowiki>module switch module/2</nowiki>
This works if both modules have the same base path.

== External links ==
* http://modules.sourceforge.net
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Population variant calling pipeline

2023-06-16T08:59:21Z

Dawes0011:

Author: Carolina Pita Barros 
Contact: carolina.pitabarros@wur.nl 
ABG 

For up-to-date documentation see [https://github.com/CarolinaPB/population-variant-calling here]

= Population level variant calling =

Path to pipeline: /lustre/nobackup/WUR/ABGC/shared/PIPELINES/population-variant-calling


== First follow the instructions here ==

[https://carolinapb.github.io/2021-06-23-how-to-run-my-pipelines/ Step by step guide on how to use my pipelines] 
Click [https://github.com/CarolinaPB/snakemake-template/blob/master/Short%20introduction%20to%20Snakemake.pdf here] for an introduction to Snakemake


== ABOUT ==

This is a pipeline that takes short reads aligned to a genome (in <code>.bam</code> format) and performs population level variant calling with <code>Freebayes</code>. It uses VEP to annotate the resulting VCF, calculates statistics, and calculates and plots a PCA.

It was developed to work with the results of [https://github.com/CarolinaPB/population-mapping this population mapping pipeline]. There are a few <code>Freebayes</code> requirements that you need to take into account if you don't use the mapping pipeline mentioned above to map your reads. You should make sure that:

* Alignments have read groups
* Alignments are sorted
* Duplicates are marked

See [https://github.com/freebayes/freebayes#calling-variants-from-fastq-to-vcf here] for more details.


==== Tools used ====

* [https://github.com/freebayes/freebayes Freebayes] - variant calling using short reads
* [https://samtools.github.io/bcftools/bcftools.html bcftools] - vcf statistics
* [https://www.cog-genomics.org/plink/ Plink] - compute PCA
* R - Plot PCA

{|
!align="center"| [[File:population-var-calling-workflow.png]]
|-
|align="center"| ''Pipeline workflow''
|}


=== Edit config.yaml with the paths to your files ===

<syntaxhighlight lang="yaml">ASSEMBLY: /path/to/fasta
MAPPING_DIR: /path/to/bams/dir
PREFIX: <prefix>
OUTDIR: /path/to/outdir
SPECIES: <species>
NUM_CHRS: <number of chromosomes></syntaxhighlight>
* ASSEMBLY - path to genome fasta file
* MAPPING_DIR - path to directory with bam files to be used
** the pipeline will use all bam files in the directory, if you want to use a subset of those, create a file named <code>bam_list.txt</code> that contains the paths to the bam files you want to use. One path per line.

<syntaxhighlight lang="text">/path/to/file.bam
/path/to/file2.bam</syntaxhighlight>
* PREFIX - prefix for the created files
* OUTDIR - directory where snakemake will run and where the results will be written to 
If you want the results to be written to this directory (not to a new directory), open config.yaml and comment out <code>OUTDIR: /path/to/outdir</code>
* SPECIES - species name to be used for VEP
* NUM_CHRS - number of chromosomes for your species (necessary for plink). ex: 38


== ADDITIONAL SET UP ==


=== Configuring VEP ===

This pipeline uses VEP in offline mode, which increases performance. In order to use it in this mode, the cache for the species used needs to be installed:


==== For people using WUR's Anunna: ====

Check if the cache file for your species already exist in <code>/lustre/nobackup/SHARED/cache/</code>. If it doesn't, create it with

<pre>/usr/bin/perl /shared/apps/SHARED/ensembl-vep/INSTALL.pl --CACHEDIR /lustre/nobackup/SHARED/cache/ --AUTO c -n --SPECIES <species></pre>
When multiple assemblies are found you need to run it again with <code>--ASSEMBLY <assembly name></code>, where "assembly name" is the name of the assembly you want to use.


==== For those not from WUR: ====

You can install VEP with

<pre>conda install -c bioconda ensembl-vep</pre>
and install the cache with

<pre>vep_install --CACHEDIR <where/to/install/cache> --AUTO c -n --SPECIES <species></pre>
When multiple assemblies are found you need to run it again with <code>--ASSEMBLY <assembly name></code>, where "assembly name" is the name of the assembly you want to use.

In the Snakefile, in rule <code>run_vep</code>, replace <code>/shared/apps/SHARED/ensembl-vep/vep</code> with <code>vep</code>


=== Installing R packages ===

First load R: <code>module load R/3.6.2</code>

Enter the R environment by writing <code>R</code> and clicking enter. Install the packages:

<pre>list.of.packages <- c("optparse", "data.table", "ggplot2")

new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages)</pre>
If you get an error like this:

<pre>Warning in install.packages(new.packages) :
'lib = "/shared/apps/R/3.6.2/lib64/R/library"' is not writable</pre>
Follow the instructions on how to install R packages locally [https://wiki.anunna.wur.nl/index.php/Installing_R_packages_locally here] and try to install the packages again.


== RESULTS ==

The most important files and directories are:

* '''<run_date>_files.txt''' dated file with an overview of the files used to run the pipeline (for documentation purposes)
* '''results''' directory that contains
** '''final_VCF''' directory with variant calling VCF files, as well as VCF stats
*** {prefix}.vep.vcf.gz - final VCF file
*** {prefix}.vep.vcf.gz.stats
** '''PCA''' PCA results and plot
*** {prefix}.eigenvec and {prefix}.eigenval - file with PCA eigenvectors and eigenvalues, respectively
*** {prefix}.pdf - PCA plot

The VCF file has been filtered for <code>QUAL > 20</code>. Freebayes is ran with parameters <code>--use-best-n-alleles 4 --min-base-quality 10 --min-alternate-fraction 0.2 --haplotype-length 0 --ploidy 2 --min-alternate-count 2</code>. These parameters can be changed in the Snakefile.

Using environment modules

2023-06-16T08:59:02Z

Dawes0011:

=== Environment Modules ===
[http://modules.sourceforge.net/ Environment modules] are a simple way to allow multiple potentially clashing programs to coexist on a large shared machine such as an HPC. It allows a user to specify exactly which programs are loaded, and even which version of each program, whilst simultaneously allowing the administrator the ability to automatically configure the appropriate environment variables for the system itself.

== Viewing Modules ==
Upon logging in to Anunna, you should find that when you do:
module list

You will see something like this:
<source lang='bash'>
-bash-4.1$ module list
Currently Loaded Modulefiles:
1) shared 2) slurm/2.5.7
</source>

This is a list of all loaded modules in your shell session. To get a list of all available modules, simply
module available

And this will show you the (very exhaustive) list of modules on Anunna:

<source lang='bash'>
-bash-4.1$ module avail

---------------------------- /shared/modulefiles ----------------------------
acml/gcc/64/5.3.1 netcdf/gcc/64/4.1.3
acml/gcc/fma4/5.3.1 netcdf/gcc/64/4.3.0
acml/gcc/mp/64/5.3.1 netcdf/gcc/64/4.3.2
acml/gcc/mp/fma4/5.3.1 netcdf/gcc/64/4.3.3
acml/gcc-int64/64/5.3.1 netcdf/gcc/64/4.3.3.1
acml/gcc-int64/fma4/5.3.1 netcdf/intel/64/4.1.3
...
</source>

Let's look at each of these module names. Each module is named for the application it provides, plus a subfolder of which compiler it was compiled with (if compiled), the number of address bits or options (if compiled), and the version.

If you want to see a list for a specific module, you can
module avail netcdf

And the complete list of versions will be shown.

== Loading Modules ==
To load a module, simply
module load foo

And the most recent version of module foo will automatically be loaded. If foo is compiled, it will automatically select the gcc version. If you want to specify a certain version, then
module load foo/gcc/64/1.0.0

Will load foo version 1, compiled with gcc. Be advised that this may not always work, as some modules are not compatible with each other, but a message will be shown if this is the case. Additionally, some modules will automatically load other modules with them for them to operate.

== Unloading Modules ==
If you want to remove a module that you've loaded, then
module unload foo

Will remove all module foo's loaded.

== Example ==
Consider this simple python3 script that should calculate Pi to 1 million digits:
<source lang='python'>
from decimal import *
D=Decimal
getcontext().prec=10000000
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))
print(str(p)[:10000002])
</source>

This script will not run at all in the default 2.4 version of Python on the cluster. In order for this script to run you must use Python3. To do this, first list all versions of Python:
<source lang='bash'>
-bash-4.1$ module avail python

---------------------------- /shared/modulefiles ----------------------------
python/2.7.6 python/3.3.3 python/3.4.2
</source>

Then you can load the specific version you need:
module load python/3.3.3

Now you have access to the executable python3.

== See also ==
* [[Environment_Modules | Environment Modules]]
* [[Control_R_environment_using_modules | Control R environment using modules]]
* [[Create_shortcut_log-in_command | Create a shortcut for the ssh log-in command]]
* [[Installing_R_packages_locally | Installing R packages locally]]

== External links ==
* http://modules.sourceforge.net
* https://modules.readthedocs.io/en/latest/ (documentation)
* http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Assemble mitochondrial genomes from short read data

2023-06-16T08:58:46Z

Dawes0011:

A simple procedure for assembling mitochondrial genomes based on whole-genome re-sequencing data. The first step is to extract reads from the sequence library based on a closely related entirely assembled genome (e.g., for pig, the MT genome as present in the genome build, but could also be of a related species). The genome is then assembled using SOAPdenovo.

* a reference genome of a closely related population or species.
* a bowtie2 index (make with bowtie2_build)
* a blastable db of the reference mitochondrial genome
* a SOAPdenovo configuration file:

soapdenovo.config
[LIB]
avg_ins=450
reverse_seq=0
asm_flags=1
rank=3
q1=fq1.fq
q2=fq2.fq
Note that the avg_ins flag may vary between libraries; may have an effect on assembly efficiency.

<source lang='bash'>
#!/bin/bash
#SBATCH --time=1000
#SBATCH --mem=16000
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH --constraint=4gpercpu
#SBATCH --output=output_%j.txt
#SBATCH --error=error_output_%j.txt
#SBATCH --job-name=assemble_mito
#SBATCH --mail-type=ALL
#SBATCH --mail-user=hendrik-jan.megens@wur.nl
module load bowtie/2-2.2.1 SOAPdenovo2/r240 BLAST+/2.2.28 MUMmer/3.23

bowtie2 --phred$2 --local -p 8 -x mt_pig.fa -1 $3 -2 $4 | head -2 >$1_mito_align.sam
bowtie2 --phred$2 --local -p 8 -x mt_pig.fa -1 $3 -2 $4 | awk '$5>0' | head -10000 >>$1_mito_align.sam

java7 -jar /shared/apps/SHARED/picard-tools/picard-tools-1.109/SamToFastq.jar I=$1_mito_align.sam F=fq1.fq F2=fq2.fq INCLUDE_NON_PF_READS=True

SOAPdenovo-63mer all -K 63 -p 4 -s soapdenovo.config -o $1_mito_assembly.fa

blastn -query $1_mito_assembly.fa.scafSeq -db mt_pig.fa -outfmt 6

mummer -mum -b -c mt_pig.fa $1_mito_assembly.fa.scafSeq > mummer.mums
mummerplot -postscript -p mummer mummer.mums
</source>

Invoke like this:
<source lang='bash'>
sbatch do_mtalign_bowtie_pig.sh MA01F18 33\
/lustre/nobackup/WUR/ABGC/shared/Pig/ABGSA/ABGSA0071/ABGSA0071_MA01F18_R1.PF.fastq.gz\
/lustre/nobackup/WUR/ABGC/shared/Pig/ABGSA/ABGSA0071/ABGSA0071_MA01F18_R2.PF.fastq.gz

Globally installed software

2023-06-16T08:58:33Z

Dawes0011:

== Available as modules ==
* acml
* alphaimpute
* bamtools
* bcftools
* beagle
* blacs
* blas
* [[BLAST | BLAST+]]
* bonnie++
* boost
* [[bowtie2 | bowtie]]
* [[canu | canu]]
* cdo
* cmgui
* [[cuda | cuda]]
* diamond
* [[dmtcp | dmtcp]]
* emos
* fasttree
* ferret
* ffmpeg
* fftw2
* fftw3
* flex
* freebayes
* gcc
* gcta
* gdal
* gdb
* geos
* glibc
* glimmer
* glimmerHMM
* globalarrays
* grads
* grib
* gsl
* [[hadoop | hadoop ]]
* hdf4
* hdf5
* [[hmmer | hmmer]]
* hpl
* htslib
* hwloc
* ima2p
* intel
* intel-cluster-checker
* intel-cluster-runtime
* intel-tbb-oss
* iozone
* jasper
* java
* julia
* lapack
* [[matlab | MATLAB]]
* mixblup
* mpich
* mpiexec
* MRO
* muscle
* mvapich
* mvapich2
* ncbi-blast
* ncl
* nco
* ncview
* netcdf
* netcdf3-c++
* netperf
* octave
* open64
* openblas
* openlava
* openmpi
* oracle-instantclient
* phdf5
* picard
* plink
* prodigal
* python
* R
* ragel
* [[RAxML | RAxML]]
* RRO
* rstudio
* [[samtools | samtools]]
* scalapack
* slurm
* slurm-drmaa
* snpEff
* [[spark | SPARK]]
* szip
* torque
* [[Trinity | trinity]]
* udunits
* vcftools
* vim
* zlib

== Globally installed on all nodes ==

* Perl5.10
* pigz
* Python2.6
* BioPerl v1.61
* [http://samtools.sourceforge.net/tabix.shtml bgzip]
* [http://samtools.sourceforge.net/tabix.shtml tabix v0.2.5]

== Available as global SHARED modules ==
Software can be deposited in:
/shared/apps/SHARED/

Modules can be found in:
/shared/modulefiles/SHARED/

{| width="90%"
|- valign="top"
| width="30%" |
* [[allpathslg_48961 | ALLPATHS-LG/48961]]
* [[allpathslg 51910 | ALLPATHS-LG/51910]]
* [[ANGSD_0.614 | angsd/0.614]]
* [[augustus_2.7 | augustus/2.7]]
* [[bedtools2.18 | bedtools/2.18.0]]
* [[BLAST | BLAST+/2.2.28]]
* [[blat_v35 |blat/v35]]
* [[bowtie2_v2.2.1 | bowtie/2-2.2.1]]
* [[bowtie1_v1.0.0 | bowtie/1-1.0.0]]
* [[bwa_5.9 | bwa/0.5.9]]
* [[bwa_7.5 | bwa/0.7.5a]]
* [[cegma_2.4 | cegma/2.4]]
* [[Cufflinks | cufflinks/2.1.1]]

| width="30%" |
* [[exonerate_2.2.0 | exonerate/2.2.0-x86_64]]
* [[geneid_1.4.4 | geneid/1.4.4]]
* [[genewise_2.2.3 | genewise/2.2.3-rc7]]
* [[gmap_2014-01-21 | gmap/2014-01-21]]
* [[hmmer_3.1 | hmmer/3.1b1]]
* [[jellyfish_2.1.1 | jellyfish/2.1.1]]
* [[MAFFT_7.130 | MAFFT/7.130]]
* [[maker_2.2.8 | maker/2.28]]
* [[Muscle_3.8.31 | muscle/3.8.31]]
* [[Plink_1.07 | Plink/1.07]]
* [[Plink_1.9 | Plink/1.9]]
* [[Provean_1.1.3 | provean/1.1.3]]
* [[RepeatMasker_4.0.3 | RepeatMasker/4.0.3]]

| width="30%" |
* [[RepeatModeler_1.0.7 | RepeatModeler/1.0.7]]
* [[RAxML8.0.0 | RAxML/8.0.0]]
* [[samtools v0.1.12a | samtools/0.1.12a]]
* [[samtools v0.1.19 | samtools/0.1.19]]
* [[snap | snap/2013-11-29]]
* [[soapdenovo2_r240 | SOAPdenovo2/r240]]
* [[sra_toolkit_2.3.4 | sra-toolkit/2.3.4]]
* [[TopHat_2.0.11 | tophat/2.0.11]]
* [[Trinity_r20131110 | Trinity/r20131110]]
* [[wgs_assembler_8.1 | wgs-assembler/8.1]]
|}

== Adding a module to your current session ==
Use <code>module apropos</code> to find the module you wish to use, then <code>module load</code> to enable it.

== Adding a custom module directory to your environment ==
To allow the <code>module</code> program to find the custom module directory, the location of that directory has to be added to <code>MODULEPATH</code> variable.

<pre>
export MODULEPATH=$MODULEPATH:/shared/apps/WUR/ABGC/modulefiles
</pre>
This can be made permanent by adding this line of code to the <code>.bash_profile</code> file in the root of your home folder. To then load the modified <code>MODULEPATH</code> variable you have to load <code>.bash_profile</code> again:
<pre>
source .bash_profile
</pre>
This needs to be done only for terminals that are already open. Next time you login, <code>.bash_profile</code> will be loaded automatically.

You can check if the modules are found.
<pre>
module avail
</pre>
This should give output that includes something similar to this:

---------------------------------------- /shared/modulefiles/ -----------------------------------------
ALLPATHS-LG/48961 bwa/0.7.5a jellyfish/2.1.1 RepeatMasker/4.0.3
augustus/2.7 cegma/2.4 MAFFT/7.130 RepeatModeler/1.0.7
bedtools/2.18.0 cufflinks/2.1.1 maker/2.28 samtools/0.1.12a
BLAST+/2.2.28 exonerate/2.2.0-x86_64 muscle/3.8.31 samtools/0.1.19
blat/v35 geneid/1.4.4 plink/1.07 snap/2013-11-29
bowtie/2-2.2.1 genewise/2.2.3-rc7 provean/1.1.3 SOAPdenovo2/r240
bwa/0.5.9 hmmer/3.1b1 RAxML/8.0.0 tophat/2.0.11

== See also ==
* [[installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[ABGC_modules | modules specific for ABGC ]]

== External links ==

Node usage graph

2023-06-16T08:55:36Z

Dawes0011:

There is a graphing tool that uses elements directly from sacct to display information about the current cluster usage, node_usage_graph (located at in the anunna module ).

Example:
<pre>
[user@login0 ~]# module load anunna
[user@login0 ~]# usage_graph
node: |0% 100%|
fat001: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
fat002: CCCCCCCCC
MMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node001:

node002:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmm
node003:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MM
node004:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node005:CCCCCCCCCC

node006:CCCCCCCCCC

node007:CCCCCCCCCC

node008:CCCCCCCCCCccccc
MMMMMMMMMMMMMMMMMMMMM
node009:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
node010:

node011:

node012:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node013:

node014:

node015:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node016:CCCCCCCCCCCCCCCCCCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node017:

node018:

node019:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node020:

node021:

node022:

node023:

node024:CCCCCCCCCCCCCCC

node025:CCCCCCCCCCCCCCCCCCCCC

node026:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node027:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node028:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMM
node029:

node030:

node031:

node032:

node033:

node034:

node035:

node036:

node037:

node038:

node039:

node040:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node041:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCcccccc
MMMMMMMMMMMMMMMMMMMMMM
node042:RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
node049:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node050:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node051:

node052:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMMMMMmmmmmmmmmmmmmmm
node053:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node054:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
</pre>

This gives an overview of the current per-node resource usage. There are four types of letter:
* M: Memory reserved and in use
* m: Memory reserved and not in use
* C: CPU reserved and in use
* c: CPU reserved and not in use
* D: Drained node (not available for submission for some adminstrative reason
* R: Reserved node

It cannot however give you an indication of how much the queue is right now for any node. for that, squeue is a better resource.

Node usage graph

2023-06-16T08:55:20Z

Dawes0011:

There is a graphing tool that uses elements directly from sacct to display information about the current cluster usage, node_usage_graph (located at /cm/shared/apps/accounting/node_usage_graph ).

Example:
<pre>
[user@login0 ~]# module load anunna
[user@login0 ~]# usage_graph
node: |0% 100%|
fat001: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
fat002: CCCCCCCCC
MMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node001:

node002:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmm
node003:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MM
node004:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node005:CCCCCCCCCC

node006:CCCCCCCCCC

node007:CCCCCCCCCC

node008:CCCCCCCCCCccccc
MMMMMMMMMMMMMMMMMMMMM
node009:cccccccccc
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
node010:

node011:

node012:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node013:

node014:

node015:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node016:CCCCCCCCCCCCCCCCCCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node017:

node018:

node019:CCCCC
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node020:

node021:

node022:

node023:

node024:CCCCCCCCCCCCCCC

node025:CCCCCCCCCCCCCCCCCCCCC

node026:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node027:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

node028:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMM
node029:

node030:

node031:

node032:

node033:

node034:

node035:

node036:

node037:

node038:

node039:

node040:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node041:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCcccccc
MMMMMMMMMMMMMMMMMMMMMM
node042:RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
node049:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node050:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node051:

node052:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MMMMMMmmmmmmmmmmmmmmm
node053:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
M
node054:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
</pre>

This gives an overview of the current per-node resource usage. There are four types of letter:
* M: Memory reserved and in use
* m: Memory reserved and not in use
* C: CPU reserved and in use
* c: CPU reserved and not in use
* D: Drained node (not available for submission for some adminstrative reason
* R: Reserved node

It cannot however give you an indication of how much the queue is right now for any node. for that, squeue is a better resource.

Main Page

2023-01-27T14:40:32Z

Dawes0011:

Anunna is a [http://en.wikipedia.org/wiki/High-performance_computing High Performance Computer] (HPC) infrastructure hosted by [http://www.wageningenur.nl/nl/activiteit/Opening-High-Performance-Computing-cluster-HPC.htm Wageningen University & Research Centre]. It is open for use for all WUR research groups as well as other organizations, including companies, that have collaborative projects with WUR.

= Using Anunna =
* [[Tariffs | Costs associated with resource usage]]

== Gaining access to Anunna==
Access to the cluster and file transfer are traditionally done via [http://en.wikipedia.org/wiki/Secure_Shell SSH and SFTP].
* [[log_in_to_B4F_cluster | Logging into cluster using ssh]]
* [[file_transfer | File transfer options]]
* [[Services | Alternative access methods, and extra features and services on Anunna]]
* [[Filesystems | Data storage methods on Anunna]]

== Access Policy ==
[[Access_Policy | Main Article: Access Policy]]

Access needs to be granted actively (by creation of an account on the cluster by FB-IT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. Note that the use of Anunna is not free of charge. List price of CPU time and storage, and possible discounts on that list price for your organisation, can be retrieved from Shared Research Facilities or FB-IT.

= Events =

* [[Courses]] that have happened and are happening
* [[Downtime]] that will affect all users
* [[Meetings]] that may affect the policies of Anunna

= Other Software =

== Cluster Management Software and Scheduler ==
Anunna uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]
* [[Using_Slurm | Submit jobs with Slurm]]
* [[node_usage_graph | Be aware of how much work the cluster is under right now with 'node_usage_graph']]
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]

== Installation of software by users ==

* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]
* [[Setting local variables]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]
* [[Virtual_environment_Python_3.4_or_higher | Setting up and using a virtual environment for Python3.4 or higher ]]
* [[Installing WRF and WPS]]
* [[Running scripts on a fixed timeschedule (cron)]]

== Installed software ==

* [[Globally_installed_software | Globally installed software]]
* [[ABGC_modules | ABGC specific modules]]

= Useful Notes =

== Being in control of Environment parameters ==

* [[Using_environment_modules | Using environment modules]]
* [[Setting local variables]]
* [[Setting_TMPDIR | Set a custom temporary directory location]]
* [[Installing_R_packages_locally | Installing R packages locally]]
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]

== Controlling costs ==

* [[SACCT | using SACCT to see your costs]]
* [[get_my_bill | using the "get_my_bill" script to estimate costs]]

== Management ==
Product Owner of Anunna is Alexander van Ittersum (Wageningen UR,FB-IT, C&PS). [[User:dawes001 | Gwen Dawes (Wageningen UR, FB-IT, C&PS)]] and [[User:haars001 | Jan van Haarst (Wageningen UR,FB-IT, C&PS)]] are responsible for [[Maintenance_and_Management | Maintenance and Management]] of the cluster.

* [[Roadmap | Ambitions regarding innovation, support and administration of Anunna ]]

= Miscellaneous =
* [[Mailinglist | Electronic mail discussion lists]]
* [[History_of_the_Cluster | Historical information on the startup of Anunna]]
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]
* [[Parallel_R_code_on_SLURM | Running parallel R code on SLURM]]
* [[Convert_between_MediaWiki_and_other_formats | Convert between MediaWiki format and other formats]]
* [[Manual GitLab | GitLab: Create projects and add scripts]]
* [[Monitoring_executions | Monitoring job execution]]
* [[Shared_folders | Working with shared folders in the Lustre file system]]

= See also =
* [[Maintenance_and_Management | Maintenance and Management]]
* [[BCData | BCData]]
* [[Mailinglist | Electronic mail discussion lists]]
* [[About_ABGC | About ABGC]]
* [[Computer_cluster | High Performance Computing @ABGC]]
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]

= External links =
{| width="90%"
|- valign="top"
| width="30%" |
* [https://www.wur.nl/en/Value-Creation-Cooperation/Facilities/Wageningen-Shared-Research-Facilities/Our-facilities/Show/High-Performance-Computing-Cluster-HPC-Anunna.htm SRF offers a HPC facilty]
| width="30%" |
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]
|}