Parallel R code on SLURM - Revision history

Vaend001 at 19:07, 19 February 2019

2019-02-19T19:07:30Z

← Older revision		Revision as of 21:07, 19 February 2019
Line 2:		Line 2:
	== Using R code on SLURM for embarrassingly parallel calculations ==		== Using R code on SLURM for embarrassingly parallel calculations ==

	The most well-known R packages that provide parallel functionality, e.g. doParallel or doSNOW, do not work properly on ~~the HPC~~. Using these packages will be particularly problematic when you try to run (array) jobs over multiple nodes. However, the [https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html rslurm package] allows you to do [https://en.wikipedia.org/wiki/Embarrassingly_parallel embarrassingly parallel] calculations on SLURM. The package automatically divides the computation over multiple nodes and writes the necessary submission scripts. It also includes functions to retrieve and combine the output from different nodes, as well as wrappers for common SLURM commands.		The most well-known R packages that provide parallel functionality, e.g. doParallel or doSNOW, do not work properly on Anunna. Using these packages will be particularly problematic when you try to run (array) jobs over multiple nodes. However, the [https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html rslurm package] allows you to do [https://en.wikipedia.org/wiki/Embarrassingly_parallel embarrassingly parallel] calculations on SLURM. The package automatically divides the computation over multiple nodes and writes the necessary submission scripts. It also includes functions to retrieve and combine the output from different nodes, as well as wrappers for common SLURM commands.

	=== Example code ===		=== Example code ===

Dawes001 at 09:23, 23 May 2018

2018-05-23T09:23:57Z

← Older revision		Revision as of 11:23, 23 May 2018
Line 3:		Line 3:

	The most well-known R packages that provide parallel functionality, e.g. doParallel or doSNOW, do not work properly on the HPC. Using these packages will be particularly problematic when you try to run (array) jobs over multiple nodes. However, the [https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html rslurm package] allows you to do [https://en.wikipedia.org/wiki/Embarrassingly_parallel embarrassingly parallel] calculations on SLURM. The package automatically divides the computation over multiple nodes and writes the necessary submission scripts. It also includes functions to retrieve and combine the output from different nodes, as well as wrappers for common SLURM commands.		The most well-known R packages that provide parallel functionality, e.g. doParallel or doSNOW, do not work properly on the HPC. Using these packages will be particularly problematic when you try to run (array) jobs over multiple nodes. However, the [https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html rslurm package] allows you to do [https://en.wikipedia.org/wiki/Embarrassingly_parallel embarrassingly parallel] calculations on SLURM. The package automatically divides the computation over multiple nodes and writes the necessary submission scripts. It also includes functions to retrieve and combine the output from different nodes, as well as wrappers for common SLURM commands.

			=== Example code ===
			<code lang='R'>library(rslurm)
			sjob <- slurm_apply(test_func, pars, jobname = 'test_apply',
			nodes = 2, cpus_per_node = 2, submit = FALSE)</code>

			Please be aware that new Slurm jobs will have a few seconds of lead time before executing, so try to make sure that your new tasks are appropriately long-living (~60s is a good minimum). Otherwise most of your 'compute' time will be waiting for jobs to start and stop.

	== External links ==		== External links ==
	[https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html Vignette for the rslurm package]		[https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html Vignette for the rslurm package]

Eikel011: Created the page Parallel R code on SLURM

2018-05-19T17:41:37Z

Created the page Parallel R code on SLURM

New page

== Using R code on SLURM for embarrassingly parallel calculations ==

The most well-known R packages that provide parallel functionality, e.g. doParallel or doSNOW, do not work properly on the HPC. Using these packages will be particularly problematic when you try to run (array) jobs over multiple nodes. However, the [https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html rslurm package] allows you to do [https://en.wikipedia.org/wiki/Embarrassingly_parallel embarrassingly parallel] calculations on SLURM. The package automatically divides the computation over multiple nodes and writes the necessary submission scripts. It also includes functions to retrieve and combine the output from different nodes, as well as wrappers for common SLURM commands.

== External links ==
[https://cran.r-project.org/web/packages/rslurm/vignettes/rslurm.html Vignette for the rslurm package]