Spark - Revision history

Dawes001: /* SPARK in Jupyter */

2020-10-05T12:23:46Z

SPARK in Jupyter

← Older revision		Revision as of 14:23, 5 October 2020
Line 49:		Line 49:
	== SPARK in Jupyter ==		== SPARK in Jupyter ==

	There is a kernel available for using Spark from Jupyter. All this does ~~(for now)~~ is to set up the correct path to the python version and the spark binaries for you. In order to set up your Context, your first cell for each notebook should be:		There is a kernel available for using Spark from Jupyter. All this does is to set up the correct path to the python version and the spark binaries for you. In order to set up your Context, your first cell for each notebook should be:
	<nowiki>import pyspark		<nowiki>import os, pyspark
	conf = (pyspark.SparkConf()		with open(os.environ['HOME']+'/.spark/current/master') as f:
	.setMaster(~~"spark://mysparkcluster:7077"~~)		conf = (pyspark.SparkConf()
			.setMaster(f.read().strip()))
	.setAppName("MyName"))		.setAppName("MyName"))

Line 58:		Line 59:

	Using the cluster master name from the master file in your job output as above. Subsequent cells will then have sc defined. Run this cell only once - attempting to reconnect will throw an error. That application will run until the kernel is terminated and prevent other applications from being able to be executed - you may wish to manually terminate your kernel from the top bar in Jupyter to free resources.		Using the cluster master name from the master file in your job output as above. Subsequent cells will then have sc defined. Run this cell only once - attempting to reconnect will throw an error. That application will run until the kernel is terminated and prevent other applications from being able to be executed - you may wish to manually terminate your kernel from the top bar in Jupyter to free resources.

			As a teacher, you might want to put that master file somewhere else, such as /lustre/shared, so that students can connect to your cluster.

Dawes001: /* SPARK on HPC */

2020-10-05T12:21:30Z

SPARK on HPC

← Older revision		Revision as of 14:21, 5 October 2020
Line 12:		Line 12:
	#SBATCH --qos=QOS		#SBATCH --qos=QOS

	module load ~~python~~		module load spark/3.0.1-2.7
	module load ~~spark~~		module load python/3.8.5

	source $SPARK_HOME/wur/start-spark		source $SPARK_HOME/wur/start-spark

Dawes001: /* SPARK on HPC */

2019-07-15T15:03:35Z

SPARK on HPC

← Older revision		Revision as of 17:03, 15 July 2019
Line 10:		Line 10:
	#SBATCH --tasks-per-node=<number of workers per node>		#SBATCH --tasks-per-node=<number of workers per node>
	#SBATCH --job-name="my spark cluster"		#SBATCH --job-name="my spark cluster"
	#SBATCH --~~partition~~=~~PARTITION~~		#SBATCH --qos=QOS

	module load python		module load python

Dawes001: Created page with "Apache Spark is a means of distributing compute resources across multiple worker machines. It is the successor to Hadoop, and allows for a wider distribution of code to be exe..."

2017-06-19T16:02:25Z

Created page with "Apache Spark is a means of distributing compute resources across multiple worker machines. It is the successor to Hadoop, and allows for a wider distribution of code to be exe..."

New page

Apache Spark is a means of distributing compute resources across multiple worker machines. It is the successor to Hadoop, and allows for a wider distribution of code to be executed on the clustered resources. The only requirement for Spark to be able to operate is that each worker must be able to reach each other via TCP, thus it allows for compute to be executed on very simple resources, if the code itself can be translated into the MapReduce paradigm.

== SPARK on HPC ==
In order to create a personal SPARK cluster, you must first request resources on the HPC. Use this example submission script to initialise your cluster:

<nowiki>#!/bin/bash
#SBATCH --time=<length>
#SBATCH --mem-per-cpu=4000
#SBATCH --nodes=<number of nodes>
#SBATCH --tasks-per-node=<number of workers per node>
#SBATCH --job-name="my spark cluster"
#SBATCH --partition=PARTITION

module load python
module load spark

source $SPARK_HOME/wur/start-spark

tail -f /dev/null</nowiki>

This will spawn a new cluster of your desired dimensions once resources are available. This spark module has been written to output its logs to your home directory, at:

/home/WUR/yourid/.spark/<jobid>/

In this folder you will find the raw logs of the master and all worker threads. By default the master will consume 1Gb of memory from the first process, and so a single 4Gb 'cluster' will be provided with one 3Gb worker. You can adjust the CPU/memory use by adjusting the parameters in your batch script.

Within the log file you will find two unique files: master, and master-console. master will always contain the URI of the current spark cluster master access point, and master-console the URL of the console of it.

To access the web console, the easiest solution is to use links:

<nowiki>links http://myspark:8081</nowiki>

This will nicely render the page for you in the console. Ctrl-R reloads the page, q to quit.

There are several caveats to remember with this:

* The cluster exists (and consumes resources) until you cancel it with scancel <jobid>
* There is no security at all - any user of the HPC can access both these at any point if they know the port and host.

== Instant SPARK ==

You can also spin up clusters solely to execute scripts. Simply replace the last line from the example above:
<nowiki>tail -f /dev/null</nowiki>
with
<nowiki>spark-submit myscript.py</nowiki>

And after the script has executed, the cluster will automatically terminate.

== SPARK in Jupyter ==

There is a kernel available for using Spark from Jupyter. All this does (for now) is to set up the correct path to the python version and the spark binaries for you. In order to set up your Context, your first cell for each notebook should be:
<nowiki>import pyspark
conf = (pyspark.SparkConf()
.setMaster("spark://mysparkcluster:7077")
.setAppName("MyName"))

sc = pyspark.SparkContext(conf=conf)</nowiki>

Using the cluster master name from the master file in your job output as above. Subsequent cells will then have sc defined. Run this cell only once - attempting to reconnect will throw an error. That application will run until the kernel is terminated and prevent other applications from being able to be executed - you may wish to manually terminate your kernel from the top bar in Jupyter to free resources.