Dawes0011 at 09:00, 16 June 2023

2023-06-16T09:00:40Z

← Older revision		Revision as of 11:00, 16 June 2023
Line 22:		Line 22:

	There is an example submission script at:		There is an example submission script at:
	<nowiki>~~/cm~~/shared/apps/hadoop/current/wur/example_sbatch.sh</nowiki>		<nowiki>/shared/apps/hadoop/current/wur/example_sbatch.sh</nowiki>
	Copy this to your preferred location and submit it.		Copy this to your preferred location and submit it.

Dawes001 at 11:24, 11 July 2017

2017-07-11T11:24:16Z

← Older revision		Revision as of 13:24, 11 July 2017
Line 7:		Line 7:
	And many other components.		And many other components.

	This install of Hadoop has been optimised for executing a standalone HDFS filesystem, primarily for teaching purposes		This install of Hadoop has been optimised for executing a standalone HDFS filesystem, primarily for teaching purposes.

	== WARNING ==		== WARNING ==

Dawes001: Created page with "Hadoop is another kind of architecture to promote writing code that can be executed in a parallel fashion. It's become more of an environment than a piece of software these da..."

2017-07-11T11:23:57Z

Created page with "Hadoop is another kind of architecture to promote writing code that can be executed in a parallel fashion. It's become more of an environment than a piece of software these da..."

New page

Hadoop is another kind of architecture to promote writing code that can be executed in a parallel fashion. It's become more of an environment than a piece of software these days, as it typically contains:

* HBase, flat table storage
* Hive, a form of data warehousing
* MapReduce, a style of coding to promote largely non-interacting workflows in order to leverage slow interconnects between workers
* HDFS, a distributed filesystem between workers
And many other components.

This install of Hadoop has been optimised for executing a standalone HDFS filesystem, primarily for teaching purposes

== WARNING ==
HDFS is incredibly insecure. As soon as anyone knows the location of your nameserver, they may spoof into it to access your data, as any user. This can be prevented by using Kerberos tickets, however that is currently beyond our level of complexity. Do not store anything you care about here.

=== Usage ===
load up the module, and java:
<source lang="bash">
module load hadoop
module load java
</source>

This will set your environment variables correctly.

There is an example submission script at:
<nowiki>/cm/shared/apps/hadoop/current/wur/example_sbatch.sh</nowiki>
Copy this to your preferred location and submit it.

Edit this script and adjust the settings accordingly. You can override the location that HDFS will store its blocks by altering the environment variable HADOOP_TMP_DIR, which will default to /lustre/scratch.

Once it's run, the output log will tell you the location. I'd recommend setting a static location in your job script so that it's reproducibly located, but the HADOOP_CONF_DIR will point to the correct fs.defaultFS once you've started it.

=== Mechanism details ===
The example script will run an executable called start-hdfs.sh. This simply starts by creating ~/.hadoop/$SLURM_JOB_ID, then assigning that as HADOOP_LOG_DIR. Next, it creates a subfolder conf, assigns that to HADOOP_CONF_DIR, and writes the core-site.xml and hdfs-site.xml files into it. Last, it creates a symlink from ~/.hadoop/current to point to your currently running HDFS instance. On the client side, all HADOOP_CONF_DIR's point to ~/.hadoop/current/conf so that whichever job is submitted, this symlink will point to the currently running config for you.

If you're trying to share your HDFS instance with other users, you should:
* Copy the ~/.hadoop/current/conf/* to somewhere shared
* Tell them to set the environment variable HADOOP_CONF_DIR to that location

Hadoop - Revision history

Dawes0011 at 09:00, 16 June 2023

Dawes001 at 11:24, 11 July 2017

Dawes001: Created page with "Hadoop is another kind of architecture to promote writing code that can be executed in a parallel fashion. It's become more of an environment than a piece of software these da..."