JupyterHub with GPU - Revision history

Brumm025: /* Create conda environment that we can use for a jupyter kernel */

2023-10-25T11:34:59Z

Create conda environment that we can use for a jupyter kernel

← Older revision		Revision as of 13:34, 25 October 2023
Line 25:		Line 25:
	python -m ipykernel install --user --name kernel_test</nowiki>		python -m ipykernel install --user --name kernel_test</nowiki>

	NOTE: You can specific the python version for you conda environment with python=3 Please take care what python version is compatible with you required packages.		NOTE: You can specific the python version for you conda environment with python=3.10 Please take care what python version is compatible with you required packages.

	== Install required packages ==		== Install required packages ==

Brumm025: /* Create conda environment that we can use for a jupyter kernel */

2023-10-25T11:34:43Z

Create conda environment that we can use for a jupyter kernel

← Older revision		Revision as of 13:34, 25 October 2023
Line 21:		Line 21:

	<nowiki>		<nowiki>
	conda create -y -n kernel_test python=3 ipykernel && conda activate kernel_test		conda create -y -n kernel_test python=3.10 ipykernel
			conda activate kernel_test
	python -m ipykernel install --user --name kernel_test</nowiki>		python -m ipykernel install --user --name kernel_test</nowiki>

	NOTE: You can specific the python version for you conda environment with python=3 Please take care what python version is compatible with you required packages.		NOTE: You can specific the python version for you conda environment with python=3 Please take care what python version is compatible with you required packages.

	== Install required packages ==		== Install required packages ==

Brumm025 at 11:31, 25 October 2023

2023-10-25T11:31:57Z

← Older revision		Revision as of 13:31, 25 October 2023
Line 12:		Line 12:
	<nowiki>		<nowiki>
	ln -s /lustre/[path to your lustre folder] [reference name, for example lustre_folders]</nowiki>		ln -s /lustre/[path to your lustre folder] [reference name, for example lustre_folders]</nowiki>

			To remove a link:

			<nowiki>
			rm [reference name, for example lustre_folders]</nowiki>

	== Create conda environment that we can use for a jupyter kernel ==		== Create conda environment that we can use for a jupyter kernel ==

Brumm025 at 08:47, 18 October 2023

2023-10-18T08:47:17Z

← Older revision		Revision as of 10:47, 18 October 2023
Line 5:		Line 5:
	* [https://wiki.anunna.wur.nl/index.php/Log_in_to_Anunna Connect to login node of Anunna]		* [https://wiki.anunna.wur.nl/index.php/Log_in_to_Anunna Connect to login node of Anunna]
	* [https://wiki.anunna.wur.nl/index.php/Running_Snakemake_pipelines#Installation Install miniconda]		* [https://wiki.anunna.wur.nl/index.php/Running_Snakemake_pipelines#Installation Install miniconda]

			== Link lustre path to home directory ==

			When working from Jupyterhub the default working directory is the home folder. However, it is recommended to put your data and code on the lustre pathings. To make this easier, we can create a link to lustre from our home directory:

			<nowiki>
			ln -s /lustre/[path to your lustre folder] [reference name, for example lustre_folders]</nowiki>

	== Create conda environment that we can use for a jupyter kernel ==		== Create conda environment that we can use for a jupyter kernel ==
Line 33:		Line 40:
	* <strong>Maximum execution time (hours:minutes:seconds):</strong> maximum amount of time the notebook is available		* <strong>Maximum execution time (hours:minutes:seconds):</strong> maximum amount of time the notebook is available
	* <strong>Extra options:</strong> --gres=gpu:1 (default when selecting GPU, gpu:x for x amount of GPUs)		* <strong>Extra options:</strong> --gres=gpu:1 (default when selecting GPU, gpu:x for x amount of GPUs)

			= Using multiple GPUs =

			* Select multiple GPUs in when starting jupyterhub in the extra options menu: --gres=gpu:x where x is amount of requested GPUs
			* There should be multiple GPUs available to the jupyterhub notebook. Check this by using GPU tests in the following section.

	= Test GPU availability =		= Test GPU availability =

Brumm025: /* Start jupyter notebook with GPU */

2023-10-06T11:36:38Z

Start jupyter notebook with GPU

← Older revision		Revision as of 13:36, 6 October 2023
Line 28:		Line 28:

	* <strong>Select a location for your server:</strong> on the cluster (default option)		* <strong>Select a location for your server:</strong> on the cluster (default option)
	* <strong>Partition to use:</strong> gpu
	* <strong>Partition to use:</strong> gpu		* <strong>Partition to use:</strong> gpu
	* <strong>Memory (in MB):</strong> desired memory		* <strong>Memory (in MB):</strong> desired memory

Brumm025: /* Start jupyter notebook with GPU */

2023-10-06T11:36:27Z

Start jupyter notebook with GPU

← Older revision		Revision as of 13:36, 6 October 2023
Line 27:		Line 27:
	Go [https://notebook.anunna.wur.nl here] and select:		Go [https://notebook.anunna.wur.nl here] and select:

	* Select a location for your server: on the cluster (default option)		* <strong>Select a location for your server:</strong> on the cluster (default option)
	* Partition to use: gpu		* <strong>Partition to use:</strong> gpu
	* Partition to use: gpu		* <strong>Partition to use:</strong> gpu
	* Memory (in MB): desired memory		* <strong>Memory (in MB):</strong> desired memory
	* Number of CPUs: desired CPU count		* <strong>Number of CPUs:</strong> desired CPU count
	* Maximum execution time (hours:minutes:seconds): maximum amount of time the notebook is available		* <strong>Maximum execution time (hours:minutes:seconds):</strong> maximum amount of time the notebook is available
	* Extra options: --gres=gpu:1 (default when selecting GPU, gpu:x for x amount of GPUs)		* <strong>Extra options:</strong> --gres=gpu:1 (default when selecting GPU, gpu:x for x amount of GPUs)

	= Test GPU availability =		= Test GPU availability =

Brumm025: /* Tensorflow */

2023-10-06T11:34:29Z

Tensorflow

← Older revision		Revision as of 13:34, 6 October 2023
Line 126:		Line 126:
	<nowiki>		<nowiki>
	import tensorflow as tf		import tensorflow as tf

	hasGPUSupport = tf.test.is_built_with_cuda()		hasGPUSupport = tf.test.is_built_with_cuda()
	gpuList = tf.config.list_physical_devices('GPU')		gpuList = tf.config.list_physical_devices('GPU')

	print("Tensorflow Compiled with CUDA/GPU Support:", hasGPUSupport)		print("Tensorflow Compiled with CUDA/GPU Support:", hasGPUSupport)
	print("Tensorflow can access", len(gpuList), "GPU")		print("Tensorflow can access", len(gpuList), "GPU")

Brumm025: /* Pytorch */

2023-10-06T11:34:09Z

Pytorch

← Older revision		Revision as of 13:34, 6 October 2023
Line 103:		Line 103:
	mat_mul = torch.matmul(tensor_a, cuda_twos.T)		mat_mul = torch.matmul(tensor_a, cuda_twos.T)
	print(mat_mul, '\n')		print(mat_mul, '\n')
	~~try:~~
	~~get_version()~~
	~~except Exception as e:~~
	~~print('get_version() failed, exception message below:')~~
	~~print(e)~~

	try:		try:
	~~check_cuda~~()		get_version()
	except Exception as e:		except Exception as e:
	print('~~check_cuda~~() failed, exception message below:')		print('get_version() failed, exception message below:')
	print(e)		print(e)

	#print('~~>>>> time.sleep~~(20)')		try:
	~~#time.sleep~~(20)		check_cuda()
			except Exception as e:
			print('check_cuda() failed, exception message below:')
			print(e)

	try:		try:
	check_cuda_ops()		check_cuda_ops()
	except Exception as e:		except Exception as e:
	print('check_cuda_ops() failed, exception message below:')		print('check_cuda_ops() failed, exception message below:')
	print(e)</nowiki>		print(e)</nowiki>

	== Tensorflow ==		== Tensorflow ==

Brumm025 at 11:30, 6 October 2023

2023-10-06T11:30:14Z

← Older revision		Revision as of 13:30, 6 October 2023
Line 20:		Line 20:
	As an example I use the following pytorch installation:		As an example I use the following pytorch installation:

	<nowiki>		<nowiki>
	pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118		pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</nowiki>
	</nowiki>

	= Start jupyter notebook with GPU =		= Start jupyter notebook with GPU =
Line 40:		Line 39:
	== Pytorch ==		== Pytorch ==

	<nowiki>		<nowiki>
	def check_all_cuda_devices():		def check_all_cuda_devices():
	device_count = torch.cuda.device_count()		device_count = torch.cuda.device_count()
Line 123:		Line 122:
	except Exception as e:		except Exception as e:
	print('check_cuda_ops() failed, exception message below:')		print('check_cuda_ops() failed, exception message below:')
	print(e)		print(e)</nowiki>
	</nowiki>

	== Tensorflow ==		== Tensorflow ==

	<nowiki>		<nowiki>
	import tensorflow as tf		import tensorflow as tf
	hasGPUSupport = tf.test.is_built_with_cuda()		hasGPUSupport = tf.test.is_built_with_cuda()
Line 145:		Line 143:
	# Run on the GPU		# Run on the GPU
	c = tf.matmul(a, b)		c = tf.matmul(a, b)
	print(c)		print(c)</nowiki>
	</nowiki>

Brumm025: /* Install required packages */

2023-10-06T11:29:23Z

Install required packages

← Older revision		Revision as of 13:29, 6 October 2023
Line 21:		Line 21:

	<nowiki>		<nowiki>
	pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</nowiki>		pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
			</nowiki>

	= Start jupyter notebook with GPU =		= Start jupyter notebook with GPU =