Setting up Python virtualenv - Revision history

Haars0011: Phase 1 § 5 P1.5.5: content merged into Python (built-in venv supersedes virtualenv/pyvenv) (via update-page on MediaWiki MCP Server)

2026-06-16T14:01:19Z

Phase 1 § 5 P1.5.5: content merged into Python (built-in venv supersedes virtualenv/pyvenv) (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 14:01, 16 June 2026
Line 1:		Line 1:
	With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.		#REDIRECT [[Python#Virtual environments]]

	~~NOTE: as of Python 3.3 virtual environment support is built-in. See this page for an~~ [[~~virtual_environment_Python_3.4_or_higher \| alternative set-up of your virtual environment if using~~ Python ~~3.4 or higher~~]].

	~~== Creating a new virtual environment ==~~
	It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
	~~<pre>~~
	~~module load python/my-favourite-version (e.g. 2.7.12)~~
	~~virtualenv newenv~~
	OR
	~~pyvenv newenv (For versions >3.4)~~
	~~</pre>~~

	~~When the new environment is created, one will see a message similar to this:~~
	~~<nowiki> New python executable in newenv/bin/python3~~
	~~Also creating executable in newenv/bin/python~~
	~~Installing Setuptools.........................................................................done.~~
	~~Installing Pip................................................................................done.</nowiki>~~

	~~== Activating a virtual environment ==~~
	~~Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:~~
	~~<pre>~~
	~~source newenv/bin/activate~~
	~~</pre>~~
	~~This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.~~
	~~When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.~~
	~~<nowiki> (newenv)user@host:~$</nowiki>~~

	~~== Installing modules on the virtual environment ==~~
	Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
	~~<pre>~~
	~~pip install numpy~~
	~~</pre>~~
	~~Similarly, installing packages from source works exactly the same as usual.~~
	~~<pre>~~
	~~python setup.py install~~
	~~</pre>~~

	~~== deactivating a virtual environment ==~~
	~~Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.~~
	~~<pre>~~
	~~deactivate~~
	~~</pre>~~

	~~== Virtualenv kernels in Jupyter ==~~
	~~Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:~~

	~~(an alternative way to the manual way (using conda) is described [[Using conda to install a new kernel into your notebook\|here ]])~~

	* Make sure you have the ipykernel module in your venv. Activate it and pip install it:
	~~<nowiki>source ~/path/to/my/virtualenv/bin/activate && pip install ipykernel</nowiki>~~
	* Create the following directory path in your homedir if it doesn't already exist:
	~~<nowiki>mkdir -p ~/.local/share/jupyter/kernels/</nowiki>~~
	* Think of a nice descriptive name that doesn't clash with one of the already present kernels. I'll use 'testing'. Create this folder:
	~~<nowiki>mkdir ~/.local/share/jupyter/kernels/testing/</nowiki>~~
	* Add this file to this folder:
	~~<nowiki>vi ~/.local/share/jupyter/kernels/testing/kernel.json~~
	{
	~~"language": "python",~~
	~~"argv": [~~
	~~"/home/myhome/path/to/my/virtualenv/bin/python",~~
	~~"-m",~~
	~~"ipykernel",~~
	~~"-f",~~
	~~"{connection_file}"~~
	],
	~~"display_name": "testing"~~
	~~}</nowiki>~~
	* Reload Jupyterhub page. testing should now exist in your kernels list.

	~~You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:~~
	~~<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json~~
	{
	~~"language": "python",~~
	~~"env": {~~
	~~"SPARK_HOME":~~
	~~"/shared/apps/spark/my-spark-version"~~
	},
	~~"argv": [~~
	~~"/home/myhome/my/spark/venv/bin/python",~~
	~~"-m",~~
	~~"ipykernel",~~
	~~"-c", "import findspark; findspark.init()",~~
	~~"-f",~~
	~~"{connection_file}"~~
	],
	~~"display_name": "My Spark kernel"~~
	~~}</nowiki>~~
	~~(You'll want to make sure your spark cluster has the same environment - start it after activating this venv inside your sbatch script)~~

	~~== Make IPython work under virtualenv ==~~
	~~IPython may not work initially under a virtual environment. It may produce an error message like below:~~

	~~<nowiki> File "/usr/bin/ipython", line 11~~
	~~print "Could not start qtconsole. Please install ipython-qtconsole"~~
	~~^</nowiki>~~

	~~This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.~~
	~~<pre>~~
	~~ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython~~
	~~</pre>~~

	~~== External links ==~~
	* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
	* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Dawes0011 at 09:02, 16 June 2023

2023-06-16T09:02:11Z

← Older revision		Revision as of 09:02, 16 June 2023
Line 75:		Line 75:
	"env": {		"env": {
	"SPARK_HOME":		"SPARK_HOME":
	"~~/cm~~/shared/apps/spark/my-spark-version"		"/shared/apps/spark/my-spark-version"
	},		},
	"argv": [		"argv": [

Prins0891: to

 to fix code blocks

2023-06-15T13:48:36Z

<source> to <pre> to fix code blocks

← Older revision		Revision as of 13:48, 15 June 2023
Line 5:		Line 5:
	== Creating a new virtual environment ==		== Creating a new virtual environment ==
	It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:		It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
	<~~source lang='bash'~~>		<pre>
	module load python/my-favourite-version (e.g. 2.7.12)		module load python/my-favourite-version (e.g. 2.7.12)
	virtualenv newenv		virtualenv newenv
	OR		OR
	pyvenv newenv (For versions >3.4)		pyvenv newenv (For versions >3.4)
	</~~source~~>		</pre>

	When the new environment is created, one will see a message similar to this:		When the new environment is created, one will see a message similar to this:
Line 20:		Line 20:
	== Activating a virtual environment ==		== Activating a virtual environment ==
	Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:		Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
	<~~source lang='bash'~~>		<pre>
	source newenv/bin/activate		source newenv/bin/activate
	</~~source~~>		</pre>
	This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.		This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory.
	When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.		When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string.
Line 29:		Line 29:
	== Installing modules on the virtual environment ==		== Installing modules on the virtual environment ==
	Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.		Installing modules is the same as usual. The difference is that modules are in <code>/path/to/virtenv/lib</code>, which may be living somewhere on your home directory. When working from the virtual environment, the default <code>pip</code> will belong to the python version that is currently active. This means that the executable in <code>/path/to/virtenv/bin</code> are in fact the first in the <code>$PATH</code>.
	<~~source lang='bash'~~>		<pre>
	pip install numpy		pip install numpy
	</~~source~~>		</pre>
	Similarly, installing packages from source works exactly the same as usual.		Similarly, installing packages from source works exactly the same as usual.
	<~~source lang='bash'~~>		<pre>
	python setup.py install		python setup.py install
	</~~source~~>		</pre>

	== deactivating a virtual environment ==		== deactivating a virtual environment ==
	Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.		Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment.
	<~~source lang='bash'~~>		<pre>
	deactivate		deactivate
	</~~source~~>		</pre>

	== Virtualenv kernels in Jupyter ==		== Virtualenv kernels in Jupyter ==
Line 97:		Line 97:

	This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.		This can be resolved by adding a soft link with the name <code>ipython</code> to the <code>bin</code> directory in the virtual environment folder.
	<~~source lang='bash'~~>		<pre>
	ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython		ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
	</~~source~~>		</pre>

	== External links ==		== External links ==
	* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]		* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
	* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]		* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Haars001: Add link to page describing conda alternative

2020-07-10T14:36:54Z

Add link to page describing conda alternative

← Older revision		Revision as of 14:36, 10 July 2020
Line 45:		Line 45:
	== Virtualenv kernels in Jupyter ==		== Virtualenv kernels in Jupyter ==
	Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:		Want your own virtualenv kernel in a notebook? This can be done by making your own kernel specifications:

			(an alternative way to the manual way (using conda) is described [[Using conda to install a new kernel into your notebook\|here ]])

	* Make sure you have the ipykernel module in your venv. Activate it and pip install it:		* Make sure you have the ipykernel module in your venv. Activate it and pip install it:

Dawes001: /* Virtualenv kernels in Jupyter */

2019-12-11T15:32:51Z

Virtualenv kernels in Jupyter

← Older revision		Revision as of 15:32, 11 December 2019
Line 57:		Line 57:
	"language": "python",		"language": "python",
	"argv": [		"argv": [
	"~/path/to/my/virtualenv/bin/python",		"/home/myhome/path/to/my/virtualenv/bin/python",
	"-m",		"-m",
	"ipykernel",		"ipykernel",

Dawes001: /* Virtualenv kernels in Jupyter */

2019-12-11T15:32:23Z

Virtualenv kernels in Jupyter

← Older revision		Revision as of 15:32, 11 December 2019
Line 76:		Line 76:
	},		},
	"argv": [		"argv": [
	"~/my/spark/venv/bin/python",		"/home/myhome/my/spark/venv/bin/python",
	"-m",		"-m",
	"ipykernel",		"ipykernel",

Dawes001 at 13:58, 19 March 2019

2019-03-19T13:58:57Z

← Older revision		Revision as of 13:58, 19 March 2019
Line 98:		Line 98:
	ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython		ln -s /path/to/virtenv/bin/ipython3 /path/to/virtenv/bin/ipython
	</source>		</source>

	~~== See also ==~~
	* [[B4F_cluster \| B4F Cluster]]

	== External links ==		== External links ==
	* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]		* [https://pypi.python.org/pypi/virtualenv Python3 documentation for virtualenv]
	* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]		* [http://cemcfarland.wordpress.com/2013/03/09/getting-ipython3-working-inside-your-virtualenv/ Solving the IPython hickup under virtual environment]

Dawes001 at 14:45, 5 July 2017

2017-07-05T14:45:19Z

← Older revision		Revision as of 14:45, 5 July 2017
Line 8:		Line 8:
	module load python/my-favourite-version (e.g. 2.7.12)		module load python/my-favourite-version (e.g. 2.7.12)
	virtualenv newenv		virtualenv newenv
			OR
			pyvenv newenv (For versions >3.4)
	</source>		</source>

Dawes001: virtualenv is installed for all python modules.

2017-07-05T14:40:47Z

virtualenv is installed for all python modules.

← Older revision		Revision as of 14:40, 5 July 2017
Line 6:		Line 6:
	It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:		It is assumed that the appropriate <code>virtualenv</code> executable for the Python version of choice is installed. A new virtual environment, in this case called <code>newenv</code> is created like so:
	<source lang='bash'>		<source lang='bash'>
			module load python/my-favourite-version (e.g. 2.7.12)
	virtualenv newenv		virtualenv newenv
	~~</source>~~

	~~Should virtualenv not be installed, the virtualenv script can be downloaded and accessed directly:~~
	~~<source lang='bash'>~~
	~~curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz~~
	~~tar -xzvf virtualenv-1.9.tar.gz~~
	~~python3 virtualenv-1.9/virtualenv.py testenv~~
	</source>		</source>

Dawes001: /* Virtualenv kernels in Jupyter */

2017-06-19T16:21:25Z

Virtualenv kernels in Jupyter

← Older revision		Revision as of 16:21, 19 June 2017
Line 72:		Line 72:

	You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:		You can do more complex things with this, such as construct your own Spark environment. This relies on having the module findspark installed:
	<nowiki> vi ~/.local/share/jupyter/kernels/~~testing~~/kernel.json		<nowiki> vi ~/.local/share/jupyter/kernels/mysparkkernel/kernel.json
	{		{
	"language": "python",		"language": "python",