Virtual environment Python 3.4 or higher: Difference between revisions
m (<source> to <pre> to fix code blocks) |
|||
(2 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary. | With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary. | ||
== creating a new virtual environment == | == creating a new virtual environment == | ||
Line 6: | Line 5: | ||
If you do not already have a directory in your <code>$HOME</code> dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit). | If you do not already have a directory in your <code>$HOME</code> dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit). | ||
< | <pre> | ||
mkdir ~/my_envs | mkdir ~/my_envs | ||
</ | </pre> | ||
Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work): | Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work): | ||
< | <pre> | ||
module load python/3.5.0 | module load python/3.5.0 | ||
</ | </pre> | ||
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>p35_myproj</code>. | And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example <code>p35_myproj</code>. | ||
< | <pre> | ||
pyvenv install ~/my_envs/p35_myproj | pyvenv install ~/my_envs/p35_myproj | ||
</ | </pre> | ||
== activating a virtual environment == | == activating a virtual environment == | ||
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued: | Once the environment is created, each time the environment needs to be activated, the following command needs to be issued: | ||
< | <pre> | ||
source ~/my_envs/p35_myproj/bin/activate | source ~/my_envs/p35_myproj/bin/activate | ||
</ | </pre> | ||
This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory. | This assumes that the folder that contains the virtual environment documents (in this case called <code>newenv</code>), is in the present working directory. | ||
When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string. | When working on the virtual environment, the virtual environment name will be between brackets in front of the <code>user-host-prompt</code> string. | ||
< | <pre> | ||
(p35_myproj)user@host:~$ | |||
</ | </pre> | ||
Note that like with any command you can make an alias in your <code>~/.bashrc</code>. Just add something like this line to your <code>.bashrc</code>: | Note that like with any command you can make an alias in your <code>~/.bashrc</code>. Just add something like this line to your <code>.bashrc</code>: | ||
< | <pre> | ||
alias p35myproj='source ~/my_envs/p35_myproj/bin/activate' | alias p35myproj='source ~/my_envs/p35_myproj/bin/activate' | ||
</ | </pre> | ||
== installing modules on the virtual environment == | == installing modules on the virtual environment == | ||
Line 44: | Line 42: | ||
Before you start installing modules, first update pip itself: | Before you start installing modules, first update pip itself: | ||
< | <pre> | ||
pip install --upgrade pip | pip install --upgrade pip | ||
</ | </pre> | ||
you can then install other modules as you like, for instance numpy: | you can then install other modules as you like, for instance numpy: | ||
< | <pre> | ||
pip install numpy | pip install numpy | ||
</ | </pre> | ||
< | <pre> | ||
(p35_myproj) [user@nfs01 ~]$ pip install numpy | (p35_myproj) [user@nfs01 ~]$ pip install numpy | ||
Collecting numpy | Collecting numpy | ||
Line 61: | Line 59: | ||
Running setup.py install for numpy ... done | Running setup.py install for numpy ... done | ||
Successfully installed numpy-1.10.4 | Successfully installed numpy-1.10.4 | ||
</ | </pre> | ||
Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>). | Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through <code>pip</code>). | ||
< | <pre> | ||
python setup.py install | python setup.py install | ||
</ | </pre> | ||
== deactivating a virtual environment == | == deactivating a virtual environment == | ||
Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment. | Quitting a virtual environment can be done by using the command <code>deactivate</code>, which was loaded using the <code>source</code> command upon activating the virtual environment. | ||
< | <pre> | ||
deactivate | deactivate | ||
</ | </pre> | ||
== Make IPython work under virtualenv == | == Make IPython work under virtualenv == | ||
IPython can simply be installed through pip. | IPython can simply be installed through pip. | ||
< | <pre> | ||
pip install ipython | pip install ipython | ||
</ | </pre> | ||
== See also == | == See also == | ||
* [[ | * [[Anunna | Anunna]] | ||
== External links == | == External links == | ||
* [https://docs.python.org/3/library/venv.html#module-venv Python docs on virtenv] | * [https://docs.python.org/3/library/venv.html#module-venv Python docs on virtenv] |
Latest revision as of 14:45, 15 June 2023
With many Python packages available, which are often in conflict or requiring different versions depending on application, installing and controlling packages and versions is not always easy. In addition, so many packages are often used only occasionally, that it is questionable whether a system administrator of a centralized server system or a High Performance Compute (HPC) infrastructure can be expected to resolve all issues posed by users of the infrastructure. Even on a local system with full administrative rights managing versions, dependencies, and package collisions is often very difficult. The solution is to use a virtual environment, in which a specific set of packages can then be installed. As many different virtual environments can be created, and used side-by-side, as is necessary.
creating a new virtual environment
If you do not already have a directory in your $HOME
dir where your virtual environments live, first make one (it is assumed that you will over the course of time create several virtual environments for different projects and different versions of Python side-by-side, best to organise them a bit).
mkdir ~/my_envs
Then, load either Python 3.4 or 3.5 module (Python 3.3.3 should also work):
module load python/3.5.0
And then simply create an environment with a reasonably descriptive name (remember, you may accumulate as many as you desire), in this example p35_myproj
.
pyvenv install ~/my_envs/p35_myproj
activating a virtual environment
Once the environment is created, each time the environment needs to be activated, the following command needs to be issued:
source ~/my_envs/p35_myproj/bin/activate
This assumes that the folder that contains the virtual environment documents (in this case called newenv
), is in the present working directory.
When working on the virtual environment, the virtual environment name will be between brackets in front of the user-host-prompt
string.
(p35_myproj)user@host:~$
Note that like with any command you can make an alias in your ~/.bashrc
. Just add something like this line to your .bashrc
:
alias p35myproj='source ~/my_envs/p35_myproj/bin/activate'
installing modules on the virtual environment
Installing modules is the same as usual. The difference is that modules are in /path/to/virtenv/lib
, which may be living somewhere on your home directory. An easy way of installing modules is using pip
.
Before you start installing modules, first update pip itself:
pip install --upgrade pip
you can then install other modules as you like, for instance numpy:
pip install numpy
(p35_myproj) [user@nfs01 ~]$ pip install numpy Collecting numpy Using cached numpy-1.10.4.tar.gz Installing collected packages: numpy Running setup.py install for numpy ... done Successfully installed numpy-1.10.4
Similarly, installing packages from source works exactly the same as usual (note: only relevant for modules that can't be pulled through pip
).
python setup.py install
deactivating a virtual environment
Quitting a virtual environment can be done by using the command deactivate
, which was loaded using the source
command upon activating the virtual environment.
deactivate
Make IPython work under virtualenv
IPython can simply be installed through pip.
pip install ipython