|
|
| (20 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| Python is a high-level, interpreted programming language that has gained widespread popularity for its readability, versatility, and user-friendly syntax. Created by Guido van Rossum and first released in 1991, Python was designed to emphasize code clarity and reduce the complexity often associated with other languages. Its straightforward, English-like syntax makes it a natural choice for beginners, while its power and flexibility continue to attract experienced developers in numerous industries. | | Python is a high-level, interpreted programming language popular in scientific computing for its readability and its enormous ecosystem of third-party libraries — NumPy, Pandas, scikit-learn, PyTorch, TensorFlow, and many more. This page describes how to use Python on Anunna: the provided modules, how to manage your own packages with virtual environments or Miniforge, and how to expose an environment as a Jupyter kernel. |
|
| |
|
| One of Python’s greatest strengths lies in its extensive standard library, which provides built-in modules and functions for tasks ranging from file manipulation to internet protocols. Additionally, a thriving open-source community has developed countless third-party libraries and frameworks, making Python suitable for everything from data analysis and machine learning to web development and automation. Popular libraries like NumPy, Pandas, and TensorFlow enable developers to handle massive datasets, train artificial intelligence models, and build sophisticated applications with relative ease.
| | * [[Python/Modules|Modules]] - Locally built and optimized python packages. Recommended for best performance |
| | * [[uv]] - Rust based version and environment manager. |
| | * [[Python/Conda|Anaconda]] - Popular version and environment manager. |
| | == See also == |
|
| |
|
| Anunna offers environment modules for a single version of python for each bucket. On top of the standard environment modules, bundle modules are available for more specific user cases.
| | * [[Environment Modules]] |
| | * [[Installing Personal Software]] |
| | * [[Jupyter]] |
| | * [[R]] |
| | * [[Apptainer]] |
|
| |
|
| Users can also install their own Python distributions, like [https://mamba.readthedocs.io/en/latest/ Mamba]
| | == External links == |
|
| |
|
| | | * [https://docs.python.org/3/library/venv.html Python venv documentation] |
| == Creating Custom Virtual Environments ==
| | * [https://github.com/conda-forge/miniforge Miniforge] |
| | | * [https://conda-forge.org/ conda-forge] |
| The HPC team is aware that users may need to build custom python environments for their jobs. These environments can be based off the python environment modules provided by Anunna.
| |
| | |
| Python virtual environments are self-contained directories that house a specific Python installation and its associated packages, ensuring that one project’s dependencies don’t clash with another. They isolate your software requirements, letting the user manage different versions of libraries or modules in separate, discrete environments. This prevents version conflicts and keeps your system’s base Python environment clean. Tools like ‘venv’ and ‘virtualenv’ simplify creating and activating these spaces, making it straightforward to switch between multiple projects.
| |
| | |
| | |
| * Load the module of the desired Python version | |
| * Create a virtual environment folder
| |
| * Activate virtual environment
| |
| * Install desired packages with Pip
| |
| | |
| | |
| Firstly, load the module of the desired python version. The example that hereby follows makes use of Python-3.12.3 available at the 2024 bucket.
| |
| | |
| <pre>
| |
| module load 2024
| |
| module load Python/3.12.3
| |
| </pre>
| |
| | |
| Once the Python module is loaded, the environment can be created. The environment will be stored in a folder at the location of your choosing. It is not necessary to create the folder beforehand, though in this example it is assumed that the location <code>$MYBKP/PythonEnv</code> already exists. Note that <code>$MYBKP</code> refers to the lustre backup location specified at your <code>~/.bash_aliases</code> file (see our entry on [[Aliases_and_local_variables| Aliases and local variables]])
| |
| | |
| <pre>
| |
| python -m venv $MYBKP/PythonEnv/my_env | |
| </pre>
| |
| | |
| Once created, the virtual environment needs to be activated in order to be used or modified.
| |
| | |
| <pre>
| |
| source $MYBKP/PythonEnv/my_env/bin/activate
| |
| </pre>
| |
| | |
| After the environment has been activated, you should see the name of the environment as a suffix of your prompt.
| |
| | |
| <pre>
| |
| (my_env) user001@login200:$
| |
| </pre>
| |
| | |
| While the virtual environment is active, you can use pip to install modules directly into your environment
| |
| | |
| <pre>
| |
| pip install -U numpy pandas matplotlib datetime
| |
| </pre>
| |
| | |
| The installed modules are stored at $MYBKP/PythonEnv/my_env/lib/python3.12/site-packages/.
| |
| | |
| | |
| | |
| Note
| |
| | |
| == Creating Jupyter kernels from virtual environments ==
| |
| | |
| It is often the case that users need to have a custom environment in jupyter. This can be facilitated with virtual environments. Assuming we use the virtual environment from the previous section, we just need to
| |
| | |
| * load the required modules | |
| * activate the environment
| |
| * install the '''ipython''' package
| |
| * generate the kernel
| |
| | |
| First, load the required modules.
| |
| | |
| | |
| <pre>
| |
| module load 2024
| |
| module load Python/3.12.3
| |
| </pre>
| |
| | |
| | |
| Then activate the environment
| |
| | |
| <pre>
| |
| source $myBKP/PythonEnv/my_env/bin/activate
| |
| </pre>
| |
| | |
| Install the ipykernel package
| |
| | |
| <pre>
| |
| pip install ipykernel
| |
| </pre>
| |
| | |
| Run the command below to generate a kernel. Enter the desired name for the kernel with the flag <code>--name</code>
| |
| | |
| <pre>
| |
| python -m ipykernel install --user --name=my_env_kernel_name
| |
| </pre>
| |
| | |
| The kernel will be written to your home folder, more precisely <code>~/.local/share/jupyter/kernels/</code>.
| |
| This location will be monitored by jupyter, which should display your custom kernel as one of the kernel options.
| |
| '''Note''', in order to use this kernel the modules above will need to be loaded with jupyter-lmod.
| |
Python is a high-level, interpreted programming language popular in scientific computing for its readability and its enormous ecosystem of third-party libraries — NumPy, Pandas, scikit-learn, PyTorch, TensorFlow, and many more. This page describes how to use Python on Anunna: the provided modules, how to manage your own packages with virtual environments or Miniforge, and how to expose an environment as a Jupyter kernel.
- Modules - Locally built and optimized python packages. Recommended for best performance
- uv - Rust based version and environment manager.
- Anaconda - Popular version and environment manager.
See also
External links