Python

Python is a high-level, interpreted programming language that has gained widespread popularity for its readability, versatility, and user-friendly syntax. Created by Guido van Rossum and first released in 1991, Python was designed to emphasize code clarity and reduce the complexity often associated with other languages. Its straightforward, English-like syntax makes it a natural choice for beginners, while its power and flexibility continue to attract experienced developers in numerous industries.

One of Python’s greatest strengths lies in its extensive standard library, which provides built-in modules and functions for tasks ranging from file manipulation to internet protocols. Additionally, a thriving open-source community has developed countless third-party libraries and frameworks, making Python suitable for everything from data analysis and machine learning to web development and automation. Popular libraries like NumPy, Pandas, and TensorFlow enable developers to handle massive datasets, train artificial intelligence models, and build sophisticated applications with relative ease.

Anunna offers environment modules for a single version of python for each bucket. On top of the standard environment modules, bundle modules are available for more specific user cases.

Users can also install their own Python distributions, like Mamba

Creating Custom Virtual Environments

The HPC team is aware that users may need to build custom python environments for their jobs. These environments can be based off the python environment modules provided by Anunna.

Python virtual environments are self-contained directories that house a specific Python installation and its associated packages, ensuring that one project’s dependencies don’t clash with another. They isolate your software requirements, letting the user manage different versions of libraries or modules in separate, discrete environments. This prevents version conflicts and keeps your system’s base Python environment clean. Tools like ‘venv’ and ‘virtualenv’ simplify creating and activating these spaces, making it straightforward to switch between multiple projects.

• Load the module of the desired Python version
• Create a virtual environment folder
• Activate virtual environment
• Install desired packages with Pip

Firstly, load the module of the desired python version. The example that hereby follows makes use of Python-3.12.3 available at the 2024 bucket.

module load 2024
module load Python/3.12.3

Once the Python module is loaded, the environment can be created. The environment will be stored in a folder at the location of your choosing. It is not necessary to create the folder beforehand, though in this example it is assumed that the location $MYBKP/PythonEnv already exists. Note that $MYBKP refers to the lustre backup location specified at your ~/.bash_aliases file (see our entry on Aliases and local variables)

python -m venv $MYBKP/PythonEnv/my_env

Once created, the virtual environment needs to be activated in order to be used or modified.

source $MYBKP/PythonEnv/my_env/bin/activate

After the environment has been activated, you should see the name of the environment as a suffix of your prompt.

(my_env) user001@login200:$ 

While the virtual environment is active, you can use pip to install modules directly into your environment

pip install -U numpy pandas matplotlib datetime

The installed modules are stored at $MYBKP/PythonEnv/my_env/lib/python3.12/site-packages/.

Note

Creating Jupyter kernels from virtual environments

It is often the case that users need to have a custom environment in jupyter. This can be facilitated with virtual environments. Assuming we use the virtual environment from the previous section, we just need to

• activate the environment
• install the ipython package
• generate the kernel

source $myBKP/PythonEnv/my_env/bin/activate

pip install -U numpy pandas matplotlib datetime ipykernel

python -m ipykernel install --user --name=my_env_kernel_name

The kernel will be written to your home folder, more precisely ~/.local/share/jupyter/kernels/ . This location will be monitored by jupyter, which should display your custom kernel as one of the kernel options.