Using Python Via Conda

Conda & Anaconda & Mamba

We focus on conda via miniforge (has micromamba):

• conda -> miniforge package manager + community repository.

• ~~anaconda is a proprietary (unfree!) summary of packages.~~

• mamba -> micromamba faster sister of conda, included.

Do not use Anaconda or Conda repositories, or get sued!

Start Conda

First load the miniforge package with module:

$ module load Miniforge3

Execute the conda bash hook to fully setup the conda tool:

$ eval "$(conda shell.bash hook)"

Note how the prompt now shows the base env:

(base) $ conda
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...

conda is a tool for managing and deploying applications,
environments and packages.
...

!!! note The ASC's base env is write protected since it is the conda tools environment for it to run. In general conda's base env should never be used for user installations.

Define Your Environment

Write a yaml file defining your environment.

name: myenv
channels:
  - conda-forge
  - defaults
dependencies:
  - tensorflow=2.18

Use e.g. nano myenv.yaml to write this file.

Create Your Environment

Create your conda env:

(base) $ conda env create --file myenv.yaml
Channels:
 - conda-forge
 - defaults
 - bioconda
Platform: linux-64
Collecting package metadata (repodata.json): /
...

Depending of the number of packages this might take some time (a couple of minutes).

Warning

Conda creates many small files and might fill up your $HOME file system quickly. You might want to store your environments in your project storage. Use the --prefix <directory> argument with conda env create to use a different directory.

Activate Your Environment

Activate your conda env, note how the prompt changes to (myenv):

(base) $ conda activate myenv
...
(myenv) $ which python
~/.conda/envs/myenv/bin/python

(myenv) $ python --version
Python 3.12.11

The python version shown is the installation from conda.

Note

Instead of using the environment's name you can also specify a path.

Caution: Add A Package With Pip

If you have to use pip, specify the pip packages in the conda env file:

name: myenv
channels:
  - conda-forge
  - defaults
dependencies:
  - tensorflow=2.18
  - pip
  - pip:
    - memory-profiler

Warning: Do not install a package with pip into the activated env!

Warning

In general the manual installation of additional packages using pip should be avoided since pip packages have their own dependencies and can even pull in unwanted binaries. It is preferrable to either use a corresponding conda package or at least specify the pip packages directly in the conda env file.

Manually installing pip packages

When installing a package using pip, make sure that you have an active conda environment (e.g. myenv). If you do not have an active conda environment, the packages will be installed into the default folder of your user (~/.local/python) and could interfere with other python environments.

(myenv) $ python -m pip install mypackage

Use python -m pip -v to display the installed packages and their installations paths to make sure they end up in the right place.

Verify Your Installation

Start python interactively:

$ python -i
Python 3.12.11 | packaged by conda-forge ...
Type "help", "copyright", "credits" or "license" ...
>>>

Verify that tensorflow can see the GPUs:

>>> import tensorflow as tf;
>>> print(tf.config.list_physical_devices('GPU'))
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'),
PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'),
PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'),
PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU'),]

Warning

Mind that this package is not optimised for the CPU microcode!

Search Installed Packages

Search for a package like numpy in the current env:

(myenv) $ conda list | grep numpy
numpy   2.3.0   py312h6cf2f7f_0   conda-forge

Or search for any package that has to do with blas:

(myenv) $ conda list | grep blas
libblas       3.9.0      31_h59b9bed_openblas   conda-forge
libcblas      3.9.0      31_he106b2a_openblas   conda-forge
libcublas     12.9.1.4   h9ab20c4_0             conda-forge
liblapack     3.9.0      31_h7ac8fdf_openblas   conda-forge
libopenblas   0.3.29     pthreads_h94d23a6_0    conda-forge

Submit Your Conda Env To Slurm

Create a slurm batch script starting your conda env and running python:

#!/bin/bash
#SBATCH --job-name=slurm_conda_example
#SBATCH --time=00-00:05:00
#SBATCH --ntasks=2
#SBATCH --mem=2GB

module load Miniforge3
eval "$(conda shell.bash hook)"
conda activate myenv

echo "Using python: $( python --version ) from $( which python )"

# now run your program using python from the conda environment:
python my_program.py

Use e.g. nano my_batch_script.sh to write this file.

List All Environments

(base) $ conda env list
# conda environments:
#
myenv     /home/trainee00/.conda/envs/myenv
base   *  /opt/sw/conda
py310     /opt/sw/conda/envs/py310
py311     /opt/sw/conda/envs/py311
py312     /opt/sw/conda/envs/py312
py38      /opt/sw/conda/envs/py38
py39      /opt/sw/conda/envs/py39

The star * shows the currently active environment.

Deactivate Your Environment

Deactivate your conda environment to get back to (base):

(myenv) $ conda deactivate
(base) $ conda deactivate
$

Type conda deactivate again to get out of conda completely.

Remove Your Environment

If you don't need them anymore please remove any old environments:

(base) $ conda env remove -n myenv

Remove all packages in environment /home/fs70824/trainee00/.conda/env
s/myenv:

Everything found within the environment (/home/fs70824/trainee00/.con
da/envs/myenv), including any conda environment configurations and an
y non-conda files, will be deleted. Do you wish to continue?
 (y/[n])?