Python venv
Using Python & Venv¶
If you have a different use case or need/want to use conda you can also use so called "virtual environments" to install python packages on ASC.
There are other tools out there that utilize virtual environments (e.g. poetry, virtualenv, ...) but since python already comes with the venv module we will focus on this already available module.
Pick your Python¶
To create a virtual environment first select a python version of your choice:
- EESSI/ASC python modules
- EESSI's compatibility layer's python
- OS python
- conda ships its own python
EESSI/ASC python modules¶
Use one of the modules if you need a rather recent python version:
$ module avail python
----- /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all -----
...
Python/3.11.3-GCCcore-12.3.0
Python/3.11.5-GCCcore-13.2.0 (D)
...
EESSI's compatibility layer's python:¶
At login, the EESSI module is loaded by default, and it provides the compatibility layer's python:
$ which python
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/python
$ python --version
Python 3.11.4
Use that if you only need a stable python version.
Operating system python¶
As part of the operating system (host system) some python packages are installed. The version is bound to change when the OS changes.
Warning
We do not recommend using the the OS python.
$ /bin/python3 --version
Python 3.9.18
conda¶
Miniforge provides conda and comes with its own python that can be upgraded to a new version.
Create a virtual env¶
After python is available we can then use the venv module to create a new virtual environment in the myvenv folder.
$ python -m venv --upgrade-deps myvenv
!!! Info Specifying --upgrade-deps is optional but ensures that the contained pip installer has the most recent version.
Activate the virtual env¶
To activate the created virtual environment simply use source to execute the generated activate script in the current bash environment.
$ source myvenv/bin/activate
(myvenv) $ which python
/home/fs12345/myuser/myvenv/bin/python
Checking the location of the used python binary shows that the python binary from the virtual environment is now used.
Install packages into the virtual env¶
Once the environment has been activated packages can be installed into it. The recommended way to do this is to create a requirements.txt file and then use pip to install all packages at once.
tensorflow[and-cuda]==2.18
The packages can then be installed using pip:
(myvenv) $ pip install -r requirements.txt
...
Verify your installation¶
Start python interactively:
(myenv) $ python -i
Python 3.11.5 (main, Aug 2 2024, 07:47:33) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
Verify that tensorflow can see the GPUs:
>>> import tensorflow as tf;
>>> print(tf.config.list_physical_devices('GPU'))
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'),
PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'),
PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'),
PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU'),]
Warning
Mind that this package is not optimised for the CPU microcode!
Submit to slurm using a python virtual environment¶
!!! Warning Make sure that you submit using a clean environment. Environment variables of active environments might get taken into account when submitting slurm jobs and may produce unwanted side effects!
Create a slurm batch script that load the virtual environment and then executes a python script
#!/bin/bash
#SBATCH --job-name=slurm_venv_example
#SBATCH --time=00-00:05:00
#SBATCH --ntasks=2
#SBATCH --mem=2GB
source ~/myvenv/bin/activate
echo "Using python: $( python --version ) from $( which python )"
# now run your program using python from the virtual environment
python my_program.py
Deactivate the environment¶
To deactivate a venv use the deactivate command
(myvenv) $ deactivate
$
Removing virtual environments¶
To delete the virtual environment simply remove the folder:
$ rm -rf myvenv/