Using Containers on MUSICA

There are three use cases possible with using Containers on MUSICA

1. Default available containers.
2. Building your own container.
3. Containers with EESSI modules/packages.
   1. Using SLURM

Default available containers.

This section will contain method to pull readily available containers and use it on MUSICA.

Apptainer will be available from the login node itself.

apptainer --version
apptainer version 1.3.5-1.el9

From here, usual apptainer commands can be used to pull and run the container.

(Note that this step assumes Nvidia's portal to download containers. For doing so, an account will be required to be created with an authenticated key for the download to work.)

apptainer pull tensorflow-gpu.sif docker://nvcr.io/nvidia/tensorflow:23.07-tf2-py3 WORKED BETTER 
apptainer pull lammps-gpu.sif docker://nvcr.io/hpc/lammps:patch_15Jun2023
apptainer pull pytorch-gpu.sif docker://nvcr.io/nvidia/pytorch:25.04-py3

Note that --writable command, required to add libraries/packages to the already built containers may cause permissions issue. If this occurs, please report this to us.

Once downloaded, the container can be used as usual.

A few more things to note here, is that few commands like nvidia-smi will work on host node, but may not work with the above pre-defined, pre-built containers. In this case, the --nv flag is crucial as it binds the NVIDIA drivers into the container. For e.g.

apptainer shell --containall --nv pytorch-gpu.sif

Building your container.

This section talks about using containers if one has a separate definition file, for a set of packages that are bundled to form a container.

Name a def file as follows, (add packages that you want in your container). Save it in the same file where you keep your .sif file. (say apptainer-demo-env.yaml)

name: funky_bonker
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.12
  - pymoo
  - gpy
  - numpy
  - matplotlib
  - scipy
  - pip

Save the following file as a .def file. (my_custom_container.def)

Bootstrap: docker
From: mambaorg/micromamba

%files
    ./apptainer-demo-env.yaml /opt/

%post
    apt-get update -y
    apt-get install -y cowsay lolcat
    apt-get clean

    micromamba env create -y -f /opt/apptainer-demo-env.yaml -n funky_bonker
    micromamba clean --all --yes
    echo "eval \"\$(micromamba shell hook --shell posix)\"" >> $APPTAINER_ENVIRONMENT
    echo "micromamba activate funky_bonker" >> $APPTAINER_ENVIRONMENT

%runscript
    eval "$(micromamba shell hook --shell posix)"
    micromamba activate funky_bonker
    python --version
    pip show GPy
    pip show pymoo
    date | cowsay | lolcat

run the apptainer command

apptainer build funky_bonker.sif my_custom_container.def

Once completed, a funky_bonker.sif file will appear in the same directory. The container can then be run as usual,

apptainer shell funky_bonker.sif

Containers with EESSI modules.

This section will construct containers using some packages that are outside of EESSI and some that are sourced from EESSI.

The following is available from EESSI repository, that allows to bind the existing/installed/modified EESSI repository to run within a container as follows,

git clone https://github.com/EESSI/software-layer.git

./software-layer/eessi_container.sh --nvidia run --host-injections /cvmfs/software.eessi.io/host_injections/ --extra-bind-paths /cvmfs/software.asc.ac.at

Once this runs, the container will be loaded with EESSI already loaded, and then all the modules within EESSI can be utilised, along with the ones installed specifically for MUSICA. This is made sure by the --host-injections flag.

Note that in this case the apptainer's own --bind is replaced by --extra-bind-paths flag. This now allows to bind a directory to the container, and can be repeated several times for different directories.

This also is important when a user has a built an application locally and is needed to be used with EESSI.

By default, loading EESSI-extend will configure the variable EESSI_USER_INSTALL to point to $HOME/eessi. If you want to load modules from this directory, it should be included as an extra bind path to the container.

./software-layer/eessi_container.sh --nvidia run --host-injections /cvmfs/software.eessi.io/host_injections/ --extra-bind-paths $HOME/eessi

Note that in this case the apptainer's own --bind is replaced by --extra-bind-paths flag. This now allows to bind a directory to the container, and can be repeated several times for different directories.

Using containers with SLURM

Containers can also be used with SLURM, as apptainers acts as an exeucatble for SLURM and can be used in a similar fashion,

See for example,

srun -N1 -p zen4_0768_h100x4 --gres gpu:4 --pty apptainer shell --nv 02pytorch-gpu.sif

Or equivalently for a container with EESSI,

srun -N1 -p zen4_0768_h100x4 --gres gpu:4 --pty \
./software-layer/eessi_container.sh --nvidia run \
--host-injections /cvmfs/software.eessi.io/host_injections/ \
--extra-bind-paths /cvmfs/software.asc.ac.at/

In this way, multiple GPU nodes can be used in tandem with the containers using the features of EESSI, along with a custom build tool (if any).

The above can naturally be also done via a slurm script.