MUSICA user test phase¶

EESSI_logo

For MUSICA's test user phase the software stack will be provided via The European Environment for Scientific Software Installations (EESSI).

EESSI (pronounced as "easy") is a collaboration between different HPC sites and industry partners, it acts like a streaming service for scientific software, regardless of which flavor/version of Linux distribution or processor architecture is used, or whether it is a full-size HPC cluster, a cloud environment or a personal workstation, and without compromising on the performance of the software.

The goal of EESSI is to set up a shared stack of scientific software installations, and by doing so avoid a lot of duplicate work across HPC sites, providing for end users a uniform user experience with respect to available scientific software, regardless of which system they use.

EESSI on MUSICA¶

The basics¶

Starting EESSI¶

EESSI is automatically initialised on login, at the moment this is set to load a specific release. If you ssh into a node that is configured to load EESSI automatically, you will likely be greeted with a line of text that contains info about what version is currently loaded. If you don't see anything, don't panic! You can check what versions of EESSI are currently available on the system by running ml avail. If this doesn't work, you may need to first ensure Lmod is set up. This is relatively straight forward, as EESSI bundles a version of Lmod with each releases. For example, to set up the environment with the Lmod released from EESSI/2023.06, run:

source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash

If this succeeds, you should be able to see what EESSI releases can be loaded with ml avail. For a more detailed introduction to how the environment is set up to run EESSI, please refer to the online documentation.

Stopping EESSI¶

As EESSI gets loaded as a module, module purge will remove it from the environment. This means that it needs to be manually reloaded after a purge. To get an overview of what EESSI versions are available on the system, run

ml purge
ml avail

The load the specific version that you want, e.g.:

ml load EESSI/2023.06

Getting an overview¶

It would be a shame to introduce EESSI and not mention anything about it's clever design. The project itself consists of three layers. Each of the sections below highlights a few aspects of each layer, and how the ASC software stack is set up to follow a similar pattern. First, let's get an overview of the available modules, this can be achieved by typing ml ov. The output should look something like the following:

------------------------------------------------------ /cvmfs/software.asc.ac.at/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all ------------------------------------------------------
CUDA  (3)   GCCcore (1)   gomkl (1)   LLVM  (1)   Miniforge3 (1)   numactl (1)   psutil       (1)   STAR-CCM+ (1)   Z3 (1)
cuDNN (1)   Go      (1)   imkl  (1)   magma (1)   neovim     (1)   NVHPC   (1)   SciPy-bundle (1)   VASP      (1)

--------------------------------------------------- /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen4/modules/all ---------------------------------------------------
CUDA (2)   Go (1)

--------------------------------------------- /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all ----------------------------------------------
CUDA-Samples (1)   CUDA (2)   NCCL (2)   OSU-Micro-Benchmarks (2)   UCC-CUDA (2)   UCX-CUDA (2)

------------------------------------------------------ /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all -------------------------------------------------------
Abseil          (2)   dlb                   (1)   graphite2             (2)   libfabric        (2)   mpi4py               (2)   protobuf-python       (1)   snakemake         (1)
ALL             (1)   double-conversion     (2)   groff                 (1)   libfdf           (1)   mpl-ascii            (1)   protobuf              (1)   snappy            (2)
...

Note that Software (N) is the number of software installations present (e.g., CUDA (3) means there are three versions of CUDA available).

The filesystem layer¶

The (truncated) output above is organized into four groups, the heading of each group displays the Lmod path where the modules are being found. If a path begins with /cvmfs, this signifies that the software is being stored on CernVM File System (CVMFS). This type of file system allows the software to be distributed across sites easily by leveraging web protocols. It will also always show up as a read-only file system on the clients. In CVMFS, the term 'repositories' or 'repos' refers to collections of software organized for distribution. The multi-site nature of MUSICA meant that CVMFS was a compelling choice for distributing the ASC software stack across the three sites. The software.eessi.io repository is the publicly available software stack provided by the EESSI project. Due to it's open source nature, there are some restrictions in place around what types of software can be distributed in the EESSI repo. For example, some licenses prohibit the public distribution of software. This is one of the primary motivations for providing a second (non-public) software stack - the software.asc.ac.at repo provides another stack, built on top of EESSI and made available to ASC users through CVMFS infrastructure.

For more information on the filesystem layer, see the docs.

The compatibility layer¶

EESSI defines a minimal filesystem hierarchy that includes a few core libraries and tools. This is then used as the basis for all other software built and released by the project. The idea behind the compatibility layer is to keep interactions with the host system to an absolute minimum, thus ensuring better portability of the software stack across operating systems. For the ASC software stack, the approach is to leverage the compatibility layer provided by EESSI. Of course, there are a few low level libraries that must come from the host, typically things like graphics or networking drivers and Slurm binaries. EESSI provides for these additions by way of a special variant symlink called host_injections. This will typically point to a location on the shared storage for a specific site. It's also possible to use this as a location for installing site specific software. More on that in the section on installing software.

For more information on the compatibility layer, see the docs.

The software layer¶

The idea of EESSI's software layer is to provide a collection of scientific software installations. The software is installed using EasyBuild. Environment modules files are provided and manged using Lmod. EESSI builds and distributes software for a wide variety of architectures, but only the appropriate architecture for the current system will ever appear in the module search results. This is made possible by using the archspec library to detect information about the host system.

For more information on the compatibility layer, see the docs.

Examples¶

Environment modules¶

As mentioned above, EESSI uses Lmod. The basic commands should be quite familiar to users of other module tools:

Use module avail to find installations of a package.
Use module load to load the module.
Use module list to list all the loaded modules.
Use module purge to remove all the loaded modules.
Use module show to inspect the module.

Slurm job script¶

Load the exact EESSI module:

#!/bin/bash
#SBATCH --job-name test_job
#SBATCH --nodes 1
#SBATCH --partition zen4_0768_h100x4
#SBATCH --qos zen4_0768_h100x4

module load LAMMPS/29Aug2024-foss-2023b-kokkos

srun --mpi=pmix lmp -i in.meam

Building software¶

There are many ways to build your own application on MUSICA, see the installing software for more info.