MUSICA partitions and QoS¶

On MUSICA, nodes of the same type of hardware are grouped into partitions. The quality of service (QoS), defines the maximum run time of a job and the number and type of allocatable nodes.

Partitions¶

AMD CPU nodes: The CPU nodes are comprised of 2x AMD Epyc 9654 (96 cores each, Zen4 architecture) with 768GB DDR5 RAM and a local NVMe with 1.92TB.
GPU Hopper nodes: These GPU nodes have 2x AMD Epyc 9654 (96 cores each, Zen4 architecture) with 768GB DDR5 RAM, 4x Nvidia SXM5 H100-94G-700W GPUs, 4x NDR200 infiniband and a local NVMe with 7.68TB.
GPU Blackwell nodes: These GPU nodes have 2x AMD Epyc 9655 (96 cores each, Zen5 architecture) with 2304GB DDR5 RAM, 8x Nvidia B200 GPUs, 4x NDR400 infiniband and a local NVMe with 6.9TB. They are installed only at Vienna site and privately owned. Generally available when they are not used by the owning organisation.

These are the partitions on MUSICA:

Partition	Nodes	Architecture	CPU	Cores per CPU (physical/with HT)	GPU	RAM	Use
zen4_0768	72	AMD	2x AMD 9654	192/384	No	768 GB	The default partition
zen4_0768_h100x4	112	AMD	2x AMD 9654	192/384	4x Nvidia SXM5 H100	768 GB	GPU partition
zen5_2304_b200x8	28	AMD	2x AMD 9655	192/384	8x Nvidia B200	2304GB	private partition, partially available when not in use

Tip

Type sinfo -o %P on any node to see all the available partitions.

andsacctmgr show user $( id -u ) withassoc format=qos%200 for the available QoS.

Quality of service (QoS)¶

The QoS defines the maximum run time of a job and the number and type of allocate-able nodes.

The following QoS are available for all normal (=non private) projects:

QoS name	Gives access to Partition	Hard run time limits	Description
zen4_0768	zen4_0768	72h (3 days)	Default
zen4_0768_h100x4	zen4_0768_h100x4	72h (3 days)	H100 GPUs
zen5_2304_b200x8	zen5_2304_b200x8	72h (3 days)	B200 GPUs (Vienna only)
ista_zen5_2304_b200x8	zen5_2304_b200x8	240h (10 days)	private qos for ISTA

Devel QoS¶

The devel QoS gives fast feedback to the user when their job is running. We recommend this before sending the job to one of the compute queues.

QoS name	Gives access to Partition	Hard run time limits	Description
dev_zen4_0768	2 nodes on zen4_0768	10min	for testing
dev_zen4_0768_h100x4	2 nodes on zen4_0768_h100x4	10min	for testing

Tip

The QoS that are assigned to a specific user can be viewed with:

sacctmgr show user `id -u` withassoc format=user,defaultaccount,account,qos%40s,defaultqos%20s

To simplify the process, you can use the ASC internal tool sqos to print available quality of service (QoS) or QoS usage.

With sfree there is a second tool to get a detailed insight into the QoS usage.

Example for sfree on MUSICA-INN

$ sfree

── zen4_0768 (cpus)───────────────────────────────────-[████████████████████ n 100% r  94% ]────────
  cpu partition · 192 phys cores/node
  nodes       47 total       5 down      42 up         36 allocated       6 mixed       0 idle
  cores    18048 total    1920 down   16128 avail   15136 used          992 free

  qos                       load      limit      used    free      ~nodes  prio        wall
  ─────────────────────────────────────────────────────────────────────────────────────────
  admin_musica_inn          ░░░░░░░░  (ungated)  0       -         -       100000000   10-00:00:00
  dev_zen4_0768             ░░░░░░░░  cpu=1920   0       1920      ~5      10000000    00:10:00
  zen4_0768                 ███████░  cpu=18432  15136   3296      ~8      100000      3-00:00:00

── zen4_0768_h100x4 (gpus)────────────────────────────-[██████░░░░░░░░░░░░░░ n  29% r  25% ]────────
  gpu partition · 4x h100/node · 192 phys cores/node
  nodes     79 total    10 down    69 up       11 allocated     9 mixed    49 idle
  gpus     316 total    40 down   276 avail    70 used        206 free

  qos                       load      limit      used    free      ~nodes  prio        wall
  ─────────────────────────────────────────────────────────────────────────────────────────
  admin_musica_inn          ░░░░░░░░  (ungated)  0       -         -       100000000   10-00:00:00
  dev_zen4_0768_h100x4      ░░░░░░░░  gpu=20     0       20        ~5      10000000    00:10:00
  zen4_0768_h100x4          ██░░░░░░  gpu=300    62      238       ~59     100000      3-00:00:00

 * "idle" nodes have nothing allocated; "~nodes" is the qos-limit-to-node estimate (not a hardware count)
 * "n" = busy nodes/up, "r" = used <cores|gpus>/avail; per-qos "load" = used/limit