Slurm quality of service (QoS)

Overview

A Quality of Service (QoS) in SLURM is a named set of scheduling parameters and limits that can be attached to jobs, users, or partitions. QoS policies allow administrators to control job priority, enforce resource limits, and define preemption behavior across different classes of workloads.

QoS Definition

Each QoS entry is managed through sacctmgr and carries a set of attributes, among which:

• Priority: An integer weight used to influence job scheduling order. Higher values result in higher scheduling priority when PriorityType=priority/multifactor is configured in slurm.conf.

• Flags: Behavioral modifiers. The DenyOnLimit flag causes a job to be rejected at submission time if it would exceed any limit defined on the QoS, rather than being held in the queue.

• Preempt: A comma-separated list of QoS names that jobs running under this QoS are allowed to preempt.

• PreemptExemptTime: A grace period (in seconds) during which a job is immune from preemption after it starts. Setting this to 0 means jobs can be preempted immediately.

Preemption

SLURM preemption allows higher-priority jobs to reclaim resources occupied by lower-priority jobs. The mechanism relies on two complementary concepts:

Preemptor QoS

A QoS with a higher Priority value and a Preempt= field listing the QoS names it is allowed to preempt. Jobs submitted under this QoS can displace running jobs that belong to the listed preemptible QoS.

Preemptible QoS

A QoS with a lower Priority value and PreemptExemptTime=0. Jobs submitted under this QoS can be interrupted as soon as a preemptor job requires their resources.

The preemption action is governed by PreemptMode in slurm.conf:

Mode	Behavior
CANCEL	The preempted job is canceled immediately.
REQUEUE	The preempted job is canceled and automatically requeued so it can restart later.
SUSPEND	The preempted job is suspended (paused in memory) and resumed once resources are available again.

Additional slurm.conf parameters that influence preemption:

• PreemptType=preempt/qos: Activates QoS-based preemption.

• PreemptParameters=strict_order: Ensures that preemption candidates are evaluated in strict priority order.

• PriorityWeightQOS: Weight applied to the QoS priority component when computing the overall job priority score.

• SchedulerParameters=preempt_youngest_first: Among eligible preemption candidates, targets the most recently started jobs first, minimizing wasted compute time.

Partition Configuration

For preemption to be active on a partition, the partition definition in slurm.conf must reference both QoS entries:

• QOS=<preemptor-qos>: Sets the default and enforced QoS for the partition.

• AllowQos=<preemptor-qos>,<preemptible-qos>: Explicitly lists every QoS that jobs submitted to this partition may use.

User Access

Users must be granted access to the relevant QoS entries in the SLURM accounting database before they can submit jobs under those QoS. Access is managed with sacctmgr modify user.

Example Playbook

Note

The playbook modifies /opt/slurm/etc/slurm.conf in place and automatically calls scontrol reconfigure via a handler when any configuration change is detected. Backup copies of the configuration file are created before each modification.

A ready-to-use Ansible playbook that automates the full configuration described above (QoS creation, slurm.conf updates, partition assignment, and user access grants) is provided at:

CCME/custom/example.install-qos-head.yaml

################################################################################
# Copyright (c) 2017-2026 UCit SAS
# All Rights Reserved
#
# This software is the confidential and proprietary information
# of UCit SAS ("Confidential Information").
# You shall not disclose such Confidential Information
# and shall use it only in accordance with the terms of
# the license agreement you entered into with UCit.
################################################################################
- name: Configure SLURM partition with QoS preemption
  hosts: localhost
  vars:
    partition_name: "hpc7a"
    qos_name: "hpc7a"
    qos_preemptable_name: "hpc7a-preemptable"
    preempt_mode: "CANCEL"  # REQUEUE, CANCEL, ou SUSPEND
  tasks:
    # Configure QoS
    - name: "Check if QoS exists for {{ qos_name }}"
      ansible.builtin.command: "sacctmgr show qos {{ qos_name }} -n"
      register: qos_check
      changed_when: false
      failed_when: false

    - name: "Create QoS for {{ qos_name }}"
      ansible.builtin.command: >
        sacctmgr -i add qos {{ qos_name }}
        Priority=100
        Flags=DenyOnLimit
      when: qos_check.stdout == ""
      register: qos_created
      changed_when: false
      failed_when: false

    - name: "Check if QoS exists for {{ qos_preemptable_name }}"
      ansible.builtin.command: "sacctmgr show qos {{ qos_preemptable_name }} -n"
      register: qos_preemptable_check
      changed_when: false
      failed_when: false

    - name: "Create QoS for {{ qos_preemptable_name }}"
      ansible.builtin.command: >
        sacctmgr -i add qos {{ qos_preemptable_name }}
        Priority=10
        Flags=DenyOnLimit
        PreemptExemptTime=0
      when: qos_preemptable_check.stdout == ""
      register: qos_preemptable_created
      changed_when: false
      failed_when: false

    - name: "Configure preemption for QoS {{ qos_name }}"
      ansible.builtin.command: "sacctmgr -i modify qos {{ qos_name }} set preempt={{ qos_preemptable_name }}"
      register: qos_config
      changed_when: "'Modified' in qos_config.stdout"
      failed_when: false

    - name: "Set PreemptExemptTime for {{ qos_preemptable_name }}"
      ansible.builtin.command: "sacctmgr -i modify qos {{ qos_preemptable_name }} set PreemptExemptTime=0"
      register: qos_preemptable_config
      changed_when: "'Modified' in qos_preemptable_config.stdout"
      failed_when: false

    # Configure SLURM
    - name: Ensure PreemptType is configured
      ansible.builtin.lineinfile:
        path: /opt/slurm/etc/slurm.conf
        regexp: '^PreemptType='
        line: "PreemptType=preempt/qos"
        backup: true
      notify: Reconfigure Slurm

    - name: Ensure PreemptMode is configured
      ansible.builtin.lineinfile:
        path: /opt/slurm/etc/slurm.conf
        regexp: '^PreemptMode='
        line: "PreemptMode={{ preempt_mode }}"
        backup: true
      notify: Reconfigure Slurm

    - name: Ensure PreemptParameters is configured
      ansible.builtin.lineinfile:
        path: /opt/slurm/etc/slurm.conf
        regexp: '^PreemptParameters='
        line: "PreemptParameters=strict_order"
        backup: true
      notify: Reconfigure Slurm

    - name: Ensure PriorityType is configured
      ansible.builtin.lineinfile:
        path: /opt/slurm/etc/slurm.conf
        regexp: '^PriorityType='
        line: "PriorityType=priority/multifactor"
        backup: true
      notify: Reconfigure Slurm

    - name: Ensure PriorityWeightQOS is configured
      ansible.builtin.lineinfile:
        path: /opt/slurm/etc/slurm.conf
        regexp: '^PriorityWeightQOS='
        line: "PriorityWeightQOS=1000"
        backup: true
      notify: Reconfigure Slurm

    - name: Ensure SchedulerParameters includes preempt_youngest_first
      ansible.builtin.lineinfile:
        path: /opt/slurm/etc/slurm.conf
        regexp: '^SchedulerParameters='
        line: "SchedulerParameters=preempt_youngest_first,bf_continue"
        backup: true
      notify: Reconfigure Slurm

    # Configure partition with both QoS
    - name: "Update QOS and AllowQos for partition {{ partition_name }}"
      ansible.builtin.replace:
        path: /opt/slurm/etc/slurm.conf
        regexp: '^(PartitionName={{ partition_name }}\s+(?:(?!QOS=|AllowQos=)\S+\s+)*)(QOS=\S+\s+)?(AllowQos=\S+\s+)?(.*?)$'
        replace: '\1QOS={{ qos_name }} AllowQos={{ qos_name }},{{ qos_preemptable_name }} \4'
        backup: true
      notify: Reconfigure Slurm

    # Grant QoS access to users
    - name: Grant QoS access to all users
      ansible.builtin.command: >
        sacctmgr -i modify user
        set qos={{ qos_name }},{{ qos_preemptable_name }}
      register: qos_access_result
      changed_when: "'Modified' in qos_access_result.stdout"

  handlers:
    - name: Reconfigure Slurm
      ansible.builtin.command: scontrol reconfigure
      changed_when: false
      failed_when: false

The playbook exposes the following variables at the top of the vars block:

Variable	Description
partition_name	Name of the SLURM partition to configure (default: hpc7a).
qos_name	Name of the preemptor QoS (default: hpc7a).
qos_preemptible_name	Name of the preemptible QoS (default: hpc7a-preemptible).
preempt_mode	Preemption action: CANCEL, REQUEUE, or SUSPEND (default: CANCEL).