License Management

Configuration

CCME provides a simple way to integrate license management with Slurm, and prevent jobs from starting if not enough licenses are available in the license server(s).

License management currently works with FlexLM license server.

To configure license management, you need to edit the CCME/custom/flexlm_licenses.yaml file (see below) or create a new file CCME/custom/ named CLUSTERNAME.flexlm_licenses.yaml (with CLUSTERNAME the name of the cluster you are about to launch): CCME will use the CLUSTERNAME.flexlm_licenses.yaml configuration file if it exists, else it will use flexlm_licenses.yaml. This allows to have different configurations for each CCME cluster that you launch. The license configuration file must contain the following information:

  • lmutil: path to the lmutil binary. lmutil binary is not included with CCME. You are required to upload it manually, or install it through a custom playbook (i.e., in CCME/custom) and update CCME/custom/flexlm_licenses.yaml with the location of your file in the lmutil variable.

  • servers: list of FlexLM servers that will be contacted to query licenses availability. Each of the servers must contain the following information:

    • myflexlm: name of the server

    • host: hostname or IP to contact FlexLM

    • port: listening port of FlexLM

    • licenses: list of licenses and total number of licenses available (as seen when querying FlexLM with lmutil -a)

Note

You do not need to install FlexLM server manager. Only lmutil binary is required.

Warning

lmutil and RHEL based distro

FlexLM may requires 32 bits libraries depending on your system. If launching lmutil returns an ELF version mismatch, simply install yum install redhat-lsb (or equivalent).

################################################################################
# Copyright (c) 2017-2024 UCit SAS
# All Rights Reserved
#
# This software is the confidential and proprietary information
# of UCit SAS ("Confidential Information").
# You shall not disclose such Confidential Information
# and shall use it only in accordance with the terms of
# the license agreement you entered into with UCit.
################################################################################
# This file can be customized to specify a list of Flexlm license servers that can be queried to
# monitor licenses.
# All licences described in this file will be added as remote licenses in Slurm
# (see https://slurm.schedmd.com/licenses.html), to be queried as job submission time.
# A script will then dynamically query the license servers to regularly update the number of available licences.
#
# IMPORTANT: lmutil is not provided by CCME, you must install it yourself (e.g., in the AMI, in a shared FS, with a custom playbook...)
#
# The format of the file is the following (all fields are mandatory)
# lmutil: PATH_TO_LMUTIL
# servers:
#   - SERVER1:
#     host: SERVER_HOST
#     port: SERVER_PORT
#     licenses:
#       LIC1: LIC1_COUNT
#       LIC2: LIC2_COUNT
#       LIC3: LIC3_COUNT
#       ...
#   - SERVER2:
#     ...
#
# Example:
# lmutil: /opt/flexlm/bin/lmutil
# servers:
#   - myflexlm:
#     host: myflexlmhost
#     port: 1055
#     licenses:
#       cfd_solve_level1: 100
#       cfd_solve_level2: 100
#       advanced_meshing: 20
#   - myflexlm2:
#     host: myflexlmhost2
#     port: 1056
#     licenses:
#       cfd_solve_level3: 200
#       cfd_solve_level4: 200
#       advanced_meshing2: 40

Note

You can update this file during a cluster update: newly added or removed licenses will be taken into account.

For each of the license declared in custom/flexlm_licenses.yaml CCME will automatically configure a corresponding license resource in Slurm with the following command (note that we specify servertype=flexlm-ccme to easily identify licenses managed by CCME):

sacctmgr add resource name=LIC cluster=CLUSTERNAME count=LIC_COUNT allowed=LIC_COUNT server=SERVER_PORT@SERVER_HOST servertype=flexlm-ccme type=license

You can check license configuration with the following commands:

$> sacctmgr show Resource  ServerType=flexlm-ccme
    Name     Server     Type  Count LastConsumed Allocated ServerType                Flags
---------- ---------- -------- ------ ------------ --------- ---------- --------------------
advanced_+ 1055@myfl+  License     20            0       100 flexlm-cc+
cfd_solve+ 1055@myfl+  License    100            0       100 flexlm-cc+
cfd_solve+ 1055@myfl+  License    100            0       100 flexlm-cc+
advanced_+ 1056@myfl+  License     40            0       100 flexlm-cc+
cfd_solve+ 1056@myfl+  License    200            0       100 flexlm-cc+
cfd_solve+ 1056@myfl+  License    200            0       100 flexlm-cc+

$> scontrol show lic
LicenseName=advanced_meshing2@1056@myflexlmhost2
    Total=40 Used=0 Free=40 Reserved=0 Remote=yes
    LastConsumed=0 LastDeficit=0 LastUpdate=2024-03-13T11:50:18
LicenseName=advanced_meshing@1055@myflexlmhost
    Total=20 Used=20 Free=0 Reserved=0 Remote=yes
    LastConsumed=0 LastDeficit=0 LastUpdate=2024-03-13T11:50:15
LicenseName=cfd_solve_level1@1055@myflexlmhost
    Total=100 Used=21 Free=79 Reserved=0 Remote=yes
    LastConsumed=0 LastDeficit=0 LastUpdate=2024-03-13T11:50:16
LicenseName=cfd_solve_level2@1055@myflexlmhost
    Total=100 Used=0 Free=100 Reserved=0 Remote=yes
    LastConsumed=0 LastDeficit=0 LastUpdate=2024-03-13T11:50:17
LicenseName=cfd_solve_level3@1056@myflexlmhost2
    Total=200 Used=0 Free=200 Reserved=0 Remote=yes
    LastConsumed=0 LastDeficit=0 LastUpdate=2024-03-13T11:50:19
LicenseName=cfd_solve_level4@1056@myflexlmhost2
    Total=200 Used=0 Free=200 Reserved=0 Remote=yes
    LastConsumed=0 LastDeficit=0 LastUpdate=2024-03-13T11:50:19

Also, CCME will automatically launch a daemon, managed by supervisord, that will query the license servers to gather the current available license counts, and update Slurm through the lastconsumed field of the license resource (see Slurm Dynamic Licenses). The logs of this daemon are located in /var/log/ccme.licenseMonitor-daemon.log

image_license_management

Submit a job with License Restrictions

Use -L <license>@<server_port>@<server_host>:<number_of_licenses> to submit a job with license restriction.

sbatch -L advanced_meshing2@1056@myflexlmhost2:5 test.sh

Slurm will ensures capacity will be provisioned only if there is at least 5 advanced_meshing2 licenses available on server 1056@myflexlmhost2. You can combine multiple license restrictions if needed:

sbatch -L advanced_meshing@1055@myflexlmhost:20,cfd_solve_level1@1055@myflexlmhost:10 test.sh

Example if you are using a job script:

#!/bin/bash
#SBATCH -J <jobname>
#SBATCH -p <partition>
#SBATCH -L advanced_meshing@1055@myflexlmhost:20,cfd_solve_level1@1055@myflexlmhost:10
/bin/sleep 600

Track license usage

Slurm is configured to track license usage through its accounting database. You can use sacct to retrieve information about license usage:

$> sacct -a --format=JobID,Licenses%80
JobID                                                                                Licenses
------------ --------------------------------------------------------------------------------
1
1.batch
2                                                        advanced_meshing2@1056@myflexlmhost2
2.batch
3                                                     advanced_meshing2@1056@myflexlmhost2:10
3.batch
4                cfd_solve_level1@1055@myflexlmhost:10,cfd_solve_level3@1056@myflexlmhost2:10
4.batch