Cluster

Configuration

Global configuration

Add a configuration to the pcluster config file in /home/$(whoami)/.parallelcluster/config or edit /home/$(whoami)/.parallelcluster/simple for a simple cluster. In order to launch a cluster, you first need to create a configuration file for the cluster. You can either create your own, or edit one of examples provided in /home/$(whoami)/.parallelcluster/ on the Management Host. We provide below an example, in which you must replace each yaml variable (e.g., {{ CCME_VARIABLE }}) by the required information. Make sure you use the correct subnets, security groups and policies.

The HeadNode.CustomActions.OnNodeStart.Args section of the ParallelCluster configuration file can contain a set of parameters that will influence the configuration of your cluster. Here is the list of supported parameters:

  • OS related parameters

    • CCME_NO_PROXY (optional): string containing a list of hosts to exclude traffic destined to certain hosts from using the proxy. It is used to set the no_proxy and NO_PROXY environment variables: content of CCME_NO_PROXY is appended to the content of those 2 variables. This is usually used in conjunction with the Proxy option in the AWS ParallelCluster configuration file.

  • S3 related parameters

    • CCME_S3FS (optional): list of S3 buckets to mount through S3FS (the policies attached to the HeadNode and Computes need to have read/write access to these buckets). Format is CSV: CCME_S3FS=bucket1,bucket2,bucket3 (or simply CCME_S3FS=bucket if there is a single bucket). If this variable is unset or equal to NONE, then no bucket is mounted through S3FS.

    • CCME_JSLOGS_BUCKET (mandatory): name of an S3 bucket on which Slurm accounting logs will be exported (see Slurm accounting logs, the policies attached to the HeadNode and Computes need to have read/write access to this bucket).

  • ALB related parameters

    • CCME_OIDC: a prefix used to locate the CCME/conf/${CCME_OIDC}.oidc.yaml file that contains configurations for OIDC authentication (see OIDC External Authentication).

    • CCME_DNS: the DNS name of the ALB (if you are not using the default ALB DNS name), e.g., my.domain.com. The cluster url is alb_url/my_cluster_name/portal/ for portal and alb_url/my_cluster_name/dcv-instance-id as default value. That default url is the the application load balancer and can be replaced by a custom Domain Name Server. If you want to configure your DNS allowing your personal.domain.com pointing to the Application Load Balancer in order to have replace your url and have a personal.domain.com/my_cluster/portal/ and personal.domain.com/my_cluster/visualization url for your cluster web access, then set the CCME_DNS to personal.domain.com instead of NONE.

  • User related parameters

    • CCME_USER_HOME: This variable allows you to use a path other than /home for the user’s home directory. You can use the %u parameter to retrieve the username (e.g., /file-system/home/%u). In case of a file system created with the cluster by parallelcluster, you must set the file system path mountpoint to /home instead of using CCME_USER_HOME

  • Remote visualization related parameters (see Prerequisites and configuration for a complete description of these parameters):

    • CCME_WIN_LAUNCH_TEMPLATE_ID: Launch template used to launch Windows EC2 instances. CCME creates a default launch template when deploying the CMH, but you can setup your own here.

    • CCME_WIN_AMI: ID of the AMI used to launch Windows EC2 instances (see Prerequisites and configuration for prerequisites).

    • CCME_WIN_INSTANCE_TYPE: Instances type used to launch Windows EC2 instances.

    • CCME_WIN_INACTIVE_SESSION_TIME, CCME_WIN_NO_SESSION_TIME and CCME_WIN_NO_BROKER_COMMUNICATION_TIME: parameters to control the lifecycle of Windows remote visualization sessions.

    • CCME_WIN_TAGS: Dictionary of additional tags to apply on the instances of the Windows fleet (see Starting a Windows DCV session for the list of default tags).

  • EnginFrame related parameters

    • CCME_EF_ADMIN_GROUP: The name of an OS group, users belonging to this group will automatically be promoted as administrators in EnginFrame (no sudo access).

    • CCME_EF_ADMIN_PASSWORD: ARN of a secret containing the password of EnginFrame admin account. The expected value in the preexisting ARN of a plaintext string stored in AWS Secrets Manager (ASM). Do not set this variable to let CCME generate and store a password in /shared/CCME/ccme.passwords.efadmin

    • CCME_EF_ADMIN_SUDOER: if true, then the efadmin user is sudoer. false is the default value.

    • CCME_EFADMIN_ID: The uid/gid to create the efadmin user locally

    • CCME_EFNOBODY_ID: The uid/gid to create the efnobody user locally

  • Remote access related parameters

    • CCME_AWS_SSM: if set to true, AWS SSM agent will be installed on all the nodes, to allow remote connection to them with AWS SSM.

  • Slurm related parameters

    • CCME_CUSTOM_SLURMDBD_SETTINGS: dictionary of specific options to add to SlurmDBD configuration See slurmdbd configuration for possible values. The format must be a valid “YAML dictionary embedded in a string”. Hence, the whole line must be enclosed in double quotes, and then the value of CCME_CUSTOM_SLURMDBD_SETTINGS must be the dict enclosed in escaped double quotes. See the following example: "CCME_CUSTOM_SLURMDBD_SETTINGS=\"{'PrivateData': 'jobs,events,accounts,reservations,usage,users', 'PurgeEventAfter': '12'}\""

  • Notifications related parameters

    • CCME_ADMIN_PHONE: can be set to a valid mobile phone number (in E. 164 format) to deliver information about the cluster when it is ready to be used. As described by the ITU, the E.164 general format must contain only digits split as follows:

      • ‘+’ sign

      • Country code (max 3 digits)

      • Subscriber number (max 12 digits)

      WARNING: This feature is restricted by an “account spend limit” that prevents you to spend more than a given amount of money on SMS sendings. See this documentation.

    • CCME_ADMIN_SNS_TOPIC_ARN: can be set to a valid SNS topic ARN you want to deliver information about the cluster when it is ready to be used. For example, you can configure your SNS topic to deliver the information by email to the administrators of the platform.

Note

No parameter must be set on the following sections, as they will be inherited from the HeadNode.CustomActions.OnNodeStart.Args parameters:

  • HeadNode.CustomActions.OnNodeConfigured:.Args

  • HeadNode.CustomActions.OnNodeUpdated:.Args

  • HeadNode.CustomActions.OnNodeStart.Args

  • HeadNode.CustomActions.OnNodeConfigured.Args

Note

The CCME solution is downloaded from S3 on the HeadNode. Then the download directory is mounted on /opt/CCME on each ComputeNodes from the HeadNode using NFS.

CCME applies its configurations and installs software on ParallelCluster clusters through a set of Bash and Ansible scripts. The entry points are the Bash scripts specified in the OnNodeStart, OnNodeConfigured and OnNodeUpdated parameters of the HeadNode.CustomActions and Scheduling.SlurmQueues[].CustomActions sections. The values presented below (and present in the generated example configuration files) must always be present.

Example ParallelCluster configuration file for CCME
  1Region: '{{ AWS_REGION }}'
  2CustomS3Bucket: '{{ CCME_CLUSTER_S3BUCKET }}'
  3Iam:
  4  Roles:
  5    LambdaFunctionsRole: '{{ CCME_CLUSTER_LAMBDA_ROLE }}'
  6  # If the role associated to the cluster includes a custom IAM path prefix,
  7  # replace "parallelcluster" by the custom IAM path prefix.
  8  ResourcePrefix: "parallelcluster"
  9Image:
 10  Os: {{ "alinux2" or "centos7" or "rhel8" or "rocky8" }}
 11  # CustomAmi: ami-id
 12Tags:
 13  - Key: Owner
 14    Value: '{{ CCME_OWNER }}'
 15  - Key: Reason
 16    Value: '{{ CCME_REASON }}'
 17SharedStorage:
 18  - Name: shared
 19    StorageType: Ebs
 20    MountDir: shared
 21HeadNode:
 22  InstanceType: t3.medium
 23  Networking:
 24    SubnetId: '{{ CCME_SUBNET }}'
 25    SecurityGroups:
 26      - '{{ CCME_PRIVATE_SG }}'
 27  Ssh:
 28    KeyName: '{{ AWS_KEYNAME }}'
 29  LocalStorage:
 30    RootVolume:
 31      Size: 50
 32      Encrypted: true
 33  CustomActions:
 34    OnNodeStart:
 35      Script: s3://{{ CCME_SOURCES }}CCME/sbin/pre-install.sh
 36      Args:
 37        - CCME_CMH_NAME={{ CCME_CMH_NAME }}
 38        - CCME_S3FS={{ CCME_DATA_BUCKET }}
 39        - CCME_JSLOGS_BUCKET={{ CCME_DATA_BUCKET }}
 40        - CCME_AWS_SSM=true
 41        - CCME_OIDC=default
 42        - CCME_USER_HOME=/file-system/home/%u
 43        - CCME_DNS="my.domain.com"
 44        - CCME_REPOSITORY_PIP="https://my.pip.domain.com/index/,https://my.pip.domain.com/index-url/"
 45        # Optional windows fleet
 46        - CCME_WIN_AMI="ami-i..."
 47        - CCME_WIN_INSTANCE_TYPE=NONE
 48        - CCME_WIN_INACTIVE_SESSION_TIME=600
 49        - CCME_WIN_NO_SESSION_TIME=600
 50        - CCME_WIN_NO_BROKER_COMMUNICATION_TIME=600
 51        - CCME_WIN_CUSTOM_CONF_REBOOT=true
 52        - CCME_WIN_LAUNCH_TRIGGER_DELAY=10
 53        - CCME_WIN_LAUNCH_TRIGGER_MAX_ROUNDS=100
 54        ## CCME_WIN_TAGS allows to add specific tags to instances of the Windows fleet
 55        # The format must be a valid "YAML dictionary embedded in a string".
 56        # Hence, the whole line must be enclosed in double quotes, and then the value
 57        # of CCME_WIN_TAGS must be the dict enclosed in escaped double quotes. See the following example:
 58        - "CCME_WIN_TAGS=\"{'MyTagKey1': 'MyTagValue1', 'MyTagKey2': 'MyTagValue2'}\""
 59        - CCME_EF_ADMIN_PASSWORD="arn:aws:secretsmanager:eu-west-1:012345678910:secret:ccme-prefix-efadmin.password-4riso"
 60        - CCME_EF_ADMIN_GROUP="Administrators"
 61        # CCME_EF_ADMIN_SUDOER defines if efadmin has a sudo role
 62        # Required: No
 63        # Patterns: true or false
 64        # CCME_EF_ADMIN_SUDOER=false
 65        # Specify EFADMIN or EFNOBODY UID/GID
 66        # - CCME_EFADMIN_ID=
 67        # - CCME_EFNOBODY_ID=
 68        ## CCME_CUSTOM_SLURMDBD_SETTINGS allows to add specific options to SlurmDBD
 69        # See https://slurm.schedmd.com/slurmdbd.conf.html for possible values
 70        # The format must be a valid "YAML/JSON dictionary embedded in a string".
 71        # Hence, the whole line must be enclosed in double quotes, and then the value
 72        # of CCME_CUSTOM_SLURMDBD_SETTINGS must be the dict enclosed in escaped double quotes.
 73        # Note: if you set PrivateData here, you must set it as well in Scheduling.SlurmSettings.CustomSlurmSettings
 74        # See the following example:
 75        - "CCME_CUSTOM_SLURMDBD_SETTINGS=\"{'PrivateData': 'jobs,events,accounts,reservations,usage,users', 'PurgeEventAfter': '12'}\""
 76    OnNodeConfigured:
 77      Script: s3://{{ CCME_SOURCES }}CCME/sbin/post-install.sh
 78    OnNodeUpdated:
 79      Script: s3://{{ CCME_SOURCES }}CCME/sbin/update-install.sh
 80  Iam:
 81    InstanceProfile: '{{ CCME_CLUSTER_HEADNODE_INSTANCE_PROFILE }}'
 82Scheduling:
 83  Scheduler: slurm
 84  SlurmSettings:
 85    Dns:
 86      # If the role associated to the cluster is not authorized to use Route 53,
 87      # or if you don't want to use Route 53,
 88      # set "DisableManagedDns" to true and "UseEc2Hostnames" to true
 89      DisableManagedDns: false
 90      UseEc2Hostnames: false
 91    CustomSlurmSettings:
 92      # If you set PrivateData in CCME_CUSTOM_SLURMDBD_SETTINGS, you must set it here as well.
 93      # PrivateData must be present in both slurmdbd.conf and slurm.conf.
 94      # Note the addition of the "cloud" value as well.
 95      - PrivateData: "jobs,events,accounts,reservations,usage,users,cloud"
 96  SlurmQueues:
 97    - Name: basic-slurm
 98      CapacityType: ONDEMAND
 99      ComputeSettings:
100        LocalStorage:
101          RootVolume:
102            Size: 50
103            Encrypted: true
104      ComputeResources:
105        - Name: t2-small
106          InstanceType: t2.small
107          MinCount: 0
108          MaxCount: 2
109      CustomActions:
110        OnNodeStart:
111          Script: s3://{{ CCME_SOURCES }}CCME/sbin/pre-install.sh
112        OnNodeConfigured:
113          Script: s3://{{ CCME_SOURCES }}CCME/sbin/post-install.sh
114      Iam:
115        InstanceProfile: '{{ CCME_CLUSTER_COMPUTE_INSTANCE_PROFILE }}'
116      Networking:
117        SubnetIds:
118          - '{{ CCME_SUBNET }}'
119        SecurityGroups:
120          - '{{ CCME_COMPUTE_SG }}'
121    - Name: dcv-basic
122      CapacityType: ONDEMAND
123      ComputeResources:
124        - Name: t3-medium
125          InstanceType: t3.medium
126          MinCount: 0
127          MaxCount: 2
128      CustomActions:
129        OnNodeStart:
130          Script: s3://{{ CCME_SOURCES }}CCME/sbin/pre-install.sh
131        OnNodeConfigured:
132          Script: s3://{{ CCME_SOURCES }}CCME/sbin/post-install.sh
133      Iam:
134        InstanceProfile: '{{ CCME_CLUSTER_COMPUTE_INSTANCE_PROFILE }}'
135      Networking:
136        SubnetIds:
137          - '{{ CCME_SUBNET }}'
138        SecurityGroups:
139          - '{{ CCME_COMPUTE_SG }}'
140DirectoryService:
141  DomainName: {{ CCME_AD_DIR_NAME }}
142  DomainAddr: ldap://{{ CCME_AD_IP_1 }},ldap://{{ CCME_AD_IP_2 }}
143  PasswordSecretArn: {{ CCME_AD_PASSWORD }}
144  DomainReadOnlyUser: cn={{ CCME_AD_READ_ONLY_USER }},ou=Users,ou={{ CCME_AD_DIR_NAME_DC1 }},dc={{ CCME_AD_DIR_NAME_DC1 }},dc={{ CCME_AD_DIR_NAME_DC2 }}
145  LdapTlsReqCert: never # Set "hard" to enable ldaps
146  # LdapTlsCaCert: /opt/CCME/conf/{{ CCME_CA_FILE }} # Set it only with ldaps
147  # LdapAccessFilter:
148  AdditionalSssdConfigs:
149    # debug_level: "0x1ff" # Uncomment for logs, can be heavy
150    ldap_auth_disable_tls_never_use_in_production: True # Don't set it with ldaps

Note

If in the configuration of your cluster you want to use an external resource such as an EFS or FSx for NetApp file system that hasn’t been deployed by CCME, you will need to ensure that the targeted resource has at least one tag which name starts with ccme. For security reasons, CCME roles only allow CCME to describe and use services that have such ccme* tags.

We recommend that you use for example an explicit tag named such as: ccme:allow (the value is not important, but for readability reasons use a value of true for example).

Without such a tag, you will get an error message when trying to launch a cluster, for example when trying to connect an FSx for NetApp ONTAP without a ccme* tag, you can get an error like:

"message": "Invalid cluster configuration: User:
arn:aws:sts::123456789012:assumed-role/CRS-myrole-ParallelClusterUserRole-10T744D833QZH/i-0423d0720df91381b
is not authorized to perform: fsx:DescribeVolumes on resource: arn:aws:fsx:eu-west-3:123456789012:volume/*/*
because no identity-based policy allows the fsx:DescribeVolumes action"

Custom Scripts

On top of CCME specific configurations, you can integrate your own custom scripts to CCME. To deploy a cluster embedding and executing your own custom scripts, you must place them in the CCME/custom directory and synchronize this directory in the S3 bucket. You can provide your own Ansible playbooks or Bash scripts to add specific configurations to the HeadNode, or to the Compute Nodes, or to all nodes. Ansible playbooks and Bash scripts are executed in this order:

  1. install-\*-all.yaml: run Ansible playbook on all nodes (Head and Compute nodes)

  2. install-\*-head.yaml: run Ansible playbook on Head Node only

  3. install-\*-compute.yaml: run Ansible playbook on Compute Nodes only

  4. install-\*-all.sh: run Bash script on all nodes (Head and Compute nodes)

  5. install-\*-head.sh: run Bash script on Head Node only

  6. install-\*-compute.sh: run Bash script on Compute Nodes only

Warning

Do not try to use any of the CCME ansible tasks in the CCME role, as there might be dependencies your custom scripts will not inherite automatically. The custom scripts have access to the following variables that you can use:

  • CCME_CONF: Path to CCME configuration file (YAML format in ansible scripts or bash format in shell scripts)

  • CCME_DEPENDENCIES: Path to CCME list of dependencies (YAML format)

You can then load these files if needed with the following tasks in your playbooks:

- name: Load CCME variables
  ansible.builtin.include_vars:
    file: "{{ CCME_CONF }}"
    name: ccme_conf

- name: Load CCME dependencies
  ansible.builtin.include_vars:
    file: "{{ CCME_DEPENDENCIES }}"
    name: ccme_deps

To update the CCME solution bucket from the Management Host, use the updateCCME.sh command.

$ ../../management/sbin/scripts/updateCCME.sh
updateCCME <S3BUCKET> <OPTIONAL: CCME.CONF> <OPTIONAL: AWS CREDENTIAL PROFILE>
 - S3BUCKET:  Name of the S3 bucket solution to upload CCME
 - CCME.CONF: Path to a ccme.conf file to replace the bucket

Note

Using the updateCCME.sh command on a Management Host does not require to specify a ccme.conf file. It will take the correct CCME conf file from: /opt/ccme/CCME/conf/ccme.conf.

Management

Note

The aws region is a required parameter, as --region option from the CLI or from the region option in the ParallelCluster configuration file used with the command.

If the aws region is specified in command line interface and in the cluster configuration file, the selected region will be the one from the CLI in priority.

Create cluster

To create a cluster, use the following command

pcluster create-cluster --cluster-name "${cluster_name}" --cluster-configuration ~/.parallelcluster/"${configuration_file}" --region "${aws_region}"

Note

If you are creating your first clusters (or a first cluster in a new environment), it is strongly recommended to create this cluster in debug mode, by setting the rollback-on-failure pcluster parameter to false with --rollback-on-failure false as shown in the command below.

pcluster create-cluster --rollback-on-failure false --cluster-name "${cluster_name}" --cluster-configuration ~/.parallelcluster/"${configuration_file}" --region "${aws_region}"

Delete cluster

pcluster delete-cluster --cluster-name "${cluster_name}" --region "${aws_region}"

List clusters

pcluster list-cluster --region "${aws_region}"

Connect to the clusters

The possibilities to connect to a deployed cluster(s) are:

  • Use the administrator account (sudoer)

    • centos or ec2-user depending of the selected OS

    • The ssh key associated to this user is selected in the cluster config file at deployment

  • Use any user from the ActiveDirectory authorized to connect to the cluster

    • With the tuple username + password

    • With the username + ssh key (after first username + password authentication) - The ssh key is available in the user home ~/.ssh/ after a first username + password authentication