Management Host

Warning

If you are updating CCME from a version prior to 6.0.0, to version 6.0.0, please review Update to 6.0.0 to understand the changes in parameters.

Prerequisites

CCME deployment can be done with the basic CCME automation configuration or manually. Even with the automated deployment, you still have the possibility to manually modify the CCME and AWS ParallelCluster configuration of your clusters.

The CCME_x.y.z directory contains the following files and directories:

deployCCME.sh: the script to start the Management Host

roles.ccme.conf: the configuration file for the CCME Roles Stack (CRS)

deployment.ccme.conf: the configuration file for the CCME Management Host (CMH)

dependencies.yaml: the version of the dependencies used in CCME

CCME/: the scripts to configure a CCME cluster

management/: the scripts to deploy the Management Host and associated services

doc/: this documentation

You will first need to untar the CCME_x.y.z.tar.gz archive to get access to the code on your local machine (which needs to have access to AWS and have the AWS CLI installed and configured):

mkdir "CCME_x.y.z"
tar xvf "CCME_x.y.z.tar.gz" -C "CCME_x.y.z"
cd "CCME_x.y.z"

`deployCCME.sh`

This is the script that you need to use to deploy the initial stacks for the CCME roles, the Management Host and the associated services.

$ ../../deployCCME.sh --help
Usage: deployCCME.sh [-h|--help] [-r|--create-roles-stack] [-m|--create-management-stack] [-cr|--conf-roles <crs_conf_file>] [-cm|--conf-management <cmh_conf_file>] [-s|--set-management-roles <stack_name>] [-n|--name <stack_name>] [-u|--update] [-nr|--no-rollback] [-p|--profile <aws_profile>]

deployCCME can be used to update the CCME bucket set in <conf_file> and/or to deploy a CCME Management Host

Options :
  -r|--create-roles-stack                           Create CCME Roles Stack (CRS) with a name built with the prefix 'CRS-' and the selected -n|--name option
  -m|--create-management-host                       Create CCME Management Host (CMH) with a name built with the the prefix 'CMH-' and the -n|--name option
  -ur|--update-roles-stack                          Update the deployed CCME Roles Stack (CRS) named with the -n|--name option
  -um|--update-management-host                      Update the deployed CCME Management Host (CMH) named with the -n|--name option
  -cr|--conf-roles              <crs_conf_file>     Reads a configuration file for the CCME Roles Stack (CRS)
  -cm|--conf-management         <cmh_conf_file>     Reads a configuration file for the CCME Management Host (CMH) and update action
  -s|--set-management-roles     <stack_name>        Replace the IAM Roles in the CCME Management Host (CMH) configuration file with the CCME Roles Stack (CRS) outputs
  -n|--name                     <stack_name>        Defines the name of the stack deployed by the script (default is CCME)
  -u|--update                                       Updates the CCME bucket
  -nr|--no-rollback                                 Disables rollback of the CMH creation stack in CloudFormation when the stack fails
  -p|--profile                  <aws_profile>       Use non-default AWS credentials profile with aws commmand
  -h/--help                                         Help

CCME Roles Stack (CRS)

CRS Configuration

The configuration of the “CCME Roles Stack” is described in the roles.ccme.conf file:

#### CONFIGURATION FILE for the CCME Roles Stack (CRS) automatic setup ####

#### IAM Configuration
# iam_path_prefix defines the prefix of the IAM roles created by the stack
# The expected value is a string prefix like "ccme" or "parallelcluster".
# Setting this variable is MANDATORY.
# Default value: parallelcluster
iam_path_prefix=parallelcluster


#### Global
# global_tags defines a list of tags associated to the CMH stack
# Pattern: ("Key=your_key,Value=your_value" "Key=your_key,Value=your_value")
# Setting this variable is OPTIONAL.
# Default value: no tag
# Update Policy: This setting can be changed during an update and requires to update the CMH and deployed clusters with the updated setting.
# global_tags=("Key=your_key,Value=your_value" "Key=your_key,Value=your_value")

# global_sources_path is the BucketName:BucketKey where BucketName is the name of an existing s3 bucket
# and where CCME machinery will be stored at the "BucketKey" path to be used to manage clusters.
# E.g.: "ccme-bucket:subfolder1/subfolder2/subfolder3/"
# Subfolder must end with a '/'
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy for BucketName: If this setting is changed, the update is not allowed for the CMH stack and the clusters.
# Update Policy for BucketKey: If this setting is changed, the update is allowed for the CMH stack, but not the clusters.
global_sources_path=

# global_data_bucket is the Name of an existing s3 bucket where common user data is accessed by users on any cluster.
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
global_data_bucket=

# global_read_bucket_prefix defines the prefix of bucket allowing read only from CMH and Clusters.
# Setting this variable is OPTIONAL.
# No default value exists.
# Update Policy: This setting can be changed during an update.
# global_read_bucket_prefix=

# global_read_bucket_prefix defines the prefix of bucket allowing read-write from CMH and Clusters.
# Setting this variable is OPTIONAL.
# No default value exists.
# Update Policy: This setting can be changed during an update.
# global_read_write_bucket_prefix=

# global_secrets_prefix defines the prefix of the secrets stored in the Secrets Service Manager
# CCME requires AWS Secrets, it requires to register each secret with a name starting by the same prefix.
# Setting this variable is MANDATORY.
# No default value exists.
# E.g.: "ccme-prefix"
# E.g. of secret using this prefix: "arn:aws:secretsmanager:eu-west-1:012345678910:secret:ccme-prefix-adreadonlyuser.password-7dFt5"
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
global_secrets_prefix=


#### Network
# network_route53_access defines the possibility of Route53 creation and usage
# for each cluster.
# Setting this variable is OPTIONAL.
# Value: true or false
# Default value: true
# network_route53_access=true

# fsx_s3_access defines the possibility of FSx to access to s3 buckets
# Setting this variable is OPTIONAL.
# Value: true or false
# Default value: false
# network_fsx_s3_access=false

# network_fsx_s3_buckets_arn_list defines the list of S3 buckets to allow the lambda function
# to import/export from/to S3 when creating an FSx filesystem.
# The expected value is a comma separated list of S3 bucket ARNs,
# NOTE - The setting is used only when network_fsx_s3_access is set to true.
# E.g.: arn:aws:s3:::<S3_BUCKET_1>,arn:aws:s3:::<S3_BUCKET_2>)
# Setting this variable is OPTIONAL.
# Default value: arn:*:s3:::ccme-fsx-*
# network_fsx_s3_buckets_arn_list='arn:*:s3:::ccme-fsx-*'

#### Clusters
# cluster_prefix defines the prefix of the future clusters
# Used to restrict rights to CloudWatch dashboards
# Setting this variable is MANDATORY.
# Default value is "ccme-".
cluster_prefix="ccme-"

# cluster_admin_sns_topic_arn define the admin SNS topic ARN you want to
# deliver information about the cluster when it is ready to be used.
# For example, you can configure your SNS topic to deliver the information
# by email to the administrator of the platform
# Setting this variable is OPTIONAL.
# No default value exists.
# E.g.: arn:aws:sns:REGION:ACCOUNT:TOPICNAME
# cluster_admin_sns_topic_arn=


#### KMS Encryption
# The following variables define the ARN of KMS keys used to encrypt storages or
# communications on multiple services used by CCME.
# Setting these variables is MANDATORY.
# Value: ARN of existing KMS key
# Default value is '*' (allow all keys)

# CMH EBS encryption
kms_cmh_ebs=*

# DCV Proxy encryption
kms_dcv_proxy_ebs=*

# Clusters EBS encryption
kms_clusters_ebs=*

# Additional KMS key for CMH and Clusters encryption.
# kms_additional_key=NONE

# FSx encryption
kms_file_systems=*

# S3 encryption
kms_s3=*

# SNS encryption
kms_sns=*

# Secrets Manager encryption
kms_secrets=*

# CloudWatch encryption
kms_cloudwatch=*

# Lambda encryption
kms_lambda=*

You can modify and use this example configuration file or you can create a new file based on it. This configuration file is then used to deploy the CCME Management Host through the script named deployCCME.sh.

Deployment

The following command will create all the IAM roles needed for CCME and on a CCME Roles Stack.

$ ./deployCCME.sh --create-roles-stack --conf-roles "my-ccme.roles.ccme.conf" --name "my-ccme"
**** CCME Roles Deployment - START ****
{
    "StackId": "arn:aws:cloudformation:eu-west-1:012345678910:stack/CRS-my-ccme/af11b1e0-9a5a-11ed-ab69-06562ac4f907"
}
**** CCME Roles Deployment - END ****

CCME Management Host (CMH)

CMH Configuration

The configuration of the “CCME Management Host” is described in the deployment.ccme.conf file:

#### CONFIGURATION FILE for the CCME Management Host (CMH) automatic setup ####

#### IAM
# iam_management_stack_role defines the IAM role ARN that will be attached to the Management Stack.
# The expected value in the ARN of an IAM role. Check the documentation for more information about this role.
# This parameter corresponds to the output "ParallelClusterUserRole" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_management_stack_role=

# iam_management_instance_profile defines the CCME Management Host (CMH) AWS IAM Instance Profile ARN
# This parameter corresponds to the output "ParallelClusterUserInstanceProfile" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_management_instance_profile=

# iam_dcv_proxy_lambda_certificate_role defines the DCV Certificate Lambda AWS IAM Role ARN
# This parameter corresponds to the output "CustomLambdaResourcesRoleDcvProxyCertificate" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_dcv_proxy_lambda_certificate_role=

# iam_tagging_lambda_role defines the Tagging Lambda AWS IAM Role ARN
# This parameter corresponds to the output "CustomLambdaResourcesRoleTagging" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_tagging_lambda_role=

# iam_dcvsm_ddb_cleanup_lambda_role defines the DCVSM DynamoDB Cleanup Lambda AWS IAM Role ARN
# This parameter corresponds to the output "CustomLambdaResourcesRoleDcvsmDdbCleanup" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_dcvsm_ddb_cleanup_lambda_role=

# iam_dcv_asg_lambda_role defines the DCV ASG Lambda AWS IAM Role ARN
# This parameter corresponds to the output "CustomLambdaResourcesRoleDcvAsg" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_dcv_asg_lambda_role=

# iam_application_load_balancer_lambda_role defines the ApplicationLoadBalancer Lambda AWS IAM Role ARN
# This parameter corresponds to the output "CustomLambdaResourcesRoleApplicationLoadBalancer" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_application_load_balancer_lambda_role=

# iam_dcv_proxy_instance_profile defines the DCV Proxy AutoScaling Group instances AWS IAM Role ARN
# This parameter corresponds to the output "DcvProxyInstanceProfile" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
iam_dcv_proxy_instance_profile=

# iam_cluster_lambda_role defines the ParallelCluster Lambda AWS IAM Role ARN
# This parameter corresponds to the output "CustomLambdaResourcesRoleSlurm" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: If this setting is changed, the update is not allowed.
iam_cluster_lambda_role=

# iam_cluster_headnode_instance_profile defines the HeadNode AWS IAM Instance Profile ARN
# This parameter corresponds to the output "HeadNodeInstanceProfileSlurm" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: If this setting is changed, the update is not allowed.
iam_cluster_headnode_instance_profile=

# iam_cluster_compute_instance_profile defines the Compute Nodes AWS IAM Instance profile ARN
# This parameter corresponds to the output "ComputeNodeInstanceProfileSlurm" of the CCME Roles Stack (CRS)
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: If this setting is changed, the update is not allowed.
iam_cluster_compute_instance_profile=

#### Global
# global_debug is a boolean variable. When set to true, the stack does not automatically "rollback" after any failure.
# Setting this variable is MANDATORY.
# Default value: false
# Update Policy: This setting can be changed during an update.
global_debug=false

# global_timezone is a string variable. It configures the timezone for the CMH and the clusters
# You can list available timezones as following :
# - Typing: ``timedatectl list-timezones`` to list your linux system timezone.
# - See: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# Setting this variable is MANDATORY.
# Default value: 'Europe/Paris'
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
global_timezone='Europe/Paris'

# global_sources_path is the BucketName:BucketKey where BucketName is the name of an existing s3 bucket
# and where CCME machinery will be stored at the "BucketKey" path to be used to manage clusters.
# Subfolder must end with a '/'
# Setting this variable is MANDATORY.
# E.g.: "ccme-bucket:subfolder1/subfolder2/subfolder3/"
# No default value exists.
# Update Policy for BucketName: If this setting is changed, the update is not allowed.
# Update Policy for BucketKey: If this setting is changed, the update is allowed for the CMH stack, but not the clusters.
global_sources_path=

# global_data_bucket is the Name of an existing s3 bucket where common user data is accessed by users on any cluster.
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
global_data_bucket=

# global_logging_bucket is the Name of an existing s3 bucket where S3 access logs are stored
# for the ccme deployed s3 buckets.
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
global_logging_bucket=

# global_logs_retention_in_days defines the retention period of CCME logs in days.
# Setting this variable is OPTIONAL.
# Pattern: 1|3|5|7|14|30|60|90|120|150|180|365|400|545|731|1096|1827|2192|2557|2922|3288|3653
# Default value: 14
# Update Policy: This setting can be changed during an update.
# global_logs_retention_in_days=14

# global_tags defines a list of tags associated to the CMH stack
# Pattern: ("Key=your_key,Value=your_value" "Key=your_key,Value=your_value")
# Setting this variable is OPTIONAL.
# Default value: no tag
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# global_tags=("Key=your_key,Value=your_value" "Key=your_key,Value=your_value")


#### Network
# network_vpc defines the VPC where the CMH should be setup. The expected value is an existing VPCId.
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: If this setting is changed, the update is not allowed.
network_vpc_id=

# network_alb_subnets defines the Primary and Secondary subnets (created in 2 different AZs)
# where the Application Load Balancer should be setup.
# The expected values is a pair of existing SubnetId.
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
network_alb_subnets=( "" "" )

# network_private_subnets defines the Primary and Secondary subnets (created in 2 different AZs)
# where the Active Directory and CCME Management Host should be setup.
# The expected values is a pair of existing SubnetId.
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: If this setting is changed, the update is not allowed.
network_private_subnets=( "" "" )

# network_ingress_security_groups_ssh_cidr_block specifies the CIDR-formatted IP range for ingress connections to:
#   - Private Security Group
#   - Headnode Security Group
# If this setting is not set or set to NONE disables the ssh authorization in the security groups.
# Setting this variable is OPTIONAL.
# Pattern: NONE or x.x.x.x/x
# Default value: NONE
# Update Policy: This setting can be changed during an update.
# network_ingress_security_groups_ssh_cidr_block=NONE

# network_security_groups_login_nodes_ingress_ssh_cidr_block specifies the CIDR-formatted IP range for ingress connections to:
#   - LoginNodes Security Group
# Setting this variable is OPTIONAL.
# Pattern: NONE or x.x.x.x/x
# Default value: NONE
# Update Policy: This setting can be changed during an update.
# network_security_groups_login_nodes_ingress_ssh_cidr_block=NONE

# network_additional_security_group defines one optional additional security group
# to the CMH and clusters.
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# network_additional_security_group=NONE

# network_proxy defines the optional proxy for CMH and clusters
# Setting this variable is OPTIONAL.
# Pattern: protocol://host:port
# E.g.: "https://url_of_proxy:443"
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# network_proxy=NONE

# network_no_proxy defines the optional list of no_proxy delimited by comma for CMH and clusters
# Setting this variable is OPTIONAL.
# Pattern: host
# E.g.: "my.domain1,my.domain2"
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# network_no_proxy=NONE

# network_pip_repository defines a private pip repository
# Setting this variable is OPTIONAL.
# Pattern: protocol://host
# E.g.: "https://url_of_pip_repo"
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# network_pip_repository=NONE


#### Application Load Balancer
# application_load_balancer_arn defines the application load balancer used to forward the
# portal and the visualization of CCME to the end users.
# The expected value is the ARN of the choosen Application Load Balancer
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# application_load_balancer_arn=NONE

# application_load_balancer_scheme defines if the Application Load Balancer (ALB) is "internet-facing" or "internal"
# Setting this variable is MANDATORY.
# Default value: 'internet-facing'
# Update Policy: If this setting is changed, the update is not allowed.
application_load_balancer_scheme=internet-facing

# application_load_balancer_ingress_cidr_block specifies the CIDR-formatted IP range for ingress connections to the ALB.
# Setting this variable is OPTIONAL.
# Default value: 0.0.0.0/0
# Update Policy: This setting can be changed during an update.
# application_load_balancer_ingress_cidr_block=0.0.0.0/0

# application_load_balancer_ssl_certificate defines the SSL certificate used by the Application Load Balancer being the entry point to CCME.
# The expected value in the ARN of a SSL certificate stored in AWS Certificate Manager (ACM).
# Setting this variable is MANDATORY.
# No default value exists.
# Update Policy: This setting can be changed during an update.
application_load_balancer_ssl_certificate=


#### DCV Proxy
# dcv_proxy_asg_min_size defines the DCV Proxy AutoScalingGroup (asg) minimum size
# Setting this variable is MANDATORY.
# Pattern: Number
# Default value: 1
# Update Policy: This setting can be changed during an update.
dcv_proxy_asg_min_size=1

# dcv_proxy_asg_max_size defines the DCV Proxy AutoScalingGroup (asg) maximum size
# Setting this variable is MANDATORY.
# Pattern: Number
# Default value: 5
# Update Policy: This setting can be changed during an update.
dcv_proxy_asg_max_size=5

# dcv_proxy_asg_desired_size defines the DCV Proxy AutoScalingGroup (asg) desired size at launch
# Setting this variable is MANDATORY.
# Pattern: Number
# Default value: 1
# Update Policy: This setting can be changed during an update.
dcv_proxy_asg_desired_size=1

# dcv_proxy_instance_type defines the instance type for the DCV Proxy
# Only AWS EC2 x86_64 instance type is allowed.
# Recommended value: m5a.large
# Default value: t2.micro
# Update Policy: This setting can be changed during an update.
dcv_proxy_instance_type=t2.micro

# dcv_proxy_instance_ami defines the AMI for the DCV Proxy instance
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update.
# dcv_proxy_instance_ami=NONE

# dcv_proxy_instance_volume_size defines the volume size in GB for DCV Proxy instances
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update. WARNING: A new DCV Proxy instances will be created.
# dcv_proxy_instance_volume_size=NONE


#### ActiveDirectory
# active_directory_id defines the id of the Microsoft or Simple Active Directory.
# This variable allows to manage user accounts from the CCME Management Host.
# Setting this variable is OPTIONAL.
# Pattern: d-xxxxxxxxxx
# Default value: NONE
# Update Policy: This setting can be changed during an update.
# active_directory_id=NONE

# active_directory_uri defines the list of URI of the Active Directory.
# The format is a comma separated list of ldap:// or ldaps:// addresses.
# Setting this variable is MANDATORY ONLY IF using ActiveDirectory.
# Pattern: protocol://host
# E.g., ldap://10.0.0.1,ldap://10.0.0.2
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# active_directory_uri=NONE

# active_directory_name defines the Domain Name of your Active Directory.
# If the value is NONE, then no Active Directory is binded to CCME.
# Setting this variable is MANDATORY ONLY IF using ActiveDirectory.
# Default value: ccme.ad
# Update Policy: This setting can be changed during an update only if the previous and the new values are set for internal Active Directory.
# active_directory_name=NONE

# active_directory_readonlyuser_dn defines the distinguished name of ReadOnlyUser account in your AD.
# E.g., "CN=string,OU=string,DC=string,DC=string".
# Setting this variable is MANDATORY ONLY IF using ActiveDirectory.
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# active_directory_read_only_user_dn=NONE

# active_directory_read_only_user_password_secret_arn defines the password of ReadOnlyUser accounts in your AD.
# The expected value in the preexisting ARN of a plaintext string stored in AWS Secrets Manager (ASM).
# E.g.: active_directory_read_only_user_password_secret_arn=arn:aws:secretsmanager:eu-west-1:012345678910:secret:ccme-prefix-adreadonlyuser.password-3nSOf
# Setting this variable is MANDATORY ONLY IF using ActiveDirectory.
# Default value: NONE
# Update Policy: This setting can be changed during an update and requires to update the deployed clusters with the updated setting.
# active_directory_read_only_user_password_secret_arn=NONE

# active_directory_admin_password_secret_arn defines the password of Admin accounts in your AD.
# The expected value in the preexisting ARN of a plaintext string stored in AWS Secrets Manager (ASM).
# E.g.: active_directory_admin_password_secret_arn=arn:aws:secretsmanager:eu-west-1:012345678910:secret:ccme-prefix-adadmin.password-7dFt5
# Setting this variable is MANDATORY ONLY IF using ActiveDirectory deployed by CCME.
# Default value: NONE
# Update Policy: If this setting is changed, the update is not allowed.
# active_directory_admin_password_secret_arn=NONE

# active_directory_ssh_public_key_attribute defines the ActiveDirectory ssh public key.
# If the value is NONE, then the ssh to the CCME ManagementHost with a ssh public key stored in
# the ActiveDirectory will be disabled.
# Setting this variable is MANDATORY ONLY IF using ActiveDirectory.
# Default value: ccme.ad
# Update Policy: This setting can be changed during an update only if the previous and the new values are set for internal Active Directory.
# active_directory_name=NONE

#### Management Host
# management_host_instance_type defines the instance type that will be configured as a CMH
# Only AWS EC2 x86_64 instance type is allowed.
# Setting this variable is MANDATORY.
# Default value: t3a.medium
# Update Policy: This setting can be changed during an update.
management_host_instance_type=t3a.medium

# management_host_os defines the operating system of the CCME Management Host (CMH)
# Allowed values for this variable are the following:
# - al2023
# - rhel8
# - rhel9
# Setting this variable is MANDATORY.
# Default value: al2023
# Update Policy: This setting can be changed during an update. WARNING: A new CMH instance will be created.
management_host_os=al2023

# management_host_ami defines the AMI for the CMH instance
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update. WARNING: A new CMH instance will be created.
# management_host_ami=NONE

# management_host_volume_size defines the volume size in GB for the CMH instance
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update. WARNING: A new CMH instance will be created.
# management_host_volume_size=NONE

# management_host_keypair_name is the Name of an existing EC2 key pair that should be used to connect to the CMH with SSH.
# If you do not specify a key pair or set the value to NONE, you can't connect to the instance
# unless you choose an AMI providing another way to log in.
# Setting this variable is OPTIONAL.
# No default value exists.
# Update Policy: If this setting is changed, the update is not allowed.
# management_host_keypair_name=

# management_host_elastic_ip defines the address of an Elastic IP (if you already have one to use for your CMH)
# If the default value is used, CCME will request a new Elastic IP to attach it to the CMH.
# Set "NONE" to not assign a public IP.
# Setting this variable is OPTIONAL.
# Default value: 0.0.0.0
# Update Policy: This setting can be changed during an update.
# management_host_elastic_ip=0.0.0.0

# management_host_sudoer_group_name defines the group of users which is allowed to use sudo on your CMH.
# The group can be local or from Active Directory.
# If the default value is used, no group is authorized to use sudo.
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update.
# management_host_sudoer_group_name=NONE

# management_host_authorized_group_name defines the AD groups of users which are allowed to login to your CMH.
# If the default value is used, no filter is applied and everyone in AD is authorized to login.
# Setting this variable is OPTIONAL.
# Default value: NONE
# Update Policy: This setting can be changed during an update.
# management_host_authorized_group_name=NONE


#### KMS Encryption
# The following variables define the ARN of KMS keys used to encrypt storages or
# communications on multiple services used by CCME.
# Setting these variables is MANDATORY.
# Value: ARN of existing KMS key
# No default value exists.

# CMH EBS encryption
# Update Policy: If this setting is changed, the update is not allowed.
kms_cmh_ebs=

# DCV Proxy encryption
# Update Policy: This setting can be changed during an update if the CCME Role Stack allows access to the resource ARN.
kms_dcv_proxy_ebs=

# Clusters EBS encryption
# Update Policy: If this setting is changed, the update is not allowed.
kms_clusters_ebs=

# EFS and FSx encryption
# Update Policy: If this setting is changed, the update is not allowed.
kms_file_systems=

# S3 encryption
# Update Policy: This setting can be changed during an update if the CCME Role Stack allows access to the resource ARN.
kms_s3=

# SNS encryption
# Update Policy: This setting can be changed during an update if the CCME Role Stack allows access to the resource ARN.
kms_sns=

# CloudWatch encryption
# Update Policy: If this setting is changed, the update is not allowed.
kms_cloudwatch=

# Lambda encryption
# Update Policy: This setting can be changed during an update if the CCME Role Stack allows access to the resource ARN.
kms_lambda=

You can modify and use this example configuration file or you can create a new file based on it. This configuration file is then used to deploy the CCME Management Host through the script named deployCCME.sh.

Resources

The resources deployed in your AWS account by the CCME Management Host are:

An AWS Lambda with a log group in /ccme/ccme-lambda-alb-${StackId}
An AWS EC2 instance as CCME Management Host
Optional
- An Active Directory
- An Application Load Balancer
- Multiple AWS EC2 security-groups

Those resources are named using ccme as prefix and -Stack_id as suffix. Example for the resource LambdaALB, the resource is named: ccmeLambdaALB-7225f980-0a8a-11ee-81f0-06c4c07a0d95:

ccme as prefix
A variable information related to the resource: LambdaALB in this case
The stack_id of the CMH stack and a suffix preceded by a -: -7225f980-0a8a-11ee-81f0-06c4c07a0d95 in our case

The only exception for this rule is related to the Application Load Balancer (ALB). The ALB is using -Stack_name instead of the -Stack_id as suffix due to restrictions on the length of the name of the ALB

Deployment

The following command will create all the environment needed for CCME and create a Management Host named my_ccme_mgt_host. First it will upload all the source code of CCME to the ccme_bucket_path as specified in the deployment.ccme.conf file, this is mandatory on a first deployment. For further deployments, you can directly reuse the same bucket if no code change has been made.

$ ./deployCCME.sh --update --create-management-host --conf-management "my-ccme.deployment.ccme.conf" --name "my-ccme"
**** CCME Bucket Update - START ****
**** CCME Bucket Update - END ****
**** CCME ManagementHost Deployment - START ****
{
    "StackId": "arn:aws:cloudformation:eu-west-1:012345678910:stack/CMH-my-ccme/bf11b1e0-9a5a-11ed-ab69-06562ac4f907"
}
**** CCME ManagementHost Deployment - END ****

Customization

Custom Ansible Scripts

On top of CCME specific configurations, you can integrate your own custom scripts to CCME Management Host. To deploy a CMH embedding and executing your own custom scripts, you must place them in the management/custom directory and synchronize this directory in the S3 bucket. You can provide your own Ansible playbooks to add specific configurations to the CMH, they must have the following naming convention: management/custom/install-*-management.yaml.

Custom ParallelCluster configuration files

On top of the ParallelCluster example configuration files provided with CCME, you can provide your own Jinja template along with a set of parameters to generate your own configuration files.

To do so, you just need to:

add your templates in management/custom and name them with the following pattern *.pcluster.config.j2,
add any specific configuration variable in the management/custom/pcluster.vars.yaml file (YAML format). All variables in this file will be assigned under the custom variable. This means that if you declare myvar: myval, it will be available as custom.myvar in you *.pcluster.config.j2 files.

management/custom/custom.pcluster.config.j2

Region: '{{ AWS_REGION }}'
CustomS3Bucket: '{{ CCME_CLUSTER_S3BUCKET }}'
Iam:
  Roles:
    LambdaFunctionsRole: '{{ CCME_CLUSTER_LAMBDA_ROLE }}'
  # If the role associated to the cluster includes a custom IAM path prefix,
  # replace "parallelcluster" by the custom IAM path prefix.
  ResourcePrefix: "{{ CCME_CLUSTER_HEADNODE_INSTANCE_PROFILE.split('/')[1] }}"
Image:
  Os: alinux2
  # CustomAmi:
Tags:
{% for key, value in CCME_TAGS.items() %}
{% if "aws:" not in key and key != "Name" %}
  - Key: {{ key }}
    Value: "{{ value }}"
{% endif %}
{% endfor %}
SharedStorage:
  - Name: shared
    StorageType: Efs
    MountDir: shared
    EfsSettings:
{% if "NONE" not in CCME_KMS_FILE_SYSTEMS %}
      Encrypted: true
      KmsKeyId: {{ CCME_KMS_FILE_SYSTEMS }}
{% else %}
      Encrypted: false
{% endif %}
HeadNode:
  InstanceType: {{ custom.headinsttype }}
  Networking:
    SubnetId: '{{ CCME_SUBNET }}'
    SecurityGroups:
      - '{{ CCME_HEADNODE_SG }}'
{%if CCME_PROXY is defined and CCME_PROXY and CCME_PROXY != "NONE"%}
    Proxy:
      HttpProxyAddress: '{{ CCME_PROXY }}'
{% endif %}
  Ssh:
    KeyName: '{{ AWS_KEYNAME }}'
  CustomActions:
    OnNodeStart:
      Script: s3://{{ CCME_SOURCES }}CCME/sbin/pre-install.sh
    OnNodeConfigured:
      Script: s3://{{ CCME_SOURCES }}CCME/sbin/post-install.sh
    OnNodeUpdated:
      Script: s3://{{ CCME_SOURCES }}CCME/sbin/update-install.sh
      Args:
        - CCME_CMH_NAME={{ CCME_CMH_NAME }}
        - CCME_S3FS={{ CCME_DATA_BUCKET }}
        - CCME_JSLOGS_BUCKET={{ CCME_DATA_BUCKET }}
        # CCME_SHARED_DIR is a mandatory parameter: it must the the mountdir of one of the SharedStorage
        # It will be used to store CCME shared files with the compute nodes. Especially EnginFrame spoolers
        # which may require a large storage space depending on how it is used.
        # Update Policy: this parameter must not be changed during an update
        - CCME_SHARED_DIR=shared
{%if CCME_NO_PROXY is defined and CCME_NO_PROXY and CCME_NO_PROXY != "NONE"%}
        - CCME_NO_PROXY={{ CCME_NO_PROXY }}
{% endif %}
        # - CCME_OIDC=default
        # - CCME_USER_HOME=/shared-filesystem/home/%u
        # - CCME_DNS=NONE
        - CCME_WIN_LAUNCH_TEMPLATE_ID={{ CCME_WIN_LAUNCH_TEMPLATE_ID }}
        # - CCME_WIN_AMI=NONE
        # - CCME_WIN_INSTANCE_TYPE=NONE
        # - CCME_WIN_INACTIVE_SESSION_TIME=600
        # - CCME_WIN_NO_SESSION_TIME=600
        # - CCME_WIN_NO_BROKER_COMMUNICATION_TIME=600
        # - CCME_LIN_INACTIVE_SESSION_TIME=3600
        # - CCME_EF_ADMIN_GROUP=
        # - CCME_EFADMIN_PASSWORD=
{%if CCME_REPOSITORY_PIP is defined and CCME_REPOSITORY_PIP and CCME_REPOSITORY_PIP != "NONE"%}
        - CCME_REPOSITORY_PIP={{ CCME_REPOSITORY_PIP }}
{% endif %}
  Iam:
    InstanceProfile: '{{ CCME_CLUSTER_HEADNODE_INSTANCE_PROFILE }}'
Scheduling:
  Scheduler: slurm
  SlurmSettings:
    Dns:
      # If the role associated to the cluster is not authorized to use Route 53,
      # set "DisableManagedDns" to true.
      DisableManagedDns: False
  SlurmQueues:
{% for queue in custom.queues %}
    - Name: {{ queue.Name }}
      CapacityType: ONDEMAND
      ComputeResources:
{% for cr in queue.InstanceTypes %}
        - Name: {{ cr | replace('.', '') }}
          InstanceType: {{ cr }}
          MinCount: 0
          MaxCount: 10
{% endfor %}
      CustomActions:
        OnNodeStart:
          Script: s3://{{ CCME_SOURCES }}CCME/sbin/pre-install.sh
        OnNodeConfigured:
          Script: s3://{{ CCME_SOURCES }}CCME/sbin/post-install.sh
      Iam:
        InstanceProfile: '{{ CCME_CLUSTER_COMPUTE_INSTANCE_PROFILE }}'
      Networking:
        SubnetIds:
          - '{{ CCME_SUBNET }}'
        SecurityGroups:
          - '{{ CCME_COMPUTE_SG }}'
{%if CCME_PROXY is defined and CCME_PROXY and CCME_PROXY != "NONE"%}
        Proxy:
          HttpProxyAddress: '{{ CCME_PROXY }}'
{% endif %}
{% endfor %}
LoginNodes:
  Pools:
    - Name: login-pool-1
      Count: 1
      InstanceType: t3a.small
      # GracetimePeriod: integer
      # Image:
      #   CustomAmi:
      Ssh:
        KeyName: '{{ AWS_KEYNAME }}'
      Networking:
        SubnetIds:
          - '{{ CCME_SUBNET }}'
        SecurityGroups:
          - '{{ CCME_LOGIN_SG }}'
      Iam:
        InstanceProfile: '{{ CCME_CLUSTER_COMPUTE_INSTANCE_PROFILE }}'
{%if CCME_AD_DIR_NAME is defined and CCME_AD_DIR_NAME != "NONE" and CCME_AD_URI is defined and CCME_AD_URI != "NONE"%}
DirectoryService:
  DomainName: {{ CCME_AD_DIR_NAME }}
  DomainAddr: {{ CCME_AD_URI }}
  PasswordSecretArn: {{ CCME_AD_READ_ONLY_USER_PASSWORD }}
  DomainReadOnlyUser: {{ CCME_AD_READ_ONLY_USER_DN }}
  LdapTlsReqCert: hard
  LdapTlsCaCert: {{ CCME_CLUSTER_DIR }}/conf/{{ CCME_AD_URI.split('/')[2] }}.crt
  # LdapAccessFilter:
  # AdditionalSssdConfigs:
    # debug_level: "0x1ff"
{% endif %}
Imds:
  ImdsSupport: v2.0
DevSettings:
  Timeouts:
    HeadNodeBootstrapTimeout: 2400
    ComputeNodeBootstrapTimeout: 1800

management/custom/pcluster.vars.yaml

################################################################################
# Copyright (c) 2017-2025 UCit SAS
# All Rights Reserved
#
# This software is the confidential and proprietary information
# of UCit SAS ("Confidential Information").
# You shall not disclose such Confidential Information
# and shall use it only in accordance with the terms of
# the license agreement you entered into with UCit.
################################################################################
# This file can be used to set custom variables used to render the *.pcluster.conf.j2 templates
# All variables in this file will be assigned under the "custom" variable.
# This means that if you declare "myvar: myval", it will be available as custom.myvar
headinsttype: t3.xlarge
queues:
  - Name: basic-slurm
    InstanceTypes:
      - c5n.18xlarge
      - c5n.xlarge
  - Name: dcv-gpu
    InstanceTypes:
      - g4dn.xlarge

Connection

There are two network possibilities to connect to the Management Host: - From authorized IP / CIDR with a Public IP attributed to the CCME Management Host - From authorized IP / CIDR with a Private IP, using an instance in the same VPC (e.g.: bastion, proxy/bounce server…)

There are two user possibilities to connect to the Management Host:

CCME Admin user: ec2-user
User from the ActiveDirectory
- Must be in the group authorized to connect to the Management Host, as defined by the variable “management_host_authorized_group” of the configuration file described in the section “Configuration”

The required information are:

Username (ManagementHostUser): available in the CloudFormation output of the CCME ManagementHost stack
- ssh key: For the local user named ec2-user
- password: For user from the Active Directory
IP address: are available in the CloudFormation output of the CCME ManagementHost stack
- Public IP (ManagementHostPublicIP)
- Private IP (ManagementHostPrivateIP)

You can retrieve the output from the Management Host stack from the AWS Console as in this example:

Key	Value	Description
`ManagementHostPrivateIP`	10.0.0.32	Private IP Address of the Management Host
`ManagementHostPublicIP`	54.122.171.48	Public IP address of the ManagementHost
`ManagementHostUser`	ec2-user	User of the Management Host

Management Host

Prerequisites

deployCCME.sh

CCME Roles Stack (CRS)

CRS Configuration

Deployment

CCME Management Host (CMH)

CMH Configuration

Resources

Deployment

Customization

Custom Ansible Scripts

Custom ParallelCluster configuration files

Connection

`deployCCME.sh`