Troubleshooting

Management Host Deployment

Deploy Management Host in debug mode

To debug deployment of Management Host you have to add -nr to disable rollback in AWS Cloudformation

$ ./deployCCME.sh -n ClusterName -cm deployment.ccme.conf -m -nr

See deployment Logs

Installation logs location: /var/log/

Accessing CMH logs

All logs are available on the CMH in /var/log, and also in the AWS CloudWatch LogGroup created with the CMH stack (/ccme/ccme-cmh-<stackID>).

You should have a look at the following logs:

• CCME logs: /var/log/ccme.*.log

• Cloud init and cfn init logs: /var/log/cloud-init.log, cloud-init-output.log, cfn-init.log

• System logs: /var/log/messages and syslog

• SSSD logs: /var/log/sssd/sssd.log and sssd_default.log

Accessing lambda logs

All the CCME Application Load Balancer logs are available in the AWS CloudWatch LogGroup created with the CMH stack: /ccme/ccme-lambda-alb-<stackID>.

Cluster deployment

Deploy Cluster in debug mode

To debug deployment of Cluster you have to add –rollback-on-failure false to disable rollback in Cloudformation

$ pcluster create-cluster --cluster-name ClusterName --cluster-configuration configuration_file.yaml --region $REGION --rollback-on-failure false

See deployment Logs

Installation logs location: /var/log/

You can also see the logs in AWS CloudWatch under your cluster name for more information you can see in AWS Parallel Cluster in CloudWatch CCME logs are also present in the same CloudWatch group: search for ccme. in the log group to find all CCME logs.

Accessing CCME logs

All logs are available on the Headnode and compute nodes of the cluster in /var/log, and also in the AWS CloudWatch LogGroup created with the cluster (see Amazon CloudWatch Logs cluster logs).

All CCME logs start with ccme.. For example to see why the pre-install script failed to run you have to see in /var/log/ccme.pre-install.log and then to see for post-install script /var/log/ccme.post-install.log

Prevent compute nodes from being killed when there is a problem

AWS ParallelCluster will kill any compute node that has an issue. If this happens too often, or at every startup of a node, you need to access the logs in /var/log/ccme*.log to see what is the problem. To prevent nodes from being killed, you need to log in your AWS console, go to EC2, and identify the instance you want to protect. You then need to set the following configurations to the instance (Actions/Instance settings):

• Change termination protection: Enable

• Change stop protection: Enable

• Change shutdown behavior: Stop

Accessing Slurm logs

Slurm logs are available on the Headnode of the cluster in:

• /var/log/slurmctld.log

• /var/log/parallelcluster/slurm* (every files starting by slurm)

Slurm logs are available on the Compute Nodes of the cluster in:

• /var/log/slurmd.log

Active Directory users

Depending of the Active Directory configuration and security, the users of the AD can have issue at connection. When the issue appears, we strongly recommend to check the sssd status with a systemctl status sssd.

SSSD Configuration

• The SSSD configuration is automatically created by the cluster and is available in the file /etc/sssd/sssd.conf.

• The SSSD parameters are based on the DirectoryService section of the cluster configuration file on the management host.

• The issues solved manually must be reported to the DirectoryService section of the cluster configuration file.

SSSD with ldaps

• The Certified Authority (CA) key file should be placed in the CCME/conf/ directory on S3

• The following parameters must be set in the DirectoryService section of the cluster configuration file on the management host.

  • LdapTlsCaCert: /opt/CCME/conf/<your_ca_file>

  • LdapTlsReqCert: hard

SSSD Logs

SSSD logs location: /var/log/sssd/

Here you can find multiple log files of SSSD, including but not limited to:

• ldap_child.log

• sssd_default.log

• sssd.log

• sssd_nss.log

• sssd_pam.log

• sssd_ssh.log

Access logs

Multiple logs can be reviewed to follow CCME access:

• ALB logs: they are placed in the ccme-alb-logs-${StackId} (with ${StackId} the CMH CloudFormation Stack ID). Elastic Load Balancing publishes a log file for each load balancer node every 5 minutes. For more details on ALB logs and their content see this documentation.

• EnginFrame logs: on each cluster, on the Headnode, EnginFrame also produces logs in /opt/nice/enginframe/logs/${hostname} (with ${hostname} the hostname of the Headnode).