[Proposal] Migrate SOCA configuration to AWS Parameter Store and add support for Jinja2

[mcrozes]

SOCA configuration is currently stored on AWS Secrets Manager as a single Secret, which make configuration updates challenging, as any typo/syntax error have the potential to bring the entire environment down.

Our proposal is to continue to store application secrets (Redis, OpenLDAP ...) on AWS Secrets Manager and migrate the rest of the configuration tree to AWS System Manager Parameter Store:

• /soca/<CLUSTER_ID>/configuration: Store SOCA environment specific parameters such as BaseOS, ClusterId, AuthProvider etc ..
  -/soca/<CLUSTER_ID>/system: Store SOCA system specific variables such as the list of packages to install, the link to download Python, OpenMPI, CloudWatch Log Agent, EFA etc ...

This new architecture will help us transition our new node setup logic to fully support Jinja2 templating, making bootstrap customization easier.

Example: Configure OpenLDAP or Microsoft AD

{% if context.get("/configuration/AuthProvider") == "openldap" %}
  # OpenLDAP configuration
  {% include "templates/linux/openldap_client.j2" %}
{% elif context.get("/configuration/AuthProvider") == "activedirectory" %}
  # Active Directory configuration
  {% include "templates/linux/join_activedirectory.j2" %}
{% else %}
  log_error "AuthProvider must be openldap or activedirectory, detected {{ context.get("/configuration/AuthProvider") }}"
  exit 1
{% endif %}

Example: Mount /data based on the FileSystem configured:

{% if context.get("/configuration/FileSystemDataProvider") == "efs" %}
	{% include "templates/linux/shared_storage/mount_efs.j2" %}
	mount_efs "{{ context.get("/configuration/FileSystemData") }}:/" "/data"

{% elif context.get("/configuration/FileSystemDataProvider") == "fsx_openzfs" %}
	{% include "templates/linux/shared_storage/mount_fsx_openzfs.j2" %}
	mount_fsx_openzfs "{{ context.get("/configuration/FileSystemData") }}:/fsx" "/data"

{% elif context.get("/configuration/FileSystemDataProvider") == "fsx_lustre" %}
	{% include "templates/linux/shared_storage/mount_fsx_lustre.j2" %}
	mount_fsx_lustre "{{ context.get("/configuration/FileSystemData") }}" "/data"

{% elif context.get("/configuration/FileSystemDataProvider") == "fsx_ontap" %}
	{% include "templates/linux/shared_storage/mount_fsx_ontap.j2" %}
	mount_fsx_ontap "{{ context.get("/configuration/FileSystemData") }}:/vol1" "/data"
{% endif %}

Additionally, we will expose ephemeral /soca/<CLUSTER_ID>/job/* hierarchy tree for job specific informations (JobID, JobOwner). This tree is not stored on AWS System Manager Parameter Store but made available temporarily during the bootstrap sequence on all EC2 nodes provisioned for the given job

Example: Add EFA configuration if efa_support=True is specified during job submission

{% if context.get("/job/Efa") %}
  {% include "templates/linux/efa.j2" %}
{% endif %}

We think this change will drastically simplify the way our end-users customize the Compute or/and Visualization nodes setup logic.

Additionally, any configuration update will be easier. As an example, let's assume you are currently using EFA version 1.31.0 and you want to bump it to 1.32.0. You could proceed to hotpatch your cluster by updating the value of /soca/<CLUSTER_ID>/system/efa/url key to https://efa-installer.amazonaws.com/aws-efa-installer-1.32.0.tar.gz

AWS System Manager Parameter Store includes native versioning, making version rollback easy in case a key was misconfigured or not updated correctly.

We are also developing a utility which will give you the ability to update these value automatically, without having to manually change the content via the AWS System Manager Parameter store console

[mcrozes]

Closing discussion, pushed to prod on 24.10 -> https://awslabs.github.io/scale-out-computing-on-aws-documentation/documentation/architecture/node-bootstrap/customize-node-boostrap/