Use SSM ResumeSession API to support EC2 instance reboot/'NoReboot' EC2 Spot CreateImage AMI via AWS SSM session-manager-plugin

[mjnowen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Please search the existing issues for relevant feature requests, and use the
reaction feature
(https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/)
to add upvotes to pre-existing requests.

Description

Currently, Packer building via "session-manager-plugin" (AWS SSM Session Manager) will report several similar, related error message(s). They are cosmetic and do not affect the resulting AMI, but it generates wastage in terms of the SSM session having to be re-created (potentially multiple times in my case) and the eventual session will always fail to be terminated as a result.

Some example error messages:

==> amazon-ebs.<name>: Provisioning with shell script: /tmp/packer-shell896848410
==> amazon-ebs.<name>: SessionId: <id> : document process failed unexpectedly: ipc messaging received timeout signal , check [ssm-document-worker]/[ssm-session-worker] log for crash reason
==> amazon-ebs.<name>: Error terminating SSM Session "<id>", this does not affect the built AMI. Please terminate the session manually: operation error SSM: TerminateSession, https response error StatusCode: 400, RequestID: ad72a001-b511-4bbb-bc88-f899966fe7a2, api error ValidationException: Session is not in a valid state

==> amazon-ebs.<name>: Waiting for AMI to become ready...
==> amazon-ebs.<name>: SessionId: <id> : document process failed unexpectedly: ipc messaging received timeout signal , check [ssm-document-worker]/[ssm-session-worker] log for crash reason
==> amazon-ebs.<name>: Error terminating SSM Session "<id>", this does not affect the built AMI. Please terminate the session manually: operation error SSM: TerminateSession, https response error StatusCode: 400, RequestID: 0b8c0f68-e2b3-4ccb-a540-1518e5ff39de, api error ValidationException: Session is not in a valid state

These messages are caused by the build instance restarting. The AWS SSM agent currently does not survive a restart. It tears down the port forwarding session, the session-manager-plugin reports the loss, and Packer's follow-up ssm:TerminateSession call is rejected because the session is already terminating. Packer then opens a replacement session and continues.

To help a reader understand the order of events and why Packer should be enhanced to support the build instance being 'rebooted' can be seen in this image:

^In the above image, you can see that I need to compile/install a custom kernel, requiring a machine reboot before I proceed with another Packer provisioner with downstream further config/installation that I must carry out as part of the Packer build process. I'm also using EC2 Spot to save cost and this needs a reboot as well during the AMI creation process. So I get a 2nd session failure as a result of the 2nd reboot and the new tunnel leaks, so the terminate session fails. I currently have to run a post-processor with "shell-local" to execute a cleanup of this SSM session.

All of this could be fixed in Packer... :)

Use Case(s)

The Packer SSM communicator should handle reboots gracefully by creating a new session after the instance comes back up (rather than expecting the existing session to survive, which is not supported now).

Packer possibly needs to:
   1. Detect the session died due to reboot (vs. a real failure)
   2. Wait for the instance to pass SSM connectivity checks again
   3. Try to start a new session

Potential Solution

So, this is the important bit....

session-manager-plugin v1.2.835.0 added this feature for client-side automatic reconnection: https://github.com/aws/session-manager-plugin/releases/tag/1.2.835.0

Fix port forwarding sessions silently dying after WebSocket reconnection

and if you read the commit notes here: aws/session-manager-plugin@29db684

Fix data channel reconnection: use StreamUrl and refresh credentials
Fix long-lived port forwarding sessions silently dying after WebSocket
reconnection. GetResumeSessionParams() now returns and propagates the
fresh StreamUrl, and ResumeSessionHandler() updates credentials on the
WebSocket channel before reconnecting.

What we are missing is integration in Packer to:

• Store url/token on successful session connection
• Detect due to reboot. Do we even need to do this, as standard SSH auto-recovers nicely in Packer already.
• Resume Session API
• Build websocket with returned url and token (from failed session)
• Once data channel is reconnected, the session can come back
• add configurable timeout only applicable when using SSM/session-manager-plugin

and thus, we no longer see all these horrible ERROR messages when the build instance is rebooted and/or restarted for AMI creation (when EC2 Spot is used and Packer does not pass NoReboot for consistent snapshot).

This would then bring the customer experience inline with standard SSH communicator during an Ec2 instance reboot and/or NoReboot of ec2:CreateImage AMI creation (where the Ec2 Spot instance is rebooted).

Potential References

Advice from AWS SDE expert on SSM Session Manager on how to achieve this:
aws/session-manager-plugin#55 (comment)