Skip to content

Support retrying interrupted FARGATE task #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
okkez merged 2 commits into from
Jul 21, 2025
Merged

Support retrying interrupted FARGATE task #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
25e3fab
Organize Ruby versions to run CI
okkez Jul 18, 2025
9f6bc49
Retry interrupted spot task
okkez Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,9 @@ jobs:
fail-fast: false
matrix:
ruby:
- 2.7
- 3.0
- 3.1
- 3.2
- 3.3
- 3.4
os:
- ubuntu-latest
name: Ruby ${{ matrix.ruby }} unit testing on ${{ matrix.os }}
Expand Down
6 changes: 6 additions & 0 deletions lib/wrapbox/runner/ecs.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ class LackResource < StandardError; end
WAIT_DELAY = 5
TERM_TIMEOUT = 120
HOST_TERMINATED_REASON_REGEXP = /Host EC2.*terminated/
SPOT_TASK_INTERRUPTED = /Your Spot Task was interrupted/
Copy link

Copilot AI Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex pattern should end with a word boundary or be more specific to avoid potential false matches if AWS changes the message format slightly. Consider using /Your Spot Task was interrupted\b/ or anchoring the pattern.

Suggested change
SPOT_TASK_INTERRUPTED = /Your Spot Task was interrupted/
SPOT_TASK_INTERRUPTED = /Your Spot Task was interrupted\b/

Copilot uses AI. Check for mistakes.

attr_reader \
:name,
Expand Down Expand Up @@ -243,12 +244,17 @@ def run_task(task_definition_arn, command, environments, parameter)

# If exit_code is nil, Container is force killed or ECS failed to launch Container by Irregular situation
error_message = build_error_message(task_definition_name, task.task_arn, task_status)
if task_status[:stopped_reason] =~ SPOT_TASK_INTERRUPTED
@logger.warn("#{log_prefix}#{task_status[:stopped_reason]}")
raise ContainerAbnormalEnd, error_message
end
raise ContainerAbnormalEnd, error_message unless task_status[:exit_code]
raise ExecutionFailure, error_message unless task_status[:exit_code] == 0

true
rescue ContainerAbnormalEnd
retry if task_status[:stopped_reason] =~ HOST_TERMINATED_REASON_REGEXP
retry if task_status[:stopped_reason] =~ SPOT_TASK_INTERRUPTED

if execution_try_count >= parameter.execution_retry
raise
Expand Down