Skip to content

feat(ingest): EC2 fleet with the WAL on instance-store NVMe - #937

Draft
Makisuo wants to merge 1 commit into
mainfrom
feat/ingest-ec2-nvme
Draft

Makisuo wants to merge 1 commit into
mainfrom
feat/ingest-ec2-nvme

Conversation

@Makisuo

@Makisuo Makisuo commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Depends on #936. Instance-store NVMe is wiped when an instance is replaced, so the WAL's S3 tier has to work before anything runs here.

What

An ECS-on-EC2 fleet for the ingest gateway, beside the Fargate one. MAPLE_INGEST_FLEETS picks which fleets run: fargate, ec2, or fargate,ec2. Unset means fargate, so merging this changes nothing in prd.

  • Hosts: c7gd.large (Graviton3, 2 vCPU / 4 GiB, 118 GB NVMe instance store), ECS-optimized AL2023 arm64 AMI, in an ASG behind an ECS capacity provider with managed scaling and managed draining. Access is through Session Manager, with no key pair and no port 22.
  • WAL on NVMe: userData formats the instance store as XFS, mounts it at /mnt/wal, and only then writes ECS_CLUSTER. A host whose disk didn't come up never gets a task. The task bind-mounts it at the gateway's default WAL dir.
  • Host networking, one task per instance: an awsvpc task on EC2 can't take a public IP, and this VPC has no NAT gateway by design. With host networking the task uses the instance's public IP. The ALB targets instances, and the instance security group admits only the ALB on 3474. A rolling deploy puts the new task on a fresh host (the old task holds the port) and drains the old one through the existing SIGTERM WAL drain. New AMIs roll out with deploys the same way.
  • Collector: its security group also admits the EC2 hosts' group.

alchemy patch (patches/alchemy@2.0.0-beta.77.patch)

  • ECS.Service: for tasks that aren't awsvpc, omit networkConfiguration (ECS rejects it) and create an instance target group instead of ip.
  • AutoScalingGroup: an update with no desiredCapacity leaves the live value alone. Before, every deploy reset it to minSize, which would undo ECS managed scaling.

Cutover plan

  1. Merge fix(ingest): send Content-Length on S3 PUTs so the WAL heartbeat lands #936, then this.
  2. Verify on a preview: label a PR preview + preview:collector + preview:ingest-ec2. ingest-preview-verify.sh targets the EC2 service when that label is set.
  3. prd: set MAPLE_INGEST_FLEETS=fargate,ec2 in Infisical prd and deploy. The EC2 ALB comes out as the ingestEc2ServiceUrl output.
  4. Point the proxied ingest.maple.dev CNAME at the EC2 ALB. Cloudflare flips it instantly, and pointing it back is the rollback.
  5. Once it's stable, set MAPLE_INGEST_FLEETS=ec2 to delete the Fargate service and its ALB.

Not verified

  • No alchemy plan or deploy has run yet; the local AWS session can't reach the deploy account. Step 2 is the first real check.
  • Unknown whether AWS_DEPLOY_ROLE_ARN has autoscaling:*, ec2:CreateLaunchTemplate*, iam:PassRole on the instance role, and ecs:CreateCapacityProvider.
  • The userData ordering (mount, then ecs.config) relies on the AL2023 ECS AMI starting ecs.service after cloud-init runs user data. That's the documented pattern, but it hasn't been run here yet.

Follow-ups

  • A per-host monitoring daemon (the Maple Docker agent) for host and container resource metrics in Maple.
  • Batched WAL fsyncs (group commit). NVMe makes the lane-mutex convoy from 2026-09-18 much less likely, but doesn't fix the structural problem.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Adds an ECS-on-EC2 fleet for the gateway beside the Fargate one, selected
by MAPLE_INGEST_FLEETS (fargate | ec2 | fargate,ec2; unset = fargate).

- c7gd.large (Graviton3, 118 GB NVMe) in an ASG behind an ECS capacity
  provider with managed scaling and managed draining. userData mounts the
  instance store at /mnt/wal and only then joins the cluster.
- Host networking, one task per instance: an awsvpc task on EC2 cannot
  take a public IP, and this VPC has no NAT by design. The ALB targets
  instances; the instance SG admits only the ALB on the gateway port.
- The WAL dir is a bind mount of the NVMe, so per-frame fsync no longer
  goes to network-backed Fargate storage.
- alchemy patch: ECS.Service omits awsvpcConfiguration and uses an
  instance target group for non-awsvpc tasks; an ASG update with no
  desiredCapacity leaves the live value to ECS managed scaling.
- Previews opt in with the preview:ingest-ec2 label.
@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant