shardVPN v2: Tailscale exit node with a Lambda control plane

[jerridan]

Replaces v1 (Terraform 0.11 + OpenVPN + a hand-run certificate authority) with
an on-demand Tailscale exit node driven from a phone. Nothing runs between
trips.

v1 is deleted, not migrated — its HCL predates 0.12 and won't parse against a
current Terraform, and every AMI, instance type and provider block in it was
stale. It survives in git history on master.

How it works

Three actions — up, down, status — sent from an iPhone over HTTPS to a
Lambda that is the whole control plane. up launches a bare AL2023 arm64
instance that turns itself into a Tailscale exit node with no inbound ports
before it ever advertises as selectable. An hourly EventBridge sweep is the
only thing running unattended: it reaps expired nodes, terminates duplicates,
reconciles state, and emails about idle ones.

What's here

lambda/shardvpn/	10 modules — stdlib plus runtime-provided boto3, no build step
tests/	182 tests, fully offline (botocore.stub.Stubber, patched urlopen)
terraform/	one root module: Lambda, Function URL, IAM, Scheduler, SNS, SSM, alarms, budget
docs/ios/shardvpn.js	Scriptable client with vendored HMAC-SHA256
.github/workflows/ci.yml	4 offline jobs — no AWS credentials in CI

Design decisions worth knowing

• Auth is HMAC + timestamp on a NONE-auth Function URL. SigV4 from a
  phone was rejected as unworkable. Every verification failure returns a
  byte-identical 403 from a single return site; SSM/KMS unavailability is a
  distinct 503, with a written argument for why that isn't an oracle.
• TTL defaults to no expiry. Forgotten nodes are surfaced by a 24h idle
  email, not killed. Duplicates are killed.
• Cloud-init fails closed. It verifies IP forwarding and the exit-node
  advertisement before advertising, and on failure withdraws, leaves the
  tailnet, and self-terminates — because a node that joins and advertises but
  doesn't forward looks perfectly healthy from the phone.
• No secret reaches Terraform state. aws_ssm_parameter stores values in
  plaintext even for SecureString, so the three secrets are placeholders
  with ignore_changes and are populated out of band.
• IAM is least-privilege where AWS permits it: RunInstances split into a
  request-tag-conditioned instance statement plus supporting resources, bounded
  to t4g.*; TerminateInstances conditioned on the role tag. Describe* and
  CloudWatch metrics are Resource: "*" because they support nothing narrower,
  and that's documented rather than hidden.

docs/superpowers/specs/2026-08-01-shardvpn-v2-design.md records the reasoning
behind every tradeoff, including the ones rejected.

Before this is usable

Two things need a human and live AWS, both in the README:

1. Decommission v1. Its IAM user carries IAMFullAccess plus a long-lived
   access key, and the shard-vpn-keys bucket still exists. Until that's done,
   "no long-lived AWS access keys anywhere" isn't true.
2. End-to-end verification. Including a deliberate broken-forwarding run,
   measuring the real idle NetworkOut baseline to replace the guessed
   threshold, and confirming Self.ExitNodeOption — the one key in
   userdata.sh that is still an unverified guess.

CLAUDE.md lists what's settled, what's open, and the accepted known issues.

🤖 Generated with Claude Code

[jerridan]

Code review — v2-tailscale-exit-node (whole branch vs master)

Baseline verified before reviewing: uv run pytest -q → 182 passed, ruff check clean. Findings ordered by severity. Items already recorded under Known and accepted in CLAUDE.md are not repeated here.

---

1. HIGH — a swallowed per-region scan failure clears the pointer for a still-running node

lambda/shardvpn/watchdog.py:184

scan() catches every exception and returns (region, []) with no entry in summary["failures"]. Step 4 then treats that as "no nodes exist" and at line 312 does write_pointer(..., None).

The module's own docstring (lines 137–142) argues that valid_regions must abort the sweep because undiscovered absence is indistinguishable from real absence. The identical argument applies to a per-region scan failure, and is not honoured there.

Failure scenario: ec2:DescribeInstances throttles in eu-west-1, where the live node is running. The sweep clears /shardvpn/current-node. A subsequent down takes the pointer-is-None branch (handler.py:179), which only scans the default region, returns absent, and the eu-west-1 node keeps billing. A subsequent up launches a second node. Nothing is emailed, because the failure was never recorded, so _notify_failures stays quiet.

test_sweep_survives_a_region_that_errors pins the swallow, but places the node in the other region, so it never exercises this path.

Minimum fix: record the failure in summary["failures"] and set a scan_complete flag that gates the pointer-clearing branch.

---

2. MEDIUM — no ERR trap, so every failure before tailscale up leaves a running, billing instance

lambda/shardvpn/userdata.sh:8

The fail-closed logic (tailscale logout + shutdown -h now) exists only at lines 115+, i.e. after the tailnet join at line 98. With set -euo pipefail and no trap, a non-zero exit from dnf install, dnf config-manager --add-repo, or systemctl enable --now shardvpn-gro.service (which fails whenever ethtool -K <dev> rx-udp-gro-forwarding on is unsupported on the attached NIC) aborts cloud-init while the instance stays up.

_up has already written the pointer, so the sweep sees pointer.instance_id == instance_id and raises no orphan alert; with the default TTL of none nothing reaps it; and should_alert's young-node guard suppresses the idle email for the first 24h. Net: a silent, indefinitely billing instance.

The file's own comment at lines 19–22 identifies exactly this risk for dnf-plugins-core, but only reorders the install rather than closing the failure mode.

Fix: trap 'shutdown -h now' ERR at the top — InstanceInitiatedShutdownBehavior=terminate is already set, so that covers all of it.

---

3. MEDIUM — boto3.client used as the client factory inside a ThreadPoolExecutor

lambda/shardvpn/watchdog.py:191

The default client_factory is boto3.client (line 126), i.e. the module-level default session, and scan creates ~34 clients concurrently across 10 threads. Creating clients from a single session across threads is the pattern AWS documents as not thread-safe (botocore loader / data-cache races).

Any such race surfaces as an exception inside scan, which is silently converted to "no nodes in this region" — directly feeding finding #1.

Fix: use a per-thread boto3.session.Session(), or pre-create the region clients on the main thread before submitting.

---

4. MEDIUM — SNS service principals granted Publish with no aws:SourceAccount / aws:SourceArn condition

terraform/sns.tf:18

aws_sns_topic_policy replaces the default owner-only policy with one allowing cloudwatch.amazonaws.com and budgets.amazonaws.com from any account. The topic ARN is exported as a non-sensitive output (outputs.tf:7).

Anyone who learns the ARN can point a CloudWatch alarm in their own account at this topic and inject arbitrary text into the single channel this system uses to tell you a node is still billing. This is the same confused-deputy class that iam.tf:157 explicitly guards against for the scheduler role.

Separately, the budgets.amazonaws.com grant is dead weight: alarms.tf:36–48 sends budget notifications to subscriber_email_addresses directly and never references the topic. The sns.tf comment ("AWS Budgets ... publish to this topic") is inaccurate, and the grant can simply be dropped.

---

5. LOW — the Tailscale auth key is minted before four fallible calls

lambda/shardvpn/handler.py:241

mint_auth_key runs at line 241. The keyword arguments to ec2ops.launch (lines 246–249) — resolve_ami, get_parameter, ensure_security_group — are all evaluated afterwards, as is run_instances itself. Any of them failing leaves a minted, pre-authorized, ephemeral, tag-scoped key live for 600s with nothing to revoke it.

ensure_security_group in particular raises a deliberate RuntimeError (ec2ops.py:65) when the SG has hand-added ingress — a repeatable condition, so every retry mints another orphan key.

Fix: move the mint to immediately before run_instances, or revoke on failure.

---

6. LOW — the HMAC is computed over str(int(raw_ts)), not the header value as sent

lambda/shardvpn/auth.py:66

_TIMESTAMP_RE is \A[0-9]{1,11}\Z, which accepts leading zeros, but the signed message is built with f"{int(raw_ts)}.". A client sending 01700000000 signs 01700000000.<body> while the server verifies 1700000000.<body>. The result is an indistinguishable 403 with no way for a legitimate client author to diagnose it.

The shipped Scriptable client can't hit this, but any reimplementation can.

Fix: either sign raw_ts verbatim (the regex already constrains it), or reject a leading zero.

---

7. LOW — _describe uses find_nodes(...)[0] and ignores pointer.instance_id

lambda/shardvpn/handler.py:120

When the region holds an unresolved duplicate, DescribeInstances ordering — not the pointer — decides which node status describes. The response can report the untracked node's public_ip / hostname, and the tailnet lookup runs against that hostname.

Worse: _up's idempotency short-circuit (lines 195–198) then returns that node and, unlike the recovery branch at line 233, never repairs the pointer.

Self-heals within an hour via the sweep, but the reported document is wrong in the meantime.

---

8. LOW — the kms:Decrypt verification recipe misdiagnoses the placeholder case

README.md:147

handler.py:66–68 returns a byte-identical 503 {"error":"cannot verify request"} when the secret still equals PLACEHOLDER_SIGNING_SECRET. The README states unconditionally that 503 "means the role needs an explicit kms:Decrypt statement".

A reader who runs the curl before the put-parameter above it — or whose put-parameter went to the wrong region/account — is sent to edit iam.tf for a non-problem, while the real state is that the control plane's signing secret is a literal published in this public repo.

Fix: distinguish the two, e.g. return a different marker for the placeholder branch, or add the caveat to the README step.

---

9. LOW — the backoff sleep runs after the final attempt

docs/ios/shardvpn.js:162

The await new Promise((resolve) => Timer.schedule(...)) is unconditional at the end of the loop body, so a request that fails all four attempts waits an extra 8s (15s of backoff total) before returning an error the user could have seen immediately.

Fix: guard with if (attempt < MAX_ATTEMPTS - 1). The inline comment ("1s, 2s, 4s") also understates the actual schedule.

---

Checked and judged correct

HMAC message framing vs. the JS client (byte-identical), extract_body base64 handling, string.Template.safe_substitute leaving $(...) / ${NETDEV} / ${advertised} intact, is_expired / should_alert / parse_ttl boundaries, _order_survivors' refusal-to-guess, archive_file excludes, and IAM action/resource coverage for every call the Lambda actually makes.