Competitive-gap and hardening sweep + live verification close-out (supersedes #2)

.github/workflows/ci.yml

@@ -10,15 +10,17 @@ on:
 # and a superseded push to the same branch cancels its own stale run.
 permissions:
   contents: read
+# One Nucleus image version for every job, the staticcheck pin, and the
+# concurrency policy. NUCLEUS_VERSION is the engine version CI tests
+# against and production runs (teploy.yml pins the same tag;
+# docker-compose.yml defaults to it). One place to bump.
+env:
+  NUCLEUS_VERSION: v1.2.1
+  STATICCHECK_VERSION: 2025.1.1
 concurrency:
   group: ci-${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
-# The engine version CI tests against and production runs (teploy.yml pins
-# the same tag; docker-compose.yml defaults to it). One place to bump.
-env:
-  NUCLEUS_VERSION: v1.2.1
-
 # This repo embeds the built UI at cmd/observe/ui/dist/ as part of the
 # repo (single-binary story). CI only needs to build + test the Go code
 # — no pnpm, no Rust toolchain, no Nucleus build. Local devs run
@@ -39,14 +41,61 @@ jobs:
         run: go vet ./...
       - name: Validate installer
         run: sh -n scripts/install.sh
+      - name: gofmt
+        # Non-vendor, non-Neutron Go files only; any output fails the job.
+        run: |
+          out=$(git ls-files '*.go' | grep -v -e '^vendor/' -e '^Neutron/' | xargs gofmt -l)
+          if [ -n "$out" ]; then
+            echo "unformatted files (run gofmt -w):" >&2
+            echo "$out" >&2
+            exit 1
+          fi
       - name: Test
         run: go test ./...
-      # AUD-056 (round 2): the concurrency-heavy packages get the race
-      # detector — the batch-admission, WAL, replay-lock, and audit-head
-      # regressions from this audit series are exactly the class of bug a
-      # plain `go test` passes.
-      - name: Race regression tests
-        run: go test -race ./internal/ingest/... ./internal/audit/... ./internal/replays/... ./internal/auth/... ./internal/backup/...
+
+  # Full race detector over the main module (supersedes the old partial
+  # list in the go job). Separate job so it runs in parallel with the plain
+  # suite; DB-backed tests skip here (no Nucleus), which keeps it bounded.
+  go-race:
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+      - name: Race
+        run: go test -race ./...
+
+  # Known-vulnerability scan over the module graph. Separate job so a new
+  # advisory is its own red check rather than hiding in `go`. Intended to
+  # be BLOCKING once the first run is triaged (stdlib findings clear by
+  # bumping the Go toolchain, dependency findings by go.mod).
+  govulncheck:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+      - name: govulncheck
+        run: |
+          go install golang.org/x/vuln/cmd/govulncheck@latest
+          govulncheck ./...
+
+  staticcheck:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+      - name: staticcheck
+        run: |
+          go install honnef.co/go/tools/cmd/staticcheck@${{ env.STATICCHECK_VERSION }}
+          staticcheck ./...
 
   # AUD-056 (round 2): the classic replay tracker is a separately shipped
   # surface; these contract tests execute the real served script in a vm
@@ -64,6 +113,21 @@ jobs:
       # dependency-free pure modules — pinned here without an npm install.
       # (smoke.mjs / player-smoke.mjs need system Chrome and stay manual.)
       - run: node --test cmd/observe/tracker/replay-delta/test/sanitizer.test.mjs
+      # Survey widget and replay-capture contract tests (vm sandbox, no deps).
+      - run: node --test cmd/observe/tracker/test/*.test.mjs
+
+  # Pure-logic UI unit tests (helpers, API shapes). Needs Node >= 22.6 for
+  # --experimental-strip-types; no npm install required.
+  ui-unit:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+      - working-directory: ui
+        run: node --test --experimental-strip-types 'src/**/*.test.ts'
 
   sdk-go:
     runs-on: ubuntu-latest
@@ -235,7 +299,7 @@ jobs:
           go-version: '1.25'
       - name: Start pinned Nucleus
         run: |
-          docker run -d --name nucleus -p 55432:5432 \
+          docker run -d --name nucleus -p 127.0.0.1:55432:5432 \
             -e NUCLEUS_ALLOW_NO_AUTH=1 \
             -e NUCLEUS_ALLOW_INSECURE_CLUSTER=1 \
             -e NUCLEUS_ALLOW_INSECURE_REPLICATION=1 \
@@ -283,7 +347,7 @@ jobs:
           go-version: '1.25'
       - name: Start pinned Nucleus
         run: |
-          docker run -d --name nucleus -p 55432:5432 \
+          docker run -d --name nucleus -p 127.0.0.1:55432:5432 \
             -e NUCLEUS_ALLOW_NO_AUTH=1 \
             -e NUCLEUS_ALLOW_INSECURE_CLUSTER=1 \
             -e NUCLEUS_ALLOW_INSECURE_REPLICATION=1 \
@@ -340,11 +404,11 @@ jobs:
           node-version: '20'
       - name: Start pinned Nucleus
         run: |
-          docker run -d --name nucleus -p 55432:5432 \
+          docker run -d --name nucleus -p 127.0.0.1:55432:5432 \
             -e NUCLEUS_ALLOW_NO_AUTH=1 \
             -e NUCLEUS_ALLOW_INSECURE_CLUSTER=1 \
             -e NUCLEUS_ALLOW_INSECURE_REPLICATION=1 \
-            ghcr.io/neutron-build/nucleus:v1.1.0 \
+            ghcr.io/neutron-build/nucleus:${{ env.NUCLEUS_VERSION }} \
             start --host 0.0.0.0 --port 5432 --cluster-port 5433 --data /data --max-memory 512
           for i in $(seq 1 60); do
             if (exec 3<>/dev/tcp/127.0.0.1/55432) 2>/dev/null; then exec 3>&-; exit 0; fi

.github/workflows/helm.yml

@@ -0,0 +1,82 @@
+name: Helm chart
+
+# Lints and renders packaging/helm/observe and validates the rendered manifests
+# against the Kubernetes API schemas. Separate from ci.yml on purpose; only runs
+# when the chart (or this workflow) changes.
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'packaging/helm/**'
+      - '.github/workflows/helm.yml'
+  pull_request:
+    paths:
+      - 'packaging/helm/**'
+      - '.github/workflows/helm.yml'
+
+permissions:
+  contents: read
+
+env:
+  CHART: packaging/helm/observe
+  KUBERNETES_VERSION: 1.30.0
+
+jobs:
+  chart:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: azure/setup-helm@v4
+        with:
+          version: v3.16.2
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+      - name: Install kubeconform
+        run: go install github.com/yannh/kubeconform/cmd/kubeconform@v0.6.7
+
+      - name: helm lint (secrets supplied)
+        run: helm lint "$CHART" --strict -f "$CHART/ci/values-ci.yaml"
+
+      - name: helm template | kubeconform (all features on)
+        run: |
+          helm template obs "$CHART" -f "$CHART/ci/values-ci.yaml" \
+            | "$(go env GOPATH)/bin/kubeconform" -strict -summary \
+                -kubernetes-version "$KUBERNETES_VERSION"
+
+      - name: helm template | kubeconform (existingSecret, external database)
+        run: |
+          helm template obs "$CHART" \
+            --set secrets.existingSecret=my-observe-secrets \
+            --set nucleus.enabled=false \
+            --set nucleus.externalUrl=postgres://db.example:5432/observe \
+            | "$(go env GOPATH)/bin/kubeconform" -strict -summary \
+                -kubernetes-version "$KUBERNETES_VERSION"
+
+      - name: Render fails without JWT secret and session salt
+        run: |
+          if helm template obs "$CHART" >/tmp/out.txt 2>/tmp/err.txt; then
+            echo "chart rendered without OBSERVE_JWT_SECRET/OBSERVE_SESSION_SALT" >&2
+            exit 1
+          fi
+          grep -q OBSERVE_JWT_SECRET /tmp/err.txt
+
+      - name: Schema rejects unpinned Nucleus and unknown keys
+        run: |
+          if helm template obs "$CHART" -f "$CHART/ci/values-ci.yaml" --set nucleus.image.tag=latest >/dev/null 2>&1; then
+            echo "nucleus.image.tag=latest was accepted" >&2; exit 1
+          fi
+          if helm template obs "$CHART" -f "$CHART/ci/values-ci.yaml" --set replicaCount=2 >/dev/null 2>&1; then
+            echo "unknown key replicaCount was accepted" >&2; exit 1
+          fi
+
+      - name: Invariants (single replica, Nucleus isolated)
+        run: |
+          helm template obs "$CHART" -f "$CHART/ci/values-ci.yaml" > /tmp/all.yaml
+          test "$(grep -c '^  replicas: 1$' /tmp/all.yaml)" = 2
+          grep -q 'kind: NetworkPolicy' /tmp/all.yaml
+          # Nucleus is never given an Ingress or non-ClusterIP Service.
+          if awk '/kind: Ingress/,/^---/' /tmp/all.yaml | grep -q 'obs-observe-nucleus'; then
+            echo "an Ingress references Nucleus" >&2; exit 1
+          fi

.gitignore

@@ -71,3 +71,6 @@ __pycache__/
 
 # Local compose overrides are, by definition, local.
 docker-compose.override.yml
+
+# Runtime data dir (audit.key, queues) when the server runs from the repo
+data/

AUDIT_OPEN.md

@@ -14,6 +14,126 @@ report lives outside the repo) — remediation record below. Round 4: audit
 passes 1-5) are closed history; their one surviving item is folded into F16
 below.
 
+## S1 (2026-10-03/04) - Competitive-gap and hardening sweep: verified live 2026-10-04
+
+Scope: the full list is in CHANGELOG `[Unreleased]`. The sweep itself was
+built without Nucleus (the sandbox's network policy blocked the image),
+so 214 integration tests were skipped at commit time. The live
+verification ran 2026-10-04 against ghcr.io/neutron-build/nucleus:v1.1.1
+(docker, the CI shape; image digest f8d4362c, binary banner v1.0.2 - the
+known upstream tag/version mismatch), and was RE-RUN after the v1.2.1 pin
+merge against ghcr.io/neutron-build/nucleus:v1.2.1: fresh ladder boot plus
+the full serial suite, 53/53 packages green on both engines.
+
+**Live verification (2026-10-04, green):**
+- Migration ladder applies fresh (001-063) AND on a populated store
+  upgraded from 047 (`TestMigrationsUpgradePopulatedStore`, added this
+  session: the six ALTER-ADD tables rebuilt in their pre-048 shape,
+  populated, then upgraded; columns, backfills, row survival and the
+  write-shaped probe all checked).
+- Full serial suite green against live v1.1.1 (`OBSERVE_NUCLEUS_URL`,
+  `OBSERVE_REQUIRE_NUCLEUS=1`, `-p 1`), including the previously-skipped
+  integration coverage: span links, issue merge/assign/unmerge, static
+  cohorts + cohort_id on funnels/retention, persons
+  properties/alias/erase, experiment metrics, replay list filters, log
+  `lq` (keyset paging + attribute terms), trace attribute filters +
+  orphan traces, the new alert metrics, retention incl. the
+  `metric_points` nanosecond cutoff, and backup/restore round-trips.
+- `scripts/audit.sh` 121/121 against a seeded live server.
+- UI rebuilt from the pinned submodule revision (pnpm 9.15.4, the `.ci`
+  pin) and committed; the delta-recorder bundle rebuilt with
+  console/network capture. The four node CI legs green (ui unit, tracker
+  contracts, replay-delta sanitizer fixtures, tracker tests - the last
+  confirmed wired into ci.yml).
+- govulncheck clean after the dep bumps; staticcheck clean after its
+  first full triage; `go vet` clean; `helm lint` + full `helm template`
+  green on their first-ever run (the chart's own secrets guard refuses
+  an unconfigured render - by design).
+
+**Defects the live run surfaced (fixed in this session):**
+- surveys `ValidateAnswers` rejected Go-native `int` values (only
+  `float64`, the JSON-wire type); the sweep's own Nucleus-gated tests
+  were the first to exercise the path and failed.
+- The legacy-results golden is architecture-sensitive: gc fuses
+  multiply-add on arm64 but not amd64, so p-values differ by 1 ULP and
+  the byte-exact pin failed - including on the arm64 CI job. Compared
+  with 1e-12 relative tolerance on float leaves now.
+- The committed delta-recorder bundle was stale relative to its
+  sources.
+- `audit.sh` was a pre-reorg artifact (absolute paths into the old
+  checkout, a pre-policy credential, expectations for retired
+  behaviors); modernized and green.
+- Dependency advisories were symbol-reachable: grpc GO-2026-6443 and
+  GO-2026-6348 (otlpgrpc.Serve), pgx GO-2026-5004 (explorer queries);
+  bumped to grpc v1.83.2 / pgx v5.9.2, vendor regenerated from the pin.
+
+**Closed this session (was open in the sweep's close-out):** the UI
+build and `ui-freshness` expectations; the restore gap (`observe
+migrate` applies the ladder without seeding - isolated-restore runbook
+is migrate -> restore -> start, verified against a fresh database); a
+global security-header middleware (nosniff, referrer-policy,
+X-Frame-Options SAMEORIGIN, conditional HSTS when the public URL is
+https; set-if-absent so handler policies win); `OBSERVE_SMTP_PRIVATE_HOSTS`
+for internal email relays (resolve-once dial-IP retained); the Sentry
+wire `timestamp` honored (client time stored; more than 24h of future
+skew refused at admission with a 400).
+
+**Open items (owner decisions, upstream, or documented posture):**
+- L9 remains open, reframed by the runner contract - see the L9 entry:
+  applied migrations are checksum-frozen, so in-place conversion of
+  048-056/010 is refused on every store that already ran them. The live
+  `llm_traces` repair on infra-home is still the owner call.
+- Surveys unique index on (site_id, survey_id, client_id): UPSTREAM-
+  BLOCKED - Nucleus v1.1.1 parses `CREATE UNIQUE INDEX` but does not
+  enforce it (duplicate keys accepted; logged in the umbrella
+  `_internal/UPSTREAM_BUGS.md`, 2026-10-04). Single-process dedupe
+  stays via the submit lock; `once` stays client-side; authoring UI has
+  no targeting editor.
+- Integrations: issue-fire delivery rides the in-memory queue (the
+  alert path has the durable outbox); alert recovery cannot resolve
+  PagerDuty/Jira/GitHub tickets (senders are create-only); no issue
+  deep link in payloads.
+- Notification buckets, spike windows, persons-properties limiter and
+  survey limiters stay per-process (reset on restart, per replica) -
+  the standing single-process posture (AUD-018).
+- `NUCLEUS_ALLOW_NO_AUTH=1` posture unchanged (network isolation is the
+  control; engine auth is an upstream question).
+- Persons residuals unchanged: erase writes a tombstone but event rows
+  are not deleted (no verified per-key DELETE), ingest does not check
+  tombstones, browser `identify()` is not wired to the properties
+  endpoint, no un-merge endpoint, and the properties route's 404-vs-200
+  reveals whether a guessed id has events.
+- Replay: per-session capture caps are enforced client-side only.
+- Trace/log attribute filters still verify in Go over a bounded
+  candidate window (the engine has no JSONB extraction on TEXT and no
+  bind parameters in JSON operands - upstream notes).
+- Experiments: fixed-horizon only; auto-exposure from /flags/evaluate
+  not wired; secondary metrics not multiplicity corrected;
+  `RecordMetricKeyed` dedupe is check-then-insert. Flags: `/flags/config`
+  exposes full targeting rules to any telemetry key (product decision);
+  modulo bias documented; per-flag fail-open override deferred.
+- Sentry endpoint: `timestamp` now honored; no OPTIONS route (SDKs send
+  simple requests).
+- Alerts: no per-service filter and no `metric_value` metric (need an
+  `alert_rules` schema change); `uptime_failures` is a window count, not
+  consecutive failures; query slots (global 8 / site 4) are shared by
+  stats, logs, metrics, traces and cohorts, so a dashboard firing many
+  heavy panels at once can see 429s - tune
+  `OBSERVE_QUERY_SITE_CONCURRENCY`.
+- `experiment_settings` and `cohort_members` are deliberately excluded
+  from backup version-collapsing (tie semantics).
+- Multi-tenancy/orgs/SCIM, per-site RBAC, SAML, HA/clustering, PromQL:
+  design notes only (docs/design/); owner decisions.
+- Helm chart linted and fully rendered for the first time (2026-10-04,
+  0 failures); the Nucleus container user/filesystem layout inside a
+  real cluster remains unverified.
+- Watch item: legacy ORDER BY on an aliased text cast in `logs.go`
+  legacySearchQuery and `tracing/query.go` - if a live run shows a
+  textual sort, log the Nucleus conformance bug upstream.
+- Flaky once under load: `internal/ingest`
+  TestDiskQueue_AppendAfterCheckpointIsStillFlushed (125 ms sleep vs
+  25 ms fsync loop) - passed 20/20 alone.
+
 ## L10 (2026-09-27) — Fixed in source: LLM traces omitted from retention
 
 LLM traces, including prompt/completion payloads, were never expired by the
@@ -46,18 +166,30 @@ regenerable LLM traces - owner call), (b) whether new ALTER-ADD migrations
 should be refused by a test, and (c) a migration test against a populated,
 upgraded store rather than a fresh one.
 
+**Update 2026-10-04 (close-out session):** (b) and (c) are DONE - the guard
+test refuses ALTER-ADD from 057 on, and
+`TestMigrationsUpgradePopulatedStore` replays the populated upgrade path.
+The "convert 048/049/051/052/054/056 and 010 in place" plan is additionally
+REFUSED BY THE RUNNER: every applied script is checksum-verified at each
+Migrate (`migration %d has been modified since it was applied ... restore
+the applied script or write a new migration`), pinned executably by
+`TestAppliedMigrationsAreChecksumFrozen`. Rewriting those files would break
+the boot of every store that already applied them (infra-home included).
+
 Upstream resolution (2026-10-04): the engine defect is fixed in Nucleus
 v1.2.1 (atomic ALTER rewrites; half-applied ALTERs no longer recorded as
 applied; a new engine-side refusal rejects ALTERs that follow writes to the
 same table in one transaction). This repo pins v1.2.1 in teploy.yml, CI
 (NUCLEUS_VERSION) and docker-compose; observe's full suite plus a fresh
 migration boot verified against that image. Re-scoped: (b) is no longer
-needed for correctness on v1.2.1+ — the engine itself enforces the unsafe
-ordering; (c) is satisfied by `TestMigrationsUpgradePopulatedStore` on the
-S1 branch; (a) remains open as an owner op and now has an in-place path
-(the `catalog.json` repair in Neutron's
+needed for correctness on v1.2.1+ - the engine itself enforces the unsafe
+ordering (the guard test stays as belt-and-braces and for older engines);
+(c) is satisfied by `TestMigrationsUpgradePopulatedStore` on this branch;
+(a) remains open as an owner op and now has an in-place path (the
+`catalog.json` repair in Neutron's
 `nucleus/docs/HANDOVER_TEPLOY_OBSERVE_ADD_COLUMN.md`) that preserves the
-rows instead of losing regenerable traces.
+rows instead of losing regenerable traces - the rename-aside fallback that
+loses the 12,080 traces stays documented above as the last resort.
 
 ## Round-4 register (2026-09-19 audit, 45 findings R01..R45)