Repository navigation
Competitive-gap and hardening sweep + live verification close-out (supersedes #2) - #4
Merged
Merged
Conversation
…eck; pin Nucleus; hardening docs
- queryguard: Limiter.Acquire read l.global / l.sites[siteID] after
unlocking l.mu to format refusal messages (concurrent map access).
Capture under the lock; add multi-site regression test.
- CI: full go test -race job, gofmt check, govulncheck and staticcheck
jobs, single NUCLEUS_VERSION (v1.1.1) for all jobs, loopback-only
published Nucleus port.
- compose: pin Nucleus to ${NUCLEUS_VERSION:-v1.1.1}; Caddy profile now
uses packaging/caddy/Caddyfile with HSTS, nosniff, Referrer-Policy and
frame-ancestors. systemd nucleus binds 127.0.0.1.
- docs/operations/hardening.md.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…_issues, service_dependencies These tables grew unbounded. RetentionPolicy gains a Unit (ms default, ns for metric_points.ts_ns) so the cutoff is scaled to the column; a ms cutoff against ns would delete nothing. New validated env knobs OBSERVE_METRICS_RETENTION_DAYS (30), OBSERVE_INFRA_RETENTION_DAYS (30), OBSERVE_UPTIME_RETENTION_DAYS (90), all >= 1. performance_issues (90d by last_seen) and service_dependencies (30d by ts_bucket, matching service_stats) use fixed windows. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
… event; cap cohort size resolveFilters used to log and drop the cohort filter when the resolver failed, so charts labeled with a cohort showed all traffic. It now returns an error mapped to HTTP: 503 for resolver/store failure (raw error logged, not echoed), 422 for a cohort over MaxFilterMembers (30000; the filter expands to one SQL parameter per member, wire limit 65535), and the refusal's own status for queryguard refusals (new internal/guardmap). Property rules with operator "!=" matched any user with at least one non-matching event. They now mean "no event has this value", computed as (all identified users) minus (users matching "="), with two plain DISTINCT queries and a Go-side subtraction. SQL builders are unit tested. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…comes Upload now stores maps under a canonical name (no query, fragment, host or leading slash) and lookup tries the exact name, the stripped URL, the canonical path, shorter path suffixes and the basename in a fixed, bounded order (existing maps stay reachable via the exact-name candidate). Resolved frames keep the minified position in orig_filename/orig_line/ orig_col with resolved:true; lineno/colno still carry the original source position. Resolved frames are skipped on re-resolution (idempotent). Resolution outcomes (hit, miss_no_map, miss_no_mapping, error) are counted and surfaced at /healthz under "sourcemaps"; errors are logged at debug, rate limited, instead of being swallowed. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…window Every enabled /flags/evaluate wrote a row. Nothing reads flag_evaluations (experiment exposure and conversion counts come from experiment_exposures / experiment_conversions, written only by the explicit RecordExposure and RecordConversion calls), so the table is an audit/volume trail and one row per key per window is enough. Bounded in-memory LRU+TTL (hashed 32-byte keys, default 5 min / 50k entries; OBSERVE_FLAG_EVAL_DEDUP_SECONDS, 0 disables; OBSERVE_FLAG_EVAL_DEDUP_MAX). A variant change is always recorded; a failed write is forgotten and retried; written/deduped/failed/evicted counters surface under /healthz "flags". The INSERT itself is unchanged, moved behind a seam. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…ooks and design notes Fix Sentry source-map curl field, shim/breadcrumb claims, state no Sentry protocol ingest; PostHog cohorts/flags/surveys truth; README integrations, SAML, alert metrics, surveys. Add install/upgrade/backup-restore runbooks and NOT-BUILT design notes (per-site RBAC, SAML, HA, PromQL). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…tDistinctID helper RecordConversion stored the variant of the user's LATEST exposure, so a user exposed to several arms was attributed to whichever exposed last and the answer moved as exposures arrived. It now reads the FIRST exposure (documented on the method). Results does not read the stored variant, so reported counts are unchanged. Exposure/conversion rows carry the caller's raw user_id while events carry the hashed distinct_id. ExperimentService.EventDistinctID derives the event-pipeline id (per-site salt, raw opt-out, global-salt fallback, via internal/identity) so callers can join; stored values are not rewritten. Wired with WithPrivacy(siteSvc.PrivacyConfig, salt) in main.go. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…ation 057) AttrsToMap (tracing, metrics) and otlpAny.text (logs) tested the scalar fields against their zero values, so attributes set to false, 0, 0.0 or "" were dropped. AnyValue now carries an unexported kind tag set by the JSON and protobuf decoders; kind-less literals keep the old fallback. Tracing also folds array/kvlist/bytes values to deterministic text (sorted compact JSON / hex, identical across wire formats, depth-bounded) and marshals AnyValue by kind so span-event JSON keeps zero values. Span links were dropped on ingest. Both OTLP trace paths now parse links (trace_id, span_id, trace_state, attributes), bounded to 128 links per span and 32 attributes per link, with truncations counted (IngestResponse links_truncated + warn log). Stored in new span_links table (migration 057, CREATE TABLE IF NOT EXISTS) inserted via chunked INSERT in the same ingest transaction as spans. QueryService.GetTraceLinks/GetSpanLinks added; trace detail spans gain an optional `links` field (omitted when empty). span_links retention is not wired (internal/jobs not touched here). UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
IntegrationService.Fire had no caller outside the test/replay endpoints, so configured integrations never fired for real issues. IssueService now emits IssueEvents (first occurrence; resolved issue reopened, detected from the real status on a cache miss and from a resolved flag on the grouphash cache entry on a hit) through an IssueNotifier callback. internal/errors does not import internal/integrations; cmd/observe/integrations_wiring.go connects them. Delivery runs on a bounded in-memory queue (256, drop-oldest with counter and loud log), 4 workers, a per-delivery timeout, and a per-(integration, issue) cooldown of 15 minutes so a flapping issue cannot spam. Counters are surfaced at /healthz under issue_notifications. Each attempt is still recorded in integration_deliveries. Ingest never waits: Notify only enqueues and a notifier panic is recovered. Hardening on the shared fire path: transport errors are redacted (Slack webhook URL is the credential) before being logged or stored, Slack/Jira URLs are scheme-validated, and the email channel now resolves smtp_host once, refuses non-public addresses and dials the pinned IP (SSRF and DNS rebinding). Alert title text is stripped of control characters. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
SearchLogs, LogStats, Histogram, ListMetrics and QuerySeries now take a concurrency slot, run under the time budget, and refuse with the labeled queryguard refusal (429 concurrency/rows, 504 time) instead of running unbounded. QuerySeries used to load every raw point in range into Go memory: its SQL now carries LIMIT budget+1 and one row past the ceiling is a refusal, never a truncated series. Histogram and ListMetrics get the same cap; SearchLogs refuses a page (limit+offset) beyond the row budget. internal/guardmap gains Guard (Begin / RowRefusal / DeadlineError, the pattern of internal/query/guard.go packaged for reuse, nil-safe with default budgets) and HTTPError, which maps refusals to problem responses. Log handlers and the metrics list/query/series handlers use it; other errors keep their historical statuses. Shared limiter wired in main.go. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…attribution, log/metric guards
UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
Chart packaging/helm/observe (Observe + Nucleus StatefulSets, replicas fixed at 1), NetworkPolicy isolating the unauthenticated Nucleus, secret handling that fails templating without JWT secret/salt, values.schema.json, ingest split Service/Ingress, docs/operations/kubernetes.md and a helm.yml workflow (lint + kubeconform). Chart was NOT rendered or linted locally: helm is not available in this environment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
Case-insensitive recursive key denylist plus value patterns (Luhn cards, JWT, Bearer, query-string keys) applied in ErrorBuffer.Push before the record is frozen for the WAL. Grouping hash is pinned from the raw input (PreGroupHash, server-set) so fingerprints are unchanged; digest is over the scrubbed canonical payload so inbox retries still dedupe. Env: OBSERVE_SCRUB_KEYS, OBSERVE_SCRUB_DISABLE. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…ntegrations Adds trace_error_rate, trace_p95_ms, log_error_count and uptime_failures rule metrics (site-wide, window-bounded, parameterised, CAST AS BIGINT). Empty windows are no-data, never 0. API whitelist now derives from platform.AlertMetricSet; UI dropdown and docs updated. Alert FIRE edges can route to site integrations behind OBSERVE_ALERTS_TO_INTEGRATIONS=true (default off, detached goroutine, never blocks the engine). UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…agent-aadeaf29cbe3176b8
…agent-a7552b844edb345f3
…agent-a16d5035d16eefcb6
…agent-a8239223aeb271e33
Register issue_merges, issue_assignments, person_properties, person_aliases and person_tombstones in replacingKeys so backup dumps one latest row per key. Writers take max(version)+1 so argMax is deterministic and a latest-version tombstone/soft-delete row is preserved. cohort_members (tie semantics differ) and experiment_settings (plain mergetree) are deliberately left out, documented in the registry. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…ordMetric readMetricEvents and the metric-event INSERT use dbutil.IntParam with CAST($n AS BIGINT). RecordMetricKeyed takes an optional event_key (<=64 chars) that derives a stable event_id and skips the insert when it already exists, so client retries do not over-count count goals (check-then-insert, not atomic; documented). RecordMetric delegates with no key. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…state Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…e target ResolveIssue now resolves a merged source to its target before bumping or notifying: a regression of a resolved target is notified for the target, a hidden source is never notified or bumped, and spikeExempt looks at the target's status (snoozed issues are status=resolved). The per-event COUNT(*) over error_events in attributeMerged is replaced by a TTL-bounded in-process count. Lifecycle storage sits behind a small interface so the decisions are unit tested without Nucleus. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…umeric ordering, backup collapse, experiment idempotency
…etric events 730 days Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
Persons properties (telemetry key is public): never return stored values (only keys_written), refuse replace on the key route, require the person to already have events in the site (404), 20 writes/min/person limiter, 50-key cap documented. Replace moves to editor+ POST /persons/properties/replace. Surveys: respond handler maps only PublicError to 400 with fixed text and returns a generic 500 otherwise (logged server-side); dedupe now keyed by (site, survey, client) under a striped in-process lock; user_id capped at 128 bytes and validated; per-site limiter buckets only for existing sites (TTL cache), unknown sites get per-IP limit only. UNVERIFIED against live Nucleus (integration tests skipped in this environment). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…c auth, JSON-in-string) Keys are NFKC-folded with a confusables map before matching. Free-text 'password: x', 'Authorization: Basic ...' and secrets inside JSON-in-a-string values (depth and node bounded) are redacted. Fingerprint, Selector, ReleaseTag, Breadcrumb.Category and StackFrame.Function get token patterns only; grouping is unaffected because the hash is pinned from the raw input before scrubbing (golden test still passes). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…ntrol syntax Per-integration and per-site token buckets (OBSERVE_ISSUE_NOTIFY_PER_HOUR, default 10/hour burst 10, 0 = unlimited) bound a distinct-fingerprint flood; denials are counted as suppressed_rate and one bounded summary notification is sent when suppression starts. Slack text escapes &, <, > and backticks so <!channel>, <@U..> and link syntax in attacker-controlled fields are inert; bidi/zero-width characters are stripped and field lengths capped. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…nt ids Pre-auth the envelope DSN peek now reads at most 64 KiB; the rest of the body is read only after the key validates. New internal/authguard wraps the key validator with a 30s sha256-keyed negative cache (10k bounded, successes never cached) and a per-IP failed-attempt limiter (60/min, then 429). Events lacking their own event_id derive a deterministic per-item id (header id, then header_<idx>) so events 2..N of an envelope no longer collide. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
Key validation moves to grpc.InTapHandle so an unauthenticated peer's message is never read or unmarshalled; the unary interceptor remains as a backstop. Default receive cap is 4 MiB (OBSERVE_OTLP_GRPC_MAX_RECV_MB, ceiling 10), MaxHeaderListSize 16 KiB, 128 streams per connection, plus total (1024) and per-IP (64) connection caps. The validator is wrapped by authguard (negative cache + per-IP failure limit). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
…ndler, refuse ALTER ADD COLUMN in new migrations, run new JS tests in CI Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rpvf5mrWnTXrcooQQyQCD3
….sh dist path ui-sync against the pinned Neutron submodule revision (pnpm 9.15.4, the .ci pin); the delta recorder bundle was stale relative to its sources (console/network capture wiring). audit.sh DIST pointed at a pre-reorg absolute path; now resolved from the script location.
…sum pin (L9)
TestMigrationsUpgradePopulatedStore rebuilds every table migrations
048-063 ALTER (pre-048 shape derived from information_schema, fixture
rows copied in, marker rows seeded), drops what those migrations create,
rewinds the ledger past 047, and re-applies the chain - the upgrade path
the live L9 corruption rode on, which empty-table runs cannot see.
Asserts added columns, backfill values, marker survival, and the
write-shaped column probe that failed on the live store.
TestAppliedMigrationsAreChecksumFrozen pins the runner constraint that
rules out rewriting applied migrations in place: editing an applied
script is refused before any mutation ('has been modified since it was
applied'), so the 048-056/010 ALTER-ADD conversion is only possible via
new migrations or owner-led ledger surgery on a quiesced store.
…/5004 All three advisories are symbol-reachable from this code (otlpgrpc.Serve, explorer's pgxpool queries). Vendor regenerated from the pinned submodule revision per the go.mod procedure; parity against the pin verified.
…pts Go-native numbers staticcheck had never run; this clears its first full pass: dead code (ReplayFilter.active, sentrycompat logger, sourcemaps abs/genID, decodeWALLine, unmarshalJSON, correlation handler pair, bench leftovers), struct-conversion rewrites, nil-Context tests, determinism pins in two-call form, ST style findings. infraReportHandler stays as the F07 reference twin with a lint:ignore. ValidateAnswers now accepts int/int64 besides float64 - SubmitResponse callers outside the JSON wire path pass Go-native numbers; the sweep's own Nucleus-gated tests were the first to exercise it. The legacy-results golden compares floats with 1e-12 relative tolerance: gc fuses multiply-add on arm64 but not amd64, so p-values differ by 1 ULP across architectures (the arm64 CI job runs this suite).
The script was a point-in-time receipt tool for the pre-reorg workspace: absolute paths into the old checkout, single-quoted bash -c wrappers that broke on the space in 'Code Projects', migration paths from before the schema package existed, a hardcoded admin password that predates the 8-character policy, and expectations for retired behaviors (query-string tokens on streams and exports). Path-resolved via REPO/DIST/OBSERVE_BASE + OBSERVE_USER/OBSERVE_PASS env; retired behaviors now assert their refusal (401) and the positive check rides the stream-ticket flow; the marketing page check became the app-shell check. 121/121 against a live seeded server on Nucleus v1.1.1.
…imestamps, observe migrate - securityHeadersMiddleware: nosniff, referrer-policy, X-Frame-Options SAMEORIGIN on every response, HSTS when the public URL is https; set-if-absent so share-page and replay policies stay authoritative. - OBSERVE_SMTP_PRIVATE_HOSTS: exact-hostname allowlist for internal email relays (resolve-once dial-IP retained; refusal message points at the knob). - Sentry wire: the event timestamp is honored (float seconds / ms / RFC 3339 via toMillis); >24h future skew refused at admission with a 400 (ErrBadTimestamp), the same rule re-enforced on the apply path. - observe migrate: apply the ladder without seeding - unblocks the isolated-restore runbook (migrate -> restore -> start); drill doc updated, verified against a fresh database on Nucleus v1.1.1. - docs: AUDIT_OPEN S1 rewritten as the live-verified close-out; L9 updated with the checksum-freeze finding and the owner options; CHANGELOG records the session; README gains the SMTP env row.
…brahmagupta-k33zkw # Conflicts: # AUDIT_OPEN.md # docker-compose.yml
…e on the pinned engine)
…s refuses duplicate keys Keep one env block: NUCLEUS_VERSION v1.2.1 (the pin) + the staticcheck version the sweep added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #2 (closed after #3 moved main to the Nucleus v1.2.1 pin). Same branch, now merged with main: takes the v1.2.1 pin, reconciles the L9 record with the upstream resolution (engine ALTER-ADD defects fixed in v1.2.1; the migration-runner checksum contract and the populated-upgrade test stay as belt-and-braces), and re-verifies on the new engine.
Close-out summary
Review notes: behavior changes to review before upgrading are in CHANGELOG [Unreleased] ### Changed; AUDIT_OPEN.md S1 is the live-verified close-out register.