Make interview engine tuning numbers runtime-configurable

[Dustymon111]

Problem

Every tuning number in the interview engine was hardcoded. Changing something as basic as "minimum 2 probes before a skill can advance" meant a code change and a deploy — and some numbers were duplicated across the state engine, the analyzer prompt and the system prompt, so they could silently drift apart.

Approach

A single registry (app/lib/app_config.rb) declares every tunable with a type, range and code default, backed by a new app_settings table.

tenant override (app_settings.tenant_id = <tenant>)
  → global override (app_settings.tenant_id IS NULL)
    → legacy ENV var (only where a setting declares one)
      → code default

• Reads are cached in Redis for 30s and the cache is invalidated on write, so a change reaches every Puma and Sidekiq pod within seconds — no deploy, no restart, no configmap edit.
• If the DB or cache is unavailable, lookups fall back to the code default. Config can never take an interview down.
• All defaults match the previously hardcoded values, so behaviour is unchanged until something is overridden.

How to change a value

Per tenant, via API (admin role) — writes are always scoped to the caller's tenant, so one tenant's admin can never change another tenant's behaviour:

GET    /api/v1/settings                      # every setting + effective value + source
PATCH  /api/v1/settings/:key   { "value": 3 }
PUT    /api/v1/settings        { "settings": { "coverage.min_probes_to_advance": 3 } }
DELETE /api/v1/settings/:key                 # drop this tenant's override

Globally, via rake (ops only — deliberately not exposed over the API):

rake app_config:list
rake app_config:set[coverage.min_probes_to_advance,3]        # global
rake app_config:set[coverage.min_probes_to_advance,3,12]     # tenant 12
rake app_config:reset[coverage.min_probes_to_advance]

What became configurable (29 settings)

coverage — probing depth

Key	Default	Was
coverage.min_probes_to_advance	2	hardcoded in StateEngine, the analyzer prompt and the system prompt
coverage.auto_advance_probe_count	4	probe_count >= 4 in CoverageAnalyzerWorker
coverage.deep_probe_ceiling	4	"only continue if pacing=ahead" rule in the system prompt
coverage.max_discovered_skills	10	discovered.count >= 10
coverage.discovery_max_exchanges	3	"2-3 exchanges maximum" in the system prompt
coverage.analyzer_turns_context	6	Analyzer::TURNS_CONTEXT

pacing — minutes remaining per uncovered skill

Key	Default
pacing.ahead_min_minutes	5.0
pacing.on_track_min_minutes	3.0
pacing.behind_min_minutes	1.5

portfolio — confidence rating

Key	Default
portfolio.confidence_high_probe_count	3
portfolio.confidence_medium_probe_count	2
portfolio.max_evidence_quotes	3

session — timing and lifecycle

Key	Default
session.time_warning_seconds	120
session.wrap_up_warning_seconds	60
session.time_ceiling_grace_seconds	60
session.wrap_up_silence_fallback_seconds	20
session.closing_phrase_fallback_seconds	15
session.end_poll_max_attempts	30
session.browser_grace_period_seconds	120

gemini — connection resilience

Key	Default
gemini.proactive_reconnect_after_seconds	510 (still honours the PROACTIVE_RECONNECT_AFTER env var as fallback)
gemini.proactive_reconnect_jitter_seconds	30
gemini.max_reconnect_attempts	3
gemini.reconnect_backoff_seconds	[1, 2, 4]
gemini.inactivity_timeout_seconds	30
gemini.gate_open_delay_seconds	0.8
gemini.silence_pump_delay_seconds	1.0

preflight — candidate device check (served to the frontend)

Key	Default
preflight.min_upload_kbps	500
preflight.min_download_kbps	500
preflight.speed_test_payload_bytes	262144
preflight.speed_test_timeout_ms	10000
preflight.block_below_minimum	true

Deliberately NOT configurable

SILENCE_PUMP_INTERVAL / SILENCE_FRAME_SAMPLES (16kHz PCM wire-format constants — a wrong value corrupts audio) and the +1-per-run probe increment cap (a correctness invariant, not a tuning knob).

When a change takes effect

Group	Applies to
coverage.*, pacing.*	next analyzer run / coverage injection (seconds)
portfolio.*	next portfolio generation
session.*, gemini.*	next WebSocket connection (read once per connection)
preflight.*	next candidate device check

The probe numbers are also baked into the compiled system prompt stored on each assessment. Assessments now carry system_prompt_config_digest; when it no longer matches the current config the prompt is recompiled automatically as the next session connects, so a settings change reaches existing assessments too.

Frontend follow-up (not in this PR)

The minimum upload speed lives in the frontend repo today — the backend /api/v1/speed_test endpoint only ever echoed the byte count and made no pass/fail judgement. This PR adds the server side of it:

GET  /api/v1/preflight/config      → { "preflight": { "preflight.min_upload_kbps": 500, ... } }
POST /api/v1/preflight/speed_test  → { "received_bytes": 262144, "preflight": { ... } }

Both are unauthenticated (a candidate only has an invite link). POST /api/v1/speed_test is left untouched so the current frontend keeps working. The frontend still needs a change to read its thresholds from /api/v1/preflight/config instead of hardcoding them.

Migrations

• 20260804000000_create_app_settings — new table, partial unique indexes on (tenant_id, key) and on key where tenant_id IS NULL (Postgres treats NULLs as distinct, so the global rows need their own index)
• 20260804001 add_system_prompt_config_digest_to_assessments — one nullable varchar(64) column

Both are additive and safe to run on a live database.

Testing

AppConfig was exercised directly against a live Rails console: set/get/reset across all four types (integer, float, boolean, integer array), range rejection (must be <= 10), type rejection (must be a whole number), unknown-key rejection, and correct source reporting (tenant / global / env / default). The gate itself was verified end-to-end: with min_probes_to_advance = 3, StateEngine.resolve_state(initiated → covered, probe_count: 2) stops at initiated, and passes to covered at probe_count: 3.

⚠️ The migrations have not been run against a correctly-migrated database. The local dev DB is out of sync with db/migrate (5 migrations show as NO FILE, and the assessments table is absent), so create_app_settings ran locally but the digest column could not. db/schema.rb was edited by hand to reflect exactly the two migrations rather than a dump of that broken DB. Please run both on staging first.

🤖 Generated with Claude Code