Merge upstream main into Prairie

.claude/skills/dev-environment-debugging/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: dev-environment-debugging
+description: Use when diagnosing a running Prairie server — playback, streaming, or transcode failures, unhealthy nodes, stuck sessions, wrong database state, errors after a deploy, or anything needing the deployment's logs, health endpoints, or SQL. Covers both a local docker compose install and a remote host over SSH. Not for building or testing code in this repo, and not for client-side bugs that reproduce without a server.
+---
+
+# Debugging a running Prairie deployment
+
+Everything here goes through `scripts/prairie-dev`, which reads `.prairie-dev.env` and runs
+commands against the deployment whether it is on this machine or behind SSH. Start with:
+
+```bash
+scripts/prairie-dev doctor
+```
+
+`doctor` reports which of API, host shell, compose stack, database, and login actually
+work. Fix what it flags before diagnosing anything else — most confusing symptoms here
+are really a misconfigured target. If `.prairie-dev.env` is missing, copy
+`.prairie-dev.env.example` and fill it in; ask the user for hosts and credentials rather
+than guessing, and never move those values into a committed file.
+
+## Orienting
+
+```bash
+scripts/prairie-dev compose ps                  # what is running
+scripts/prairie-dev api /api/v1/health          # API up
+scripts/prairie-dev api /api/v1/ready           # PostgreSQL and optional S3 reachable
+scripts/prairie-dev logs --tail 200             # recent server logs
+scripts/prairie-dev logs -f                     # follow
+```
+
+`/ready` failing usually means the database, not the API — read the service logs before
+concluding Postgres is down.
+
+Prairie also records structured logs in the database itself, which survive container
+restarts and are easier to filter than stdout. `operational_logs` holds server events,
+`activity_log` holds HTTP requests; both are keyed by `"timestamp"` (quoted — it is a
+reserved word). See [references/queries.md](references/queries.md) for the diagnostic
+queries worth having on hand, along with the schema traps that bite most often.
+
+Log level lives in `server_settings` under `server.log_level` and is read at startup, so
+raising it to `debug` means updating the row and recreating the service. Put it back
+afterwards unless the task needs it to persist.
+
+## Search
+
+Meilisearch is optional — Prairie runs without it — but it is recommended, and it is what
+makes search results good. It ships as a compose service behind the `search` profile and
+needs `MEILI_MASTER_KEY` set, so a stack brought up without that profile simply has no
+`meilisearch` service.
+
+The provider is chosen from the `catalog.search.provider` setting rather than from
+whether the container is running, and the failure mode is quiet: if Meilisearch is
+selected but the URL is empty, unreachable, or the provider fails to initialize, Prairie
+logs a warning at startup and serves Postgres results instead. Nothing surfaces to the
+client. So "search got worse" or "search stopped matching the way it used to" is usually
+the fallback being active, not a ranking regression — check which provider is live before
+touching query code:
+
+```bash
+scripts/prairie-dev psql "SELECT key, value FROM server_settings WHERE key LIKE 'catalog.search%' ORDER BY key;"
+scripts/prairie-dev logs --tail 500 | grep -i 'catalog search'
+```
+
+Provider settings, index name, timeouts, batch sizes, and the semantic-search options all
+live under `catalog.search.meilisearch.*` in `server_settings`. Changing the provider
+needs a restart to take effect.
+
+## Playback
+
+Work outward from the failure the user actually saw.
+
+**Direct play** — confirm the file exists at the path stored in `media_files`
+(`scripts/prairie-dev sh 'ls -la …'`, under `PRAIRIE_MEDIA_ROOT`), then compare the file's
+codecs against what the client advertised. A mismatch should have fallen back to remux
+or transcode; if it did not, that is the bug.
+
+**Remux and transcode** — both run ffmpeg inside the server container, so check ffmpeg
+is present and, for hardware paths, that `/dev/dri` is passed through:
+
+```bash
+scripts/prairie-dev compose "exec -T prairie sh -lc 'ffmpeg -hwaccels; ls -la /dev/dri'"
+```
+
+Transcode output lands in a bind-mounted temp directory; a full disk there looks like a
+transcode bug. If `PRAIRIE_TRANSCODE_SSH` is set, the node is a separate host with its own
+`/api/v1/health` and logs — check it before blaming the main server.
+
+**Stuck sessions** — query `playback_sessions_sync` for rows whose `updated_at` has gone
+stale. The reconciler and shutdown path clean these up; when Redis is unavailable,
+cross-node coordination degrades and stale rows are a symptom rather than the cause.
+
+## Deploying a fix
+
+`make dev-deploy` and friends come from `Makefile.local`, which is gitignored and
+per-developer — do not assume they exist. Confirm with `make -n dev-deploy` first, and
+otherwise deploy however the user's setup does. Afterwards, verify rather than assume:
+
+```bash
+scripts/prairie-dev compose ps
+scripts/prairie-dev api /api/v1/ready
+scripts/prairie-dev logs --tail 100
+```
+
+Deploys restart the container, which invalidates any token you were holding.
+
+## Rename drift
+
+Prairie was formerly Silo, and Silo was formerly Continuum. Names like `continuum-postgres`, `/opt/continuum`,
+`CONTINUUM_IMAGE`, or a stray `docker-compose.dev.yml` are usually legacy artifacts, but
+on a long-lived host they are occasionally still the live stack. Check what compose
+actually reports before deciding which one you are looking at.

.claude/skills/dev-environment-debugging/references/queries.md

@@ -0,0 +1,110 @@
+# Diagnostic queries
+
+Run with `scripts/prairie-dev psql "<sql>"`.
+
+## Schema traps
+
+- `activity_log` and `operational_logs` order by `"timestamp"` — quoted, not `created_at`.
+- `admin_jobs` uses `job_type`, `requested_at`, and `completed_at` (not `type`,
+  `created_at`, `finished_at`).
+- Login accounts (`users`) are separate from household profiles (`user_profiles`); a
+  playback session carries both a `user_id` and a `profile_id`.
+
+## Errors and requests
+
+```sql
+SELECT "timestamp", level, component, message, request_id, playback_session_id, attrs
+FROM operational_logs
+WHERE level IN ('error', 'warn')
+ORDER BY "timestamp" DESC
+LIMIT 50;
+```
+
+```sql
+SELECT "timestamp", client_ip, method, path, status_code, duration_ms, request_id
+FROM activity_log
+WHERE status_code >= 500
+ORDER BY "timestamp" DESC
+LIMIT 20;
+```
+
+Drop the `WHERE` clause on `activity_log` for a plain recent-traffic view — useful for
+confirming a client is reaching the server at all, and with which paths.
+
+## Playback
+
+```sql
+SELECT s.session_id, u.username, COALESCE(s.profile_id, '') AS profile_id,
+       s.play_method, s.reporting_node, s.transcode_node_url,
+       s.position_seconds, s.is_paused, s.target_resolution,
+       s.started_at, s.updated_at
+FROM playback_sessions_sync s
+LEFT JOIN users u ON u.id = s.user_id
+ORDER BY s.updated_at DESC NULLS LAST;
+```
+
+Stale rows only:
+
+```sql
+SELECT session_id, user_id, play_method, updated_at
+FROM playback_sessions_sync
+WHERE updated_at < now() - interval '5 minutes'
+ORDER BY updated_at;
+```
+
+## Nodes
+
+```sql
+SELECT id, name, type, url, enabled, healthy, active_jobs, last_health_check
+FROM stream_nodes
+ORDER BY type, name;
+```
+
+Read-only node state is better taken from here than from the admin endpoints, which need
+a bearer token. To force a re-check, `POST /api/v1/admin/nodes/{id}/check` — reachable as
+`scripts/prairie-dev api /api/v1/admin/nodes/{id}/check -X POST`.
+
+## Jobs
+
+```sql
+SELECT id, job_type, status, message, error_message,
+       progress_current, progress_total, requested_at, started_at,
+       completed_at, heartbeat_at
+FROM admin_jobs
+ORDER BY requested_at DESC
+LIMIT 20;
+```
+
+Filter by `job_type = 'library_refresh'` (and select `request_payload`, `result_payload`)
+when chasing a scan or refresh that did not finish.
+
+## Users and settings
+
+```sql
+SELECT id, username, email, role, enabled, permissions, max_streams, max_transcodes, created_at
+FROM users
+ORDER BY id;
+```
+
+```sql
+SELECT key, value
+FROM server_settings
+WHERE key NOT ILIKE '%secret%'
+  AND key NOT ILIKE '%password%'
+  AND key NOT ILIKE '%api_key%'
+  AND key NOT ILIKE '%token%'
+ORDER BY key;
+```
+
+Keep that filter. Encrypted `server_settings` values are GCM-bound to their key name, so
+they are unreadable here anyway — and dumping them into a transcript is a leak.
+
+## Raising the log level
+
+```sql
+INSERT INTO server_settings (key, value) VALUES ('server.log_level', 'debug')
+ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;
+```
+
+Then `scripts/prairie-dev compose "up -d --force-recreate silo"`, and set it back to `info`
+when done.