[PILOT-01] Run the base local API and durable-worker stack #488

Outcome and agreed scope

A developer on Linux or macOS can run API, durable worker, scheduler, PostgreSQL, Redis and MinIO locally, with distinct PM, two contributor and Finance identities. Hosted deployment waits for local end-to-end proof.

Current owners and dependencies

docker-compose.yml currently has a profiled backend, PostgreSQL, Redis and MinIO, but no worker/beat; backend artifact storage is disabled. Inspect backend/app/core/config.py, backend/app/workers/, backend/app/adapters/auth/dev.py and backend/scripts/external_api_drill.py. Project creation is already merged. Guide-declaration CLI is separate in-progress work; inspect its current availability before selecting CLI versus REST steps. Depend on the capability, not a particular implementation PR number. This base stack is a prerequisite, not dependent on the whole pilot.

Implementation scope

• Add backend-image worker and beat with the supported prefork topology and recovery scheduling.
• Host ports are configurable through each checkout's untracked .env for every published service (API, MinIO API/console, PostgreSQL, Redis and later proxy/runner ports). Use a distinct COMPOSE_PROJECT_NAME per agent and project-scoped containers, networks, volumes and scratch paths; document client URLs while retaining container-internal service DNS. Two stacks on one host must not share mutable data or stop/reset each other.
• Configure API/worker through existing ArtifactStore as s3_compatible, with startup bucket provisioning and health/readiness checks.
• Reuse local Flow-style HMAC authentication from existing drills. Provide an explicit local token helper and existing trust-root/AccessAdmin bootstrap to issue project-scoped grants; do not introduce Workstream login or another auth mode.
• Seed one project ready for guide declaration/upload, not a fabricated approved guide. Document reproducible clean-start, token/grant setup, CLI/API steps and teardown in docs/engineering/local-pilot.md.

Leave alone

No AWS/Terraform, frontend, production bootstrap or broad CI redesign. Never erase existing Docker volumes as part of startup. Runner/proxy compose integration belongs to PILOT-04 (#491)/#492; the complete journey belongs to PILOT-09 (#496) and does not hold this base issue open.

Acceptance and how to check

• Clean Linux and Docker Desktop/macOS starts bring up all six services with API/worker agreeing on config.
• Two stacks with different project names and host-port settings run simultaneously on one machine; submitting/restarting/tearing down one leaves the other's containers, storage and database intact.
• A real guide upload schedules durable setup work and persists its outcome; restart/recovery is demonstrated.
• Real artifact verification uses MinIO; storage-disabled configuration fails clearly.
• Distinct identities have real scoped grants; contributor task release and cross-project access are denied without effects.
• Secrets remain untracked; setup/teardown commands document which data is retained and which disposable resources are removed.

Close when the base stack and guide/storage/authority checks are proved. Do not claim the complete contributor journey is live.

Two-agent coordination

Agent 2 owns this local stack. Agent 1 owns every database migration; bootstrap helpers call approved backend operations and do not introduce private DDL. Model proxy/runtime services are later hand-offs, not requirements for closing the base stack.

Working and verification rules

Read AGENTS.md, CONTRIBUTING.md, docs/roadmap_status.md and the affected owner contracts before coding. The baseline inspected for this plan is main c0c4fe70f77e2e84347cec979e3de00e9d7f4295; recheck current main and open PRs before selecting a bounded implementation. Open/planned work is not delivered capability.

This is a planning issue and may require several bounded PRs. Record each PR's allowed files, prohibited changes, acceptance criteria and review scope using the repository's existing Commitrail process. Extend existing owner operations/typed public ports; do not add parallel legacy/new implementations or bypass authority. Preserve retained data, locked lineage, immutable evidence and caller-owned atomicity. A planning issue does not authorize deployment or merging a PR.

Run focused positive/negative tests and applicable repository checks. Use real PostgreSQL for database/locking claims and MinIO/S3 for storage claims. Add regressions that fail on the reproduced defect; do not skip failing tests or weaken CI. Run required full CI and affected independent review tracks before requesting human approval. Update affected current specs/ADRs, README/CLI contracts and roadmap in the same PR. Report exact tested head, commands, failure/cleanup evidence and remaining limitations; passing counts alone are insufficient.