feat: add database-backed distributed work runtime

[thunguo]

Which issue or RFC does this PR close?

Closes #1430.

Implements the bilingual RFC added by this PR:

• English RFC 1430
• 中文 RFC 1430

Rationale for this change

The existing background runtime depended on an APScheduler SQLite sidecar, process-local handler registration, and
process-local MCP session state. Those assumptions prevented API replicas from serving requests without affinity and
did not provide safe Scheduler failover or Worker recovery.

This PR makes OceanBase the single coordination and persistence authority for distributed execution. API replicas
enqueue durable work, Schedulers use a fenced database lease, and Workers use renewable fenced leases. Provider calls
remain at-least-once, while the final domain write, cursor advancement, attempt completion, and work success commit in
one transaction, so one logical Source window produces at most one database result.

The default deployment remains single_node/all. It now uses the same Work Ledger as distributed mode, avoiding two
different scheduling and recovery protocols.

What changes are included in this PR?

Architecture and persistence

• Adds the English and Chinese RFC for the distributed API, Scheduler, and Worker architecture.
• Adds forward-only Alembic migrations and the powercontext server migrate command.
• Adds the database-backed Work Ledger:
  • pc_work_items
  • pc_work_attempts
  • pc_work_lanes
  • pc_work_keys
  • pc_scheduler_leases
  • pc_scheduler_scans
  • pc_runtime_members
  • pc_rate_limit_windows
• Uses database time, explicit row locking, monotonic fencing tokens, bounded candidate scans, logical-key
  deduplication, and strict per-lane sequencing.
• Keeps provider/model calls outside database transactions. The existing domain cursor CAS is reused in the final
  fenced transaction.
• Replaces the APScheduler execution path in both single-node and distributed deployments.

Scheduler and Worker runtime

• Adds role-aware deployment modes:
  • single_node/all
  • distributed/api
  • distributed/scheduler
  • distributed/worker
• Adds fenced Scheduler leader election, durable keyset scan continuation, and safe standby takeover.
• Adds capacity-aware Worker claiming, lease heartbeat, expired-attempt recovery, full-jitter retry, cancellation,
  operator retry, and graceful draining.
• Migrates Memory activation and Experience incubation to versioned internal Work handlers.
• Adds durable maintenance work for bounded operation-history and rate-limit cleanup.
• Rejects unsupported distributed configurations before startup, including SQLite, seekDB, role=all, and host-local
  External Skill targets.

HTTP, Client, MCP, and authorization

• Extends POST /v1/memory/flush with synchronous wait and asynchronous operation behavior:
  • 200 when idle or completed within the wait budget
  • 202 with Location and Retry-After while pending
  • 409 operation_blocked when a failed logical window requires operator action
• Adds get, list, cancel, and retry Operation endpoints with optimistic expected_version transitions.
• Keeps PowerContextClient.flush_memory() synchronous-looking by polling to completion and adds explicit asynchronous
  operation methods.
• Uses stateless FastMCP HTTP in distributed mode, so consecutive MCP calls can hit different API replicas without
  sticky sessions.
• Propagates the authenticated PrincipalRef through HTTP and the internal MCP bridge.
• Adds an optional OceanBase-backed fixed-window rate limiter shared by API replicas.

Operations and observability

• Adds role-specific readiness, liveness, shutdown, and compatibility checks.
• Adds bounded-cardinality metrics for queue state, attempts, lease expiry, leadership changes, and live role members.
• Adds separate enqueue, claim, execute, commit, and retry spans with retry span links.
• Keeps Source content, prompts, model output, credentials, authorization headers, and complete secret URLs out of
  Work rows, attempt rows, logs, metrics, and traces.
• Adds docker/compose.distributed.yaml with one migrator, two API replicas, two Scheduler replicas, and two Worker
  replicas.
• Updates English and Chinese deployment, configuration, HTTP API, interface, troubleshooting, and architecture
  documentation.

Code cleanup

• Consolidates CLI and Runtime schema/index construction so migration and startup cannot drift.
• Consolidates duplicate Client HTTP, tracing, and error handling.
• Removes invalid intermediate DTO states, unused configuration error classes, one-line wrappers, redundant
  observability exception handling, and unused runtime configuration.
• Preserves correctness checks at trust boundaries: lease ownership, fencing, lock order, row counts, state versions,
  payload versions, and stored-data decoding.

Are there any user-facing changes?

Yes.

• New distributed deployment and role configuration is available. Distributed mode requires an OceanBase MySQL-mode
  database and one role per process.
• Operators must run powercontext server migrate before starting distributed roles. Distributed processes never run
  DDL automatically.
• The raw POST /v1/memory/flush endpoint may now return 202 for durable asynchronous execution. Existing
  PowerContextClient.flush_memory() callers continue to receive FlushMemoryResponse unless their total deadline
  expires, in which case OperationPendingError includes the operation ID.
• New Operation API and Client methods expose operation status, listing, cancellation, and explicit retry.
• Distributed MCP uses stateless HTTP and does not require load-balancer affinity.
• Scheduler and Worker roles expose only health and metrics routes; they do not expose public business routes.
• single_node/all remains the default and automatically uses the same migration chain and Work Ledger.
• The legacy scheduler_path argument remains accepted for bridge-release compatibility but is ignored. The old
  APScheduler sidecar is not deleted automatically.

The OpenAPI contract and generated clients were regenerated from openapi/powercontext.yaml.

How was this change tested?

Automated validation

The final branch was validated with:

make check
make test
make contract-test
make docs-test

Results:

• make check: passed, including Ruff, formatting, lock consistency, prek hooks, and ty checks.
• make test: 1,198 passed, 18 skipped.
• make contract-test: 29 passed.
• make docs-test: passed.

The real OceanBase acceptance tests were also run against a dedicated OceanBase 4.3.5.6 MySQL-mode database:

POWERCONTEXT_TEST_OCEANBASE_URL='mysql+aoceanbase://<user>:<password>@<host>:2881/<database>' \
  uv run pytest -q tests/e2e/test_distributed_work_oceanbase.py

Result: 2 passed. These tests verify a single claim across two real Worker processes under Read Committed isolation,
plus cross-replica HTTP operation visibility and stateless MCP round-robin behavior.

Additional tests cover:

• manual and scheduled discovery joining the same logical Memory window;
• one-lane serialization and different-lane concurrency;
• stale Worker and Scheduler fences;
• lease expiry, retry/backoff, cancellation/commit races, operator retry, and history cleanup;
• unknown payload version readiness;
• shared rate limiting and authentication;
• role route isolation and readiness;
• migration compatibility and database schema checks;
• payload, log, trace, and attempt privacy.

Actual distributed Docker Compose test

The deployment example was started against a clean OceanBase 4.3.5.6 instance using the local Server image and a
deterministic OpenAI-compatible model stub. No external model service was called.

Topology exercised:

• one one-shot migrator;
• two API replicas;
• two Scheduler replicas;
• two Worker replicas;
• no sticky HTTP or MCP session routing.

Observed results:

Scenario	Result
Empty-database migration	Alembic migration completed successfully before roles started
Role startup	2 APIs, 2 Schedulers, and 2 Workers became healthy; member counts matched the topology
Cross-replica HTTP	Work submitted through one API was visible from the other API
Logical-window deduplication	Concurrent manual/Scheduler discovery produced one logical key, one Memory cursor advancement, and one result
Worker concurrency	Different lanes overlapped across the two Workers; one lane remained serial
Scheduler failover	The standby acquired a higher fence after leader loss; the stale leader could not enqueue
Worker recovery	Attempt 1 ended as lease_expired_retry; another Worker completed attempt 2 with fence 2; the old Worker could not duplicate the commit
Stateless MCP	One client made four requests alternating between both APIs; no Mcp-Session-Id was sent or returned
Role isolation	Scheduler and Worker public business routes returned 404
Authentication	Missing bearer credentials returned 401
Shared rate limiting	The quota was shared across both APIs; after 120 protected requests the deployment returned 429 rate_limited with Retry-After

The Worker failure test used a 12-second lease, 3-second heartbeat, 9-second shutdown grace, concurrency 1, and a model
stub that delayed generation for 25 seconds. This made lease recovery observable without changing production defaults.

Basic reproduction flow

Build the local image:

POWERCONTEXT_VERSION=$(uvx --from hatchling --with hatch-vcs hatchling version)
docker build \
  --file docker/Dockerfile \
  --build-arg "POWERCONTEXT_VERSION=$POWERCONTEXT_VERSION" \
  --tag powercontext-server:local \
  .

Configure a dedicated OceanBase database and provider credentials, then start the topology:

export POWERCONTEXT_SERVER_DATABASE_URL='mysql+aoceanbase://<user>:<password>@<host>:2881/<database>'
export POWERCONTEXT_SERVER_AUTH_TOKEN='<test-token>'
export POWERCONTEXT_GENERATION_MODEL='<provider>:<model>'
export OPENAI_API_KEY='<test-key>' # or ANTHROPIC_API_KEY

docker compose -f docker/compose.distributed.yaml up -d
docker compose -f docker/compose.distributed.yaml ps -a

The migrate service should exit with status 0. Both API readiness endpoints should report ready or degraded only for
an intentionally absent optional dependency:

curl -fsS http://127.0.0.1:8001/health/ready
curl -fsS http://127.0.0.1:8002/health/ready

Exercise cross-replica operation visibility:

TOKEN='<test-token>'
SCOPE_ID='distributed-pr-smoke'

curl -fsS \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{\"scope_id\":\"$SCOPE_ID\",\"source_id\":\"source-1\",\"content\":\"distributed durable work\"}" \
  http://127.0.0.1:8001/v1/sources/content

OPERATION_ID=$(
  curl -fsS \
    -H "Authorization: Bearer $TOKEN" \
    -H 'Content-Type: application/json' \
    -H 'Prefer: respond-async' \
    -d "{\"scope_id\":\"$SCOPE_ID\"}" \
    http://127.0.0.1:8002/v1/memory/flush |
    jq -r '.operation_id'
)

curl -fsS \
  -H "Authorization: Bearer $TOKEN" \
  "http://127.0.0.1:8001/v1/operations/$OPERATION_ID" |
  jq

Poll the final command until the operation becomes succeeded. For fault injection, stop the current Scheduler and
confirm its standby takes over with a higher database fence. Stop a Worker during the delayed provider call, wait for
lease expiry, and confirm another Worker completes the operation while the original attempt is recorded as
lease_expired_retry.

Finally, inspect bounded metrics and clean up:

curl -fsS http://127.0.0.1:8001/metrics
docker compose -f docker/compose.distributed.yaml down

The temporary Compose services, network, model stub, and OceanBase test data used in the actual run were removed after
validation.

AI usage statement

[Copilot]

🔵 Needs a closer look

It introduces a broad set of coordinated changes across persistence/migrations, runtime coordination, HTTP/client behavior, and deployment modes that warrant careful human review of correctness and upgrade/rollback safety.

Pull request overview

Implements RFC #1430 distributed Server/Scheduler/Worker architecture by replacing the APScheduler sidecar with a database-backed Work Ledger + fenced leases, adding forward-only migrations + powercontext server migrate, and extending the HTTP/client/MCP surface for durable Operations, stateless multi-replica MCP, shared rate limiting, and expanded observability.

Changes:

• Introduces durable Work Ledger tables + coordination/membership/rate-limit tables, plus Alembic forward-only migrations and a Server migrate CLI.
• Adds distributed deployment roles (api/scheduler/worker), durable scheduler discovery and worker execution semantics, and corresponding tests/e2e coverage.
• Extends HTTP/client/MCP behavior for durable Operations (200/202 flush semantics, operation list/get/cancel/retry), shared fixed-window rate limiting, and updated tracing/metrics/docs + docker compose topology.

File summaries

File	Description
zensical.toml	Adds RFC 1430 to nav
tests/test_server.py	Updates scheduler sidecar assertions
tests/test_server_tracing.py	Tracing tests for new links + DB
tests/test_server_roles.py	Verifies role route isolation + stateless MCP
tests/test_server_migration_cli.py	Tests server migrate CLI
tests/test_server_metrics.py	Adds bounded work/membership metrics checks
tests/test_mcp.py	Tests stateless MCP picker behavior
tests/test_docker_contract.py	Validates distributed compose contract
tests/test_client.py	Client tests for 202 operation polling/deadline
tests/e2e/test_shared_rate_limit.py	E2E rate limiting behavior
tests/e2e/test_scheduled_experience_incubation.py	Removes scheduler sidecar from e2e
tests/e2e/test_operation_api.py	E2E tests for stateless operation API
tests/e2e/test_observability.py	Updates spans/attributes for work stages
tests/e2e/test_durable_memory_work.py	E2E durable work dedupe + commit
tests/e2e/test_distributed_work_oceanbase.py	OceanBase multi-process/replica e2e
tests/e2e/real_experience_skill/harness.py	Harness updates for durable work
tests/builtin/runtime/test_operation_maintenance.py	Durable maintenance work retention test
tests/builtin/runtime/test_membership.py	Runtime membership + single-node lease tests
tests/builtin/runtime/test_durable_scheduler.py	Durable scheduler fencing/continuation tests
tests/builtin/runtime/test_deployment_config.py	Deployment/worker config validation tests
tests/builtin/persistence/test_rate_limit.py	Rate limit repository tests
tests/builtin/persistence/test_mysql_schema.py	Column budget updates (DateTime)
tests/builtin/persistence/test_migration.py	Migration lease + schema validation tests
tests/builtin/persistence/test_experience_index.py	Legacy schema upgrade path for projections
tests/builtin/persistence/test_coordination.py	Coordination lease/scan state tests
src/powercontext/server/tracing.py	Background span linking support
src/powercontext/server/settings.py	Adds deployment/coordination/worker/ops/rate_limit settings
src/powercontext/server/rate_limit.py	New shared DB rate-limit middleware
src/powercontext/server/principal.py	New principal context binding
src/powercontext/server/middleware.py	Principal binding + auth middleware updates
src/powercontext/server/metrics.py	Adds bounded durable-work + membership metrics
src/powercontext/server/mcp.py	Stateless MCP mode + internal bridge changes
src/powercontext/server/mapping.py	Adds Operation mapping utilities
src/powercontext/server/handoff_picker.py	Elicitation gating for stateless MCP
src/powercontext/server/factory.py	Role-aware server composition + work observer + rate limit
src/powercontext/server/cli.py	Adds server migrate command
src/powercontext/http/_generated/models.py	Generated Operation models
src/powercontext/http/init.py	Re-exports new HTTP models
src/powercontext/client/errors.py	Adds operation pending/failed errors
src/powercontext/client/init.py	Exposes new client errors
src/powercontext/builtin/runtime/work_observability.py	Work observer protocol + queue refresh helper
src/powercontext/builtin/runtime/scheduler.py	Removes APScheduler adapter
src/powercontext/builtin/runtime/relational.py	Adds scope keyset paging API
src/powercontext/builtin/runtime/protocols.py	Adds RuntimeTraceContext + optional accessor
src/powercontext/builtin/runtime/membership.py	Adds runtime membership + single-node lease
src/powercontext/builtin/runtime/durable_scheduler.py	Adds fenced durable scheduler
src/powercontext/builtin/runtime/config.py	Adds deployment/coordination/worker/ops/rate-limit config
src/powercontext/builtin/runtime/init.py	Removes legacy scheduled processor exports
src/powercontext/builtin/persistence/tables.py	Adds work/coordination/rate-limit tables + indexes
src/powercontext/builtin/persistence/sqlite/profile.py	Adds optional schema creation + tx serialization
src/powercontext/builtin/persistence/seekdb/profile.py	Adds optional schema creation toggle
src/powercontext/builtin/persistence/rate_limit.py	Adds DB-time fixed-window limiter
src/powercontext/builtin/persistence/oceanbase/profile.py	Adds optional schema creation toggle
src/powercontext/builtin/persistence/oceanbase/memory_index.py	Adds verify-only checks for provisioned indexes
src/powercontext/builtin/persistence/oceanbase/experience_index.py	Adds verify-only checks for projections/indexes
src/powercontext/builtin/persistence/migrations/versions/0002_work_ledger.py	Migration adds work/coordination tables
src/powercontext/builtin/persistence/migrations/versions/0001_baseline.py	Baseline migration
src/powercontext/builtin/persistence/migrations/script.py.mako	Forward-only migration template
src/powercontext/builtin/persistence/migrations/env.py	Packaged Alembic env (connection-provided)
src/powercontext/builtin/persistence/migrations/init.py	Migrations package marker
src/powercontext/builtin/persistence/migration.py	Migration orchestration + legacy stamping/validation
src/powercontext/builtin/persistence/memory_index.py	Adds schema verification hook
src/powercontext/builtin/persistence/database.py	Adds DB-time helper + insert-if-absent
src/powercontext/builtin/persistence/cursors.py	Uses insert-if-absent for CAS insert
src/powercontext/builtin/persistence/init.py	Exports new persistence primitives
scripts/generate_api.py	Updates generator for multiple 2xx responses
pyproject.toml	Adds Alembic, removes APScheduler
integrations/pi/plugins/powercontext/src/operations.generated.ts	Adds Operation endpoints to integration
integrations/opencode/plugins/powercontext/src/operations.generated.ts	Adds Operation endpoints to integration
integrations/dsh/plugins/powercontext/src/operations.generated.ts	Adds Operation endpoints to integration
docs/zh/rfcs/0020_runtime_backed_memory_remote_access.md	Notes RFC 1430 execution update
docs/zh/rfcs/0019_local_source_memory_runtime.md	Notes RFC 1430 execution update
docs/zh/rfcs/0011_remote_access_architecture.md	Notes RFC 1430 deployment boundary
docs/zh/docs/reference/interfaces.md	Updates scheduled Experience description
docs/zh/docs/reference/http-api.md	Documents durable flush + operations + 429
docs/zh/docs/reference/configuration.md	Documents new distributed/lease/ops settings
docs/zh/docs/how-to/troubleshoot.md	Updates table list for export
docs/zh/docs/how-to/full-capability-runtime.md	Notes work state in primary DB
docs/zh/docs/how-to/deploy-server.md	Adds distributed role deployment guide
docs/zh/development/remote-access-implementation.md	Updates scheduling/ops surface docs
docs/zh/development/core-protocol.md	Updates scheduler/worker lifecycle wording
docs/en/rfcs/0020_runtime_backed_memory_remote_access.md	Notes RFC 1430 execution update
docs/en/rfcs/0019_local_source_memory_runtime.md	Notes RFC 1430 execution update
docs/en/rfcs/0011_remote_access_architecture.md	Notes RFC 1430 deployment boundary
docs/en/docs/reference/interfaces.md	Updates scheduled Experience description
docs/en/docs/reference/http-api.md	Documents durable flush + operations + 429
docs/en/docs/reference/configuration.md	Documents new distributed/lease/ops settings
docs/en/docs/how-to/troubleshoot.md	Updates table list for export
docs/en/docs/how-to/full-capability-runtime.md	Notes work state in primary DB
docs/en/docs/how-to/deploy-server.md	Adds distributed role deployment guide
docs/en/development/remote-access-implementation.md	Updates scheduling/ops surface docs
docs/en/development/core-protocol.md	Updates scheduler/worker lifecycle wording
docker/README.md	Notes compose.distributed.yaml template
docker/compose.distributed.yaml	Adds migrator + role-separated topology
.env.example	Documents new deployment/work/ops/rate-limit env vars

Review details

• Files reviewed: 112/113 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

There are a few concrete correctness/maintainability issues in the changed code (notably rate limiting behavior when unbound and duplicated exports) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

scripts/generate_api.py:354

• _success_response() now silently selects the lowest 2xx code when an operation defines multiple success responses. This can hide cases where different success codes use different schemas, leading generated clients to treat some successful responses as errors unless they are manually special-cased.

• Files reviewed: 115/117 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Critical correctness issues remain in time handling, payload validation, and rate-limit middleware, alongside unresolved moderate findings.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

Previously missed (5) — in code that hasn't changed since the last review.

docs/en/docs/how-to/troubleshoot.md:193

• Importing pc_scheduler_leases into the freshly initialized target is not safe: migration always leaves a schema-migration lease row in that table, so a current source export contains the same primary key and obloader will fail on the duplicate. The recovery procedure should provision the target with server migrate (without starting the runtime) and exclude or explicitly reconcile transient lease/member rows before import.
  docs/zh/docs/how-to/troubleshoot.md:186
• 把 pc_scheduler_leases 导入刚初始化的目标库并不安全：迁移过程一定会在该表保留一条 schema-migration lease，当前版本源库也有相同主键，因此 obloader 会因重复键失败。恢复流程应仅运行 server migrate 来建目标 schema（不要启动 Runtime），并在导入前排除或明确协调临时 lease/member 数据。
  src/powercontext/server/factory.py:111
• Distributed API replicas derive this response from their process-local inference adapters. The supplied Compose intentionally configures generation only on Workers, so APIs report memory_extraction=false even though they accept durable flush work; clients that gate flushing on this capability will skip the feature. Derive execution capabilities from the distributed worker contract/topology rather than the API process's local pipeline.
  docs/en/docs/how-to/full-capability-runtime.md:136
• This new paragraph contradicts the immediately preceding statement that SQLite still stores scheduler.db, and repeats the same directory list. Remove the obsolete sidecar statement and keep one corrected description.
  docs/zh/docs/how-to/full-capability-runtime.md:129
• 新增段落与前文仍声称会保存 scheduler.db 相矛盾，而且重复列出了数据目录。请删除旧 sidecar 说明，只保留一份更新后的描述。

src/powercontext/builtin/persistence/init.py:97

• These two public export names are each listed twice. Remove the duplicates so __all__ remains a canonical, auditable API list.

• Files reviewed: 115/117 changed files
• Comments generated: 3
• Review effort level: Balanced

[thunguo]

Addressed the remaining review feedback in b7343b4, including the summary-only findings: distributed API replicas now advertise worker-backed Memory extraction; the OceanBase restore guide provisions schema with server migrate and excludes transient coordination rows; and the obsolete scheduler.db wording was removed in both locales. The inline findings were also fixed and resolved: all generated 2xx response models, UTC database time, payload-version validation, shared public-route classification, and duplicate exports. Validation: make check; make test (1316 passed, 21 skipped); make contract-test (33 passed); make docs-test.