Skip to content

Expand observability: worker queue depth, store transaction boundaries, virtual-time tracking #3

Description

@coderabbitai

Background

Raised during review of PR #1 (Initial repo structure) by @leynos.

The PR introduces structured HTTP request logging and REST handler error logging, but observability for the deterministic worker, in-memory store, and virtual clock (specified in ADR 0004) is absent. Without this, diagnosing queue stalls, store transaction failures, or virtual-time drift in CI will be difficult.

Required work

  • Worker queue depth: emit a structured log event whenever the queue depth changes (task enqueued, task dequeued, queue drained to idle).
  • Store transaction boundaries: log transaction begin/commit/rollback with duration and affected slice names.
  • Virtual-time advancement: log each advanceClock call with the delta and resulting virtual timestamp.
  • Metrics: expose throughput, latency, and error-rate counters on the /_digitalpuddle admin surface (can be a simple JSON endpoint rather than Prometheus initially).
  • Alerts / thresholds: define documented thresholds for worker queue stall (e.g. queue non-empty for > N virtual seconds without progress) and store transaction failure rate that should trigger a test failure or warning.

References

/cc @leynos

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions