feat(observability): add production telemetry

[tiankaima]

Summary

• add production-safe REST request logging and bounded API metrics across REST handlers
• add readiness checks for database/storage and document observability operations
• add MCP tool result, storage operation, and audit write metrics
• persist structured app logs to mounted JSONL files via APP_LOG_DIR / APP_LOG_HOST_DIR
• support local loopback sibling origins/audiences for non-3000 E2E verification

Verification

• bun run verify:fast
• APP_PUBLIC_ORIGIN=http://127.0.0.1:3010 PLAYWRIGHT_HOST=127.0.0.1 PLAYWRIGHT_PORT=3010 NO_PROXY=127.0.0.1,localhost,::1 no_proxy=127.0.0.1,localhost,::1 bun run verify:full
• scripts/test-target.sh unit tests/unit/app-logger.test.ts
• docker compose -f docker-compose.prod.yml config with required prod env populated
• docker compose -f docker-compose.dev.yml --profile app config

Notes

• /api/metrics remains the Prometheus endpoint; disk persistence is for structured logs only.
• Production Compose mounts ${APP_LOG_HOST_DIR:-./logs/app} to /var/log/life-ustc.
• Local full E2E was run on port 3010 because this machine has a stale broken listener on 127.0.0.1:3000.

[Copilot]

Pull request overview

This PR introduces production-oriented observability across the Next.js server: request-safe structured logging, bounded in-memory Prometheus-style metrics for REST/MCP/storage/audit operations, and a new readiness endpoint for dependency checks. It also updates tooling (OpenAPI generation + feature-doc parity checks) and Docker Compose defaults to support log persistence and local E2E on non-3000 loopback ports.

Changes:

• Add REST request start/finish/error logging + bounded metrics via observedApiRoute(...), applied across most /api/* handlers and middleware request tracing headers.
• Add new /api/readiness endpoint (DB + storage config checks) and expand observability metrics (MCP tool results, storage ops, audit writes).
• Persist structured server logs as JSONL when APP_LOG_DIR is configured; update Compose and docs accordingly, plus broaden loopback localhost/127.0.0.1 audience/origin matching.

Reviewed changes

Copilot reviewed 82 out of 83 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/dev/check/check-features-doc.ts	Adjust REST route detection for feature-doc parity checks.
tools/build/openapi/generate-spec.ts	Teach OpenAPI generator to extract JSDoc from observedApiRoute-wrapped handlers; exclude internal routes.
tests/unit/runtime-metrics.test.ts	Add unit coverage for new storage + audit metrics.
tests/unit/readiness-route.test.ts	Add unit coverage for readiness access control + checks payload.
tests/unit/oauth-utils.test.ts	Add unit coverage for loopback resource-indicator equivalence.
tests/unit/mcp-urls.test.ts	Add unit coverage for loopback sibling MCP audience URLs.
tests/unit/mcp-observability.test.ts	Add unit coverage for MCP tool-result metrics emission.
tests/unit/auth-origins.test.ts	Add unit coverage for loopback sibling trusted origins.
tests/unit/app-logger.test.ts	Add unit coverage for JSONL file persistence via APP_LOG_DIR.
tests/unit/api-observability.test.ts	Add unit coverage for REST path normalization, logs, and metrics via wrapper.
src/proxy.ts	Propagate request timing headers and emit request-start telemetry for /api/*.
src/lib/storage/s3.ts	Wrap S3 send + signed URL generation with bounded storage-operation metrics.
src/lib/oauth/utils.ts	Allow localhost ↔ 127.0.0.1 equivalence in resource indicator matching (same port/path).
src/lib/metrics/observability-metrics.ts	Add MCP tool-result, storage-operation, and audit-write metric helpers.
src/lib/mcp/urls.ts	Include loopback sibling MCP resource URLs in valid audience sets.
src/lib/mcp/observability.ts	Record per-tool result status/duration metrics from MCP request summaries.
src/lib/log/app-logger.ts	Add optional JSONL disk persistence for structured logs (APP_LOG_DIR).
src/lib/log/api-observability.ts	New module implementing REST request normalization, logging, and bounded metrics wrapper.
src/lib/auth/auth-origins.ts	Add loopback sibling origin to trusted origins derived from APP_PUBLIC_ORIGIN.
src/lib/audit/write-audit-log.ts	Record bounded audit-write metrics (status + duration).
src/app/api/users/[userId]/calendar.ics/route.ts	Wrap handler with observedApiRoute for REST telemetry.
src/app/api/uploads/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/uploads/complete/route.ts	Wrap POST with observedApiRoute for REST telemetry.
src/app/api/uploads/[id]/route.ts	Wrap PATCH/DELETE with observedApiRoute for REST telemetry.
src/app/api/uploads/[id]/download/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/todos/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/todos/[id]/route.ts	Wrap PATCH/DELETE with observedApiRoute for REST telemetry.
src/app/api/teachers/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/teachers/[id]/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/semesters/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/semesters/current/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/sections/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/sections/match-codes/route.ts	Wrap POST with observedApiRoute for REST telemetry.
src/app/api/sections/calendar.ics/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/sections/[jwId]/schedules/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/sections/[jwId]/schedule-groups/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/sections/[jwId]/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/sections/[jwId]/calendar.ics/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/schedules/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/readiness/route.ts	Add readiness endpoint with DB/storage checks + restricted visibility.
src/app/api/openapi/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/metadata/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/me/subscriptions/homeworks/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/me/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/mcp/route.ts	Record MCP tool-result metrics alongside existing MCP metrics/logs.
src/app/api/locale/route.ts	Wrap POST with observedApiRoute for REST telemetry.
src/app/api/homeworks/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/homeworks/[id]/route.ts	Wrap PATCH/DELETE with observedApiRoute for REST telemetry.
src/app/api/homeworks/[id]/completion/route.ts	Wrap PUT with observedApiRoute for REST telemetry.
src/app/api/descriptions/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/dashboard-links/visit/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/dashboard-links/pin/route.ts	Wrap POST with observedApiRoute for REST telemetry.
src/app/api/courses/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/courses/[jwId]/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/comments/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/comments/[id]/route.ts	Wrap GET/PATCH/DELETE with observedApiRoute for REST telemetry.
src/app/api/comments/[id]/reactions/route.ts	Wrap POST/DELETE with observedApiRoute for REST telemetry.
src/app/api/calendar-subscriptions/route.ts	Wrap POST with observedApiRoute for REST telemetry.
src/app/api/calendar-subscriptions/current/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/bus/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/bus/preferences/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/auth/oauth2/token/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/auth/oauth2/device-authorization/route.ts	Wrap OPTIONS/POST with observedApiRoute for REST telemetry; refine typing.
src/app/api/admin/users/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/admin/users/[id]/route.ts	Wrap PATCH with observedApiRoute for REST telemetry.
src/app/api/admin/suspensions/route.ts	Wrap GET/POST with observedApiRoute for REST telemetry.
src/app/api/admin/suspensions/[id]/route.ts	Wrap PATCH with observedApiRoute for REST telemetry.
src/app/api/admin/homeworks/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/admin/homeworks/[id]/route.ts	Wrap DELETE with observedApiRoute for REST telemetry.
src/app/api/admin/descriptions/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/admin/comments/route.ts	Wrap GET with observedApiRoute for REST telemetry.
src/app/api/admin/comments/[id]/route.ts	Wrap PATCH with observedApiRoute for REST telemetry.
docs/observability.md	Document log persistence, request tracing, metrics, readiness, and alerting guidance.
docs/index.md	Link new observability documentation from docs index.
docs/features/upload.json	Document storage observability guarantees for upload flow.
docs/features/openapi.json	Document REST observability + readiness endpoint presence.
docs/features/oauth.json	Document OAuth token endpoint behavior notes (incl. GET + OPTIONS note).
docs/features/mcp.json	Extend MCP feature doc to include per-tool result metrics.
docs/features/_audit.json	Document audit writer now records bounded audit write metrics.
docker-compose.prod.yml	Mount log directory and set APP_LOG_DIR for production Compose.
docker-compose.dev.yml	Mount log directory and set APP_LOG_DIR for dev Compose profile.
.gitignore	Ignore ./logs directory.
.env.example	Add APP_LOG_DIR / APP_LOG_HOST_DIR examples.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

🎉 This PR is included in version 1.32.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀