Skip to content

feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts (stacked on #101) - #102

Closed
Diego Colombo (colombod) wants to merge 1 commit into
fix/metadata-json-corruption-on-enospcfrom
fix/metadata-atomic-write-and-disk-breaker
Closed

feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts (stacked on #101)#102
Diego Colombo (colombod) wants to merge 1 commit into
fix/metadata-json-corruption-on-enospcfrom
fix/metadata-atomic-write-and-disk-breaker

Conversation

@colombod

@colombod Diego Colombo (colombod) commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #101 — review that first

This PR is stacked on top of #101 (bkrabach — "recover from corrupt metadata.json on disk full"). Its base branch is fix/metadata-json-corruption-on-enospc, so the diff shown here is only the additive hardening layer on top of Brian's fix. When #101 merges, GitHub auto-retargets this to main.

Related issue: microsoft-amplifier/amplifier-support#492. Follow-up design issue for the runaway events.jsonl growth that filled the disk: microsoft-amplifier/amplifier-support#493.

What this adds (and delivers to users)

  1. A full disk no longer spins or spams. An ENOSPC/EDQUOT write failure opens a circuit breaker: disk writes are skipped for a capped exponential-backoff cooldown (5s → … → 300s) instead of being retried on every event; one event then probes for recovery. It never gives up and never hammers a full filesystem. This composes with fix: recover from corrupt metadata.json on disk full (ENOSPC) #101 because Brian's _atomic_write_text re-raises OSError, letting this layer classify ENOSPC.

  2. Disk-full fails loud, where the user can actually see it. When the disk is full the log file can't be written either, so a logger.warning is invisible. Alerts now ride HookResult.user_message (surfaced in the UI), rate-limited, with wording matched to the real severity:

    • disk full and the event reached no destination (no server configured, or its queue is also full) → error: "PERMANENT DATA LOSS …"
    • disk full but the event was still delivered to the serverwarning: local log stale, not lost.
    • recovered → one-shot info: logging resumed.
  3. Events keep flowing to the server while the disk is down. Disk and network are independent sinks; the network fan-out runs regardless of disk state (enqueue is a zero-I/O in-memory hot path). enqueue() now returns whether the event was queued, so the handler can tell "delivered" from "lost" — which is exactly the distinction that decides the alert severity above.

Scope

  • Additive only. Does not touch Brian's helpers or the four metadata functions; wraps them.
  • logging_handler.py internal logic only — no hook wiring, config, bundle-structure, or auth changes.

Verification

  • Full module suite (Brian's fix + this layer): 649 passed.
  • New test_logging_handler_disk_breaker.py (10 tests): breaker opens/skips/probes/recovers, capped backoff, non-ENOSPC does not trip it, dispatch stays independent of disk, and the three alert cases — no-destination + disk full = PERMANENT DATA LOSS (error), delivered-to-server = warning, queue-also-full = permanent loss.
  • ruff check / ruff format clean on changed files.

Not covered here (deliberately)

@colombod
Diego Colombo (colombod) force-pushed the fix/metadata-atomic-write-and-disk-breaker branch from 370ecb7 to 319377f Compare August 26, 2026 15:24
@colombod
Diego Colombo (colombod) changed the base branch from main to fix/metadata-json-corruption-on-enospc August 26, 2026 15:25
@colombod Diego Colombo (colombod) changed the title fix(hook-context-intelligence): atomic metadata writes, self-heal, and ENOSPC disk circuit breaker feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts (stacked on #101) Aug 26, 2026
…isible fail-loud alerts

Stacked on #101 (atomic metadata writes + self-heal). Adds the resilience/UX layer on top:

- ENOSPC/EDQUOT circuit breaker: skip disk writes for a capped exponential-backoff cooldown (5s..300s), then probe for recovery, instead of hammering a full disk on every event. Composes with #101 because its atomic writer re-raises OSError so this layer can classify ENOSPC.
- Fail loud to the user via HookResult.user_message (bypasses the unwritable log file), severity matched to reality: PERMANENT DATA LOSS (error) when the event reached no sink (disk full and no destination / queue also full), a milder warning when still delivered to the server, and a one-shot info on recovery.
- enqueue() now returns whether the event was queued so the handler distinguishes delivered from lost; network dispatch stays independent of disk state.

Related: microsoft-amplifier/amplifier-support#492.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod
Diego Colombo (colombod) force-pushed the fix/metadata-atomic-write-and-disk-breaker branch from 319377f to 30d2fa9 Compare August 27, 2026 09:15
@colombod

Copy link
Copy Markdown
Collaborator Author

Superseded by #105, which targets main directly and is independent of #101 (no duplicated code). GitHub does not allow re-basing a PR opened as part of a stack, so this one is closed in favour of the fresh independent PR. Same commit, same content.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant