Skip to content

feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts - #105

Open
Diego Colombo (colombod) wants to merge 1 commit into
mainfrom
feat/disk-breaker-fail-loud
Open

feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts#105
Diego Colombo (colombod) wants to merge 1 commit into
mainfrom
feat/disk-breaker-fail-loud

Conversation

@colombod

Copy link
Copy Markdown
Collaborator

Independent of #101 — targets main, no duplicated code

This PR adds the disk-pressure resilience + user-visible alerting layer to logging_handler.py. It is independent of #101 and targets main directly: it can merge in any order relative to #101 and shares none of #101's code (0 references to _atomic_write_text / _read_metadata — it only wraps methods that already exist on main).

  • fix: recover from corrupt metadata.json on disk full (ENOSPC) #101 (bkrabach) fixes the corruption: atomic metadata writes + tolerant reads + self-heal of a 0-byte metadata.json.
  • This PR fixes the behaviour under a full disk: stop the every-event failure loop, and make the failure visible to the user (the log file can't be written when the disk is full).

The two are complementary and orthogonal. Related issue: microsoft-amplifier/amplifier-support#492 (this PR is related, not a full close — #101 is the corruption fix). Runaway events.jsonl growth that filled the disk: design issue microsoft-amplifier/amplifier-support#493.

What this adds (and delivers to users)

  1. A full disk no longer spins or spams. An ENOSPC/EDQUOT write failure opens a circuit breaker: disk writes are skipped for a capped exponential-backoff cooldown (5s → … → 300s), then one event probes for recovery — instead of retrying (and failing, and logging) on every event. Never gives up, never hammers.

  2. Disk-full fails loud, where the user can see it. When the disk is full the log file can't be written either, so logger.warning is invisible. Alerts ride HookResult.user_message (surfaced in the UI), rate-limited, severity matched to reality:

    • disk full and the event reached no destination (no server configured, or its queue is also full) → error: "PERMANENT DATA LOSS …"
    • disk full but the event was still delivered to the serverwarning: local log stale, not lost.
    • recovered → one-shot info: logging resumed.
  3. Events keep flowing to the server while the disk is down. Disk and network are independent sinks; the network fan-out runs regardless of disk state (enqueue is a zero-I/O in-memory hot path). enqueue() now returns whether the event was queued, so the handler tells "delivered" from "lost" — the distinction that drives the alert severity above.

Scope / relationship to #101

Verification

  • Full module suite on main + this layer (without fix: recover from corrupt metadata.json on disk full (ENOSPC) #101): 639 passed.
  • New test_logging_handler_disk_breaker.py (10 tests): breaker opens/skips/probes/recovers, capped backoff, non-ENOSPC does not trip it, dispatch stays independent of disk, and the three alert cases — no-destination + disk full = PERMANENT DATA LOSS (error), delivered-to-server = warning, queue-also-full = permanent loss.
  • ruff check / ruff format clean on changed files.

Not covered here (deliberately)

…isible fail-loud alerts

Stacked on #101 (atomic metadata writes + self-heal). Adds the resilience/UX layer on top:

- ENOSPC/EDQUOT circuit breaker: skip disk writes for a capped exponential-backoff cooldown (5s..300s), then probe for recovery, instead of hammering a full disk on every event. Composes with #101 because its atomic writer re-raises OSError so this layer can classify ENOSPC.
- Fail loud to the user via HookResult.user_message (bypasses the unwritable log file), severity matched to reality: PERMANENT DATA LOSS (error) when the event reached no sink (disk full and no destination / queue also full), a milder warning when still delivered to the server, and a one-shot info on recovery.
- enqueue() now returns whether the event was queued so the handler distinguishes delivered from lost; network dispatch stays independent of disk state.

Related: microsoft-amplifier/amplifier-support#492.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant