Skip to content

fix: clean observation dogfood telemetry#43

Merged
cafitac merged 1 commit intomainfrom
fix/observation-dogfood-quality
Apr 30, 2026
Merged

fix: clean observation dogfood telemetry#43
cafitac merged 1 commit intomainfrom
fix/observation-dogfood-quality

Conversation

@cafitac
Copy link
Copy Markdown
Owner

@cafitac cafitac commented Apr 30, 2026

Summary

  • stop storing/emitting query previews for retrieval observations; keep hash/ref/status telemetry only
  • skip deterministic hermes hooks doctor/test pre-LLM payloads as dogfood observations while still exercising context injection
  • add observation audit data-quality warnings for low observation counts and high empty-retrieval ratios
  • lazily migrate memory_status_transitions on existing DB status update/history paths
  • add tests proving Hermes hook context can include retrieved memory content and that actual local Hermes can use injected memory

Verification

  • RED confirmed for query preview output, synthetic doctor observation pollution, missing audit quality warnings, and existing DB memory_status_transitions gap
  • uv run pytest tests/test_cli.py::test_python_module_cli_approve_fact_migrates_existing_database_without_status_transition_table tests/test_cli.py::test_python_module_cli_retrieve_observe_records_secret_safe_local_observation tests/test_cli.py::test_python_module_cli_observations_audit_reports_frequent_and_stale_refs_without_raw_queries tests/test_cli.py::test_python_module_cli_observations_audit_reports_low_signal_empty_retrievals tests/test_cli.py::test_python_module_cli_observations_list_migrates_existing_database_without_observation_table tests/test_cli.py::test_python_module_cli_hermes_pre_llm_hook_outputs_context_for_hermes_shell_hook_payload tests/test_cli.py::test_python_module_cli_hermes_pre_llm_hook_skips_synthetic_doctor_observation tests/test_cli.py::test_python_module_cli_hermes_pre_llm_hook_injects_retrieved_memory_context -q
  • uv run pytest tests/ -q -> 173 passed
  • uv run python scripts/check_release_metadata.py
  • uv run python scripts/smoke_release_readiness.py
  • npm pack --dry-run
  • git diff --check
  • node --check bin/agent-memory.js
  • temp DB dogfood quality smoke: query_preview=None, no raw secret-like query text, audit warnings reported
  • static diff secret scan: finding_count: 0

Live Hermes QA

  • Created a temporary approved local fact with marker AM_LIVE_E2E_1777567838 scoped to /Users/reddit/Project/agent-memory
  • Direct hook output contained <agent_memory_context>, Retrieved fact, and the marker
  • Actual Hermes command hermes --accept-hooks -z "What is the Hermes live E2E QA marker? Return only the marker and nothing else." answered with the marker
  • Cleanup: deprecated the temporary test fact; review explain showed it is hidden from default retrieval
  • Live QA uncovered the existing DB memory_status_transitions migration gap fixed in this PR

@cafitac cafitac merged commit a33596a into main Apr 30, 2026
2 checks passed
@cafitac cafitac deleted the fix/observation-dogfood-quality branch April 30, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant