Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,13 +137,14 @@ See the `Tracker` trait in `crates/jilog-review/src/tracker.rs` to implement you

## Signal anatomy

Four signal types are detected today. Two more (`Pattern`, `Deferral`) are reserved in the enum for forward compatibility but no detector produces them yet.
Five signal types are detected today. One more (`Pattern`) is reserved in the enum for forward compatibility but no detector produces it yet.

| Signal | What triggers it | Detector heuristic |
|---------------|---------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
| `Correction` | User pushes back on an assistant turn in 15–200 chars | `assistant → user → assistant` window with a short user message in the middle |
| `Error` | A tool call returned a structured failure | A `role: tool` message whose JSON content has `success: false` |
| `Workaround` | Assistant text admits a temporary or hacky path | First-match across `for now`, `temporary`, `workaround`, `hardcoded`, `TODO`, `FIXME`, `quick fix`, `hack` |
| `Deferral` | Assistant text postpones work to a later session | First-match across `come back to`, `defer`, `punt on`, `leave for later`, `skipping for now`, `park for now`, `next session`, `circle back` |
| `P0 Alert` | The same tool failed in **3+ distinct root sessions** in the window | Aggregation pass over `Error` signals (sub-agent sessions with the all-zero prefix are excluded) |

Each emitted signal carries the `session_id` it came from, so digest and tracker entries always link back to the conversation that produced them.
Expand All @@ -162,6 +163,7 @@ p0_count: 1
corrections: 3
errors: 2
workarounds: 2
deferrals: 0
---

# Learning Digest — 2026-05-09
Expand All @@ -185,6 +187,10 @@ workarounds: 2

- `2f1c8d77-91ab-4c5d-8e6f-1a2b3c4d5e6f` pattern=`for now`: 'Hardcoding the retry count to 3 for now until we wire it through config.'
- `8a4b1e09-3344-4abc-9def-0123456789ab` pattern=`TODO`: 'TODO: replace this fallback path once the upstream module exposes the new API.'

## Deferrals

_No deferrals detected._
````

The frontmatter is machine-parseable. A common downstream check is "did yesterday's digest go silent on P0?" — `yq '.p0_count' learning-digest-*.md` gives a per-day series.
Expand Down Expand Up @@ -224,6 +230,7 @@ The issue title is built by `signal_title()` (see `crates/jilog-review/src/track
[jilog/correction] <session_id>: <truncated context>
[jilog/error] <tool_name>: <truncated message>
[jilog/workaround] <pattern>: <truncated context>
[jilog/deferral] <session_id>: <truncated item>
```

A run that produced the digest above would land entries like these in `.beads/issues.jsonl` with `tracker.type = "beads"` (one JSON object per line, shown pretty-printed):
Expand Down
145 changes: 142 additions & 3 deletions crates/jilog-review/src/detectors.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
//! Signal detectors — heuristic analysis of transcript messages.
//!
//! All four detectors are pure functions over `&[Message]`.
//! All five detectors are pure functions over `&[Message]`.
//! Ported verbatim from opsctl/crates/opsctl/src/review_nightly.rs.

use std::collections::{BTreeSet, HashMap};
Expand All @@ -9,7 +9,7 @@ use std::sync::OnceLock;
use regex::RegexSet;

use crate::reader::Message;
use crate::signal::{Correction, ErrorSignal, Workaround};
use crate::signal::{Correction, DeferralSignal, ErrorSignal, Workaround};

// ---------------------------------------------------------------------------
// Constants
Expand Down Expand Up @@ -51,6 +51,32 @@ const WORKAROUND_LABELS: &[&str] = &[
"hack",
];

/// Deferral pattern regexes — at most ONE deferral per message; first-match wins.
const DEFERRAL_PATTERNS: &[&str] = &[
r"(?i)\bI'?ll come back to (this|that|it)",
r"(?i)\bdeferr?ing (this|that|it|until)",
r"(?i)\bdefer (this|that|it)(?: (to|until|for))?",
r"(?i)\bpunt(ing)? on (this|that|it)",
r"(?i)\bleav(e|ing) (this|that|it) for (later|now|next)",
r"(?i)\bskipping for now",
r"(?i)\bpark(ing)? (this|that|it) for now",
r"(?i)\bnext session",
r"(?i)\bcircle back (to|on)",
];

/// Human-readable labels (parallel to DEFERRAL_PATTERNS by index).
const DEFERRAL_LABELS: &[&str] = &[
"come back later",
"deferring",
"defer",
"punt",
"leave for later",
"skipping for now",
"park for now",
"next session",
"circle back",
];

/// Sub-agent session prefix (16 zeros). Sessions whose ID starts with
/// this are excluded from P0 alert counting.
const SUB_AGENT_PREFIX: &str = "0000000000000000";
Expand Down Expand Up @@ -254,7 +280,46 @@ pub(crate) fn extract_assistant_text(content: &Option<serde_json::Value>) -> Str
}

// ---------------------------------------------------------------------------
// Heuristic 4: P0 alert detection
// Heuristic 4: Deferral detection
// ---------------------------------------------------------------------------

/// Compile the deferral regex set once.
fn deferral_regex() -> &'static RegexSet {
static SET: OnceLock<RegexSet> = OnceLock::new();
SET.get_or_init(|| RegexSet::new(DEFERRAL_PATTERNS).expect("deferral patterns must compile"))
}

/// Detect deferral language in assistant messages. At most one
/// deferral per message (first-match wins, in declaration order).
pub fn detect_deferrals(messages: &[Message], session_id: &str) -> Vec<DeferralSignal> {
let regex = deferral_regex();
let mut out = Vec::new();
for msg in messages {
if msg.role.as_deref() != Some("assistant") {
continue;
}
let text = extract_assistant_text(&msg.content);
if text.is_empty() {
continue;
}
let matches = regex.matches(&text);
if !matches.matched_any() {
continue;
}
// Find the FIRST matched pattern in declaration order.
let first_idx = matches.iter().next().expect("matched_any is true");
let label = DEFERRAL_LABELS.get(first_idx).copied().unwrap_or("unknown");

out.push(DeferralSignal {
session_id: session_id.to_string(),
item: label.to_string(),
});
}
out
}

// ---------------------------------------------------------------------------
// Heuristic 5: P0 alert detection
// ---------------------------------------------------------------------------

/// Group errors by tool, count distinct ROOT sessions (skip sub-agents).
Expand Down Expand Up @@ -511,6 +576,80 @@ mod tests {
assert_eq!(out[0].context.chars().count(), 200);
}

// ---------- deferrals ----------

#[test]
fn deferrals_basic_match() {
let msgs = vec![assistant("I'll come back to this after the tests pass.")];
let out = detect_deferrals(&msgs, "s1");
assert_eq!(out.len(), 1);
assert_eq!(out[0].session_id, "s1");
assert_eq!(out[0].item, "come back later");
}

#[test]
fn deferrals_short_text_detected() {
let msgs = vec![assistant("next session")];
let out = detect_deferrals(&msgs, "s1");
assert_eq!(out.len(), 1);
assert_eq!(out[0].item, "next session");
}

#[test]
fn deferrals_first_match_wins() {
let msgs = vec![assistant("I'll come back to this next session.")];
let out = detect_deferrals(&msgs, "s1");
assert_eq!(out.len(), 1);
assert_eq!(out[0].item, "come back later");
}

#[test]
fn deferrals_user_role_skipped() {
let msgs = vec![user("please punt on this until next session")];
assert!(detect_deferrals(&msgs, "s1").is_empty());
}

#[test]
fn deferrals_tool_blocks_excluded() {
let msgs = vec![Message {
role: Some("assistant".into()),
content: Some(json!([
{"type": "tool_use", "name": "bash", "input": {"note": "next session"}},
{"type": "text", "text": "I ran the command."},
])),
name: None,
}];
let out = detect_deferrals(&msgs, "s1");
assert!(out.is_empty(), "tool_use blocks must be skipped");
}

#[test]
fn deferrals_no_match_returns_empty() {
let msgs = vec![assistant("All requested work is complete.")];
assert!(detect_deferrals(&msgs, "s1").is_empty());
}

#[test]
fn deferrals_label_emitted_correctly() {
let msgs = vec![assistant("Let me defer that until the next pass.")];
let out = detect_deferrals(&msgs, "s1");
assert_eq!(out.len(), 1);
assert_eq!(out[0].item, "defer");
}

#[test]
fn deferrals_overlap_with_workaround_both_fire() {
let msgs = vec![assistant(
"Skipping for now while we validate the migration.",
)];
let workarounds = detect_workarounds(&msgs, "s1");
let deferrals = detect_deferrals(&msgs, "s1");
assert_eq!(workarounds.len(), 1);
assert_eq!(workarounds[0].pattern, "for now");
assert_eq!(deferrals.len(), 1);
assert_eq!(deferrals[0].item, "skipping for now");
}

// ---------- p0 alerts ----------

#[test]
Expand Down
Loading
Loading