You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The MVP uses deterministic rules, not calibrated probabilities. RunSift needs a central decision policy that combines rule matches and evidence quality conservatively, resolves conflicts deterministically, and abstains when the result is not safe to state.
Checklist
Define documented rule-priority and specificity policy
Define rule_strength inputs and scale without calling them probability/confidence
Require minimum evidence quality and evidence references for non-unknown results
Detect cross-category conflicts and contradictory evidence
Implement decision_source: rule|abstain for the MVP
Implement stable abstention reasons such as insufficient evidence, conflicting rules, unsupported input, parser degradation, and policy threshold
Keep category decision separate from temporal assessment
Require explicit comparable-attempt evidence before suspected_transient
Prohibit maintainer_confirmed_flaky without maintainer confirmation
Version policy configuration and include policy/rule versions in output
Add boundary, conflict, ordering, ambiguity, and regression tests
Acceptance criteria
Consumers can branch on decision, decision source, category, and abstention reason without undocumented numeric thresholds
Every non-unknown result has at least one redacted evidence reference
Conflict resolution is deterministic and tested independently of rule registration order
Weak or contradictory evidence produces unknown/abstention
Rule scores are clearly typed and never described as calibrated probability
Policy behavior is versioned and reproduced in JSON/golden tests
A one-attempt input cannot confirm flaky behavior
Recommendations remain safe when the policy abstains
Future model boundary
A future decision_source: model may expose calibrated probabilities only after #36 authorizes ML work and a validated evaluation/calibration issue proves the semantics. Model behavior is out of scope here.
Problem statement
The MVP uses deterministic rules, not calibrated probabilities. RunSift needs a central decision policy that combines rule matches and evidence quality conservatively, resolves conflicts deterministically, and abstains when the result is not safe to state.
Checklist
rule_strengthinputs and scale without calling them probability/confidencedecision_source: rule|abstainfor the MVPsuspected_transientmaintainer_confirmed_flakywithout maintainer confirmationAcceptance criteria
unknown/abstentionFuture model boundary
A future
decision_source: modelmay expose calibrated probabilities only after #36 authorizes ML work and a validated evaluation/calibration issue proves the semantics. Model behavior is out of scope here.Dependencies