Skip to content

[Decision Policy] Implement rule priority, score semantics, conflict handling, and abstention #24

Description

@Dyu20705

Problem statement

The MVP uses deterministic rules, not calibrated probabilities. RunSift needs a central decision policy that combines rule matches and evidence quality conservatively, resolves conflicts deterministically, and abstains when the result is not safe to state.

Checklist

  • Define documented rule-priority and specificity policy
  • Define rule_strength inputs and scale without calling them probability/confidence
  • Require minimum evidence quality and evidence references for non-unknown results
  • Resolve same-category duplicate matches deterministically
  • Detect cross-category conflicts and contradictory evidence
  • Implement decision_source: rule|abstain for the MVP
  • Implement stable abstention reasons such as insufficient evidence, conflicting rules, unsupported input, parser degradation, and policy threshold
  • Keep category decision separate from temporal assessment
  • Require explicit comparable-attempt evidence before suspected_transient
  • Prohibit maintainer_confirmed_flaky without maintainer confirmation
  • Version policy configuration and include policy/rule versions in output
  • Add boundary, conflict, ordering, ambiguity, and regression tests

Acceptance criteria

  • Consumers can branch on decision, decision source, category, and abstention reason without undocumented numeric thresholds
  • Every non-unknown result has at least one redacted evidence reference
  • Conflict resolution is deterministic and tested independently of rule registration order
  • Weak or contradictory evidence produces unknown/abstention
  • Rule scores are clearly typed and never described as calibrated probability
  • Policy behavior is versioned and reproduced in JSON/golden tests
  • A one-attempt input cannot confirm flaky behavior
  • Recommendations remain safe when the policy abstains

Future model boundary

A future decision_source: model may expose calibrated probabilities only after #36 authorizes ML work and a validated evaluation/calibration issue proves the semantics. Model behavior is out of scope here.

Dependencies

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:classificationFailure classification, confidence, and explanation signalsarea:coreCore domain models and cross-cutting application logicarea:reportingCLI, Markdown, and machine-readable reportspriority:criticalCritical path or release-blocking worktype:featureUser-facing product capability

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions