rp-why: Configuration Effectiveness (CE) - Phase 1

rp-why/references/configuration-effectiveness.md

@@ -0,0 +1,73 @@
+# Configuration Effectiveness (CE)
+
+CE measures alignment between declared agent configuration (AGENTS.md) and
+observed session behavior. It feeds into ADT as empirical grounding for
+trust assessment.
+
+## What CE Measures
+
+CE answers: "Is your configuration investment paying off?"
+
+It is NOT a compliance check. It is a return-on-investment signal. Dead
+instructions are wasted tokens. Unreliable commands are friction. Untested
+boundaries are false confidence. CE makes all of this visible and actionable.
+
+## Relationship to Three Dimensions
+
+CE is not a fourth dimension. It is an input to ADT that makes trust
+empirical rather than inferred.
+
+- DOK answers: "How sophisticated is the human's thinking?"
+- TM answers: "How sophisticated is the tool orchestration?"
+- ADT answers: "How much trust is warranted?" (now informed by CE)
+- CE answers: "Is the declared configuration serving the actual workflow?"
+
+## CE Score
+
+Range: 0.0 - 1.0
+
+| Range | Band | Interpretation |
+|-------|------|----------------|
+| 0.85 - 1.0 | Optimized | High alignment between declaration and execution |
+| 0.7 - 0.85 | Well-tuned | Configuration is serving the workflow |
+| 0.5 - 0.7 | Developing | Some instructions land, others drift |
+| 0.3 - 0.5 | Under-effective | Significant gaps between intent and behavior |
+| 0.0 - 0.3 | Misconfigured | Agent is not following most instructions |
+
+## ADT x CE Quadrants
+
+| Quadrant | ADT | CE | Meaning |
+|----------|-----|-----|---------|
+| Reckless Trust | High | Low | Delegating heavily but config isn't holding |
+| Earned Autonomy | High | High | Trust justified by data. Target state. |
+| Justified Caution | Low | Low | Not delegating, config not working. Fix config first. |
+| Ready to Trust | Low | High | Config proven. Safe to extend autonomy. |
+
+## Measurement Frequency
+
+CE is event-driven, not polled every session:
+
+- AGENTS.md hash changes (file edited) - new measurement cycle
+- Rolling window boundary (20 sessions) - periodic drift check
+- Manual invocation (`/rp-why ce`) - on-demand
+
+## Phase 1: Implicit Commands
+
+Phase 1 measures implicit command adherence only. The parser extracts
+trigger/action pairs from the Implicit Commands table in AGENTS.md and
+checks session history for evidence of correct execution.
+
+Detection uses action fingerprinting: observable signals in tool calls
+and agent responses that indicate the expected action occurred.
+
+## Dead Instruction Penalty
+
+Instructions that never fire across the measurement window reduce the
+CE score. They consume context window tokens without return. CE surfaces
+them with recommendations (remove, move to project-level, or keep if
+safety-critical).
+
+## Privacy
+
+CE runs locally. No data leaves the machine. No comparison between
+practitioners. The measurement serves the practitioner, not an observer.

rp-why/scripts/goose_skill.py

@@ -3,20 +3,27 @@
 Goose Skill: rp-why - Three Dimensions of AI Collaboration
 
 Provides DOK assessments, baseline analysis, comparison reports,
-and longitudinal analysis directly within Goose conversations.
+longitudinal analysis, and configuration effectiveness measurement
+directly within Goose conversations.
 
 Commands:
     /rp-why init      - Generate baseline from all session history
     /rp-why baseline  - Same as init
     /rp-why current   - Analyze current session
     /rp-why compare   - Compare current session to baseline
     /rp-why overall   - Full longitudinal report
+    /rp-why ce        - Configuration Effectiveness report
 """
 
 from __future__ import annotations
 
 from rp_why_baseline import RPWhyAnalyzer
-from rp_why_core import aggregate_session_metadata
+from rp_why_core import aggregate_session_metadata, calculate_adt_zone, estimate_tm_tier
+from rp_why_ce import (
+    parse_implicit_commands, find_agents_md_files, compute_ce_report,
+    format_ce_report, check_config_changed, save_config_version,
+    load_runs_from_db, ConfigVersion, hash_file,
+)
 from typing import Dict
 from pathlib import Path
 import json
@@ -254,6 +261,48 @@ def token_spend_report(self) -> str:
             self.analyzer.close()
 
 
+    def ce_report(self) -> str:
+        """Configuration Effectiveness report for AGENTS.md adherence."""
+        from datetime import datetime as dt
+
+        agents_files = find_agents_md_files()
+        if not agents_files:
+            return "\nNo AGENTS.md files found. CE requires at least one configuration file."
+
+        all_commands = []
+        for fp in agents_files:
+            all_commands.extend(parse_implicit_commands(fp))
+
+        if not all_commands:
+            return "\nNo implicit commands found in AGENTS.md. CE Phase 1 measures implicit command adherence."
+
+        sessions_data = load_runs_from_db(limit=20)
+        if not sessions_data:
+            return "\nNo session history found."
+
+        baseline = self.analyzer.load_baseline()
+        adt_zone = "Expected"
+        if baseline:
+            adt_zone = baseline.get("three_dimensions", {}).get("adt_zone", "Expected")
+
+        report = compute_ce_report(all_commands, sessions_data, adt_zone)
+
+        data_dir = str(Path.home() / ".config" / "goose")
+        changed, current_hash = check_config_changed(data_dir)
+        if changed and current_hash:
+            version = ConfigVersion(
+                file_path="|".join(agents_files),
+                content_hash=current_hash,
+                first_seen=dt.now().isoformat(),
+                last_measured=dt.now().isoformat(),
+                session_count=len(sessions_data),
+                ce_score=report.overall_score,
+            )
+            save_config_version(data_dir, version)
+
+        return format_ce_report(report)
+
+
 # --- Goose skill interface functions ---
 
 def init() -> str:
@@ -337,6 +386,26 @@ def token_spend() -> str:
     return skill.token_spend_report()
 
 
+def ce() -> str:
+    """
+    Configuration Effectiveness report.
+
+    Measures alignment between declared AGENTS.md configuration and
+    observed session behavior. CE feeds into ADT (Agentic Delegation
+    Trust) as empirical grounding for trust assessment.
+
+    Usage:
+        /rp-why ce
+
+    Returns:
+        CE score, implicit command adherence rates, dead instruction
+        detection, token waste analysis, ADT x CE quadrant, and
+        growth nudges for configuration improvement.
+    """
+    skill = RPWhySkill()
+    return skill.ce_report()
+
+
 if __name__ == '__main__':
     import sys
 
@@ -356,7 +425,9 @@ def token_spend() -> str:
         print(overall())
     elif command in ('token-spend', 'tokens'):
         print(token_spend())
+    elif command == 'ce':
+        print(ce())
     else:
         print(f"Unknown command: {command}")
-        print("Usage: python goose_skill.py [init|baseline|current|compare|overall|token-spend]")
+        print("Usage: python goose_skill.py [init|baseline|current|compare|overall|token-spend|ce]")
         sys.exit(1)