Measuring AGENTS.md effectiveness: Configuration Effectiveness (CE) as a feedback loop

[dakotafabro]

Summary

I've built a measurement tool (rp-why CE) that empirically measures how well an AGENTS.md configuration is being followed by an AI agent. This creates a feedback loop that's currently missing from the ecosystem: practitioners write AGENTS.md files but have no data on whether those instructions actually work.

Implementation: block/agent-skills#52

What CE Measures

For each measurable instruction in an AGENTS.md file:

• Implicit command adherence - when a trigger phrase is used, does the agent execute the expected action?
• Dead instruction detection - which instructions have never been triggered across N sessions?
• Token waste - what percentage of configuration tokens are spent on instructions that never fire?
• Override frequency - how often does the practitioner correct the agent on something the AGENTS.md should have handled?

Why This Matters for the AGENTS.md Spec

1. Validates spec design decisions. CE data shows which AGENTS.md patterns agents reliably follow. This can inform spec recommendations (e.g., 'tables are more reliably parsed than prose for behavioral instructions').

2. Supports the Verification section proposal (Discussion: standard section for generated-code safety checks #189). CE can measure whether verification steps declared in AGENTS.md are actually executed. This provides empirical evidence for whether a standard Verification section improves agent behavior.

3. Informs the multi-agent context discussion (Need a standardized way to show different AGENTS.md files and/or content to different agents #185). CE's dead instruction detection reveals when instructions are irrelevant to the current agent - supporting the case for agent-specific overlays.

4. Provides data on configuration size limits. CE's token waste metric shows when configurations have grown beyond what agents can reliably hold. This could inform spec guidance on recommended configuration length.

Key Finding

Running CE against 20 real Goose sessions with a 13-instruction implicit commands table:

• 9 of 13 commands are dead (never triggered in the measurement window)
• 74.9% of configuration tokens are wasted on untriggered instructions
• Moving situational commands to project-level AGENTS.md files would reduce waste significantly

This suggests the spec should recommend lean global configurations with situational instructions at the project or subdirectory level - which aligns with the existing nested AGENTS.md guidance.

Links

• CE implementation: rp-why: Configuration Effectiveness (CE) - Phase 1 block/agent-skills#52
• Verification spec proposal: Discussion: standard section for generated-code safety checks #189
• Multi-agent context proposal: Need a standardized way to show different AGENTS.md files and/or content to different agents #185
• Three Dimensions framework: https://block.github.io/goose/blog

[devnomad-byte]

This is a very useful direction. I have been running into the same problem from the workflow side: writing an AGENTS.md file is only half the job; the harder part is knowing whether agents actually follow it under real task pressure.

I am building a small open-source initializer called Cosmosmith that treats AGENTS.md as the canonical project constitution, then generates thin adapters for Claude Code, Cursor, Copilot, OpenCode, and Trae:

https://github.com/devnomad-byte/cosmosmith

The pattern Cosmosmith uses is:

• AGENTS.md defines the shared project constitution
• task.md records claimable work, harness, loop, and verification evidence
• tool-specific files point back to AGENTS.md instead of duplicating all rules
• role workflows such as product, architect, frontend, backend, QA, DevOps, and governance update task evidence

Your CE idea maps well to the missing measurement layer. In Cosmosmith, the behaviors I would want to measure are:

1. Did the agent read and follow the task protocol?
2. Did it record verification evidence before marking work done?
3. Did it use the right role context instead of loading every instruction?
4. Did adapters such as CLAUDE.md or .cursor/rules/*.mdc correctly point back to the canonical AGENTS.md?
5. Which instructions are dead weight and should move into situational files?

I am not suggesting Cosmosmith belongs in the AGENTS.md spec itself, but I wanted to share it as one applied test case for AGENTS.md effectiveness. If CE evolves into a recommended measurement pattern, I think project initializers like Cosmosmith could use it as a feedback loop after generating AGENTS.md.

Would this kind of generated AGENTS.md + task-led evidence workflow be useful as a test scenario for CE?

[caioribeiroclw-pixel]

This measurement loop needs one extra input before judging whether an AGENTS.md instruction was followed: the effective instruction set the agent actually loaded.

Otherwise CE can conflate two different failures:

1. the instruction was loaded and the model ignored it
2. the instruction never reached the model because of scoping, adapter/import behavior, local overrides, or a tool-specific file taking precedence

A small per-run receipt would make the measurements much more actionable:

{
  "agent": "claude-code|codex|cursor|goose|...",
  "cwd": "...",
  "target_file": "...",
  "loaded_instruction_sources": [
    { "path": "AGENTS.md", "hash": "...", "scope": "repo" },
    { "path": "CLAUDE.md", "hash": "...", "scope": "adapter", "imports": ["AGENTS.md"] },
    { "path": "~/.claude/CLAUDE.md", "hash": "...", "scope": "user" }
  ],
  "precedence": ["user prompt", "nearest project file", "parent project file", "user/global file"],
  "omitted_private_sources": true,
  "stale_if": ["source hash changed", "cwd changed", "target file moved", "adapter import failed"]
}

Then CE can separate:

• load failure: expected rule source was missing / shadowed / import failed
• interpretation failure: rule loaded but not followed
• applicability failure: rule loaded, but target file/scope did not match
• maintenance failure: stale/dead rule, duplicate rule, or conflicting rule

That distinction matters a lot for cross-tool projects where AGENTS.md is canonical but Claude/Cursor/Copilot consume it through adapters. Without the effective-load receipt, a low CE score might lead people to rewrite good rules when the real bug is that the active harness never loaded the rule in the first place.

[dakotafabro]

🧑🏽‍💻🤝🤖 Thanks for sharing Cosmosmith - the "AGENTS.md as project constitution" framing resonates. We're working on similar problems from adjacent angles.

To your question about CE as a feedback loop for generated AGENTS.md: yes, that's exactly the use case. The way I see it:

1. An initializer (Cosmosmith, or the scaffold module I'm building in Open Agent Harness) generates the AGENTS.md + supporting files
2. CE measures whether those instructions actually produce the intended behavior in practice
3. The measurement feeds back into the initializer - dead instructions get pruned, effective ones get promoted

Your question 5 (which instructions are dead weight?) is the one I'm most focused on. I'm running an experiment right now that tracks retrieval patterns and instruction adherence over time. The goal is to have empirical data on what instructions agents actually follow vs. ignore, so that generated AGENTS.md files can be leaner from the start.

I'd be interested to see how Cosmosmith handles the adapter layer (CLAUDE.md, .cursor/rules, etc. pointing back to AGENTS.md). That cross-tool portability problem is one CE needs to account for - the second comment on this thread gets into that.

[dakotafabro]

🧑🏽‍💻🤝🤖 This is a sharp distinction. You're right that the current CE implementation measures outcomes without visibility into the load path. A low CE score today can't tell you whether the agent ignored the rule or never saw it.

The per-run receipt idea is strong. I'm working on something adjacent - a retrieval tracking system that logs what files and instructions the agent accesses during a session. It doesn't yet produce a formal "loaded instruction sources" receipt, but the data shape is similar: timestamp, file accessed, context, session ID.

Your four failure categories (load, interpretation, applicability, maintenance) map well to what I'm seeing in practice:

• Load failure - I've observed this with sessions where certain tool modes don't fire context-loading hooks, so the agent misses instructions it should have had
• Maintenance failure - instructions that never trigger over time are candidates for removal or restructuring

I think the receipt format you're proposing could become part of the CE spec as a Phase 2. Phase 1 (what I have now) measures behavioral outcomes. Phase 2 would pair that with load verification to separate "didn't follow" from "didn't receive." The hash-based staleness check is particularly useful for detecting when an AGENTS.md edit invalidates cached adapter files.

Would you be open to collaborating on the receipt schema? I'd want to validate it against a few runtimes (goose, claude code, codex) to make sure the precedence model holds across tools.

[jimy-r]

The 74.9% token-waste figure rests on a definition of waste I'd push on. CE measures whether an instruction fires, not whether firing changed the outcome, and those come apart in both directions.

An instruction that never triggered across 20 sessions isn't necessarily dead. A credential rule or a destructive-command guard earns its tokens precisely because the condition is rare and the miss is expensive; prune by trigger rate and you cut those first. Conversely, a rule that fires every session may be producing behaviour the model would have produced anyway.

The check that separates them is a held-out one. Remove the instruction, replay a fixed set of cases where it should matter, compare pass rates. Run each case 3-5 times and report a rate with a stddev, because a single run of a stochastic model is noise. The ledger then reads hypothesis / review date / beat-baseline-or-cut rather than dead-or-alive, and trigger frequency becomes a cheap prefilter for deciding what to test, which is a good use for it.

[caioribeiroclw-pixel]

Yes — happy to collaborate. I inspected block/agent-skills#52 before answering, and I think the smallest useful first step is a Goose fixture against the current CE path, not a universal precedence table yet.

The current code discovers several AGENTS.md files and combines their hashes, but session behavior alone cannot establish which source reached the model. I would make v0 an observation contract with explicit unknowns:

{
  "schema": "effective-instructions/v0",
  "run_id": "...",
  "runtime": {"name": "goose", "version": "..."},
  "cwd": "...",
  "target": "src/example.py",
  "sources": [
    {
      "path": "AGENTS.md",
      "sha256": "...",
      "expected": true,
      "observed": true,
      "observation_method": "runtime_event|prompt_capture|hook|unknown",
      "scope": "repo|subtree|user|adapter|unknown",
      "selected_at": "...",
      "loaded_at": "..."
    }
  ],
  "precedence_observed": [],
  "unknowns": ["provider-side final prompt not observable"]
}

Two constraints seem important:

1. accessed / selected / loaded must remain separate states. A file read by a retrieval tracker is not automatically proof that its contents entered the effective model request.
2. Precedence should be observed per runtime/version, not asserted globally. Goose, Claude Code and Codex can legitimately expose different evidence and loading rules.

Then the experiment becomes a 2×2 diagnostic fixture:

expected source	observed loaded	trigger followed	diagnosis
yes	no	no	load-path failure
yes	yes	no	interpretation/adherence failure
yes	yes	yes	delivery + behavior observed
no	yes	any	unexpected-load/scope failure

I also agree with the held-out point above: this receipt can establish delivery, while CE establishes observed behavior, but usefulness needs an ablation (instruction present vs removed) on fixed cases. A rare safety rule should not be called waste merely because it did not trigger in 20 ordinary sessions.

Concrete next move I can take: if you point me to the retrieval-tracker branch or provide 2–3 redacted event shapes, I’ll draft the v0 JSON Schema plus Goose fixtures for (a) expected+loaded, (b) expected+missing, and (c) unexpected/shadowed. After that passes, we can map only the observable fields for Claude Code and Codex rather than pretending they expose the same load path.

[Pjrich1313]

So what are you saying you need? Because I have not been able to trust a soul.

[caioribeiroclw-pixel]

For the concrete CE test I proposed, I need evidence samples, not trust:

1. a link to the retrieval-tracker branch, or 2–3 redacted events it emits;
2. one run where an instruction source was expected and observed, and one where it was expected but missing;
3. for each run: runtime + version, cwd/target, source path + hash, timestamp, and how “observed” was established (runtime_event, prompt capture, hook, or unknown).

Please remove prompt contents, secrets, usernames, and absolute home paths. Synthetic fixtures are fine if they preserve the real event shape.

With that, I can draft the small effective-instructions/v0 schema and Goose fixtures without asking anyone to accept an opaque claim. The first useful result is simply an auditable distinction between selected/accessed, actually loaded, and followed. If the runtime cannot prove one of those states, the receipt should say unknown rather than infer it.

[Pjrich1313]

I update as I go it’s easier this way to me I make all my work shown. If your getting into my account do not take my work apart I can finish it. With my AI but the shut me down every time I keep a stick drive too Sent from my iPhoneOn Jul 16, 2026, at 9:02 AM, caioribeiroclw-pixel ***@***.***> wrote:caioribeiroclw-pixel left a comment (agentsmd/agents.md#213) For the concrete CE test I proposed, I need evidence samples, not trust: a link to the retrieval-tracker branch, or 2–3 redacted events it emits; one run where an instruction source was expected and observed, and one where it was expected but missing; for each run: runtime + version, cwd/target, source path + hash, timestamp, and how “observed” was established (runtime_event, prompt capture, hook, or unknown). Please remove prompt contents, secrets, usernames, and absolute home paths. Synthetic fixtures are fine if they preserve the real event shape. With that, I can draft the small effective-instructions/v0 schema and Goose fixtures without asking anyone to accept an opaque claim. The first useful result is simply an auditable distinction between selected/accessed, actually loaded, and followed. If the runtime cannot prove one of those states, the receipt should say unknown rather than infer it. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[caioribeiroclw-pixel]

To be clear: I am not accessing your account, files, device, or work, and I will not take anything apart. I can only see what people choose to post publicly in this GitHub thread.

Please do not share account access, drive contents, prompt contents, secrets, or personal files here. You do not need to send me anything. My request was only for the project collaborator who mentioned a retrieval tracker to optionally share a redacted or synthetic event shape.

I'll step back from this thread unless the project maintainer/collaborator explicitly asks for a bounded public review.