Skip to content

measure: statement for ASu-skills v0.3.0 (plugin, claude_code) #68

Description

@Zeming-Yuan

Following #64 — a report for a plugin I use daily: ASu-skills v0.3.0, six skills for a Chinese-language job-search workflow (open-source contribution, résumé rewriting, résumé generation, interview prep, offer tracking).

Produced with:

context-report produce --subject <asu-skills-0.3.0> --kind plugin --target claude_code

Rows

attribute result note
cost.context_tokens PASSED 4,891 tokens across 6 files (make-resume 1,566 … offer 354)
reachability, fault.scriptMissing, fault.interpreterMissing, fault.timeout, fault.malformedOutput, cost.latency_ms NotApplicable "the plugin declares no hooks"
conformance, decision, interference NotAvailable v0.1 producer does not produce these
efficacy NotAvailable paired ablation not run

What I expected vs. what the statement says

I expected the report to answer "does this plugin work in Claude Code?". Instead, ten of eleven rows say the question was not measured, and the one that was measured is a token count. The NotApplicable/NotAvailable split is doing its job — none of this reads as a defect — but the shape is worth recording: a skills-only plugin is probably the most common subject kind for this format, and this is what its report looks like today.

Two things I did not expect:

  1. conformance is NotAvailable. Whether the bundle loads at all in the target agent is the first question I would ask of a plugin report, and it is the one row v0.1 cannot produce.
  2. cost.context_tokens counts six SKILL.md files, but a skill body is only injected when the skill is invoked. The row question is "how many tokens does this artifact add to the agent context window" — for a skills bundle, the always-on surface is the frontmatter, not the full files. conditions does say files: 6, so the report is not misleading on its own terms; I raise it as a question about which surface this row is meant to name.

Statement

asu-skills-0.3.0.claude_code.json
{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    {
      "name": "0.3.0",
      "digest": {
        "sha256": "a5d9e6d6ba42e195af854e01b528c0cf10d648887458a899a8b115f7693290c8"
      }
    }
  ],
  "predicateType": "https://open-coder-ai.github.io/context-report/attestation/v0.1",
  "predicate": {
    "subjectKind": "plugin",
    "target": {
      "name": "claude_code"
    },
    "producer": {
      "id": "https://github.com/open-coder-ai/context-report#local",
      "version": {
        "context-report": "0.1.0"
      }
    },
    "attributes": [
      {
        "attribute": "conformance",
        "basis": "re-derivable",
        "result": "NotAvailable",
        "inputHash": "sha256:3e0fa8abd51afdd3bf0e15556f0e452118647e666db7a77dbd4ebd686963693e",
        "reasoning": "v0.1 producer does not validate the bundle against the target agent's plugin schema"
      },
      {
        "attribute": "reachability",
        "basis": "re-derivable",
        "result": "NotApplicable",
        "inputHash": "sha256:bab6566e7b5d3121f279980d4be04b8569893fcf30063001acbc6c1430f7cad4",
        "reasoning": "the plugin declares no hooks"
      },
      {
        "attribute": "decision",
        "basis": "re-derivable",
        "result": "NotAvailable",
        "inputHash": "sha256:28d71b81ee9019356e3eb89009bc1bfce53b9b07abf72d0fde2647c158346ac0",
        "reasoning": "no declared positive/negative cases were supplied; v0.1 has no decision replay"
      },
      {
        "attribute": "fault.scriptMissing",
        "basis": "re-derivable",
        "result": "NotApplicable",
        "inputHash": "sha256:dc2854213fd9f7ed0fe40d10e30ec46499a05d58309774b65b545bd3f9692a8b",
        "reasoning": "the plugin declares no hooks"
      },
      {
        "attribute": "fault.interpreterMissing",
        "basis": "re-derivable",
        "result": "NotApplicable",
        "inputHash": "sha256:25c8ff47101d41f25b8adc7e4be55b756a1579125c5510f76caebba0f002f47c",
        "reasoning": "the plugin declares no hooks"
      },
      {
        "attribute": "fault.timeout",
        "basis": "re-derivable",
        "result": "NotApplicable",
        "inputHash": "sha256:62bad980030257111f14c22b0016596671ae3e294c4b6aadae382613c8078458",
        "reasoning": "the plugin declares no hooks"
      },
      {
        "attribute": "fault.malformedOutput",
        "basis": "re-derivable",
        "result": "NotApplicable",
        "inputHash": "sha256:849c9c60d203dc9c932784f1d250ee024d4ce3fb5a88347e3532b879abfba73e",
        "reasoning": "the plugin declares no hooks"
      },
      {
        "attribute": "cost.latency_ms",
        "basis": "re-derivable",
        "result": "NotApplicable",
        "inputHash": "sha256:969a69179906189092eb3a1fe1d089d6d6f24a34deb73fc82dabde994a5d4ccb",
        "reasoning": "the plugin declares no hooks"
      },
      {
        "attribute": "cost.context_tokens",
        "basis": "re-derivable",
        "result": "PASSED",
        "inputHash": "sha256:380b5bef9bf748aa243b65770177128d96f6003781b59eb7efacbb3591055633",
        "conditions": {
          "tokenizer": "approx-regex-v1",
          "files": 6
        },
        "values": {
          "per_file": {
            "skills/asu-resume/SKILL.md": 707,
            "skills/asu/SKILL.md": 401,
            "skills/contributor/SKILL.md": 861,
            "skills/interview/SKILL.md": 1002,
            "skills/make-resume/SKILL.md": 1566,
            "skills/offer/SKILL.md": 354
          }
        },
        "measurement": {
          "unit": "tokens",
          "n": 1,
          "min": 4891,
          "max": 4891,
          "mean": 4891
        }
      },
      {
        "attribute": "interference",
        "basis": "re-derivable",
        "result": "NotAvailable",
        "inputHash": "sha256:e2b93680ceacb34e0ba4587ccbb6fb2396b831bdb8cfd595028efa3697d841ae",
        "reasoning": "no co-installed artifacts were declared; v0.1 producer does not measure interference"
      },
      {
        "attribute": "efficacy",
        "basis": "claimed",
        "result": "NotAvailable",
        "reasoning": "efficacy (paired ablation) was not run; use `context-report run` with models"
      }
    ],
    "metadata": {
      "startedOn": "2026-09-09T11:23:05Z",
      "finishedOn": "2026-09-09T11:23:05Z"
    },
    "configuration": [
      {
        "name": "produce-args",
        "digest": {
          "sha256": "6c4bd09df444f42285cb50bc7d4852606838e1bf024ce11b14340ffde18d25a5"
        },
        "mediaType": "application/json"
      }
    ]
  }
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions