Skip to content
View jimmyjames177414's full-sized avatar

Block or report jimmyjames177414

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
jimmyjames177414/README.md
jimmyjames177414

I build tools that measure what AI agents actually do

Most tooling around coding agents helps them do more. These mostly help you find out what they already did: what context they needed, what they changed, what they broke, and what they quietly got wrong.

Every one ships a NOVELTY.md: the prior-art search run before any code, including a table of claims the project is not allowed to make. Several were renamed or cut down by that gate. It is the most useful file in each repo.


Watching agents work

rollcall Which worktree does each running agent own, and which work does nobody own? Joins the OS process table, provider session data and git. Anthropic's docs say to remove leftover worktrees by hand; this finds them.
sincelast Your agent read that file four minutes ago. Another agent has changed it since. Read-dependency invalidation for independently-launched sessions.
claude-memory-atlas Renders a directory of Claude Code auto-memory files as one offline interactive knowledge graph.

Measuring what context is worth

stopless Which words does your prompt actually need? Learned stopwords, measured per model.
stopbench How much language does an LLM need? A contributable linguistic-ablation benchmark.
mincontext Delta debugging for AI failures. Shrink a failing agent context to the part that actually causes it.
toolsweep Which tool-schema decision is costing you accuracy? Controlled ablation over function definitions.

Making evaluation honest

errorbars The statistics your LLM eval does not do. Power analysis, paired bootstrap, minimum detectable effect.
inconclusive Your agent test says FAIL. Did the property fail, or did the trace just end? Three-valued verdicts.
knowwhen Every agent benchmark asks whether it finished. This one asks whether it should have started.

Keeping agents accountable

assumptionledger Make an agent state what it is assuming, in a form you can execute against reality.
validwhile Evidence-bound agent memory. Every fact carries an executable check for whether it is still true.

Other things

worker-king Always-on AI desktop assistant for Windows. Listens, and delegates real work to Claude Code.
local-transcription Fully offline Windows transcription with speaker memory, and an optional live meeting AI. No cloud, ever.
claude-roundtable Two Claude personas argue a topic adversarially, streamed live in your terminal.

jimmyjames177414

Everything here is Apache-2.0 or MIT, local-first, and runs without an account.

Pinned Loading

  1. errorbars errorbars Public

    The statistics your LLM eval does not do. Power analysis, paired confidence intervals, and null vs underpowered.

    Python

  2. knowwhen knowwhen Public

    Does an agent know when to ask instead of act? A benchmark for epistemic action choice under uncertainty.

    Python

  3. mincontext mincontext Public

    Delta debugging for AI failures. Shrink a failing agent context to the smallest reproducer someone else can run.

    Python

  4. rollcall rollcall Public

    Roll call for the AI coding agents already running on your machine: which worktree each one owns, and which work nobody owns. Read-only, local-first, cross-provider.

    Python

  5. sincelast sincelast Public

    Tell a coding agent what changed underneath it, before it acts. Read-dependency invalidation for independently-launched Claude Code and Codex sessions. Local-first, zero dependencies.

    Python

  6. stopless stopless Public

    Measure which words your prompt actually needs. Learned stopwords and anti-stopwords, by real ablation rather than proxy scores.

    Python