Skip to content
View ksdisch's full-sized avatar

Highlights

  • Pro

Block or report ksdisch

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
ksdisch/README.md

Kyle Disch

Data engineer in Chicago. Healthcare data by day — claims, clinical encounters, the unglamorous reconciliation work. Reproducing recent AI-failure research by night, because the fastest way to learn whether a claim holds is to try to measure it yourself.

Everything below links to a repo you can clone and run.


AI-reliability research

I built a repeatable reproduce-and-measure engine and ran it in two lanes — reproductions of recent agent-reliability papers, and a model-internals lineage built on an independently validated Jacobian lens — pre-registered, judge-free, real confidence intervals, nulls reported as headlines.

Start here → the portfolio index, which frames all eight projects as one body of work and states plainly what's unfinished.

Model internals — the J-lens lineage (build the instrument → map with it → audit with it):

dim-stage Is Anthropic's "global workspace" readable in small models? I rebuilt their Jacobian lens independently, validated it bitwise against their reference, and got a pre-registered null across Qwen2.5 0.5B–3B.
mute-map The one effect that survived every control in dim-stage: delete a concept's lens direction from the late band and the model can't say that word — 0/34 on the diagonal vs 363/374 off it. No paper behind this one, which the card says before you can.
hush-gauge Can you tell from the activations that a model is about to leak a secret it was ordered to keep — even on the trials where it never says it? In progress: M0 complete, G0 passes on all three scales.

Agent reliability — behavioral reproductions:

forge-gap How much does each reliability guardrail actually buy on multi-step tool-calling? 67.5% → 100%, +32.5pp [+17.3, +48.0]. The gap is injected, and the chart says so.
decay-pin A safety rule in context is silently abandoned once compaction evicts it — 0/20 → 20/20 violations. Re-pinning the same ~50 tokens restores it to 0/40.
lossy-wall A memory note that keeps a wrong conclusion but drops its source is worse than no memory — the model re-emits the stale answer instead of abstaining. Cross-checked against the paper author's own harness: AGREE.
ghost-patch Do code LLMs knowingly follow a wrong-location repair instruction and compound it? Two nulls, reported as headlines. $1.42 against a $5 guard.

Tooling

claude-config My version-controlled Claude Code setup — slash commands, skills, subagents, global instructions.
task-manager-mcp A local MCP server exposing Todoist to Claude. Built it after finding 87 of 102 of my own invocations were raw API calls for queries Zapier couldn't express. Tests are larger than the implementation.
constellation Asymmetric two-player co-op: a Phaser platformer on the laptop, React puzzles on your phone, glued by a websocket relay.
stopwatchlive Offline-first PWA with retroactive start, Firestore sync behind per-user security rules, and a Capacitor iOS build.

Data engineering

clinical-data-etl Three heterogeneous healthcare datasets → pandera validation → Postgres → dbt → three independent star schemas, orchestrated with Prefect. 56 pytest tests, 98 dbt tests, an SCD2 snapshot, and 10 ADRs explaining why each design call went the way it did.
personal-health-elt Apple Health → Postgres → dbt (27 models) → Streamlit. Idempotent loaders, range-joined HR zones, training-load and recovery marts. The apex mart is a versioned public API with three downstream consumers, so schema changes move in lockstep.

Stack — Python · SQL · dbt · PostgreSQL · Prefect · pandas · Streamlit · pytest · Docker · TypeScript · React

Reach meLinkedIn

Pinned Loading

  1. forge-gap forge-gap Public

    How much does each reliability guardrail close the failure gap of cheap LLMs on multi-step tool-calling? An ablation rig measures every one: 67.5% → 100%, +32.5pp (Newcombe 95% [+17.3, +48.0]). Gap…

    Python

  2. decay-pin decay-pin Public

    Reproduction: an in-context safety rule is silently abandoned once context compaction evicts it (0/20 → 20/20 violations), and re-pinning the same ~50 tokens fully restores compliance (0/40). arXiv…

    Python

  3. ghost-patch ghost-patch Public

    Reproduction: do code LLMs knowingly obey a wrong-location repair instruction and compound unrecoverable 'ghost' errors? Two nulls reported as headlines — awareness doesn't reproduce, single-pass o…

    Python

  4. hush-gauge hush-gauge Public

    Can you tell from the activations that a small model is about to leak a secret it was ordered to keep — even on trials where it never says it? J-lens audit of instructed secrecy. Complete (M0–M4): …

    Python

  5. mute-map mute-map Public

    Cartography of a late-band J-lens output off-switch in small language models — breadth, localization, dose, specificity. Follow-on to dim-stage.

    Python

  6. portfolio portfolio Public

    A portfolio of honest AI-failure reproductions: pre-registered, judge-free measurements of recent agent-reliability & interpretability papers on cheap models — nulls reported as headlines.