Skip to content

feat: build autocti_assistant — the CTI calibration assistant cell #136

Description

@Jammy2211

Overview

Build autocti_assistant, the science-assistant cell for CTI calibration with
PyAutoCTI, seeded from autolens_assistant via the Clone Agent. This follows the
CTI resurrection epic (all six phases merged 2026-07-17), which left PyAutoCTI,
autocti_workspace (118 validated scripts) and autocti_workspace_test fully
modern — making the domain assistant the natural next organ of the CTI cell line.

Planning surfaced a blocker that gates every future assistant birth, not just
this one: pyauto-brain clone --apply hard-fails (exit 4) because 39 tracked
files in the reference cell match no template-boundary pattern. Phase 0 clears it
and adds the guard that stops it recurring.

Scope note (intake trap): this is a new assistant cell. PyAutoCTI is a
consumed dependency, not an edit target — bare "PyAutoCTI" mentions must not
steal the classification Target. Filed on PyAutoBrain per the clone-epic
precedent (pyautoscientist-3b-clone → PyAutoBrain#73).

Plan

  • Phase 0 — unblock the birth. Classify the 39 unclassified reference files
    (the euclid mode, the JOSS paper/, .mcp.json, the cosmos_web_ring scripts —
    all shipped after the boundary was last written) in both places that must
    agree: the owning prose and the executable patterns. Add a guard test so the
    next feature landing in the reference can't silently re-block births.
  • Phase 1 — birth the seed. Human gate on repo name/owner/visibility, then
    --apply --mode lightweight-seed. Born private; flips public after Heart's
    publish gate. Register in the body map.
  • Phase 2 — the ac_* skills. Grounded per-skill against the 118 validated
    workspace scripts, never API memory.
  • Phase 3 — wiki + project profile. core/ reference pages and a
    literature/ corpus rebuilt from public sources with verified citations.
  • Phase 4 — demonstrations + validation. End-to-end fits that recover input
    trap models, plus citation/link/cold-chat gates.

Each phase is its own PR (the CTI resurrection phase0..phase5 precedent).

Clone Agent decision

bin/pyauto-brain clone PyAutoCTI --workspace autocti_workspace --reference autolens_assistant:

  • Library: PyAutoCTI @ eca130ca (74 public symbols) · Workspace: 118 scripts
  • Partition: generic 51 · domain 257 · mixed 93 · unclassified 39
  • Mode: lightweight-seedforced, not merely preferred. _clone.py gates
    --apply on --mode lightweight-seed; exact-clone and
    differentiated-sibling are v2 and unimplemented (exit 5). The agent's own
    next-action recommends seed-first, and the prompt's scope sketch (no
    benchmarks/, paper/, modes/, hpc/) is already seed-shaped.

Two decisions worth recording

  1. Visibility. The human answered public, and public is the end state — but
    PyAutoBuild/autobuild/clone_seed.py creates the newborn private and flips
    it public only after Heart's publish gate
    (PyAutoHeart/docs/newborn_validation.md, legs 1–3). That is the machinery's
    own gate and matches this prompt's constraint that a newborn is not announced
    before its validation legs pass. Because the repo ends public, the privacy
    seam stays fully in force
    : PyAutoMemory wiki/cti/ is consulted for pointers
    and structure only, never copied.
  2. PyAutoMemory wiki/cti/ is thin — 1 concept, 2 entities, 8 sources. It is a
    bibliography pointer, not a corpus. Phase 3 is real research, not transcription.
Detailed implementation plan

Affected Repositories

  • PyAutoBrain (primary) — Clone Agent profile + guard test + stale docs
  • autolens_assistant — the reference cell's template boundary (Phase 0)
  • autocti_assistant — the newborn (does not exist until Phase 1)
  • PyAutoMind — body-map registration (Phase 1)
  • PyAutoCTI, autocti_workspace — read-only grounding sources

Branch Survey

Repository Current Branch Dirty?
./PyAutoBrain main clean
./autolens_assistant main clean
./PyAutoMind main clean
./PyAutoCTI main clean
./autocti_workspace main clean

No worktree claims collide (ep-projection-weights, potential-correction-port
are unrelated).

Suggested branch: feature/autocti-assistant
Worktree: ~/Code/PyAutoLabs-wt/autocti-assistant/

Implementation Steps

Phase 0 — unblock the birth (autolens_assistant + PyAutoBrain)

The gate, in agents/conductors/clone/_clone.py:

sets = partition(reference_root, profile)
if sets["unclassified"]:
    fail(4, "unclassified reference files — fix the boundary before a birth")

The boundary lives in two places that must agree: the prose in
autolens_assistant/modes/maintainer.md ("Assistant-as-template", which owns
the boundary) and the executable patterns in _clone.py's REFERENCE_PROFILES.

  1. PyAutoBrain/agents/conductors/clone/_clone.py — extend the
    autolens_assistant profile:
    • skills/euclid_*.md, .claude/skills/euclid_*.md, wiki/euclid/*
      domain (Euclid strong-lensing science; a newborn regrows its own)
    • paper/*domain (autolens's own JOSS paper)
    • scripts/*_cosmos_web_ring.pydomain (lensing science scripts)
    • .mcp.jsonmixed (results-inspector MCP: structure generic, values domain)
  2. autolens_assistant/modes/maintainer.md — mirror the same four judgments into
    the "Assistant-as-template" prose.
  3. PyAutoBrain/tests/test_clone_conductor.pyadd the guard: assert
    partition() over the real autolens_assistant checkout yields zero
    unclassified. No such guard exists today; its absence is exactly why three
    separate features (euclid mode, JOSS paper, MCP) silently re-blocked births.
    Make the trap loud at PR time rather than documenting it.
  4. Docs drift riderPyAutoBrain/AGENTS.md and
    agents/conductors/clone/AGENTS.md both claim the Clone Agent is
    "analysis-only v0 … writes nothing". --apply --mode lightweight-seed is
    implemented and hands a plan to Build. Correct both.

Phase 1 — birth the seed (autocti_assistant)

  • Human interactive gate first — confirm name / owner / visibility (default
    PyAutoLabs/autocti_assistant). Never gh repo create unprompted.
  • bin/pyauto-brain clone PyAutoCTI --workspace autocti_workspace --apply --mode lightweight-seed
    → 51 generic + 93 mixed files copied with substitutions
    (autolens_assistantautocti_assistant, al_ac_, PyAutoLensPyAutoCTI,
    autolensautocti); 257 domain files become the PENDING.md growth queue.
    The al_ac_ derivation is pkg[0]+pkg[4]autocti[4] is c, so the
    prompt's ac_* naming falls out correctly.
  • Heart publish gate legs 1–3 (symbol audit, link sweep, wiki-currency) → flip public.
  • Register in PyAutoMind/repos.yaml (category: assistant), then
    python3 scripts/repos_sync.py --write to regenerate the routing tables.

Phase 2 — the ac_* skills

Grounded per-skill against the 118 validated autocti_workspace/scripts/ and
PyAutoCTI/AGENTS.md — never API memory. Ground truth the skills must teach
(all post-resurrection):

  • Plotting is the matplotlib function API (aplt.subplot_dataset_1d,
    aplt.subplot_imaging_ci, aplt.plot_cti_1d, *_list). There are no
    *Plotter / MatPlot* / Visuals* objects.
  • Multi-dataset fits are factor graphs: af.AnalysisFactor(prior_model=model, analysis=...) per dataset → af.FactorGraphModel(*factors)
    search.fit(model=fg.global_prior_model, analysis=fg); results come back as a
    list. Analysis summing (analysis + analysis) no longer exists.
  • arcticpy install (a guaranteed top-3 user question): needs libgsl-dev + a
    C++ toolchain; pip install numpy cython first, then
    pip install arcticpy==2.6 --no-build-isolation --no-deps — a naive pip install
    downgrades numpy below 2.0. No-root workaround in PyAutoCTI/AGENTS.md; CI
    form in autocti_workspace_test/.github/scripts/smoke_install.sh.
  • PYAUTO_TEST_MODE is the knob (2 = sampler bypass). PYAUTOFIT_TEST_MODE
    does not exist.
    Known artifact: identical-prior ordered-trap models tie at
    prior medians under the bypass and raise their own assertion; real runs are fine.
  • Clocker1D(express=, roe=) vs Clocker2D(parallel_express=, parallel_roe=)
    — different kwargs.
  • Domain surface: Dataset1D/ImagingCI, Layout1D/Layout2DCI + Region
    objects, FPR/EPER extraction (layout.extract, "fpr"/"eper" in 1D,
    "parallel_fpr"… in 2D), CTI1D/CTI2D, trap species + CCDPhase,
    remove_cti, noise scaling / hyper fits, ac.agg.Dataset1DAgg over the
    consolidated dataset.fits format.

Skills: install (incl. arcticpy), simulate 1D + charge injection, compose+fit
(single and factor-graph), masking + FPR/EPER extraction, correction, plotting,
results + aggregator, workspace navigation, test-mode conventions.

Phase 3 — wiki + project profile

  • wiki/core/ — what CTI is, trap physics, FPR/EPER anatomy, the arctic
    algorithm, calibration strategy, parallel vs serial.
  • wiki/literature/ — Massey/Israel-line CTI correction, the arctic papers,
    Euclid VIS calibration, HST ACS CTI history, trap pumping, CTI as a weak-lensing
    shape systematic. Public sources only; every citation verified; bib entries
    verbatim from the literature (the autolens_assistant PR#41/Checkpoint-and-continue: supervised runs batch questions instead of blocking #42 pattern).
  • wiki/project/profile.md — resurrection state, what is/isn't wired (release
    train pending: draft/release/autocti/cti_release_train_wiring.md), Euclid
    heritage in autocti_workspace_test/legacy/.

Phase 4 — demonstrations + validation

Demonstrations double as validation — the assistant must drive these end-to-end:

  1. simulate + calibrate a 1D dataset, recovering input trap density/timescale;
  2. simulate + calibrate a small charge-injection image;
  3. correct a dataset and show the residual improvement;
  4. load results through the aggregator and plot.

Gates: --check-citations clean, link CI green, the demonstration fits actually
run, and Heart leg 4 — the cold-session smoke ("new Opus session + this repo
only" answers an install question and a calibration question correctly).

Key Files

  • PyAutoBrain/agents/conductors/clone/_clone.pyREFERENCE_PROFILES
    (boundary patterns), partition(), apply_seed()
  • PyAutoBrain/tests/test_clone_conductor.py — the new unclassified guard
  • autolens_assistant/modes/maintainer.md — "Assistant-as-template" (owns the boundary)
  • PyAutoBuild/autobuild/clone_seed.py — the Hands primitive that executes the birth
  • PyAutoHeart/docs/newborn_validation.md — the publish gate (legs 1–4)
  • PyAutoMind/repos.yaml + scripts/repos_sync.py — body-map registration
  • autocti_workspace/scripts/ — the 118 validated grounding scripts

Verification

  • Phase 0: bin/pyauto-brain clone PyAutoCTI --workspace autocti_workspace
    reports unclassified 0 and exits 0; the new guard test fails when a file is
    unclassified; --apply --mode lightweight-seed --no-push builds a seed tree in
    scratch without creating a repo.
  • Phase 1: newborn passes Heart legs 1–3; repos_sync.py --write clean.
  • Phase 2/3: python autoassistant/audit_skill_apis.py clean;
    --check-citations clean; link CI green.
  • Phase 4: the four demonstration fits run and recover input trap models; the
    cold chat-surface smoke answers install + calibration questions correctly.

Original Prompt

Click to expand starting prompt
# Build autocti_assistant — the CTI calibration domain assistant cell

Type: feature
Target: autocti_assistant
Repos:
- @autocti_assistant
- @PyAutoBrain
Difficulty: large
Autonomy: supervised
Priority: normal
Status: formalised

Build **autocti_assistant**, the science-assistant cell for CTI calibration with
PyAutoCTI, seeded from the mature reference assistant cells. This follows the
CTI resurrection epic (all six phases merged 2026-07-17 — PyAutoCTI,
autocti_workspace and autocti_workspace_test are fully modern:
`complete/2026/07/cti-resurrection-phase{0..5}.md`), which makes the domain
assistant the natural next organ of the CTI cell line.

## Original request (verbatim)

> can you draft the prompt to write the autocti_assistant, which il will run in
> opus

## Routing / gates (read before starting)

- **Route through the Clone Agent** (`/clone`) at start_dev time for the
  CloneDecision: exact-clone vs sibling vs seed. `autolens_assistant` is the
  mature reference cell (library-domain assistant, closest shape);
  `autofit_assistant` and the ic50_assistant build
  (`complete/2026/07/build_ic50_assistant_from_autofit_assistant.md`) are the
  precedent runs of this playbook.
- **Repo creation is a human interactive gate** — ask name / org / visibility
  as a dedicated question before any `gh repo create`. Suggested default:
  `PyAutoLabs/autocti_assistant`, public (matching autolens_assistant /
  autofit_assistant).
- **Privacy seam**: PyAutoMemory (`wiki/cti/` — trap physics, the arctic
  algorithm, Euclid VIS / HST ACS heritage, per-topic source bibliographies) is
  the user's private knowledge base. Consult it (via the memory faculty) for
  *pointers and structure*, but a public assistant wiki must be **rebuilt from
  the public literature** with verifiable citations — never copy PyAutoMemory
  pages into a public repo. Bib entries verbatim from the literature; validate
  with the assistant `--check-citations` tooling + link CI (the
  autolens_assistant PR#41/#42 pattern).
- **Intake-classification trap** (hit on the ic50 build): bare library mentions
  steal Target — this is a **new assistant cell**, not a PyAutoCTI change.
  PyAutoCTI is a consumed dependency.
- **Judgment tier throughout**: skills and wiki prose are the product (this is
  why it runs in Opus). Mechanical validation runs may be delegated.

## Ground truth the assistant must teach (post-resurrection API only)

Everything is grounded against the live stack — grep
`autocti_workspace/scripts/` (all 118 scripts validated 2026-07-17) and
`PyAutoCTI/AGENTS.md`, never memory of the old API:

- **Plotting is the matplotlib function API**: `aplt.subplot_dataset_1d`,
  `aplt.subplot_imaging_ci`, `aplt.figure_*`, `aplt.plot_array`/`plot_yx`,
  `aplt.plot_cti_1d`, `*_list` combined subplots. There are **no**
  `*Plotter` / `MatPlot*` / `Visuals*` objects.
- **Multi-dataset fits are factor graphs**: `af.AnalysisFactor(prior_model=
  model, analysis=...)` per dataset → `af.FactorGraphModel(*factors)` →
  `search.fit(model=factor_graph.global_prior_model, analysis=factor_graph)`;
  results come back as a list (`result_list[0].max_log_likelihood_fit`).
  Analysis summing (`analysis + analysis`) no longer exists.
- **arcticpy install recipe** (a top-3 user question, guaranteed): needs
  `libgsl-dev` + a C++ toolchain; `pip install numpy cython` first, then
  `pip install arcticpy==2.6 --no-build-isolation --no-deps` — a naive pip
  install **downgrades numpy below 2.0**. No-root header workaround in
  `PyAutoCTI/AGENTS.md`; CI form in
  `autocti_workspace_test/.github/scripts/smoke_install.sh`.
- **Test/fast-mode conventions**: the knob is `PYAUTO_TEST_MODE` (2 = sampler
  bypass; `PYAUTOFIT_TEST_MODE` does not exist). Known artifact: identical-
  prior ordered-trap models tie at prior medians under the bypass and raise
  their own assertion (filed autofit issue) — real runs are fine.
- **Domain surface**: `Dataset1D` / `ImagingCI`, `Layout1D`/`Layout2DCI` +
  Region objects, FPR/EPER extraction (`layout.extract`, region strings
  `"fpr"/"eper"` in 1D and `"parallel_fpr"`… in 2D), `Clocker1D(express=, roe=)`
  vs `Clocker2D(parallel_express=, parallel_roe=)` (different kwargs!),
  `CTI1D`/`CTI2D`, trap species + `CCDPhase`, correction
  (`remove_cti`), noise scaling / hyper fits, the aggregator
  (`ac.agg.Dataset1DAgg` etc. over the consolidated `dataset.fits` format).

## Scope sketch (to be firmed at start_dev / clone-agent time; expect phases)

1. **Seed the cell** from the CloneDecision's reference: AGENTS.md (persona:
   CTI-calibration assistant; include the chat handshake — "chat use requires
   the GitHub connector; tell me if you can actually read the repo"),
   CLAUDE.md pointer, config, citation/link tooling, Makefile/activate
   conventions (mind the venv trap: clear PYTHONPATH).
2. **Skills** (`skills/ac_*.md`, grounded per-skill against
   autocti_workspace's validated scripts): install (incl. arcticpy),
   simulate 1D + charge injection, compose + fit a CTI model (single and
   factor-graph multi-dataset), masking + FPR/EPER extraction, correction,
   plotting (function API), results + aggregator, workspace navigation,
   test-mode/fast-run conventions.
3. **Wiki**: `wiki/core/` (what CTI is, trap physics, FPR/EPER anatomy, the
   arctic algorithm, calibration strategy, parallel vs serial) and
   `wiki/literature/` (concepts/entities/sources for: Massey/Israel-line CTI
   correction, the arctic papers, Euclid VIS CTI calibration, HST ACS CTI
   history, trap pumping, CTI as a weak-lensing shape systematic) — public
   sources only, every citation verified.
4. **Project profile** (`wiki/project/profile.md`): the resurrection state,
   what is and is not wired (release-train wiring pending —
   `draft/release/autocti/cti_release_train_wiring.md`), Euclid heritage in
   `autocti_workspace_test/legacy/`.
5. **Demonstrations** (the assistant must be able to drive these end-to-end,
   and they double as its validation): simulate + calibrate a 1D dataset
   (recover the input trap density/timescale), simulate + calibrate a small
   charge injection image, correct a dataset and show the residual
   improvement, load results through the aggregator and plot.
6. **Validation**: `--check-citations` clean, link CI green, the demonstration
   fits run, and a cold-session smoke ("new Opus session + this repo only"
   answers an install question and a calibration question correctly).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions