diff --git a/docs/design/agc-download-command.md b/docs/design/agc-download-command.md index c8d4f77..82c18e5 100644 --- a/docs/design/agc-download-command.md +++ b/docs/design/agc-download-command.md @@ -4,6 +4,17 @@ - **Branch:** `feat/agc-osf-test` (WIP — **do not merge to main**; no push/PR without confirmation) - **Status:** Design approved → implementation +> **Update (v0.18.1):** This rename **shipped** and `atb agc download` is now +> officially released - it graduated from preview in v0.18.0 (2026-08-03) and is +> current in **v0.18.1** (2026-08-04, the Latest release). The "WIP / do not merge" +> and "Branch" notes above are the original point-in-time record and no longer apply. +> The `agc` group has since gained a `locate` subcommand and accession-mode download +> over OSF, and the default batch index is the hosted OSF index TSV (crawling the +> v202505 collection nodes is the fallback). See +> [`agc-osf-balanced-migration.md`](./agc-osf-balanced-migration.md) for the current +> design and [`agc-osf-accession-search.md`](./agc-osf-accession-search.md) for +> `locate`. The surface table below predates `locate`. + ## Problem The genome-retrieval surface mixed two vocabularies. `atb download` (released) diff --git a/docs/design/agc-osf-accession-search-plan.md b/docs/design/agc-osf-accession-search-plan.md index 7ac8485..0a32852 100644 --- a/docs/design/agc-osf-accession-search-plan.md +++ b/docs/design/agc-osf-accession-search-plan.md @@ -1,5 +1,14 @@ # AGC accession → batch search and selective extraction Implementation Plan +> **Status (v0.18.1):** Implemented and officially released. `atb agc locate` and +> accession-mode `atb agc download` shipped through the beta line and are now in the +> current **v0.18.1** release (2026-08-04). All eight tasks below are complete; the +> as-built code is reconciled in the design doc +> ([`agc-osf-accession-search.md`](./agc-osf-accession-search.md)). The collection +> layout has since moved to the balanced v202505 batches with a hosted-index default - +> see [`agc-osf-balanced-migration.md`](./agc-osf-balanced-migration.md). This plan is +> retained as the build record, not a live checklist. + > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Route `atb agc download` accession-mode onto the full OSF v202505 AGC collection and add a read-only `atb agc locate` search command, so users can find which batch holds an accession and extract only the samples they want. diff --git a/docs/design/agc-osf-accession-search.md b/docs/design/agc-osf-accession-search.md index c275129..b76031a 100644 --- a/docs/design/agc-osf-accession-search.md +++ b/docs/design/agc-osf-accession-search.md @@ -1,9 +1,23 @@ # Design: AGC accession → batch search and selective extraction over the full OSF collection -- Date: 2026-07-10 -- Status: Approved (brainstorming), ready for implementation plan +- Date: 2026-07-10 (reconciled with as-built code 2026-07-11) +- Status: **Implemented and released** - `atb agc locate` and accession-mode + `atb agc download` shipped through the beta line and are now in the current + **v0.18.1** release (2026-08-04). The §5 component signatures were reconciled with + the code as merged at beta.3; §3 "Current state" is the pre-change baseline the + design started from, kept for context. - Scope: `internal/sources`, `internal/osf`, `internal/agc`, `internal/cli`, docs +> **Update (v0.18.1):** Since this design was reconciled at beta.3, the collection was +> finalized by the balanced-v202505 migration: the shipped collection nodes are +> `4jq8u`/`jmeqg`/`kzcnr` (not the `6g8by`/`9fqeh`/`xrzub` preview nodes named in §4), +> the folder is `agc_batches` (the `AGCArchivesFolder` constant now holds `agc_batches`, +> not `agc_archives`), and the default index is a single hosted OSF TSV rather than a +> live three-node crawl (which is now the fallback). The §4-§5 node IDs, folder names, +> and loader details are the beta.3 as-built record; see +> [`agc-osf-balanced-migration.md`](./agc-osf-balanced-migration.md) for the current +> design. + ## 1. Motivation Upstream has finished uploading the full **ATB v202505** AGC collection (2.76M genomes, @@ -106,18 +120,21 @@ accession ──(map: atb202505_files_list)──▶ batch name ──(index: 3- - Add a multi-node crawl that, for each `AGCCollectionNodes` entry, resolves the `agc_archives/` folder (`findFolderURL`) and crawls it (`CrawlAGCNode`), stamping `ProjectID` with the node ID. Concatenate into one `osf.Index`. - Suggested signature: `CrawlAGCCollection(nodes []sources.AGCNode) (*Index, error)`. -- Extend `FetchAGCIndex` to drive the multi-node crawl and cache the combined TSV - under `/agc/atb_agc_files.tsv` with the existing `.source` marker + As built: `CrawlAGCCollection(rootURLFor func(nodeID string) string, nodes []sources.AGCNode) (*Index, error)` - + the `rootURLFor` closure (`sources.OSFNodeFilesURL`) is injected so the crawl is testable. +- A dedicated `FetchAGCCollection(cacheDir, rootURLFor, nodes, force)` drives the + multi-node crawl and caches the combined TSV under + `/agc/atb_agc_files.tsv` with the existing `.source` marker (source key = a stable digest of the node set, so changing the node list - invalidates the cache). + invalidates the cache). The single-node `FetchAGCIndex` stays for `--osf-node`. - Folder is chosen **per node**: collection nodes crawl `AGCArchivesFolder` (`agc_archives`); the legacy `z7q5y` node keeps `AGCBatchesFolder` (`agc_batches`) when reached via `--osf-node`. -- `loadAGCIndex`/`useHostedAGCIndex` in `internal/cli` gain a default (no - `--osf-node`, no `--agc-index`) that returns the combined crawl; the single-node - hosted/crawl paths stay for explicit `--osf-node`. **Both** download modes call - this one loader, so `--species` and accession-mode see the same collection. +- As built, `loadAGCBatchIndex` in `internal/cli` is the default dispatcher (no + `--osf-node`, no `--agc-index`): it returns the combined `FetchAGCCollection` + crawl, and delegates to the single-node `loadAGCIndex`/`useHostedAGCIndex` path + for an explicit `--osf-node`. **Both** download modes call this one loader, so + `--species` and accession-mode see the same collection. - No `Entry` schema change: `ProjectID` already carries the node; `part` is derived via `sources.PartForNode`. `SpeciesFromArchive` is unchanged. - Graceful partial availability: a node that is still uploading simply contributes @@ -128,10 +145,13 @@ accession ──(map: atb202505_files_list)──▶ batch name ──(index: 3- - `FetchMap` downloads the artifact and caches it **as downloaded** (the ~11 MB zip) rather than the 148 MB expansion. Cache filename keyed to the artifact (e.g. `agc_file_list.txt.zip`), 7-day + `--refresh` semantics unchanged. -- Introduce a reader that transparently decompresses: if the cached bytes are a - ZIP (magic `PK\x03\x04`), open the single entry via `archive/zip` and stream it; - otherwise read as plain text. This keeps `ParseMap` untouched and stays robust - if upstream later publishes the list uncompressed. +- Introduce a reader that transparently decompresses: if the cached bytes start + with the ZIP magic `PK` (a 2-byte sniff, so a valid-but-empty `PK\x05\x06` + archive still routes to the zip reader and surfaces as an error instead of a + silently empty map), open the single entry via `archive/zip` and stream it; + otherwise read as plain text. The map's first column is always an accession, + never `PK`, so plain text stays unambiguous. This keeps `ParseMap` untouched and + stays robust if upstream later publishes the list uncompressed. - `ParseMap` and `ResolveArchives` are unchanged in behaviour (two-column, whitespace-delimited, exact accession lookup, order-preserving groups). - Scale note: `ParseMap` loads the full map into a `map[string]string` @@ -177,8 +197,9 @@ input conventions. **Does not require the agc binary** and never downloads. - Default TSV columns: `accession`, `batch`, `part`. `json` additionally includes the resolved OSF `url`. Unresolved accessions print `batch = `; known-but-unavailable batches print `part = `. -- Register in `newAGCCmd()`; document alongside `download` (the search half of the - same workflow). +- As built, registered via `newAGCLocateCmd()` in the `agc` command group + (`agc_cmd.go`); documented alongside `download` (the search half of the same + workflow). ### 5.7 Extraction @@ -190,7 +211,7 @@ requested samples via `agc getset`. It inherits OSF support from the bridge (5.4 **`atb agc locate SAMEA2247573 --from ids.txt`** 1. Gather accessions (args + `--from`/stdin), dedupe. 2. `FetchMap` (zip-aware) → `ParseMap` → `ArchiveMap`. -3. `FetchAGCIndex` (3-node crawl, cached) → `RefsFromIndex`. +3. `FetchAGCCollection` (3-node crawl, cached) → `RefsFromIndex`. 4. `ResolveArchives` → groups + unresolved; join with refs for part/URL. 5. Print `accession, batch, part[, url]`. diff --git a/docs/design/agc-osf-balanced-migration.md b/docs/design/agc-osf-balanced-migration.md index 6c40d10..03bc44a 100644 --- a/docs/design/agc-osf-balanced-migration.md +++ b/docs/design/agc-osf-balanced-migration.md @@ -20,6 +20,20 @@ --- +> **As-built (v0.18.1):** This migration shipped as designed - the balanced v202505 +> collection, the hosted-index default with the three-node crawl as fallback, and the +> v0.18.0 graduation all landed - with one deviation from the plan below: the combined +> index was published to OSF node **`h7wzy`**, not **`z7q5y`** as §1, §3.2, §4 (step 3), +> and §6 state. `h7wzy` is the primary ATB project node, which also hosts the parquet +> metadata and sketchlib. The shipped `AGCIndexURL` (`sources.go:148`) is +> `/download/6a719381a2e9e3d202b91f7d/` - node `h7wzy`, folder +> `metadata_balanced_batches/atb_agc_files.tsv`, md5 +> `8aea3d79da3e2a0af10c9904d0c3a10f`, 1,268 batches. The source files `gtqrx` (map) and +> `8y9r2` (batch metadata) are unaffected; only the combined index's home node differs. +> Current authority: the OSF Link Registry in +> [`../reference/data-sources.md`](../reference/data-sources.md) and the +> [v0.18.1 smoke test](./agc-v0.18.1-smoke-test.md). + ## 1. Context: what the collaborator shipped (verified against OSF) The finalized balanced ATB **v202505** AGC collection is published and its figures diff --git a/docs/design/agc-osf-migration.md b/docs/design/agc-osf-migration.md index 57d2aed..09a632e 100644 --- a/docs/design/agc-osf-migration.md +++ b/docs/design/agc-osf-migration.md @@ -14,6 +14,15 @@ > the original point-in-time record; the command and file names below describe the > pre-rename CLI. +> **Update (v0.18.1):** The default AGC batch-index source has moved on from the +> single `z7q5y` test node described here. atb now defaults to a hosted OSF index TSV +> over the finalized balanced **v202505** collection (crawling its three OSF nodes is +> the fallback; `--agc-index`/`--osf-node` override). Accession-mode `atb agc download` +> and the read-only `atb agc locate` are officially released - graduated from preview +> in v0.18.0 (2026-08-03), current in **v0.18.1** (2026-08-04). See +> [`agc-osf-balanced-migration.md`](./agc-osf-balanced-migration.md) for the current +> design and [`agc-osf-accession-search.md`](./agc-osf-accession-search.md) for `locate`. + --- ## 1. Context diff --git a/docs/design/agc-osf-test-implementation.md b/docs/design/agc-osf-test-implementation.md index ce05d32..7dc1e15 100644 --- a/docs/design/agc-osf-test-implementation.md +++ b/docs/design/agc-osf-test-implementation.md @@ -11,6 +11,14 @@ > the original point-in-time record; the command and file names below describe the > pre-rename CLI. +> **Update (v0.18.1):** The single-node `z7q5y` crawl this spec implements is no +> longer atb's default index source. The default is now a hosted OSF index TSV over +> the finalized balanced **v202505** collection (crawling its three OSF nodes is the +> fallback), with accession-mode `atb agc download` and a read-only `atb agc locate` +> added on top - officially released, current in **v0.18.1** (2026-08-04). See +> [`agc-osf-balanced-migration.md`](./agc-osf-balanced-migration.md) for the current +> design and [`agc-osf-accession-search.md`](./agc-osf-accession-search.md) for `locate`. + ## Goal Make `atb fetch-genomes` work **today** against the AGC test archives staged on diff --git a/docs/design/agc-osf-test-summary.md b/docs/design/agc-osf-test-summary.md index 4dab58f..8d642ba 100644 --- a/docs/design/agc-osf-test-summary.md +++ b/docs/design/agc-osf-test-summary.md @@ -11,6 +11,14 @@ > as the original point-in-time record; the command and file names below describe > the pre-rename CLI. +> **Update (v0.18.1):** What this summary delivered against the single `z7q5y` test +> node has been superseded. atb now defaults to a hosted OSF index TSV over the +> finalized balanced **v202505** collection (crawling its three OSF nodes is the +> fallback), and accession-mode `atb agc download` plus the read-only `atb agc locate` +> are officially released, current in **v0.18.1** (2026-08-04). See +> [`agc-osf-balanced-migration.md`](./agc-osf-balanced-migration.md) for the current +> design and [`agc-osf-accession-search.md`](./agc-osf-accession-search.md) for `locate`. + ## Goal (met) Make `atb fetch-genomes` work **today** against the AGC test archives staged on