Skip to content

feat(query): filter by sequencing run accession; fix NA reaching download --from - #43

Merged
thanhleviet merged 3 commits into
mainfrom
feat/query-by-run-accession
Aug 5, 2026
Merged

thanhleviet merged 3 commits into
mainfrom
feat/query-by-run-accession

Conversation

@thanhleviet

@thanhleviet thanhleviet commented Aug 5, 2026 •

Copy link
Copy Markdown
Collaborator

Answers a community question: "is it possible to query ATB using SRR/ERR accessions? Or only biosample IDs? I only have SRR/ERR accessions for ~6000 genomes."

Today it is sample accessions only. run.parquet already ships the mapping and is already a core table, so the data was there - nothing exposed it.

atb query --runs / --run-file

atb query --run-file runs.txt --has-assembly \
  --columns sample_accession,aws_url -o urls.tsv
atb download --from urls.tsv --parallel 8

Run accessions are resolved to sample accessions in the CLI layer, before the engine is chosen, so the SQLite index fast path and the parquet scan both see a plain sample filter. No schema change, no reindex, no second code path to keep in sync. Verified: both engines return identical rows for the same run list.

The resolution is reported rather than silently applied:

Resolved 6000 run accession(s) to 5981 sample accession(s)
5,840 result(s)

Those three numbers differ for real reasons, all covered in the guide:

  • A run may carry several samples. Multiplexed runs are stored as a comma-joined list in sample_accession - 3,651 rows, hiding 10,037 distinct samples. All of them are expanded.
  • Not every resolved sample is in ATB. So the result count can be lower.
  • If nothing resolves, the query fails rather than returning all 3.5M rows. An unresolved filter that silently collapses to "no constraint" is the worst possible failure mode for a filter.

Runs combine with --samples as a union, and with every other filter as an AND. The pass column in run.parquet is deliberately ignored - resolving an accession is a lookup, quality filtering belongs to the query flags.

Fix: NA reaching download --from

Found while testing the above. 464,368 rows carry the literal string NA in aws_url for samples that were never assembled. parseCSVURLs treated it as a value, so atb download --from requested .../NA.fa.gz once per row - a guaranteed 404 each. Confirmed by HEAD probe that those samples have no object on S3 at all.

Now skipped and counted:

Skipping 1065 row(s) with no assembly (aws_url is NA)
Downloading 4775 file(s) to /home/ubuntu/atb/genomes

Verification

Built from source and run against the live database:

Check Result
6,000 runs → samples → URLs 5,981 samples → 5,840 in ATB → 4,775 downloadable, 4.6 s
Index path vs parquet path identical rows
Multiplexed run ERR016606 1 run → 12 samples
--has-assembly vs skip count 5,840 − 4,775 = 1,065, matches exactly
Spot-checked assembly URL HTTP 200
All-bogus run list exits 1, does not dump the database

Tests are TDD throughout: 22 new tests across internal/runs, internal/query and internal/cli, each watched fail first. Full suite green (21 packages), go vet clean, mkdocs build --strict passes, make docs regenerated.

ATB is keyed by sample accession, so a run accession (ERR/SRR/DRR) could
not be used as a filter. --runs and --run-file resolve them through the
run.parquet mapping table before the query runs, so both the SQLite index
and the parquet scan see a plain sample filter.

A run that carries several samples contributes all of them. Accessions
the table does not list are named in a warning; if none resolve the query
fails rather than returning every row.
The published tables record a sample with no assembly as the literal
string NA in aws_url, on 464,368 rows. parseCSVURLs treated that as a
value and turned it into a request for .../NA.fa.gz, which 404s once per
row. Those rows are now skipped and the count reported.
@thanhleviet
thanhleviet merged commit 97969e8 into main Aug 5, 2026
6 checks passed
@thanhleviet
thanhleviet deleted the feat/query-by-run-accession branch August 5, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant