Repository navigation
feat(query): filter by sequencing run accession; fix NA reaching download --from - #43
Merged
Merged
Conversation
ATB is keyed by sample accession, so a run accession (ERR/SRR/DRR) could not be used as a filter. --runs and --run-file resolve them through the run.parquet mapping table before the query runs, so both the SQLite index and the parquet scan see a plain sample filter. A run that carries several samples contributes all of them. Accessions the table does not list are named in a warning; if none resolve the query fails rather than returning every row.
The published tables record a sample with no assembly as the literal string NA in aws_url, on 464,368 rows. parseCSVURLs treated that as a value and turned it into a request for .../NA.fa.gz, which 404s once per row. Those rows are now skipped and the count reported.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers a community question: "is it possible to query ATB using SRR/ERR accessions? Or only biosample IDs? I only have SRR/ERR accessions for ~6000 genomes."
Today it is sample accessions only.
run.parquetalready ships the mapping and is already a core table, so the data was there - nothing exposed it.atb query --runs/--run-fileRun accessions are resolved to sample accessions in the CLI layer, before the engine is chosen, so the SQLite index fast path and the parquet scan both see a plain sample filter. No schema change, no reindex, no second code path to keep in sync. Verified: both engines return identical rows for the same run list.
The resolution is reported rather than silently applied:
Those three numbers differ for real reasons, all covered in the guide:
sample_accession- 3,651 rows, hiding 10,037 distinct samples. All of them are expanded.Runs combine with
--samplesas a union, and with every other filter as an AND. Thepasscolumn inrun.parquetis deliberately ignored - resolving an accession is a lookup, quality filtering belongs to the query flags.Fix:
NAreachingdownload --fromFound while testing the above. 464,368 rows carry the literal string
NAinaws_urlfor samples that were never assembled.parseCSVURLstreated it as a value, soatb download --fromrequested.../NA.fa.gzonce per row - a guaranteed 404 each. Confirmed by HEAD probe that those samples have no object on S3 at all.Now skipped and counted:
Verification
Built from source and run against the live database:
ERR016606--has-assemblyvs skip countTests are TDD throughout: 22 new tests across
internal/runs,internal/queryandinternal/cli, each watched fail first. Full suite green (21 packages),go vetclean,mkdocs build --strictpasses,make docsregenerated.