Repository navigation
feat(amr): add --species-like wildcard filter - #41
Merged
Merged
Conversation
Mirror the atb query flag so AMR results can be filtered by a wildcard species pattern. The literal text before the first % selects which genus partitions are read, so an anchored pattern such as "Campylobacter_D jej%" stays on the fast path; a pattern starting with % scans the full dataset and prints a note explaining why. Move LIKE matching into internal/match so the parquet and SQLite paths share one definition: % matches any sequence of characters, and every other character, including _, matches itself. This resolves two divergences from that rule: - The Go matcher fell back to an exact comparison for patterns with an interior wildcard, such as "Enterococcus%faecium". - atb query's index path treated _ as a single-character wildcard, so "Escherichia_coli" matched Escherichia coli. It is now literal, which is what GTDB clade names such as Campylobacter_D require. Refs #33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #33.
atb amrgains--species-like, matching the flag of the same name onatb query, so the command from the issue now works as written:atb amr --hq-only --species-like "Campylobacter_D jej%" --format tsv --output raw_AMRFP.tsvPattern rule
%matches any sequence of characters. Every other character, including_, matches itself. GTDB splits genera into lettered clades (Campylobacter_D,Enterococcus_B), so an underscore has to be literal for those names to be typeable.Partition narrowing
The AMR store is partitioned per genus, so the species filter does double duty: a row predicate and a file selector. The literal text before the first
%decides which partition files can hold a match:Campylobacter_D jej%(literal contains a space, so the genus is exact)Campylobacter_D.parquet, else_other.parquetStreptococcus%(genus prefix)Streptococcus*.parquetplus_other.parquet%coli(leading wildcard)_other.parquetis always included for prefix patterns because it sweeps up every genus below the 10k-row partition threshold. On the real dataset that is 2,617 of 2,658 genera but only ~2% of samples (64,492 of 3,227,665), so including it is what makes the narrowing complete rather than silently lossy, and it stays cheap.--species-likealso counts as a filter for the "scan everything?" confirmation prompt, so it no longer demands--yes.Beyond the issue
Two divergences from the documented rule turned up while wiring this in, both now fixed by routing every path through one shared
internal/matchpackage:internal/query/filter.gofell back to an exact string comparison for patterns with an interior wildcard.docs/guides/query.mdhas been teaching--species-like "Enterococcus%faecium"as the way to search across GTDB naming variants, and that only worked when the query happened to be answered from a SQLite index rather than from parquet.internal/index/query.go(atb query's index path) passed the pattern to SQLLIKEunescaped, so_acted as a single-character wildcard."Escherichia_coli"matchedEscherichia coli. This one is a behavior change to an existing command and was outside the change set originally scoped for this issue; it is here because the docs added by this PR assert "_is literal" for both commands, and that has to be true of both. Real-data impact is near-nil: a_wildcard also matches a literal_, and no GTDB name has the shapeCampylobacter?D, so the only thing lost is accidental matches such as"Escherichia_coli"→Escherichia coli. Called out under Fixed in the changelog.Verification
go build ./... && go vet ./... && go test ./...— 20/20 packages okmkdocs build --strict— clean;make docsregenerated the CLI referenceEscherichia.parquetand deletes its sidecar index so reading an unnecessary partition fails loudlyCampylobacter_D jejuni,jejuni_A, andjejuni_D(the two sub-clades an exact--speciesmatch silently drops),genuscame back asCampylobacter_Dconfirming the partition filename derived from the pattern, and no leading-wildcard note was printed. That run had noamr/partition directory, so it exercised the monolithic worst case; with partitions built the same pattern reads one fileatb query --species-like "Campylobacter_D jej%"against the live 3.2M-row index still resolves in 0.46s after the escaping change