Skip to content

feat(amr): add --species-like wildcard filter - #41

Merged
thanhleviet merged 1 commit into
mainfrom
feat/amr-species-like
Aug 5, 2026
Merged

thanhleviet merged 1 commit into
mainfrom
feat/amr-species-like

Conversation

@thanhleviet

Copy link
Copy Markdown
Collaborator

Closes #33.

atb amr gains --species-like, matching the flag of the same name on atb query, so the command from the issue now works as written:

atb amr --hq-only --species-like "Campylobacter_D jej%" --format tsv --output raw_AMRFP.tsv

Pattern rule

% matches any sequence of characters. Every other character, including _, matches itself. GTDB splits genera into lettered clades (Campylobacter_D, Enterococcus_B), so an underscore has to be literal for those names to be typeable.

Partition narrowing

The AMR store is partitioned per genus, so the species filter does double duty: a row predicate and a file selector. The literal text before the first % decides which partition files can hold a match:

Pattern Files read
Campylobacter_D jej% (literal contains a space, so the genus is exact) Campylobacter_D.parquet, else _other.parquet
Streptococcus% (genus prefix) every Streptococcus*.parquet plus _other.parquet
%coli (leading wildcard) full 1.18 GB scan, with a stderr note explaining why

_other.parquet is always included for prefix patterns because it sweeps up every genus below the 10k-row partition threshold. On the real dataset that is 2,617 of 2,658 genera but only ~2% of samples (64,492 of 3,227,665), so including it is what makes the narrowing complete rather than silently lossy, and it stays cheap.

--species-like also counts as a filter for the "scan everything?" confirmation prompt, so it no longer demands --yes.

Beyond the issue

Two divergences from the documented rule turned up while wiring this in, both now fixed by routing every path through one shared internal/match package:

  1. internal/query/filter.go fell back to an exact string comparison for patterns with an interior wildcard. docs/guides/query.md has been teaching --species-like "Enterococcus%faecium" as the way to search across GTDB naming variants, and that only worked when the query happened to be answered from a SQLite index rather than from parquet.

  2. internal/index/query.go (atb query's index path) passed the pattern to SQL LIKE unescaped, so _ acted as a single-character wildcard. "Escherichia_coli" matched Escherichia coli. This one is a behavior change to an existing command and was outside the change set originally scoped for this issue; it is here because the docs added by this PR assert "_ is literal" for both commands, and that has to be true of both. Real-data impact is near-nil: a _ wildcard also matches a literal _, and no GTDB name has the shape Campylobacter?D, so the only thing lost is accidental matches such as "Escherichia_coli" → Escherichia coli. Called out under Fixed in the changelog.

Verification

  • go build ./... && go vet ./... && go test ./... — 20/20 packages ok
  • mkdocs build --strict — clean; make docs regenerated the CLI reference
  • Unit tests cover partition selection, both execution paths returning identical rows for 7 patterns, and a test that corrupts Escherichia.parquet and deletes its sidecar index so reading an unnecessary partition fails loudly
  • Real data, the issue's exact command against ~58.5M AMR rows: 277,638 rows in 3m09s, 922 MB RSS. Species returned were Campylobacter_D jejuni, jejuni_A, and jejuni_D (the two sub-clades an exact --species match silently drops), genus came back as Campylobacter_D confirming the partition filename derived from the pattern, and no leading-wildcard note was printed. That run had no amr/ partition directory, so it exercised the monolithic worst case; with partitions built the same pattern reads one file
  • atb query --species-like "Campylobacter_D jej%" against the live 3.2M-row index still resolves in 0.46s after the escaping change

Mirror the atb query flag so AMR results can be filtered by a wildcard
species pattern. The literal text before the first % selects which genus
partitions are read, so an anchored pattern such as "Campylobacter_D
jej%" stays on the fast path; a pattern starting with % scans the full
dataset and prints a note explaining why.

Move LIKE matching into internal/match so the parquet and SQLite paths
share one definition: % matches any sequence of characters, and every
other character, including _, matches itself. This resolves two
divergences from that rule:

- The Go matcher fell back to an exact comparison for patterns with an
  interior wildcard, such as "Enterococcus%faecium".
- atb query's index path treated _ as a single-character wildcard, so
  "Escherichia_coli" matched Escherichia coli. It is now literal, which
  is what GTDB clade names such as Campylobacter_D require.

Refs #33
@thanhleviet
thanhleviet merged commit 7963bfe into main Aug 5, 2026
6 checks passed
@thanhleviet
thanhleviet deleted the feat/amr-species-like branch August 5, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exposing --species-like for amr queries

1 participant