Skip to content

feat(query): add atb columns and validate --columns names - #42

Merged
thanhleviet merged 3 commits into
mainfrom
feat/columns-command
Aug 5, 2026
Merged

thanhleviet merged 3 commits into
mainfrom
feat/columns-command

Conversation

@thanhleviet

Copy link
Copy Markdown
Collaborator

Closes #34.

--columns had no discoverable list of names, and a name that was not a column silently produced a blank column and exit status 0. Auditing the accepted names to answer the issue turned up a second problem: nine of them never returned data on the parquet path.

atb columns

A new command lists all 40 columns, grouped by the table each comes from, with * marking the ones the SQLite index cannot answer:

$ atb columns
...
sylph
  * Adjusted_ANI         Nucleotide identity to the species reference, coverage-adjusted
  * Taxonomic_abundance  Percentage of the sample's cells estimated to be the species
  * Sequence_abundance   Percentage of the sample's reads assigned to the species
  * Median_cov           Median k-mer coverage of the species reference

mlst
    mlst_scheme   MLST scheme applied, - when the species has none
    mlst_st       Sequence type assigned by the scheme
...
40 columns. * marks one the SQLite index cannot answer: asking for it
sends the query to the parquet files, which is slower.

--format tsv|csv|json prints the same list for scripts. The command reads no data, so it works before anything has been downloaded.

Validation (breaking)

An unrecognised name is now an error, raised before the index is opened:

$ atb query --species "Escherichia coli" --columns sample_accession,N5O
Error: unknown column "N5O" - did you mean "N50" or "N90"? Run 'atb columns' to list all
$ echo $?
1

Names from a TOML filter file are checked too, after the merge, since they never pass through the flag. Scripts that pass a name that is not a column will start failing where they previously exited 0 with a blank column.

Suggestions use a new suggest.Identifiers. suggest.Suggest's dist*4 <= maxLen gate is tuned for species names and demands an exact match at three characters, so N5O would have suggested nothing. Identifiers accepts a distance of 2, the bound cobra uses for command names.

Fix: sylph and mlst returned nothing on the parquet path

Plan() reads sylph.parquet and mlst.parquet when one of their columns is requested, but buildRow() never carried the values onto the result. The map lookup in Execute turns a missing key into "", so the query exited 0 and looked like a sample with no MLST record rather than a gap in the executor.

Before / after, on real data:

$ atb query --species "Escherichia coli" --limit 3 --columns sample_accession,mlst_st,mean_length
sample_accession   mlst_st  mean_length          sample_accession  mlst_st  mean_length
SAMN40589622                18645.92        ->   SAMN40589622      14141    18645.92
SAMN41155940                8233.3               SAMN41155940      152      8233.3
SAMN41388190                8258.6               SAMN41388190      152      8258.6

The two execution paths now agree. Asking the SQLite fast path and the parquet path for the same two samples returns identical rows, including a genuinely absent MLST record:

sample_accession   mlst_scheme      mlst_st  mlst_status
SAMEA117870366
SAMEA7472386       ecoli_achtman_4  52       PERFECT

Guarding against the next drift

The defect class here is two lists of the same thing that nothing compared: Plan() decided which files to read from the column list, buildRow() decided which keys to emit from a hardcoded struct. Three tests now hold the registry to the code:

  • Plan reads the source file of every registered column.
  • Every column Plan routes is registered, under the same source.
  • InIndex agrees with userColToSQL, in both directions.

Each was verified by breaking the invariant deliberately and checking the failure message before being kept.

Docs

docs/reference/columns.md is generated from the registry by make docs, moving it out of the hand-written tier that drifts silently. The CI staleness gate now covers it as well as docs/reference/cli. The query guide links to it.

output.DefaultColumns is deleted in its own commit: it listed organism, tissue, age, sex and disease, none of which are columns in this dataset, and nothing referenced it.

Plan() reads sylph.parquet and mlst.parquet when one of their columns is
requested, but buildRow() never carried the values onto the result, so
Adjusted_ANI, Taxonomic_abundance, Sequence_abundance, Median_cov and the
five mlst_* columns came back as empty strings whenever a query could not
be answered from the SQLite index. The map lookup in Execute turns a
missing key into "", so the query exited 0 and looked like a sample with
no MLST record rather than a gap in the executor.

Carry both tables through sampleData the way checkm2 and assembly_stats
already are, and add a sylph fixture so the executor tests cover the join
rather than the file being opened.
The list named organism, tissue, age, sex and disease, none of which are
columns in this dataset, and nothing referenced it. Default output comes
from InferColumns, which reads the keys the query actually produced.
There was no way to find out which names --columns accepts, and a name
that was not a column produced a blank column and exit status 0, so a
typo looked like missing data.

Add internal/columns, one registry of the 40 columns with the table each
comes from and whether the SQLite index can answer it. atb columns prints
it grouped by table, or as tsv/csv/json for scripts, and reads no data so
it works before anything has been downloaded. docs/reference/columns.md is
generated from the same registry by make docs, and the CI staleness gate
now covers it.

atb query rejects an unrecognised name and exits 1 before opening the
index, naming the closest columns when any are close. Names from a TOML
filter file are checked after the merge, since they never pass through the
flag. This is a breaking change for scripts that pass a name that is not
a column.

Suggestions use a new suggest.Identifiers rather than suggest.Suggest,
whose dist*4 <= maxLen gate is tuned for species names and demands an
exact match at three characters, so "N5O" would suggest nothing.
Identifiers accepts a distance of 2, the bound cobra uses for command
names.

Two tests keep the registry honest: one checks that Plan reads the source
file of every registered column and that every column Plan routes is
registered, the other that InIndex agrees with userColToSQL.

Closes #34
@thanhleviet
thanhleviet merged commit 064bd79 into main Aug 5, 2026
6 checks passed
@thanhleviet
thanhleviet deleted the feat/columns-command branch August 5, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Is there a way to get a list of column headers available for use with --columns?

1 participant