feat(fetch,pull): --csv target lists; drop dt get --filter (0.22.0)

[johnyaku]

Adds --csv/--path-col to dt fetch and dt pull, and removes --filter from dt get --csv so every CSV-consuming command agrees on the same two flags.

dt pull --csv samples.csv
dt pull --csv samples.csv --path-col fq_dir
dt fetch --csv samples.csv --regular --dry

Design record: docs/csv-targets.md.

Why this is narrower than dt get --csv

dt get/dt import read two meaningful columns — a source path in a remote repository, and an optional output naming where to put it locally. dt fetch/dt pull act on the current repo, where the destination is fixed by the repo layout. So --csv here is only a way to build the target list, and every column but --path-col is ignored — output no differently from sample_id.

A sheet written for dt get stays usable: if its path column holds registry paths and output the local ones, --path-col output is the invocation. No new flag, just the existing one pointed elsewhere.

The hazard worth the code

Both commands read targets=None as every stage — correct for a bare invocation, disastrous for a CSV. A sheet that yielded nothing would silently widen into a whole-repo pull.

Two layers, deliberately redundant:

• read_csv_target_list cannot return an empty list. A header-only file is already an error, and a blank cell in the path column is now one too (an empty-string target resolves to the working directory). It names the line, since a hand-edited sheet is the usual source.
• _resolve_targets refuses to build one anyway. x if x else None already appears three times in cli.py and is exactly what a later edit would reach for by reflex.

TestBadCsvNeverWidensTheOperation parametrises every way a CSV can be wrong × both commands, and asserts the command never calls through.

Removing --filter

Row selection belongs upstream. csvgrep/awk/pandas already do it well, and doing it beforehand leaves an artifact recording exactly what was transferred. Carrying a filter grammar cost its own error cases and a second way to end up with a selection you didn't intend.

It reached only dt get --csv — dt import --csv never took it — and landed one day before this branch in 9323eef (0.18.0), so there is no meaningful exposure and no case for deprecating rather than deleting. parse_row_filters and row_matches go with it; nothing else called them.

Also

_csv_context appends the CSV, the column, and the target count to FetchError/PullError. DVC names the first unresolvable target and stops; over a 400-row sheet that's one run per typo, with nothing saying the path came from a file.

Not built: per-target pre-validation. Reporting all misses at once means collect_stages(targets=None) first, and a full collect on a large repo is exactly the work that passing targets avoids. Revisit if one-typo-per-run proves annoying.

Cut from scope: dt push --csv. In simple mode push never parses targets at all — ctx.args goes to dvc push verbatim — so --csv would make it the first place push builds its own argv, with validation landing in DVC. The use case is also thin. Reasoning recorded in the design doc's non-goals.

Tests

2100 passed, 2 skipped, run under PBS via qxub, not on the login node. 38 new tests in tests/unit/test_csv_targets.py.

Two pre-existing failures under PBS, not from this branch: TestFetchUrlImport::test_successful_update and ::test_failed_update. fetch._fetch_url_import makes a real check_network_connectivity probe that the tests don't mock, so they pass on the login node and fail on a compute node. dt/fetch.py and dt/pull.py are byte-identical to main on this branch — the fix is a two-line patch in those tests, worth doing separately.

🤖 Generated with Claude Code