Declarative file routing for multi-file ingestion

[Iwan-Dyke]

Problem

process_file handles a single file. process_directory handles all files with the same options. There's no middle ground for the common case: a directory of files where different filename patterns need different read options, column names, and target tables.

Every consumer that ingests multiple file types rebuilds the same logic:

1. List files in a directory
2. Match each filename against a set of patterns
3. Apply pattern-specific read options (delimiter, header, etc.)
4. Apply column names
5. Route to the correct target table
6. Track what's been processed

This is boilerplate that belongs in the library.

Proposed solution

A declarative FileRouter that maps filename patterns to processing config:

from file_processing import FileRouter

router = FileRouter([
    {
        "pattern": r"transactions_\d{8}\.csv",
        "target": "catalog.schema.transactions_raw",
        "read_options": {"delimiter": ",", "header": "true"},
    },
    {
        "pattern": r"balances_\d{8}\.dat",
        "target": "catalog.schema.balances_raw",
        "read_options": {"delimiter": ",", "header": "false"},
        "columns": ["AccountID", "Balance", "Date", "Currency"],
    },
])

results = router.process_directory("/Volumes/landing", spark)

Design questions

1. Config format — Python dicts only? Or also support loading from YAML/JSON?
2. Unmatched files — Report in results, skip silently, or fail?
3. Column application — Should the router rename columns, or leave that to a transform?
4. Tracking — Per-route tracking config, or a single tracking table for the whole router?
5. Write integration — Should the router also write to target tables, or just return processed results with route metadata?
6. Transforms — Should routes support transforms: [fn1, fn2] for post-read logic (renames, redaction, casting)?
7. Ordering — First match wins? Explicit priority? Fail on ambiguous matches?

What this is not

• Not a workflow orchestrator — it doesn't manage task dependencies or scheduling
• Not a replacement for process_file — single-file processing stays as-is
• Not opinionated about write behaviour — consumers decide what to do with results

Acceptance criteria

• Declarative config maps filename patterns to read options and (optionally) column names and targets
• process_directory returns per-file results with route metadata (which pattern matched, which target)
• Unmatched files are reported in results, not silently dropped
• Pattern ordering is deterministic and documented
• Existing process_file / process_directory APIs unchanged