Feature/ndit 655 refdata arrow integration#37

Open

stevenhsd wants to merge 24 commits intodevelop_v06from

feature/ndit-655_refdata_arrow_integration

Commits on Jan 26, 2026

feat: Change how error messages are generated (by writing in batches). Duckdb no long relies on pandas- use of pyarrow, multiprocessing and background thread batch writing to avoid memory pressure
stevenhsd
committed

Commits on Jan 27, 2026

style: run black, isort and resolve linting issues
stevenhsd
committed

Commits on Jan 28, 2026

Commits on Jan 30, 2026

Commits on Feb 3, 2026

refactor: merge in main and resolve conflicts and linting issues
stevenhsd
committed

Commits on Feb 6, 2026

Commits on Feb 9, 2026

feat: added reference data loading of arrow ipc files including enhanced test coverage for reference data loaders
stevenhsd
committed