feat(huggingface): support local_path in download/data and bypass split verification for data_files

[VictorMorand]

Description

Overview

This PR improves HuggingFaceDataset and hf_download_and_prepare to properly handle local dataset mirrors / task output directories (local_path) and custom file patterns / single-split downloads.

---

Key Changes

1. local_path Handling in HuggingFaceDataset (src/datamaestro/data/huggingface.py):

   • download(): Added an explicit check if self.local_path is not None: return to skip hf_download_and_prepare when dataset bytes are already materialized locally on disk (e.g. from task output directories like FlattenAndShuffleDataset).
   • data property: When self.local_path is set, dataset access now loads the Arrow dataset directly from disk via datasets.load_from_disk(str(self.local_path)) (falling back to load_dataset(str(self.local_path))), preventing unnecessary Hub re-downloads and FileNotFoundError on generated Arrow directories.

2. Split Verification Bypass for Filtered Downloads (src/datamaestro/download/huggingface.py):

   • Updated hf_download_and_prepare() to automatically pass verification_mode="no_checks" whenever data_files or split is specified.
   • Prevents datasets.exceptions.ExpectedMoreSplitsError when downloading specific data file patterns or single splits whose recorded splits differ from total repository metadata (such as datasets where some splits like validation are omitted).

---

Verification

• Tested materializing preprocessed task output datasets with local_path set — prepare() completes immediately as a no-op, and data cleanly loads the Arrow shards from disk.
• Tested downloading single splits and filtered data_files patterns without triggering split verification errors.

[bpiwowar]

A few things

• update the documentation
• add a test
• don't forget to run pre-commit hooks