Skip to content

Add schema references and extensible converters - #59

Merged
timkpaine merged 1 commit into
mainfrom
schema-references
Aug 6, 2026
Merged

Add schema references and extensible converters#59
timkpaine merged 1 commit into
mainfrom
schema-references

Conversation

@timkpaine

Copy link
Copy Markdown
Member

Description

Adds independently resolvable provided and requested schemas with fsspec, Arrow IPC, structural JSON Schema, and Confluent Schema Registry providers. Schema credentials remain scoped to each reference and provenance is recorded before source data is opened.

Adds a native converter registry with lazy entry-point discovery plus optional Xarray-to-Zarr and Zarr-to-Xarray routes. Updates API, how-to, and explanation docs with the canonical data, schema, projection, codec, and native-converter flow.

Confluent support currently accepts JSON Schema documents without external references. Zarr remains a native multi-object resource conversion rather than a DataFileSystem single-file codec.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactor / code cleanup
  • CI / build configuration
  • Other (describe below)

Checklist

  • Linting passes (make lint)
  • Tests pass (make test)
  • New tests added for new functionality
  • Documentation updated (if applicable)
  • Changelog / version bump (not applicable; no release requested)

Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Test Results

89 tests  +29   88 ✅ +28   0s ⏱️ ±0s
 1 suites ± 0    1 💤 + 1 
 1 files   ± 0    0 ❌ ± 0 

Results for commit 06c1e22. ± Comparison against base commit f2b7102.

@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.95298% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.74%. Comparing base (f2b7102) to head (06c1e22).

Files with missing lines Patch % Lines
fsspec_data/confluent.py 64.04% 19 Missing and 13 partials ⚠️
fsspec_data/json_schema.py 73.33% 10 Missing and 10 partials ⚠️
fsspec_data/schema.py 90.20% 8 Missing and 6 partials ⚠️
fsspec_data/converters.py 83.78% 7 Missing and 5 partials ⚠️
fsspec_data/xarray_zarr.py 67.74% 5 Missing and 5 partials ⚠️
fsspec_data/tests/test_converters.py 92.00% 6 Missing ⚠️
fsspec_data/tests/test_schema.py 98.50% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #59      +/-   ##
==========================================
- Coverage   95.08%   89.74%   -5.34%     
==========================================
  Files          11       18       +7     
  Lines         631     1239     +608     
  Branches       30      102      +72     
==========================================
+ Hits          600     1112     +512     
- Misses         28       84      +56     
- Partials        3       43      +40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timkpaine
timkpaine merged commit 51dcc2f into main Aug 6, 2026
4 of 6 checks passed
@timkpaine
timkpaine deleted the schema-references branch August 6, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant