Add post: Lumen now speaks SQL on N-dimensional scientific data#100
Add post: Lumen now speaks SQL on N-dimensional scientific data#100ghostiee-11 wants to merge 8 commits into
Conversation
Writeup of the new XArraySQLSource for the xarray / Zarr / Pangeo audience. Covers the architecture, a real ARCO-ERA5 query end to end, where coordinate metadata actually lives today (source schema, not the result DataFrame), and the design decisions behind per-variable table registration and the xarray-sql + DataFusion stack. Adds 9 files under posts/2026-06-05-lumen-xarray-source/ (index.qmd + 8 PNGs in images/). No project-level config or CSS changes.
|
Did you see my other comments in my previous PR? |
|
Yupp, i did and going to do that only |
|
For reference you can view the dev version of blog here: |
|
Yup.. Thanks!! |
|
@ahuang11, By the way, I was planning to lead Post 2 (the AI wiring post on #1791) with the Lumen AI chat angle. Keeping this one focused on the Source layer, so Post 2 has the machinery to plug into. Your views on this? |
|
Sure. I think we should still advertise Lumen from Lumen AI lens though if that makes sense (e.g. users probably don't need to know about Pipeline maybe) |
|
Yeah okay.. Makes sense |
|
Basically I imagine this description is outdated since Lumen has pivoted to AI (90% of docs advertise the AI features) but the rest of the blog can elaborate how the XArrat SQL engine works. declarative data application framework in the HoloViz stack. You describe a Source (where data comes from), a Pipeline (how to filter and transform it), and a View (how to render it). Lumen wires the three together into a deployable Panel app, an embeddable component, or, since the Lumen AI 1.0 release earlier this year, a chat-driven exploration surface. |
|
Okay sure!! Will do this |
- Reframe recap from the Lumen AI lens (chat -> agents). - New section "From a dropped file to a SQL agent" covers PR #1791 wiring with the actual upload-handler snippet. - Replace mocked notebook stills with eight real Lumen AI captures. - Rework diagram-3 (Lumen AI architecture). Add diagram-5 (upload flow). - Embed hero video from YouTube. - Pair PR #1741 (source) and PR #1791 (wiring) in Get started and Credits.
|
Is this ready for review? |
|
Yupp... It is @ahuang11 |
|
Hey @ahuang11, for now I have removed the example images you flagged. I am investigating that from my end!! |
2 participants
New post: "Lumen now speaks SQL on N-dimensional scientific data". A writeup of the recently-landed
XArraySQLSource(lumen#1741).The post explains the new capability for an xarray / Zarr / Pangeo audience and walks through the architecture, a real query end-to-end on ARCO-ERA5, the metadata that survives the SQL round-trip, and the design decisions behind per-variable table registration and the xarray-sql plus DataFusion stack.
Assets