Repository navigation
feat: add experimental SDP streaming/materialized tables and blob publishing - #293
Merged
Merged
Conversation
Keep existing profiles default-off and unchanged; support session-backed native streaming table creation and completion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Add SDP |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Raki (mdrakiburrahman)
marked this pull request as ready for review
September 28, 2026 21:47
Raki (mdrakiburrahman)
deleted the
dev/mdrrahman/experimental-stream
branch
September 28, 2026 21:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this change is needed
Fabric Spark doesn't natively support streaming tables because it requires Spark Connect:
https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html#creating-a-streaming-table-in-sql
However....creatively inclined individuals are able to use Networking innovations like this to create a reverse ingress tunnel to take advantage of these upstream Spark features by establishing a Spark Connect tunnel.
In order to put mileage on the adapter for WHEN Fabric officially supports the
CREATE STREAMING TABLEthrough Livy or otherwise, the streaming behavior is being added as explicitly experimental and disabled by default so existing profiles and materializations remain unchanged.How
streaming_tablematerialization gated byenable_experimental_unstable: trueandmethod: session.CREATE STREAMING TABLESQL, and waits on the owning Spark session for the exact run identity to become stopped and inactive.npx nx run dbt-fabricspark:publish-blob, which builds the wheel and uploads it with Azure CLI using destination settings fromtest.env.--full-refreshis rejected. Normal runs resume the checkpoint, while intentional query changes can opt intoon_query_change='rebuild'.dist/and an authenticated Azure CLI session.Test
Ran: