Skip to content

feat: add experimental SDP streaming/materialized tables and blob publishing - #293

Merged
Raki (mdrakiburrahman) merged 15 commits into
mainfrom
dev/mdrrahman/experimental-stream
Sep 28, 2026
Merged

Raki (mdrakiburrahman) merged 15 commits into
mainfrom
dev/mdrrahman/experimental-stream

Conversation

@mdrakiburrahman

@mdrakiburrahman Raki (mdrakiburrahman) commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator

Why this change is needed

Fabric Spark doesn't natively support streaming tables because it requires Spark Connect:

https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html#creating-a-streaming-table-in-sql

image

However....creatively inclined individuals are able to use Networking innovations like this to create a reverse ingress tunnel to take advantage of these upstream Spark features by establishing a Spark Connect tunnel.

In order to put mileage on the adapter for WHEN Fabric officially supports the CREATE STREAMING TABLE through Livy or otherwise, the streaming behavior is being added as explicitly experimental and disabled by default so existing profiles and materializations remain unchanged.

How

  • Adds an opt-in streaming_table materialization gated by enable_experimental_unstable: true and method: session.
  • Validates streaming model configuration, emits native CREATE STREAMING TABLE SQL, and waits on the owning Spark session for the exact run identity to become stopped and inactive.
  • Fails explicitly on missing or changed run metadata, runtime failures, connection changes, unsupported configurations, and polling timeouts.
  • Documents the experimental profile flag, runtime assumptions, execution semantics, and model usage.
  • Adds npx nx run dbt-fabricspark:publish-blob, which builds the wheel and uploads it with Azure CLI using destination settings from test.env.
  • Updates the adapter version for the new functionality.
  • --full-refresh is rejected. Normal runs resume the checkpoint, while intentional query changes can opt into on_query_change='rebuild'.
  • Streaming execution must retain the same open dbt connection and Spark context until completion.
  • Blob publishing expects exactly one wheel in dist/ and an authenticated Azure CLI session.

Test

Ran:

npx nx run dbt-fabricspark:lint --output-style=stream
npx nx run dbt-fabricspark:build --output-style=stream
npx nx run dbt-fabricspark:test --output-style=stream

Keep existing profiles default-off and unchanged; support session-backed native streaming table creation and completion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mdrakiburrahman

Raki (mdrakiburrahman) commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator Author

Add SDP MATERIALIZED VIEW as well?

@mdrakiburrahman Raki (mdrakiburrahman) changed the title feat: add experimental streaming tables and blob publishing feat: add experimental SDP streaming/materialized tables and blob publishing Sep 26, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mdrakiburrahman
Raki (mdrakiburrahman) marked this pull request as ready for review September 28, 2026 21:47
@mdrakiburrahman
Raki (mdrakiburrahman) merged commit c8ca549 into main Sep 28, 2026
2 checks passed
@mdrakiburrahman
Raki (mdrakiburrahman) deleted the dev/mdrrahman/experimental-stream branch September 28, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant