Zerobus demo for ingest

[nkarpovdb]

Pretty natural for our JSON data...

https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/zerobus-ingest

[nkarpov]

WIP findings — linked to #58

PR #58 has a working implementation of Zerobus ingestion but is blocked from merging due to a platform constraint:

Blocker: Zerobus requires tables with a managed storage location, but serverless workspaces only support Default Storage — which Zerobus does not support. INGEST_MODE=zerobus cannot be used in a serverless context until this is resolved upstream.

See: Zerobus limitations – Workspace and target table

What's implemented in the WIP branch:

• Dual-read bronze ingestion — Lakeflow reads from both volume JSON stream and raw events Delta table simultaneously
• Runtime-selectable writer mode via INGEST_MODE param (volume default, zerobus optional) — cutover requires no pipeline reconfiguration
• Auto-bootstrap of Zerobus credentials in stages/canonical_data: creates/reuses service principal, secret scope/keys, and UC permissions
• Bronze layer refactored: all_events split into bronze_volume_events + bronze_zerobus_events views for DAG visibility
• New runtime params: ZEROBUS_REGION, optional ZEROBUS_ENDPOINT

Holding on merge until the serverless storage limitation is resolved.