Skip to content

Conversation

@teodordelibasic-db
Copy link
Contributor

What changes are proposed in this pull request?

New Features

  • Type-Safe Stream Classes: New ProtoZerobusStream<T> and JsonZerobusStream with compile-time type safety
  • Stream Builder API: Fluent builder pattern with
    sdk.streamBuilder(tableName).clientCredentials(...).compiledProto(...).build()
  • JSON Record Support: Direct JSON ingestion without protobuf encoding via JsonZerobusStream
  • Batch API with Wrapper Types: MessageBatch, BytesBatch, StringBatch, MapBatch for consistent batch ingestion
  • SDK Builder Pattern: ZerobusSdk.builder(serverEndpoint, unityCatalogEndpoint).executor(...).build()
  • Headers Provider Support: HeadersProvider interface for custom authentication strategies
  • TLS Configuration Support: TlsConfig abstract class for flexible TLS settings
  • Extended Proto Type Support: Added TINYINT, BYTE, TIMESTAMP_NTZ, VARIANT, and nested STRUCT support

API Improvements

  • New Ingestion API: ingest() and ingestBatch() return offset ID directly (previously CompletableFuture<Void>)
  • Wait for Offset: New waitForOffset(long) method for explicit acknowledgment control
  • Type Widening: ProtoZerobusStream.ingest(byte[]) and JsonZerobusStream.ingest(Map<String, ?>)
  • Unchecked Exceptions: ZerobusException now extends RuntimeException
  • Configuration Renames: maxInflightRequests() replaces maxInflightRecords(), offsetCallback() replaces
    ackCallback()

Project Structure

  • Multi-Module Maven Project: Reorganized into common/, sdk/, cli/, and sdk/examples/ modules
  • New CLI Module: Standalone zerobus-cli JAR with generate-proto command
  • New Common Module: Shared HTTP client and JSON utilities
  • Runnable Examples: Examples as Maven module with proto compilation

Internal Improvements

  • gRPC channel reuse across streams
  • SDK-level cached thread pool
  • Graceful executor shutdown
  • Parameterized SLF4J logging
  • JSR-305 nullability annotations

Dependency Updates

  • Protocol Buffers: 3.24.0 → 4.33.0
  • gRPC: 1.58.0 → 1.76.0
  • SLF4J: 1.7.36 → 2.0.17

Deprecations

  • ZerobusStream<T> → use ProtoZerobusStream<T> or JsonZerobusStream
  • sdk.createStream() → use sdk.streamBuilder()
  • TableProperties<T> → use ProtoTableProperties or JsonTableProperties

How is this tested?

Describe any tests you have done; especially if test tests are not part of
the unit tests (e.g. local tests).

ALWAYS ANSWER THIS QUESTION: Answer with "N/A" if tests are not applicable
to your PR (e.g. if the PR only modifies comments). Do not be afraid of
answering "Not tested" if the PR has not been tested. Being clear about what
has been done and not done provides important context to the reviewers.

Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants