feat: add kernel_reducers, framework engine_error, and Sidecar Clone#2642
Draft
OussamaSaoudi wants to merge 3 commits into
Draft
feat: add kernel_reducers, framework engine_error, and Sidecar Clone#2642OussamaSaoudi wants to merge 3 commits into
OussamaSaoudi wants to merge 3 commits into
Conversation
This was referenced May 28, 2026
|
PR title does not match the required pattern. Please ensure you follow the conventional commits spec. Your title should start with Title: |
a60651b to
048d32d
Compare
3564434 to
1bcca88
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2642 +/- ##
==========================================
- Coverage 88.67% 88.65% -0.02%
==========================================
Files 207 216 +9
Lines 65204 65906 +702
Branches 65204 65906 +702
==========================================
+ Hits 57818 58428 +610
- Misses 5173 5252 +79
- Partials 2213 2226 +13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
62ee1db to
85359fc
Compare
1ceb2d0 to
fd8f0c2
Compare
Introduces a prost-generated protobuf layer mirroring the kernel's plan,
schema, and expression IR. Engines that need to ship plans across a
process or language boundary (Rust kernel -> JVM consumer, etc.)
serialise via these messages; the Rust kernel produces them but does
not interpret them on receipt.
- kernel/proto/{schema,expressions,plan}.proto -- one .proto file per
kernel module. proto3, oneof-discriminated, one-to-one with the Rust
enums (DataType, Expression, Predicate, Scalar, NodeKind). New IR
variants land as new oneof cases at the next unused field number;
removing a variant requires \`reserved\`-ing the old number so
out-of-date binaries fail loudly.
- kernel/build.rs -- invokes prost-build over the .proto files only
when \`declarative-plans\` is enabled, so off-by-default consumers
don't pay the protoc / codegen cost.
- kernel/src/plans/proto/mod.rs -- re-exports the generated modules
(\`schema\`, \`expressions\`, \`plan\`) with rustdoc-lint suppression
for comments prost forwards from the .proto files.
A follow-up PR will add From/TryFrom conversions between the kernel IR
and the proto messages.
Co-authored-by: Isaac
Adds the leaf errors module that drives kernel state machine failure reporting. DeltaError is the SM-facing error type; DeltaErrorCode is the catalog of structured failure causes the engine can route on. KernelErrAsDelta plus the delta_error!/bail_delta! macros lift internal crate::Error sources into DeltaError without losing typed-code context. Includes the declarative-plans feature scaffold: - kernel/Cargo.toml: `declarative-plans = ["internal-api", "arrow"]` base feature. Consumer stacks layer additional transitive deps (e.g. `dyn-clone`, `genawaiter2`) on top as they introduce them. - lib.rs gating (feature-gated `pub mod plans`). - build.yml CI job for the feature. - plans/mod.rs adds `pub mod errors` alongside `pub mod ir` from the basic-IR PR. No semantic dep on the basic-IR PR -- pure data type definitions plus their construction/extension macros. Co-authored-by: Isaac
Adds the KernelReducer trait and the four canonical reducers (MetadataProtocol, CheckpointHint, SidecarCollector, and Handle) that drive log-replay aggregations. Each reducer is a small stateful visitor over a partition of input rows; KernelReducerToken plus the FinishedHandle/Extractor pair give the executor a typed channel for shipping finished reductions back to the SM driver. Also lands: - framework/engine_error.rs: the EngineError typed failure the engine surfaces to the SM (distinct from the kernel-to-caller DeltaError). Bundled here because handle.rs depends on it. - actions/mod.rs + actions/visitors.rs: Sidecar: Clone, SidecarVisitor: Debug + Clone, and Protocol::try_new_from_data annotated #[internal_api]. Folded in because the kernel reducers depend on these (separating them out would be three trivial-LOC PRs). Cross-sub-chain plans/mod.rs conflict expected when the basic-IR PR lands first: resolve by syncing this sub-chain after that lands. The state_machines module here declares only `framework::engine_error`; the remaining framework members (state_machine, coroutine, plan_context) land in follow-up PRs. Depends on the errors PR for DeltaError (in chain ancestry).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🥞 Stacked PR
Use this link to review incremental changes.
What changes are proposed in this pull request?
Adds the KernelReducer trait and the four canonical reducers (MetadataProtocol, CheckpointHint, SidecarCollector, and Handle) that drive log-replay aggregations. Each reducer is a small stateful visitor over a partition of input rows; KernelReducerToken plus the FinishedHandle/Extractor pair give the executor a typed channel for shipping finished reductions back to the SM driver.
Also lands:
#[internal_api]. Folded in because the kernel reducers depend on these (separating them out would be three trivial-LOC PRs).
Cross-sub-chain plans/mod.rs conflict expected when the basic-IR PR lands first: resolve by syncing this sub-chain after that lands. The state_machines module here declares only
framework::engine_error; the remaining framework members (state_machine, coroutine, plan_context) land in follow-up PRs.Depends on the errors PR for DeltaError (in chain ancestry).
How was this change tested?