Skip to content

feat: add kernel_reducers, framework engine_error, and Sidecar Clone#2642

Draft
OussamaSaoudi wants to merge 3 commits into
delta-io:mainfrom
OussamaSaoudi:stack/c3
Draft

feat: add kernel_reducers, framework engine_error, and Sidecar Clone#2642
OussamaSaoudi wants to merge 3 commits into
delta-io:mainfrom
OussamaSaoudi:stack/c3

Conversation

@OussamaSaoudi

@OussamaSaoudi OussamaSaoudi commented May 28, 2026

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


What changes are proposed in this pull request?

Adds the KernelReducer trait and the four canonical reducers (MetadataProtocol, CheckpointHint, SidecarCollector, and Handle) that drive log-replay aggregations. Each reducer is a small stateful visitor over a partition of input rows; KernelReducerToken plus the FinishedHandle/Extractor pair give the executor a typed channel for shipping finished reductions back to the SM driver.

Also lands:

  • framework/engine_error.rs: the EngineError typed failure the engine surfaces to the SM (distinct from the kernel-to-caller DeltaError). Bundled here because handle.rs depends on it.
  • actions/mod.rs + actions/visitors.rs: Sidecar: Clone, SidecarVisitor: Debug + Clone, and Protocol::try_new_from_data annotated
    #[internal_api]. Folded in because the kernel reducers depend on these (separating them out would be three trivial-LOC PRs).

Cross-sub-chain plans/mod.rs conflict expected when the basic-IR PR lands first: resolve by syncing this sub-chain after that lands. The state_machines module here declares only framework::engine_error; the remaining framework members (state_machine, coroutine, plan_context) land in follow-up PRs.

Depends on the errors PR for DeltaError (in chain ancestry).

How was this change tested?

@github-actions

github-actions Bot commented May 28, 2026

Copy link
Copy Markdown

PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: feat: add kernel_reducers, framework engine_error, and Sidecar Clone

@OussamaSaoudi OussamaSaoudi force-pushed the stack/c3 branch 19 times, most recently from a60651b to 048d32d Compare June 1, 2026 22:10
@OussamaSaoudi OussamaSaoudi force-pushed the stack/c3 branch 4 times, most recently from 3564434 to 1bcca88 Compare June 2, 2026 23:12
@codecov

codecov Bot commented Jun 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.51648% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.65%. Comparing base (721faee) to head (bddc0b4).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
kernel/src/plans/errors/mod.rs 59.37% 39 Missing ⚠️
...ernel/src/plans/kernel_reducers/checkpoint_hint.rs 82.81% 5 Missing and 6 partials ⚠️
kernel/src/plans/kernel_reducers/handle.rs 90.10% 8 Missing and 1 partial ⚠️
...nel/src/plans/kernel_reducers/metadata_protocol.rs 88.75% 4 Missing and 5 partials ⚠️
...nel/src/plans/kernel_reducers/sidecar_collector.rs 87.50% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2642      +/-   ##
==========================================
- Coverage   88.67%   88.65%   -0.02%     
==========================================
  Files         207      216       +9     
  Lines       65204    65906     +702     
  Branches    65204    65906     +702     
==========================================
+ Hits        57818    58428     +610     
- Misses       5173     5252      +79     
- Partials     2213     2226      +13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@OussamaSaoudi OussamaSaoudi force-pushed the stack/c3 branch 3 times, most recently from 62ee1db to 85359fc Compare June 3, 2026 02:37
@OussamaSaoudi OussamaSaoudi changed the title feat: add kernel_reducers, framework engine_error, and Sidecar Clone feat: add kernel_reducers, framework engine_error, and Sidecar Clone SC-232995 Jun 3, 2026
@OussamaSaoudi OussamaSaoudi changed the title feat: add kernel_reducers, framework engine_error, and Sidecar Clone SC-232995 feat: add kernel_reducers, framework engine_error, and Sidecar Clone SC-233023 Jun 3, 2026
@OussamaSaoudi OussamaSaoudi changed the title feat: add kernel_reducers, framework engine_error, and Sidecar Clone SC-233023 feat: add kernel_reducers, framework engine_error, and Sidecar Clone Jun 3, 2026
@OussamaSaoudi OussamaSaoudi force-pushed the stack/c3 branch 4 times, most recently from 1ceb2d0 to fd8f0c2 Compare June 5, 2026 19:10
Introduces a prost-generated protobuf layer mirroring the kernel's plan,
schema, and expression IR. Engines that need to ship plans across a
process or language boundary (Rust kernel -> JVM consumer, etc.)
serialise via these messages; the Rust kernel produces them but does
not interpret them on receipt.

- kernel/proto/{schema,expressions,plan}.proto -- one .proto file per
  kernel module. proto3, oneof-discriminated, one-to-one with the Rust
  enums (DataType, Expression, Predicate, Scalar, NodeKind). New IR
  variants land as new oneof cases at the next unused field number;
  removing a variant requires \`reserved\`-ing the old number so
  out-of-date binaries fail loudly.
- kernel/build.rs -- invokes prost-build over the .proto files only
  when \`declarative-plans\` is enabled, so off-by-default consumers
  don't pay the protoc / codegen cost.
- kernel/src/plans/proto/mod.rs -- re-exports the generated modules
  (\`schema\`, \`expressions\`, \`plan\`) with rustdoc-lint suppression
  for comments prost forwards from the .proto files.

A follow-up PR will add From/TryFrom conversions between the kernel IR
and the proto messages.

Co-authored-by: Isaac
Adds the leaf errors module that drives kernel state machine failure
reporting. DeltaError is the SM-facing error type; DeltaErrorCode is
the catalog of structured failure causes the engine can route on.
KernelErrAsDelta plus the delta_error!/bail_delta! macros lift internal
crate::Error sources into DeltaError without losing typed-code
context.

Includes the declarative-plans feature scaffold:
- kernel/Cargo.toml: `declarative-plans = ["internal-api", "arrow"]`
  base feature. Consumer stacks layer additional transitive deps
  (e.g. `dyn-clone`, `genawaiter2`) on top as they introduce them.
- lib.rs gating (feature-gated `pub mod plans`).
- build.yml CI job for the feature.
- plans/mod.rs adds `pub mod errors` alongside `pub mod ir` from the
  basic-IR PR.

No semantic dep on the basic-IR PR -- pure data type definitions plus
their construction/extension macros.

Co-authored-by: Isaac
Adds the KernelReducer trait and the four canonical reducers
(MetadataProtocol, CheckpointHint, SidecarCollector, and Handle) that
drive log-replay aggregations. Each reducer is a small stateful
visitor over a partition of input rows; KernelReducerToken plus the
FinishedHandle/Extractor pair give the executor a typed channel for
shipping finished reductions back to the SM driver.

Also lands:
- framework/engine_error.rs: the EngineError typed failure the engine
  surfaces to the SM (distinct from the kernel-to-caller DeltaError).
  Bundled here because handle.rs depends on it.
- actions/mod.rs + actions/visitors.rs: Sidecar: Clone, SidecarVisitor:
  Debug + Clone, and Protocol::try_new_from_data annotated
  #[internal_api]. Folded in because the kernel reducers depend on
  these (separating them out would be three trivial-LOC PRs).

Cross-sub-chain plans/mod.rs conflict expected when the basic-IR PR
lands first: resolve by syncing this sub-chain after that lands. The
state_machines module here declares only `framework::engine_error`;
the remaining framework members (state_machine, coroutine,
plan_context) land in follow-up PRs.

Depends on the errors PR for DeltaError (in chain ancestry).
@github-actions github-actions Bot removed the breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump. label Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant