Skip to content

Selective CDR parsing (deserialize_selected); fix non-leading @key extraction - #47

Merged
sqt merged 4 commits into
mainfrom
fix/get-key-value-full-deserialize
Sep 2, 2026
Merged

Selective CDR parsing (deserialize_selected); fix non-leading @key extraction#47
sqt merged 4 commits into
mainfrom
fix/get-key-value-full-deserialize

Conversation

@sqt

@sqt sqt commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Motivation

get_key_value and ContentFilteredTopic/QueryCondition field access both need a few members out of a serialized sample, but zidl had no way to parse a subset. So each backend worked around it:

  • get_key_value ran the key-only deserializer over a full stored sample. That only works when the key is a leading, contiguous prefix — a non-leading @key member (e.g. the 3rd field) reads a preceding field's bytes as the key. Wrong key, all four backends.
  • get_field_from_cdr fully deserialized the sample — allocating and freeing every string/sequence — just to read one field, even when the referenced field sits behind a large blob member.

The fix both needed is the same primitive: a parser that decodes the wanted members and skips the rest.

Description of changes

  • deserialize_selected(want: FieldMask) — new mask-driven selective parser generated for every topic struct, all four backends. Decodes the members whose bit is set in want; skips the others. Companions: KEY_FIELD_MASK (bits for the @key members), field_index(name) (name → bit index), and deinit_selected(want) for Zig/C (owned-allocation cleanup; C++/Java rely on RAII/GC). Falls back to a full decode for structs with a base type or > 64 members.
  • skipPrimitives(count, elem_wire_size) added to zidl-rt (cdr.zig) and zidl-cdr (zidl_cdr.c) — steps a primitive array/sequence in one aligned cursor advance instead of a per-element discard loop. Used by the skip paths.
  • Shared src/backend/cdr_skip.zig — the member/array/typeref skip codegen, previously copy-pasted between the C and C++ CdrGenerators, lifted into one module both import (Zig backend gains the same fast paths inline).
  • get_key_value (writer + reader emitters) rewired to deserialize_selected(KEY_FIELD_MASK) with matching deinit_selected on the error path.
  • get_field_from_cdr rewired to deserialize_selected(fieldIndex(field) mask) — decodes only the referenced field, skips the rest; no allocation for unreferenced members.
  • Goldens regenerated; new runtime integration tests (zig_wrapper_contract, java/Test.java) covering non-leading-key extraction and single-field access; CHANGELOG.md entry for v0.3.12-zig.0.16.0.
  • All zig build test / check-goldens / integration-test / zig fmt --check green; verified end-to-end against zzdds with a local path pin.

@greptile-apps

greptile-apps Bot commented Sep 2, 2026

Copy link
Copy Markdown

Greptile Summary

The PR adds mask-driven selective CDR deserialization across the C, C++, Java, and Zig backends, allowing key and field extraction without fully decoding unrelated members.

  • Adds generated field masks, field-name indexing, selective decode paths, and allocation cleanup where required.
  • Introduces shared primitive and composite skip generation plus optimized primitive-run skipping in both runtimes.
  • Rewires key extraction and content-filter field access to use selective parsing.
  • Hardens the C runtime’s cursor alignment, multiplication, and bounds handling in response to the prior review findings.
  • Adds runtime, integration, and regenerated golden coverage for non-leading keys and selective field access.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the previously reported cursor, alignment, or overflow paths.

No blocking failure remains.

Important Files Changed

Filename Overview
packages/zidl-cdr/src/zidl_cdr.c Adds checked primitive-run skipping and fixes the previously reported C cursor alignment and overflow paths.
packages/zidl-rt/src/cdr.zig Adds checked primitive-run skipping with correct zero-length sequence handling.
src/backend/cdr_skip.zig Centralizes C and C++ code generation for skipping unselected CDR members.
src/backend/c.zig Generates mask-driven selective deserialization and cleanup for C bindings.
src/backend/cpp.zig Generates selective C++ deserialization using shared skip emission and RAII cleanup.
src/backend/java.zig Adds generated Java field masks, indexing, and selective member parsing.
src/backend/zig.zig Adds selective Zig parsing and routes skipped primitive collections through the optimized runtime helper.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CDR[Serialized CDR sample] --> Mask[Key or field mask]
  Mask --> Parser[Generated deserialize_selected]
  Parser -->|wanted member| Decode[Decode member]
  Parser -->|unwanted member| Skip[Generated skip path]
  Skip --> Primitive[Runtime skipPrimitives]
  Decode --> Result[Selected output]
  Primitive --> Result
Loading

Reviews (4): Last reviewed commit: "resolving review comments" | Re-trigger Greptile

Comment thread packages/zidl-cdr/src/zidl_cdr.c
Comment thread packages/zidl-cdr/src/zidl_cdr.c
@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.11976% with 66 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/backend/c.zig 77.47% 25 Missing ⚠️
src/backend/zig.zig 88.69% 19 Missing ⚠️
src/backend/cdr_skip.zig 88.49% 13 Missing ⚠️
src/backend/java.zig 92.85% 5 Missing ⚠️
src/backend/cpp.zig 95.08% 3 Missing ⚠️
packages/zidl-cdr/src/zidl_cdr.c 90.90% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Comment thread packages/zidl-cdr/src/zidl_cdr.c
Comment thread packages/zidl-cdr/src/zidl_cdr.c
@sqt
sqt merged commit fba6b93 into main Sep 2, 2026
8 checks passed
@sqt
sqt deleted the fix/get-key-value-full-deserialize branch September 2, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant