Skip to content

feat: add chunk row accessors for typed and string reads#129

Open
mrchypark wants to merge 2 commits into
duckdb:mainfrom
mrchypark:feature/upstream-query-chunks-accessors
Open

feat: add chunk row accessors for typed and string reads#129
mrchypark wants to merge 2 commits into
duckdb:mainfrom
mrchypark:feature/upstream-query-chunks-accessors

Conversation

@mrchypark
Copy link
Copy Markdown

@mrchypark mrchypark commented Apr 3, 2026

Summary

  • add ChunkRows plus Conn.QueryChunksContext and Stmt.QueryChunksContext for chunk-by-chunk result access
  • add typed DataChunk accessors for INTEGER, BIGINT, DOUBLE, TIMESTAMP, and zero-copy VARCHAR reads
  • add tests and benchmarks covering chunk iteration, null handling, type mismatches, and chunk-oriented performance

Motivation

The existing database/sql row path is still the right default, but some consumers need a lower-overhead read path when they already process results chunk-wise. This change keeps the existing API untouched and adds an opt-in chunk API on top of the existing DataChunk abstraction.

The typed accessors are intentionally narrow:

  • IsNull
  • Int32Slice
  • Int64Slice
  • Float64Slice
  • TimestampMicrosSlice
  • StringRefs

All returned views are documented as valid only until the next NextChunk call or Close.

Benchmarks

On an Apple M1 with:

go test ./... -run '^$' -bench 'BenchmarkQueryChunksContext(TypedNumericScan|GetValueNumericScan|TypedStringScan|Count)$' -benchmem -benchtime=1x
  • BenchmarkQueryChunksContextGetValueNumericScan: 37.57 ms/op, 1.06 MB/op, 131162 allocs/op
  • BenchmarkQueryChunksContextTypedNumericScan: 4.66 ms/op, 13.9 KB/op, 346 allocs/op
  • BenchmarkQueryChunksContextTypedStringScan: 13.51 ms/op, 1.07 MB/op, 444 allocs/op
  • BenchmarkQueryChunksContextCount: 4.71 ms/op, 18.1 KB/op, 411 allocs/op

The main improvement is removing per-cell boxing and GetValue overhead when the caller can already consume chunk-oriented data.

Validation

  • go test ./...
  • go test ./... -run '^$' -bench 'BenchmarkQueryChunksContext(TypedNumericScan|GetValueNumericScan|TypedStringScan|Count)$' -benchmem -benchtime=1x

@cypark-conalog
Copy link
Copy Markdown

I opened this as a draft because I wanted to get early feedback on the interface shape before treating it as a finished proposal.

If you think this should be smaller, split differently, or exposed through a different API, I would be happy to follow that direction. In particular, if there is a preferred way to introduce the chunk access path or the typed accessors, please let me know and I can adjust the PR accordingly.

I would appreciate any guidance on how you would like this to be shaped before a full review starts.

@mlafeldt
Copy link
Copy Markdown
Member

I would appreciate any guidance on how you would like this to be shaped before a full review starts.

Thanks for the draft. I think the overall shape looks good. Keeping it as an explicit low-level, opt-in path is the right call. Please keep it focused. We can add more typed accessors later. Maybe add a test for empty and non-inlined strings as well.

@mlafeldt mlafeldt added the feature / enhancement Code improvements or a new feature label Apr 21, 2026
@mrchypark
Copy link
Copy Markdown
Author

I would appreciate any guidance on how you would like this to be shaped before a full review starts.

Thanks for the draft. I think the overall shape looks good. Keeping it as an explicit low-level, opt-in path is the right call. Please keep it focused. We can add more typed accessors later. Maybe add a test for empty and non-inlined strings as well.

Thanks, that makes sense. I'll keep this PR focused and add coverage for empty strings and non-inlined strings in the StringRefs tests.

@mrchypark mrchypark marked this pull request as ready for review April 21, 2026 09:51
@taniabogatsch taniabogatsch requested a review from mlafeldt May 28, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature / enhancement Code improvements or a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants