The problem / use case
Context: Chunkers live in src/retrieval_lab/text.py (FixedSizeChunker,
RecursiveChunker, SemanticChunker, ParentChildChunker). A sliding-window /
overlap chunker is a natural, widely-used addition.
What: Add a SlidingWindowChunker(size, overlap) next to the others, exported from
retrieval_lab, and wire it into the CLI --chunkers parser (e.g. sliding:400x100).
Acceptance:
Pointers: copy the shape of FixedSizeChunker; remember gold is a source-document
span, so a "hit" is any retrieved chunk overlapping the gold span — overlap chunks are fine.
Proposed idea
Add an overlap-based chunker that emits fixed-size chunks with a configurable stride, so adjacent chunks share context — a common strategy the sweep can compare against the existing ones.
The problem / use case
Context: Chunkers live in
src/retrieval_lab/text.py(FixedSizeChunker,RecursiveChunker,SemanticChunker,ParentChildChunker). A sliding-window /overlap chunker is a natural, widely-used addition.
What: Add a
SlidingWindowChunker(size, overlap)next to the others, exported fromretrieval_lab, and wire it into the CLI--chunkersparser (e.g.sliding:400x100).Acceptance:
--chunkersoption and runs end-to-end inretrieval-lab demotests/test_chunking.py(correct sizes + overlap; spans still map to gold)CHANGELOG.mdupdatedPointers: copy the shape of
FixedSizeChunker; remember gold is a source-documentspan, so a "hit" is any retrieved chunk overlapping the gold span — overlap chunks are fine.
Proposed idea
Add an overlap-based chunker that emits fixed-size chunks with a configurable stride, so adjacent chunks share context — a common strategy the sweep can compare against the existing ones.