mine: --jsonl and save FILE.jsonl -- JSON Lines with header, streamed records and footer (0.18.1) - #26
Merged
Conversation
…ed records, and a footer with the counts (0.18.1)
One object per line for results too large to parse as a document: line 1
the header (material, filter, max), then one record per position exactly as
--json's positions[] element, then {"positions": N, "skipped_saturated": M}.
On the command line the records stream as the scan finds them and nothing is
held, so a --max infinity scan runs in constant memory; the shell's
`save FILE.jsonl` writes the same three parts from the held set. --json and
--jsonl are mutually exclusive. The --json record is now built in one place
(position_json) shared by both formats; MineSet gained make_hit/enrich as the
streaming building blocks.
Version 0.18.1 in VERSION, the three pyproject files, both package __init__
modules and the helpmate-api pin. USAGE (JSON Lines subsection with a real
example), README, CHANGELOG.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DFQB6YogM3gaauunTfMuxW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JSON Lines output for
mineresults too large to parse as one document.--jsonl: line 1 is the header (material,filter,max), then one object per position, byte-for-byte the record--jsonputs inpositions[](same optional fields under--themes/--solutions), then a footer{"positions": N, "skipped_saturated": M}. A reader tells the lines apart by their keys (fen/material/positions);jq -c 'select(.fen)'yields the positions.--themes: 115k records written at 16 MB resident. The counts live in the footer because that is when a streaming writer knows them; a missing footer marks a truncated file.save FILE.jsonlwrites the same three parts from the held set, same facet rule as.json.--jsonand--jsonlare mutually exclusive (exit 3).MineSet: the--jsonrecord is now built in one place (position_json) shared by both formats;make_hit/enrichare the streaming building blocks;jsonl_header/jsonl_record/jsonl_footer/to_jsonl.Test plan
to_jsonlheader/records/footer with records equal toto_json's positions, the streaming builders agree with the held-set writer, empty set; shellsave .jsonl--jsonl --max 2line shapes,--max infinity --jsonl= 582 lines with the footer,--json --jsonlrejectedmake test335/335,make format-check BASE=mainexit 0,make lintclean--max infinity --jsonl --themes, every line parses, constant RSS🤖 Generated with Claude Code
https://claude.ai/code/session_01DFQB6YogM3gaauunTfMuxW