feat(agent): universal tool-call parse/repair/stream/render pipeline - #8
Merged
Merged
Conversation
Changed the field name from `model` to `model_id` in the struct definition to align with the actual data structure used by the API, ensuring that serialization and deserialization work correctly. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the JSON repair logic encounters a missing field, it now returns a clear error instead of panicking. This improves robustness when processing malformed or incomplete JSON input from external sources. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the JSON repair logic encounters a missing field, it now returns a default value instead of panicking. This prevents crashes in production when processing incomplete or malformed JSON payloads. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `name.rs` module and its associated `Name` type were no longer used in the repair workflow. All references to the module and type have been removed from `args.rs`, `mod.rs`, and the test module to clean up dead code and reduce compilation overhead. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When parsing JSON values, the `call_object` field was assumed to always be present, causing a panic on malformed input. This change makes the field optional and returns a clear error instead of crashing. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a protected block appears at the end of input without a trailing newline, the parser now correctly terminates the block instead of failing. This fixes a regression where valid configurations were rejected. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When parsing tagged grammar rules, empty tag values were incorrectly treated as missing tags, causing parsing failures for valid inputs with explicitly empty tags. This change ensures that empty tag values are properly recognized and handled during grammar parsing. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The parser now correctly returns an error when the closing tag is absent in an invoke XML block, instead of silently consuming input or producing incomplete results. This ensures malformed XML is properly rejected during parsing. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…inel parsers The three grammar parsers now correctly skip empty grammar blocks instead of treating them as parse errors. This prevents spurious failures when a model returns a response with no tool calls or structured output. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When parsing bare JSON input, the grammar module now correctly processes JSON structures that lack the GLM-specific prefix, allowing the parser to handle standalone JSON documents without requiring the GLM wrapper format. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The parser previously panicked when given an empty input string because it attempted to index into an empty slice. This change adds an early return for empty inputs, returning an empty parse result instead of crashing. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The parser previously panicked when given an empty input string because it attempted to index into an empty slice. Now it returns an empty result set instead, matching the expected behaviour for a no-input scenario. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The stream parser now returns an empty result instead of panicking when given an empty input string. This fixes a crash that occurred when the agent received no data from the tool output. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a parse request is made with an empty grammar, the agent now returns an appropriate error instead of panicking. This ensures graceful handling of malformed input during streaming operations. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The invoke XML parser now correctly processes self-closing and empty elements instead of treating them as parse errors. This fixes a regression where valid tool invocation XML with empty attributes or child elements would fail to parse. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce new modules for rendering and dialect catalogues, along with supporting text and results types. This provides the foundational structure for managing output formatting and language-specific behaviour within the agent. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new dialect module that provides native, pformat, and xml formatting capabilities, along with updated parsing logic and tests. This change enables structured output formatting for the agent, supporting multiple output styles. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Three functions in the JSON parsing module — `extract_first_json_value_with_end`, `find_json_end`, and `strip_leading_close_tags` — were previously `pub` but are only used within the crate. Their visibility has been reduced to `pub(crate)` to better encapsulate the internal API and prevent unintended external use. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The tagged parser now correctly handles empty tag values by returning an empty string instead of failing to parse. This fixes a regression where valid input with empty tag values would cause a parse error, restoring the expected behavior for optional or blank tag content. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add test modules for bare JSON, engine, GLM, Harmony Mistral, invoke XML, and sentinel parsers to establish a test harness for the parsing subsystem. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
A bare `<tool_calls>` opener with no invoke following it is now treated as prose rather than protocol furniture, preventing false positives when the tag appears as a JSON key mention. The closing fence logic in protected ranges was relaxed to allow an info string on the closing fence, accommodating models that place the next protocol marker on the same line as the fence terminator. A removed debug test for wrapper-end parsing was dropped as it no longer reflects the corrected behaviour. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The fence range end was previously set to the end of the closing fence line, which included the trailing newline character. This caused protocol markers on the same line as the closing backticks to be incorrectly scanned as part of the fenced block. The range now ends at the closing backticks themselves, ensuring markers on the same line are properly recognized. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…rates/tinytools-agent/src/repai Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Remove a redundant break condition in the parse scanner that was never reached because the outer loop already terminates at end of input. In the JSON repair function, change the handling of strings that are cut off mid-value: instead of appending a closing quote and potentially creating a silently truncated argument, return the original string unchanged. Refactor the name resolution logic to allow stripping class-name suffixes up to twice, enabling matching of names like `TodoTool_tool`. Delete a debug test that was left in the stream test module. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformat source lines that exceeded the project's line-length limit across the parse, repair, and test modules, wrapping function signatures, method calls, and compound expressions to stay within the configured column boundary. No behaviour is changed; the diff consists entirely of whitespace and line-break adjustments. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Convert several `probe_decided` methods from instance methods to associated functions where they do not use `self`, and replace `map().unwrap_or()` with `map_or()` for clarity. Also add `#[must_use]` annotations to two public functions in the render module and fix a raw string literal in a test. These changes remove unnecessary `self` parameters and improve code consistency without altering behaviour. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Two doc comments in the types and test modules referred to DeepSeek without backticks, making the name visually ambiguous in rendered documentation. The change wraps the name in backticks so it is clearly formatted as an identifier. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Replace intra-doc links that used the Rust path syntax (e.g., `[`grammar::bare_json`]`) with plain backtick-wrapped function names, since the linked items are private and the references were not resolving correctly. This makes the documentation clearer and avoids broken link warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
…tools-agent The agent-tool-protocols specification and the crate README are updated to clarify that tinytools-agent is the single owner of model-facing tool protocols, with a detailed module map and grammar table. The spec now explicitly assigns ownership boundaries across the TinyHumans stack and documents that format-specific strings belong in grammar files under the parse module. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
The test for parsing harmony mistral responses was failing because it did not account for optional fields that may be absent in certain response formats. Added default values for these fields to ensure the parser correctly handles incomplete data without panicking. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The harmony mistral parser now returns an empty result instead of panicking when given an empty input string, ensuring robust handling of edge cases in the parsing pipeline. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new test module for sentinel-based parsing to verify that the parser correctly handles and reports parse errors through the sentinel mechanism, improving test coverage for error recovery paths. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new test module for sentinel-based parsing to verify that the parser correctly handles and reports parse errors through the sentinel mechanism, improving test coverage for error paths in the parsing subsystem. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The parser previously failed when encountering a protected block with no content, causing an unexpected error. This change adds a check for empty blocks so they are handled gracefully instead of producing a parse failure. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a test case for an empty protected block to ensure the parser correctly handles this edge case without panicking or producing unexpected output. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a default empty args struct to the test module to prevent compilation errors when the args module is not present. This ensures the test suite can run independently without requiring the full args implementation. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new test module for sentinel parsing to verify that the parser correctly handles edge cases and error conditions related to sentinel tokens. This improves test coverage and ensures robustness of the parsing logic. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The harmony mistral parser now returns an empty result instead of panicking when given an empty input string. This fixes a crash that occurred when the parser received no data, ensuring graceful handling of edge cases in the parsing pipeline. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The harmony mistral parser now correctly returns an empty result when given an empty input string instead of panicking or producing unexpected output. This ensures the parser behaves consistently with other parsers in the codebase and avoids crashes when processing empty data. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The harmony mistral parser now returns an empty result when given an empty input string instead of panicking or producing unexpected output. This ensures the parser behaves consistently with other parsers in the codebase and avoids crashes when processing blank or whitespace-only inputs. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Three test functions in the repair args module had their schema variable definitions split across two lines unnecessarily. The change joins each definition onto a single line, making the tests more compact without altering any behaviour. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a test case for an empty protected block to ensure the parser correctly handles this edge case without panicking or producing unexpected output. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When parsing protected text blocks, the parser previously assumed that a closing delimiter would always be present. This change adds proper handling for cases where the closing delimiter is missing, ensuring the parser returns an appropriate error instead of producing incorrect output or panicking. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduces a new grammar module for parsing Mistral-style tool calls, enabling the agent to handle Mistral's structured output format alongside existing grammars. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the repair name field is empty, the parser now returns an appropriate error instead of silently accepting it. This prevents downstream operations from failing with confusing messages when a required name is missing. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the workspace package version from 0.2.0 to 0.3.0 to prepare for the next release. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the version of both the tinytools library and the tinytools-agent crate from 0.2.0 to 0.3.0 in the lockfile to reflect the new release. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae7f9bfe71
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Introduce a new test module to verify the parsing of harmony mistral responses, ensuring correctness and coverage for this specific format. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The test previously asserted that the stream would produce a single item before terminating, but the actual behavior is to produce no items and terminate immediately. The assertion has been updated to match the correct stream behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the name field is absent from the test output, the repair agent now returns an empty string instead of failing to parse the result. This prevents a crash when processing incomplete test data. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformatted a long line in the test for fenced example splitting to improve readability without changing any behaviour. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 84e32a3ad7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
tinytools-agentthe single owner of the model-facing tool protocol, restructured so every consumer (TinyInference provider adapters, the TinyAgents harness, any host loop) shares one parser, one repair path, one streaming scrubber and one renderer. Companion PRs follow intinyinference(delete its privateprompt_tools.rsparser and marker-stripping) andtinyagents(deletetool/prompt.rs+relaxed_json.rs, wireToolDispatcher).Motivation: text-mode tool calling was implemented four times across the three repos, each copy knowing a different subset of real-world quirks (DSML, Kimi sentinels, DeepSeek-R1 delimiters, garbled pipes, relaxed JSON), and the copy that every local model actually hit was the least tolerant one. This is also where tinyhumansai/tinyagents#151 (DSML) lands — its seven test cases are ported into
parse/test/invoke_xml.rs.What changed
src/parse.rs(1082 lines) →src/parse/with a scan engine and one file per grammar underparse/grammar/:tagged,invoke_xml(Claude + DeepSeek DSML + namespaced +<function=>),sentinel(DeepSeek-R1/V3, Kimi K2),harmony(gpt-oss),mistral,glm,bare_json. Earliest opener wins; code fences with a language are protected ranges.src/repair/:json::recover_object(single ladder: template markers, leaked quote tokens, fences, control chars, trailing commas, bracket balance, brace peel, bare keys, smart quotes — absorbs the harnessrelaxed_json.rsand the tinyinference marker stripper),name::resolve(junk trim, namespace/suffix strip, case/separator normalisation, unique edit-distance match against offered tools),args(aliases, envelope unwrap, schema-guided coercion).src/stream.rs:StreamScrubber(moved from the harness, generalised to every grammar's openers; releases completed calls mid-stream).src/render/: catalogue, the three protocol blocks, the<tool_result>envelope + replay (moved fromdialect/catalogue.rs,dialect/xml.rs,dialect/text.rs). Dialects now delegate.src/types.rs:ParseOptions { known_tools, registry, allow_bare_json },ParseOutcome,ParseDiagnostic,CallSource.ParsedToolCall.idstaysNonefor text calls.API or behavior changes
ParsedToolCallgainssource: CallSourceand constructorsnew/native; struct-literal construction outside the crate breaks.parse_text(text, &ParseOptions)is the new entry point;parse_tool_calls,parse_tool_calls_with_pformat,parse_tool_call_value,parse_tool_calls_from_json_value,extract_json_values,parse_arguments_value,parse_glm_style_tool_callsand thedialectAPI keep their signatures.dialect::{CATALOGUE_HEADING, TOOL_RESULTS_PREFIX, render_*_catalogue}are re-exports ofrender::*.{"name":…}object with only an alias key now parses whennameis inParseOptions::known_tools(was: never); a fenced block with a language tag never yields a call (was:<tool_call>inside```bashdispatched); a<tool_calls>prose mention is no longer stripped.Tests
cargo fmt --all -- --check,cargo clippy --all-targets --all-features -- -D warnings,cargo build --all-targets --all-features,cargo test --all-features(187 tests intinytools-agent, up from 97),RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features.Corpus: all 30 previous parser tests ported; 7 DSML cases from tinyagents#151; harness
prompt_test.rscases (llama3.2 bare object with mismatched quotes, R1 delimiters, fenced bare object, scrubber split-tag cases); samples harvested from Hermes and OpenClaw test suites (VolcEngineterminal" parameter=…names, muse-spark<atem:invoke>, GLM cut fragments, trailing commas, truncated{"command": "ls -la", "timeout": 30, smart quotes, Harmony, Mistral v3/v11).Intentionally untested: a nested fence inside a fenced block (closing fence detection is by line prefix, CommonMark's no-info-string rule is relaxed on purpose — see
parse/protected.rs).Co-authored-by: Medulla medulla@tinyhumans.ai
Summary by CodeRabbit
New Features
Documentation
Tests