Skip to content

fix(runtime): reject non-boolean goal evaluation flags - #4737

Open
Totoro-qaq wants to merge 2 commits into
apache:mainfrom
Totoro-qaq:fix/goal-evaluation-types
Open

fix(runtime): reject non-boolean goal evaluation flags#4737
Totoro-qaq wants to merge 2 commits into
apache:mainfrom
Totoro-qaq:fix/goal-evaluation-types

Conversation

@Totoro-qaq

Copy link
Copy Markdown
Contributor

Summary

Fixes #4736

Boolean("false") currently turns an invalid judge response into a valid Goal verdict. Reject non-boolean met, impossible, progress, or waiting fields as a neutral evaluator failure. Missing flags still default to false, and valid boolean judgments are unchanged.

This keeps malformed evaluations from terminating or waiting a Goal, or changing its no-progress counter. The change adds no dependencies, model calls, or state. JSON extraction, prompts, and Todo behavior are unchanged.

Verification

  • Rebuilt the runtime before testing. The evaluator/continuation regression run was 68 passed, 33 failed before the fix → 101 passed, 0 failed after it.
  • The regressions feed raw synthetic judge output through the real continuation coordinator, checking that the Goal stays active, keeps its existing no-progress count, and admits the next turn.
  • Full runtime suite: 3,243 passed, 0 failed, 13 skipped. This PR adds no skips.
  • Runtime Host Goal coordinator/root-authority/protocol tests: 23 passed.
  • npm run lint, npm run format:check, npm run build, npm run typecheck: passed.
  • Both documented Knip workspace checks passed. ASF header check passed on a clean worktree of the proposed commits.
  • Evaluator coverage: 95.77% lines, 92.11% branches, 90% functions.

Minimal reproduction output for a string "false" flag:

Before: parsedMet=true,  evaluatorFailed=false
After:  parsedMet=false, evaluatorFailed=true

Not run: the full all-workspace test matrix or live-provider E2E. The issue contains the standalone reproduction command; no API credentials are needed.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Codex assisted with investigation, implementation, regression tests, and local verification. AI-authored commits include Generated-by: Codex.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

RED: rebuilt runtime and ran goal-evaluator plus goal-continuation suites. Invalid flag types are accepted as judgments and corrupt goal settlement or stall accounting.

Generated-by: Codex
Treat invalid flag types as a neutral evaluator failure while preserving missing-field defaults. GREEN: 101/101 evaluator and continuation tests pass. Evaluator coverage: 95.77% lines, 92.11% branches, 90% functions.

Generated-by: Codex
@github-actions github-actions Bot added the effort/S Under 100 readable lines label Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/S Under 100 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Goal evaluator coerces non-boolean flags into valid judgments

1 participant