fix: intent-agnostic entity-question wording, clarify script/scene action - #17
Merged
AboveColin merged 2 commits intoSep 23, 2026
Conversation
…tion
Two small wording fixes to interpret.py's build_questions(), both backed
by measured evidence:
1. The `entity` question was hardcoded to command-oriented phrasing
("which device should receive this?"), but it's answered in the same
fan-out call as `action`, before get_state vs. turn_on/off is known.
Measured in a comparable Jev-backed integration: for a get_state-style
question, identical entity candidates, command phrasing put
none_of_these ahead of the one real match at 58% confidence; "which
device is this about?" put the real match ahead at 69%. Reworded to
be intent-agnostic at no cost to the single-call design.
2. The `turn_on` action description didn't address a script/scene whose
own name reads like a question. Reproduced directly against this
repo's real question shapes with a live TypeSafe call against a real
house's `script.get_weather` entity: "run the get weather script"
scored action=none_of_these at 0.98 confidence and fell back silently
instead of running the script. Added a clarifying clause; the
equivalent fix in a sibling Jev integration resolved the same failure
mode (information_request -> smarthome_command misclassification on
the identical phrase).
No test hardcodes the changed strings (checked all of tests/*.py).
AboveColin
requested changes
Sep 23, 2026
AboveColin
left a comment
Owner
There was a problem hiding this comment.
Thanks for this. I want the entity change. That question gets its answer in the same request as action, so it must read right for a status check as well as a command. Before I merge, please make these changes:
- Shorten the
turn_ondescription. This text goes out with every request, so each word costs tokens on every call. One short line is enough, for example:"turn_on": "Switch something on, open it, start it, or run a script or scene",
- Cut the comment above
entitydown to one line that says why the wording fits both intents, for example:The numbers in it were measured outside this repo, and a code comment here should only hold what this code shows.# Answered in the same request as action, so it must fit a status check too. - No em dashes in code, comments or strings. Use a period or a comma.
- Run
ruff format.ruff format --checkfails oninterpret.py:161now, because ruff wants single quotes around the string with"get_weather". After change 1 that string is gone anyway.
About the test run: on this branch, ruff check and mypy --strict pass. Three tests in test_ai_task.py and test_init.py fail in the early morning (UTC) because they compare against the system date. main has the same failure, and your change does not cause it. I will fix it separately. With the dates in line, the suite gives 407 passed, 12 skipped on your branch.
When the changes are in, I will review the branch again.
- Shortened the turn_on description to one line, per-request token cost - Cut the entity-question comment to one line explaining why the wording fits both intents; the measured numbers belonged in the PR description, not a code comment about what this code doesn't show - Removed em dashes - ruff format --check and ruff check both pass on the changed file
Contributor
Author
|
All four addressed:
Ready for another look. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two small wording-only fixes to
interpret.py'sbuild_questions(), both backed by measured evidence rather than guesswork — found while building a separate TypeSafe/Jev-backed Home Assistant conversation agent and comparing notes against this project's approach (which is considerably more mature in most respects — the confidence gating, the lock/entrance-cover exclusions, and the single-call fan-out design are all things I ended up converging toward independently after hitting the same problems the hard way).1.
entityquestion wording was command-only, but it's answered before intent is known.The
entityquestion is hardcoded to"Which device should receive this?", but it's asked in the same fan-out call asaction— before the router knows whether the resolved action will beget_state(a status question) orturn_on/turn_off(a command). In a sibling Jev-backed integration I measured this exact framing mismatch directly: for a get_state-style question, identical entity candidates (one real match +none_of_these), the command-phrased instructions putnone_of_theseahead at 58% confidence, while"Which device is this about?"put the real entity ahead at 69% — wording alone flipped the answer. Rewording costs nothing against the single-call design since the new phrasing reads naturally for both a command and a status question.2.
turn_ondoesn't address a script/scene whose own name reads like a question.Reproduced directly against this repo's real question shapes (
action/compound/free_text/target_type/entity/area/domain, built exactly asbuild_questions()does) with a live TypeSafe call, using a real house'sscript.get_weatherentity: "run the get weather script" scoredaction=none_of_theseat 0.98 confidence and would fall back silently instead of running the script — the model reads "get weather" as a question rather than a script name to execute. The equivalent fix (explicitly telling the model that a script/scene name sounding like a question doesn't make running it a question) resolved the identical failure mode in the sibling integration.Verification
tests/*.pyfor hardcoded assertions on the two changed strings — none found.python3 -m py_compileclean on the changed file.pytest-homeassistant-custom-componentenvironment set up to run this repo's actual test suite, so I'd appreciate a maintainer runningpytestto confirm nothing else depends on the exact wording.Test plan
pytestpassesaction=turn_oninstead of falling backget_state-style query against a named entity still resolves correctly with the rewordedentityquestion