Skip to content

fix: intent-agnostic entity-question wording, clarify script/scene action - #17

Merged
AboveColin merged 2 commits into
AboveColin:mainfrom
sriramsv:fix/intent-agnostic-entity-question-and-script-action-wording
Sep 23, 2026
Merged

AboveColin merged 2 commits into
AboveColin:mainfrom
sriramsv:fix/intent-agnostic-entity-question-and-script-action-wording

Conversation

@sriramsv

Copy link
Copy Markdown
Contributor

Summary

Two small wording-only fixes to interpret.py's build_questions(), both backed by measured evidence rather than guesswork — found while building a separate TypeSafe/Jev-backed Home Assistant conversation agent and comparing notes against this project's approach (which is considerably more mature in most respects — the confidence gating, the lock/entrance-cover exclusions, and the single-call fan-out design are all things I ended up converging toward independently after hitting the same problems the hard way).

1. entity question wording was command-only, but it's answered before intent is known.

The entity question is hardcoded to "Which device should receive this?", but it's asked in the same fan-out call as action — before the router knows whether the resolved action will be get_state (a status question) or turn_on/turn_off (a command). In a sibling Jev-backed integration I measured this exact framing mismatch directly: for a get_state-style question, identical entity candidates (one real match + none_of_these), the command-phrased instructions put none_of_these ahead at 58% confidence, while "Which device is this about?" put the real entity ahead at 69% — wording alone flipped the answer. Rewording costs nothing against the single-call design since the new phrasing reads naturally for both a command and a status question.

2. turn_on doesn't address a script/scene whose own name reads like a question.

Reproduced directly against this repo's real question shapes (action/compound/free_text/target_type/entity/area/domain, built exactly as build_questions() does) with a live TypeSafe call, using a real house's script.get_weather entity: "run the get weather script" scored action=none_of_these at 0.98 confidence and would fall back silently instead of running the script — the model reads "get weather" as a question rather than a script name to execute. The equivalent fix (explicitly telling the model that a script/scene name sounding like a question doesn't make running it a question) resolved the identical failure mode in the sibling integration.

Verification

  • Checked all of tests/*.py for hardcoded assertions on the two changed strings — none found.
  • python3 -m py_compile clean on the changed file.
  • I don't have a full pytest-homeassistant-custom-component environment set up to run this repo's actual test suite, so I'd appreciate a maintainer running pytest to confirm nothing else depends on the exact wording.

Test plan

  • pytest passes
  • Manually verify "run the get weather script" (or an equivalent script whose name reads like a question) now resolves to action=turn_on instead of falling back
  • Manually verify a get_state-style query against a named entity still resolves correctly with the reworded entity question

…tion

Two small wording fixes to interpret.py's build_questions(), both backed
by measured evidence:

1. The `entity` question was hardcoded to command-oriented phrasing
   ("which device should receive this?"), but it's answered in the same
   fan-out call as `action`, before get_state vs. turn_on/off is known.
   Measured in a comparable Jev-backed integration: for a get_state-style
   question, identical entity candidates, command phrasing put
   none_of_these ahead of the one real match at 58% confidence; "which
   device is this about?" put the real match ahead at 69%. Reworded to
   be intent-agnostic at no cost to the single-call design.

2. The `turn_on` action description didn't address a script/scene whose
   own name reads like a question. Reproduced directly against this
   repo's real question shapes with a live TypeSafe call against a real
   house's `script.get_weather` entity: "run the get weather script"
   scored action=none_of_these at 0.98 confidence and fell back silently
   instead of running the script. Added a clarifying clause; the
   equivalent fix in a sibling Jev integration resolved the same failure
   mode (information_request -> smarthome_command misclassification on
   the identical phrase).

No test hardcodes the changed strings (checked all of tests/*.py).

@AboveColin AboveColin left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. I want the entity change. That question gets its answer in the same request as action, so it must read right for a status check as well as a command. Before I merge, please make these changes:

  1. Shorten the turn_on description. This text goes out with every request, so each word costs tokens on every call. One short line is enough, for example:
    "turn_on": "Switch something on, open it, start it, or run a script or scene",
  2. Cut the comment above entity down to one line that says why the wording fits both intents, for example:
    # Answered in the same request as action, so it must fit a status check too.
    The numbers in it were measured outside this repo, and a code comment here should only hold what this code shows.
  3. No em dashes in code, comments or strings. Use a period or a comma.
  4. Run ruff format. ruff format --check fails on interpret.py:161 now, because ruff wants single quotes around the string with "get_weather". After change 1 that string is gone anyway.

About the test run: on this branch, ruff check and mypy --strict pass. Three tests in test_ai_task.py and test_init.py fail in the early morning (UTC) because they compare against the system date. main has the same failure, and your change does not cause it. I will fix it separately. With the dates in line, the suite gives 407 passed, 12 skipped on your branch.

When the changes are in, I will review the branch again.

- Shortened the turn_on description to one line, per-request token cost
- Cut the entity-question comment to one line explaining why the wording
  fits both intents; the measured numbers belonged in the PR description,
  not a code comment about what this code doesn't show
- Removed em dashes
- ruff format --check and ruff check both pass on the changed file
@sriramsv

Copy link
Copy Markdown
Contributor Author

All four addressed:

  1. Shortened turn_on to one line: "Switch something on, open it, start it, or run a script or scene"
  2. Comment above entity cut to one line: # Answered in the same request as action, so it must fit a status check too.
  3. Em dashes removed
  4. ruff format --check and ruff check both pass on the changed file (ruff 0.16.8)

Ready for another look.

@AboveColin
AboveColin merged commit 1a6c968 into AboveColin:main Sep 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants