Docs for calibration, the LLM tools, the ask-back and a month of cost

custom_components/jev/manifest.json

@@ -10,5 +10,5 @@
   "iot_class": "cloud_polling",
   "issue_tracker": "https://github.com/AboveColin/HA-Jev/issues",
   "requirements": ["jevclient==1.2.0"],
-  "version": "1.15.1"
+  "version": "1.16.0"
 }

mkdocs.yml

@@ -62,7 +62,7 @@ plugins:
         A Home Assistant custom integration, installed through HACS. Questions become
         sensors: a probability, one of a set of options, or a number on a scale. Four
         actions (jev.noul, jev.choice, jev.score, jev.ask) answer inside an
-        automation, an AI Task entity answers ai_task.generate_data, and a
+        automation, jev.calibrate fits a threshold from history, an AI Task entity answers ai_task.generate_data, and a
         conversation agent handles Assist. Every call is billed in input tokens
         against an optional daily budget.
       sections:
@@ -79,10 +79,13 @@ plugins:
           - actions.md: The four actions, their fields and their response variables
           - ai-task.md: The ai_task.generate_data entity and the structures it accepts
           - examples.md: Worked automations
+          - calibrate.md: jev.calibrate, a threshold measured from recorder history
         Voice:
           - conversation.md: The Assist conversation agent and what it will not control
+          - llm-tools.md: The jev__noul and jev__choice tools for other LLM agents
         Reference:
-          - cost.md: Token billing, the usage sensors and the daily budget
+          - bayesian.md: When to use Jev and when the Bayesian binary sensor
+          - cost.md: Token billing, the usage sensors, a month worked out and the daily budget
           - measurements.md: Latency and answer quality measured against the live API
           - troubleshooting.md: Debug logging and the common failures
           - limitations.md: What Jev does not do
@@ -119,9 +122,12 @@ nav:
       - Actions: actions.md
       - AI Task: ai-task.md
       - Examples: examples.md
+      - Tuning a threshold: calibrate.md
   - Voice:
       - Conversation agent: conversation.md
+      - Tools for other LLM agents: llm-tools.md
   - Reference:
+      - Jev and the Bayesian sensor: bayesian.md
       - What it costs: cost.md
       - Measurements: measurements.md
       - Troubleshooting: troubleshooting.md

quality_scale.yaml

@@ -87,7 +87,7 @@ rules:
       agent all start it when TypeSafe rejects the key.
   test-coverage:
     status: done
-    comment: 97% measured across 407 tests, enforced in CI.
+    comment: 97% measured across 474 tests, enforced in CI.
 
   # Gold
   devices: done
@@ -133,12 +133,12 @@ rules:
       placeholder. A question sensor is named by the user's own question text, which
       is theirs to write and cannot be translated by anyone else. The conversation
       agent and the AI Task entity take the device name, which is what every core
-      provider of either does, and the ten sentences the agent speaks itself go
+      provider of either does, and the twelve sentences the agent speaks itself go
       through the common translation category, so a Dutch pipeline answers in Dutch.
   exception-translations:
     status: done
     comment: >-
-      Twenty-eight keys, including the library's own limit errors, which are
+      Thirty-four keys, including the library's own limit errors, which are
       re-raised with the real numbers as placeholders rather than passed through in
       English. The coordinator's UpdateFailed and setup's ConfigEntryNotReady and
       ConfigEntryAuthFailed are translated too, since both reach the integrations
@@ -148,7 +148,7 @@ rules:
   icon-translations:
     status: done
     comment: >-
-      icons.json covers every translated entity and all four actions, and a test
+      icons.json covers every translated entity and all five actions, and a test
       asserts that. The conversation entity takes the conversation domain's own icon,
       as core's agents do.
   reconfiguration-flow:
@@ -176,4 +176,4 @@ rules:
       Assistant ships aiohttp and httpx, not httpx2.
   strict-typing:
     status: done
-    comment: mypy --strict passes on all 20 modules, enforced in CI.
+    comment: mypy --strict passes on all 22 modules, enforced in CI.

site-docs/actions.md

@@ -109,6 +109,12 @@ the five you might need and discarding four is cheaper than two round trips.
 An object arrives as an object, not as a stringified dict. The model reads the field
 names as labels, so name them for what they hold.
 
+## Tune a threshold
+
+`jev.calibrate` is a fifth action that asks nothing. It compares the recorded history
+of a noul sensor with an entity that shows what was really true, and returns the
+threshold that fits best. It costs no tokens. See [tuning a threshold](calibrate.md).
+
 ## Errors name the limit
 
 Every budget failure names the budget, the limit and your ask, because an agent

site-docs/bayesian.md

@@ -0,0 +1,73 @@
+# Jev and the Bayesian sensor
+
+Home Assistant already has a sensor that turns several readings into one probability,
+the [Bayesian binary sensor](https://www.home-assistant.io/integrations/bayesian/).
+Both give you a number from 0 to 1 and a threshold that turns it into on or off. They
+get the number in different ways, and that decides which one fits.
+
+## How each gets its number
+
+The Bayesian sensor multiplies probabilities you give it. You set a prior, and for
+each observation you say how often it is seen when the answer is yes
+(`prob_given_true`) and when it is no (`prob_given_false`). An observation is a state,
+a numeric range or a template. The sensor combines them with Bayes' rule and treats
+them as independent of each other.
+
+Jev reads the states and your question in words, and the model gives the probability.
+You write no probabilities. You write what the question means, and optionally what
+counts as yes and as no.
+
+## Side by side
+
+| | Bayesian sensor | Jev noul |
+|---|---|---|
+| You supply | a prior and two probabilities per observation | a question in words |
+| Where it runs | in Home Assistant | a call to the TypeSafe API |
+| Cost | nothing | input tokens, see [what it costs](cost.md) |
+| Time to update | when a state changes | 250 to 580 ms warm, measured from the Netherlands |
+| Same input, same output | yes | no. One sentence gave 0.25, 0.28 and 0.31 in three runs |
+| Reads free text | only through a template you write | yes, a note or a transcript is ordinary input |
+| Observations that depend on each other | counted as independent, so they count twice | read together |
+| Why it said what it said | each observation's share is visible | a number with no reasoning |
+
+## Which to use
+
+Use the Bayesian sensor when you know, or can measure, how often each reading goes with
+the answer, and the readings are few and mostly independent. "Somebody is home" from a
+phone, a door and motion in the hall is the textbook case. It costs nothing, it works
+without the internet, and you can see why it changed.
+
+Use Jev when the probabilities are the part you cannot write down. A washing machine
+whose power draw rises and falls through a programme, a doorbell transcript, or three
+readings that only mean something together are hard to express as
+`prob_given_true`. Jev takes them as they are.
+
+If you already have a working Bayesian sensor, keep it. Jev does not make one wrong.
+
+## Using both
+
+A Jev noul sensor is a number, so a Bayesian sensor can use it as a `numeric_state`
+observation, next to readings that it handles well on its own:
+
+```yaml
+binary_sensor:
+  - platform: bayesian
+    name: Laundry waiting
+    prior: 0.2
+    probability_threshold: 0.8
+    observations:
+      - platform: numeric_state
+        entity_id: sensor.jev_laundry_done
+        above: 0.6
+        prob_given_true: 0.9
+        prob_given_false: 0.1
+      - platform: state
+        entity_id: binary_sensor.laundry_door
+        to_state: "off"
+        prob_given_true: 0.95
+        prob_given_false: 0.5
+```
+
+The probabilities in that example are placeholders. Measure your own, for example with
+[jev.calibrate](calibrate.md), which reports precision and recall for a Jev sensor
+against an entity that shows what was really true.

site-docs/calibrate.md

@@ -0,0 +1,80 @@
+# Tuning a threshold
+
+A noul is a probability, and the threshold that turns it into yes or no is your
+choice. 0.5 is where the model says it cannot tell. It is not where your washing
+machine is done. `jev.calibrate` finds a threshold from what really happened in your
+house.
+
+## What it needs
+
+- a probability sensor, such as a Jev noul sensor
+- an entity that shows what was really true, such as a door contact, or a smart plug
+  that reports when a machine runs
+- the recorder, which keeps the history of both
+
+It reads the recorder and nothing else. It does not call TypeSafe, so it costs no
+tokens.
+
+```yaml
+- action: jev.calibrate
+  response_variable: fit
+  data:
+    entity_id: sensor.jev_laundry_done
+    truth_entity_id: binary_sensor.laundry_door
+    truth_state: "off"
+    days: 7
+```
+
+| Field | Default | Meaning |
+|---|---|---|
+| `entity_id` | required | The probability sensor |
+| `truth_entity_id` | required | The entity that shows what was true |
+| `truth_state` | `on` | The state of that entity that means yes |
+| `days` | 7 | How far back to read. The recorder keeps 10 days unless you changed `purge_keep_days` |
+
+## What it returns
+
+```yaml
+threshold: 0.31
+precision: 1.0
+recall: 1.0
+f1: 1.0
+hours: 4.0
+hours_true: 2.0
+times_true: 1
+probability_changes: 4
+table:
+  - {threshold: 0.1, precision: 0.5, recall: 1.0, f1: 0.667}
+  # ... one row for each tenth up to 0.9
+```
+
+The action cuts the window at every change of either entity, and counts time, not
+state changes:
+
+- **precision** is the part of the time the threshold said yes that was really true.
+  A low precision means false alarms.
+- **recall** is the part of the true time where the threshold said yes. A low recall
+  means missed cases.
+- **f1** balances the two. The action tries every hundredth from 0.01 to 0.99 and
+  returns the one with the highest f1. When several tie, it returns the middle one, so
+  a probability a little off its usual values still lands on the same side.
+
+Use `table` when you care more about one side. For a notification you would rather
+miss than repeat, pick a row with higher precision.
+
+## How much to trust it
+
+Read `times_true` before you read `f1`. One wash is one occasion, and an f1 of 1.0 over
+one occasion tells you almost nothing. A week with a machine that runs every other day
+gives three or four. Collect more occasions before you move a threshold far.
+
+Time when the probability sensor has no number, for example while the API was down,
+is left out.
+
+## When it refuses
+
+| Error | Why |
+|---|---|
+| The recorder is not running | The action has nothing to read |
+| No history | No time in the window has both a number and a truth state |
+| Never true, or always true | With only one side, every threshold gets the same score |

site-docs/conversation.md

@@ -52,14 +52,42 @@ hallway" would reach a lock exposed there and unlock it.
 | An entity you did not expose to Assist | Never described to the model at all |
 | No room and no device named | Refused, unless you allow it. `turn_off` is exempt. Below the confidence floor, fallback |
 | Two kinds of device, no kind named, whole house | Asks which kind. With one kind exposed, it acts on that kind |
-| A spent budget, a rejected key or no answer | Fallback. With no fallback agent it says which of the three it was |
+| Two devices whose names fit the command equally well | Asks which one, see below |
+| Too little budget left for the command, a rejected key or no answer | Fallback. With no fallback agent it says which of the three it was, as an error reply |
 
 !!! info "It only sees what Assist sees"
     The agent describes only entities you exposed to Assist. You already decided
     which entities a voice assistant may touch, and a question is not a reason to
     widen that. This is the voice path only. The [four actions](actions.md) send
     whatever an automation targets, exposed or not.
 
+## When two devices fit the name
+
+The agent compares what you said with the names of the exposed devices of the kind
+the model picked. The whole name said wins: with a "Lamp" and a "Desk lamp", "turn on
+the lamp" is the Lamp and "turn on the desk lamp" is the Desk lamp. When two names fit
+equally well, as "the lamp" does for a Desk lamp and a Floor lamp, the agent asks
+**"Do you mean Desk lamp or Floor lamp?"** and keeps the conversation open. When the
+two have the same name, it adds the room: "Lamp (Office) or Lamp (Bedroom)". A room
+you name settles it first, so "the lamp in the office" acts. Next comes the room of
+the satellite that heard you, as for Home Assistant's own agent: "turn on the lamp"
+said to the office satellite turns on the office Lamp.
+
+The names decide this, not the model's confidence. On a test instance with two lights
+both called "Lamp", the model put 1.00 on one of them in one session. In another it put
+0.55 on "none of these", 0.44 on one Lamp and 0.01 on the other. Neither split showed
+that there were two. With three or more that fit equally well, the command goes to the
+fallback agent.
+
+Your reply, such as "the desk one", is one more request, and it counts against the
+budget. It asks which of the two the reply picks, and if the reply asks for something
+of its own. A confident pick carries out the first command on that device. A reply
+that picks neither, or that is a new instruction, is handled as a new command. So
+"never mind, turn off the lamp in the bedroom" turns that lamp off, and does not run
+the first command on it.
+The question expires after five minutes, the same time Home Assistant keeps a
+conversation open.
+
 ## What it costs
 
 Every command counts against the same daily token budget as your sensors. A
@@ -118,6 +146,13 @@ only sees what Jev could not route, which is the cheap arrangement.
 
 ## Diagnostics
 
+In the Assist dialog, each reply from Jev has a note under it with what Jev answered:
+the decision and its reason, the slots, each answer with its top three options, the
+model, the input tokens and the time the call took. The pipeline's debug view
+(**Settings**, **Voice assistants**, the pipeline's menu, **Debug**) keeps the same
+note as an `intent-progress` event of the run. The note goes to the pipeline only. It
+is not added to the conversation, so a fallback agent does not read it.
+
 The last 20 decisions the agent made are in the integration's diagnostics, with the
 reason for every decision and the action distribution behind it. The sentence itself
 is redacted, because the file is meant to be pasted into an issue.

site-docs/cost.md

@@ -43,6 +43,29 @@ roughly 110 tokens per entity per call.
 thousands of tokens on every evaluation, which is why it is off by default. Access
 tokens, entity pictures and coordinates are never sent, even with it on.
 
+## A month, worked out
+
+The formula is calls per month, times input tokens per call, times the price. The
+examples use the published $0.042 per million input tokens and a 30 day month.
+
+| Setup | Calls a month | Tokens a call | A month |
+|---|---|---|---|
+| One context on 2 entities, every 5 minutes | 8,640 | about 405 | $0.15 |
+| 20 spoken commands a day, 5 entities exposed | 600 | 1,371 at most | $0.035 |
+| One context on 250 entities, every 5 minutes | 8,640 | about 16,500 | $5.99 |
+
+Where the token counts come from:
+
+- A request with one entity measured 339 input tokens, and each entity record adds
+  65.8, so two entities are about 405. See [what an entity costs](measurements.md#what-an-entity-costs).
+- A spoken command with five entities exposed measured 1,329 to 1,371 input tokens.
+- 250 entities is the cap on a target, at 65.8 tokens each.
+
+Every 5 minutes is the default scan interval, and it is the most a context on a
+schedule asks. A context that wakes on entity changes asks at most once every 30
+seconds, so a busy one can ask up to 10 times as often. Your own numbers are in
+`sensor.jev_input_tokens_today` and `sensor.jev_estimated_cost_today`.
+
 ## The budget is a tripwire
 
 Set **Daily input token budget** in the integration options. The check runs before
@@ -60,7 +83,9 @@ When it trips:
 - the answer sensors of that context go unavailable, because Jev was not asked and
   there is no answer for right now. The last answers are not thrown away, and the
   next call that fits replaces them
-- `binary_sensor.jev_daily_budget_exceeded` turns on
+- `binary_sensor.jev_daily_budget_exceeded` turns on. It shows that contexts have
+  stopped, so a spoken command, an action or an AI Task that the budget refuses
+  does not turn it on. Each of those says so to whoever started it
 - a repair issue explains it, naming the budget, what has been used, and the context
   that was refused with the tokens it needed
 
@@ -89,10 +114,10 @@ that either of those cleared would not be a daily budget.
 - YAML contexts name no entry, so they belong to the first enabled Jev entry. A
   second entry does not ask them again and bill them twice.
 
-The two one-off paths, a conversation command and the preview in the question editor,
-still check what has already been spent rather than estimating the call ahead. Both
-are started by a person and both have somewhere to fall back to, so the worst case is
-one request over the line rather than a runaway.
+A conversation command, an action and an AI Task estimate their call ahead, like a
+context does. The preview in the question editor checks only what has already been
+spent. A person starts it and it has nothing to fall back to, so the worst case is one
+request over the line rather than a runaway.
 
 ### How the estimate is worked out
 

site-docs/limitations.md

@@ -61,6 +61,27 @@ Locks are refused on purpose: Home Assistant reads turn_on on a lock as `lock.lo
 which is the opposite way round from the spoken command. Garage, gate and door covers
 are refused for the same reason.
 
+## No local model
+
+Jev needs the TypeSafe API. There is no local mode.
+
+A llama.cpp fork adds a `POST /v1/decision` endpoint that answers a schema of enum,
+boolean and number fields in one batched pass, with a probability for each answer. I
+read its interface to see whether Jev could use it as a local backend. It cannot, as
+the fork is now:
+
+- It returns only the probability of the value it chose, not the distribution over
+  all values.
+- A noul maps exactly, because the probability of yes follows from the probability
+  of the chosen value.
+- A choice needs the probability of every option, and a score's level is a weighted
+  average over all levels, so neither can be built from one probability.
+- The request shape is different, so Jev would need a second client.
+
+If that endpoint returns the full distribution for each field, a local backend is
+worth doing. Until then, only the yes/no questions could run locally, with the other
+two types still going to TypeSafe.
+
 ## It is not a core integration
 
 `quality_scale.yaml` tracks this against Home Assistant's quality scale at 47 done