feat(perception+web): 本地视觉感知通路 —— 与云端 API 并列的第二条路径(免 Key / 画面不出本地 / 零 token 成本) - #488
Open
LeonJoeeee wants to merge 38 commits into
Open
feat(perception+web): 本地视觉感知通路 —— 与云端 API 并列的第二条路径(免 Key / 画面不出本地 / 零 token 成本)#488LeonJoeeee wants to merge 38 commits into
LeonJoeeee wants to merge 38 commits into
Conversation
…ception backend Adds a second perception path that needs no model-provider API key: video frames are encoded to H.264 and handed to a local GPU sidecar, which returns a scene description plus per-rule judgments. Footage never leaves the home and the path costs zero tokens. Why not just point base_url at a local OpenAI-compatible server: the value of a codec-native model comes from consuming H.264 motion vectors and residuals directly, which the OpenAI chat protocol cannot carry (the reference model's own online client sends sampled frames as image_url). Measured on real home footage, the codec path uses 736 prompt tokens vs 7338 for uniform frame sampling (-90%) and runs 3.2x faster. Design: - LocalVisionEngine implements the existing BasePerceptionEngine ABC, so the pipeline skeleton is untouched; PerceptionEngineProxy picks the backend from perception.engine_backend (default cloud — no behavior change for anyone who does not opt in). - GPU inference lives in services/local-vision, a standalone sidecar outside the uv workspace: miloco's target hardware is CPU-only (Mac mini, Pi) and must never pull in torch/CUDA. miloco never downloads weights or manages the model process (see upstream XiaoMi#144, where 1.x owning the model container spread the failure surface into GPU passthrough and container issues). - Vision-only by construction: speeches/env_sounds stay empty rather than letting a model that cannot hear invent them (same rationale as the existing requires_audio gating). - The local path never drives devices: rule hits always go to the agent, so STATIC rules' direct execution is inactive. That is surfaced in the log and the WebUI instead of letting existing rules silently stop firing. Rule parsing is fail-closed — an unparseable verdict counts as no-hit, since a missed reminder is cheaper than the agent acting on a fact that never happened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the cloud-vs-local comparison to the perception-pipeline capability boundary section and points the architecture overview at the BasePerceptionEngine seam, per the knowledge base's L2 rule (record the why of key design decisions, not the parameter values). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The admin perception-backend endpoint echoes the sidecar's /health body back to the caller, and base_url is user-supplied — echoing the raw body would turn that endpoint into a probe that can read any URL's response body (SSRF echo). Only known fields pass through, strings are length-capped, and a non-dict body yields an empty payload. Mirrors the existing anti-SSRF posture around omni credentials, where a fetch target is never trusted just because an admin supplied it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t line
Live testing surfaced a fourth output shape the parser missed: the model
restates the condition on the rule line and puts the verdict underneath --
规则1: 有人在客厅沙发上
否 - 依据: 沙发上没有人物
Fail-closed kept this safe (no false hit), but a real verdict was being read
as "no verdict", so a positive answer in that shape would have been a silent
missed hit. Now an unparseable rule line looks ahead to the next non-empty
line, stopping at the next rule line so a neighbour's verdict is never
borrowed.
Also stop back-filling the reason from the rule line's own text: when the
model only echoes the condition, that produced a hit=False row whose reason
read like the condition was satisfied -- exactly backwards to anyone reading
the record.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The core promise to upstream is that anyone who does not opt in sees no change at all. Pin it directly on the dispatch in _init_engine: with default settings the local branch must not be entered (otherwise every existing deployment without a sidecar would land in PREREQ_MISSING and perception would stop), and with engine_backend=local it must be entered rather than silently falling back to the key-requiring cloud path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A clean-context review of the branch found four defects that unit tests calling the engine directly could never have caught, because none of them went through PerceptionEngineProxy. 1. The backend could not start with engine_backend=local. The proxy and the pipeline processor call set_tierc_frame_provider / set_main_loop / apply_omni_fps / get_input_config on the engine unconditionally, but the ABC declared only the two perceive methods, so those lived on the cloud engine alone. Any second implementation would AttributeError at startup. Fixed at the source: BasePerceptionEngine now declares them with harmless defaults, so the ABC is an honest contract for every implementation. 2. Rules fired once and then went silent forever. device_rule_map was never populated, and the rule state machine is edge-triggered — with no per-cycle False the last state stayed True for the process lifetime, so a rule that hit once never fired again, state-mode rules never EXITed, and duration windows only ever accumulated. Now populated per successfully-perceived device, and deliberately not for devices whose inference failed (registering those would retract rules on no evidence). The gate now suppresses only the narration, never the rule verdicts, for the same reason. 3. The "switch takes effect immediately" path was a silent no-op: it reached for manager.perception_engine_proxy, which does not exist, and getattr's default swallowed it. A user switching away from cloud saw a success toast while the cloud engine kept running and kept billing. Now goes through perception_service.stop_to_unconfigured, the same path omni activation uses. 4. The parser read 不是 as a hit. 不是 contains 是, and the miss-word list had no entry for it, so "厨房有明火? 不是 - 灶台已关闭" produced a hit whose reason said the opposite — precisely the false positive the fail-closed design promises cannot happen. Negation is now checked first, on the verdict head only, so a reason containing 不 no longer drags a genuine hit closed. Also from the same review: blocking sync health probes moved off the event loop and given short timeouts (they run every tick and would stall frame ingestion against a firewalled sidecar); a stored token is no longer carried to a changed base_url, and probe errors are reduced to a coarse code, since base_url is user-supplied and the video itself would be the real loss; unresolvable rule hits are dropped instead of being credited to the index-matched rule; encoder setup moved inside try/finally so a PyAV build without libx264 degrades per device instead of killing the cycle and leaking the container; payload budget (max_frames, short_edge) added to match what the cloud path already does before inference; empty-device cycles no longer book an inference error; and non-latency timing keys moved under the underscore namespace. The sidecar's tests now run in CI (services/ was covered by no job), the caption no longer falls back to raw machine-formatted output, and token comparison is constant-time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fixed An adversarial re-review of the previous commit found that two of the four "fixed" blockers still failed in realistic cases, and that the commit had introduced new problems of its own. **Rule verdicts were still read out of prose.** Checking negation first only moved the boundary; the scan was still "does this 12-char window contain 是", so any 但是 / 于是 / 总是 won, and the next-line lookahead handed arbitrary prose to the same scanner. "画面中看到一个人影,但是不能确定是否在沙发上" came back as a HIT whose reason said the opposite — the exact artifact fail-closed exists to prevent. Verdicts are now recognised only as a word at the start of the verdict head, with the asymmetry the safety argument requires: negation stays loose (over-matching only loses a report), affirmation must end on a word boundary (so a restated condition beginning with 有 is no longer a hit). 是否 is caught before the affirmative branch, since it is the model repeating the question rather than answering it. Verified over the reviewer's full corpus: zero false hits, zero dropped hits. **device_rule_map was populated but then discarded.** skipped was derived from "no caption and no matched rules", and the consumer returns before reading the map when skipped is set — so a device that judged its rules and simply found nothing still fed the state machine nothing, and rules stayed pinned exactly as before. The previous commit's own caption fix made this the common case. skipped now means "no evidence": it stays false whenever any rule was actually judged. Also fixed from the same review: the CI job added last commit failed on every run (pytest from the repo root cannot import local_vision — needs the package dir as cwd); switching backends left the fresh engine without the tier_c frame provider, because the rebuild now lands directly in ready and the tick path that re-attaches only handles non-ready states; local engine construction had no try/except, so a failure left status=ready with engine=None — ready forever false and try_reinit refusing to retry, i.e. perception permanently stopped with no self-heal; the health probe still blocked the main event loop every tick against a firewalled sidecar (measured 1.5s per 4s cycle), now on a cooldown after failure; uniform floor sampling never selected the last frame, silently dropping the end of every window — the part most likely to contain the event; the ABC's realtime_perceive signature omitted the on_early_* hooks that the caller passes unconditionally, so a third implementation written to the ABC would TypeError on its first cycle; and the STATIC-rule capability flag was dead code, now a class attribute the admin endpoint actually reads instead of hardcoding the same fact twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Everything the second review left on the table, rather than shipping a known-issue list: - Health probing no longer happens on the event loop at all. The tick now awaits a threaded refresh before the synchronous rebuild, so the rebuild path never touches the network; a firewalled sidecar can no longer stall camera ingestion, SSE and the API. The cooldown stays as a backstop. - Persisting a base_url now always probes it first, on both branches. The cloud branch previously accepted an arbitrary address with no validation, which the GET endpoint would then go and probe. - The sidecar caps in-flight inferences and returns 503 when busy. With three cameras each sending a request per window, queued requests would otherwise pile up holding threadpool threads on the GPU lock until /health itself timed out — miloco would then declare the sidecar dead over what is purely queueing. - The sidecar loads weights on a background thread, so /health answers during the tens of seconds of loading. Previously the 'loading' state was unreachable and a cold start looked identical to a missing service. - A 描述 written after the rule lines is recovered instead of leaving the caption empty with no way back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…arallel set An audit for "did I redefine something that already exists" found three hits, one of which was worse than duplication — it silently dropped a feature. - **Per-camera 感知须知 was being ignored.** Users write per-camera guidance in the dashboard (CAMERA_PROMPT_MAP_KEY, surfaced as perception_prompt); the cloud path injects it every window. The local path never read it, so switching backends silently voided whatever the user had written, with nothing in the UI to hint at it. Now injected the same way. - `physical_did` and the per-device rule filter were copy-pasted from the cloud engine. Both now live in perception/rule_scope and are used by both paths — two copies of "which rules go to which camera" is exactly the kind of thing that drifts silently and leaves the two backends disagreeing. - `local_vision.video_short_edge` duplicated `perception.engine.input.video_short_edge`, which already has an API and a UI. It now defaults to following the shared value, so tuning resolution in the dashboard affects both paths; an explicit override remains for the rare case that needs it. Two defects the live deployment caught that no unit test could: - Making short_edge default to None (to follow the shared value) sent None into the encoder, which compares it against an int — every window raised TypeError and fell back. Only visible when the proxy actually builds the engine, which the unit tests bypass; now pinned by a test that goes through the proxy. - Greedy decoding loops on this model: a real caption repeated "房间的角落里还有一个 白色的物体" a dozen times, ran to the token cap and doubled generation time. The reference implementation hides this behind max_new_tokens=80; we want longer descriptions, so repetition has to be suppressed explicitly. Captions went from ~600 chars of repeated filler at 3.3s to ~140 clean chars at 1.3-1.6s. Raw H.264 passthrough was implemented and then deliberately dropped from this PR. The plumbing works (subscribe, callback, buffer), but the camera's raw packets are not the plain H.264 Annex-B this assumed — the NAL layout matches HEVC — and miloco's raw-video callback signature drops the SDK's codec_id, so the format cannot even be determined at that layer. Feeding the camera's original bitstream is a real further improvement, but it needs codec_id plumbed through first, which is its own change. The measured codec-native benefit is unaffected: it was always measured on a re-encoded stream. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mbiguous names The audit listed these and I then only reported them instead of acting, which is the same mistake as shipping a known-issue list. - **libx264 was running synchronously inside a coroutine.** The project already learned this one: miot/transcoder.py exists specifically so "callers await the async encode() so the asyncio event loop is never blocked by libx264", and it keeps a dedicated executor for it. Window encoding now goes through asyncio.to_thread, with a test that fails if it ever runs on the loop thread again. On-demand queries are served from the main loop, so this was not theoretical. - `fps` -> `container_fps`. Perception already has two frame rates that mean real things (engine.input.fps for dispatch/tracking, omni_fps for what reaches the model); a third field called `fps` reads as a third sampling rate when it only writes the mp4 container timebase. - `gate_threshold` -> `event_gate_threshold`. Perception already has a gate (frame diff + audio energy, tuned by change_threshold). Two settings both called "gate" leave no way to tell which one you are adjusting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…camera note replaced the task Two blockers, both introduced by the two commits immediately before this one. **Renaming gate_threshold missed a call site, and it took the whole feature with it.** `admin/router.py` still read `cfg.gate_threshold`; on pydantic v2 that is an AttributeError, so GET and POST /api/admin/perception-backend both returned 500. The 模型 page is the only way to switch backends, so on that branch the feature was unreachable from the UI — and no test covers either endpoint, which is why the rename looked safe. The field was dead weight anyway (no component read it), so it is gone from the payload and the TS type rather than renamed. **The per-camera 感知须知 was replacing the task instead of supplementing it.** scene_ask defaults to empty, so the branch that was supposed to append the note instead produced a prompt consisting only of the user's note — no question, no "describe the scene" instruction. That is the inverse of what the previous commit claimed to fix: the cloud path appends the note as an extra section on top of the full task prompt. Worse, the note landed *upstream* of the output-format spec, so a note like "只用一句话回答" could delete the 规则N: lines entirely — and because rule parsing is deliberately fail-closed, every rule on that camera would stop firing with no exception, no error_code, and the state machines still being fed False. The note is now its own request field, rendered in a delimited block *after* the format spec, length-capped, and explicitly told not to change the output format. Also from the same review: - The sidecar's in-flight cap defaulted to 2 while miloco allows 4 concurrent cameras and dispatches them all at once, so the same one or two cameras lost the race every window and their rules were never evaluated — silently, since a 503 just drops the device from the batch. Default is now 4. - The in-flight semaphore was acquired outside the try, so a failed temp-file write leaked the slot permanently; two of those and the sidecar answers 503 forever. - repetition_penalty / no_repeat_ngram_size count the prompt too, and the prompt contains each rule's query verbatim — a model restating the condition in its verdict would get cut off mid-sentence, and that mangled text becomes the reason shown to the agent and the user. They now apply only when there are no rules, which is where the observed repetition actually happened. - A rule hit with a blank name fell through to the index-positional rule, i.e. the exact mis-attribution the comment above it warns about. Blank names are dropped. - video_short_edge is resolved per window instead of at engine construction, so the dashboard's "takes effect next frame" contract holds for this path too. - api.py's `_physical_did` is a real delegation again rather than a snapshot alias, and the local path stopped doing an extra physical-did prompt lookup the cloud path never did — rule_scope exists to remove differences like that, not host them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…top the note from forging verdicts **The UI was stating a property the code did not have.** The backend card, the log line and both locale files all told the user "本地通路不直接控制设备:规则命中一律交 agent 决策执行". Nothing implemented it. RuleRunner is engine-agnostic, so a matched rule with `actions` still went straight to `_execute_action`. A user with 厨房明火 → 关燃气总阀 could read that bullet, leave the rule enabled believing an agent would vet it, and have the valve closed directly by a 4B model's "是" — from a path this same PR documents as loop-prone and fail-closed-only. STATIC slots are now converted to agent decisions while the local backend is active, preserving the configured action semantics as intent for the agent. The cloud path is untouched, and the rule tests now pin the backend explicitly rather than reading whatever the developer's config.json happens to say (which is how this surfaced: the suite went red only because this machine was switched to local). **A camera note could fabricate a rule hit.** The note is user-editable free text rendered next to the rule list. A note containing a line shaped like `规则1: 是 - …` is indistinguishable from the demanded output format sitting one line above, so a model echoing it produces a genuine, correctly-named `hit=True` → MatchedRule → (per the above) a device action. User-writable text conjuring a rule hit is the exact direction the design calls unacceptable. Verdict-shaped lines are now stripped from the note, and the 「」 delimiters are stripped from its contents too — otherwise a single 」 closed the block early and dropped the remainder into the strongest recency position at the end of the prompt, where "上面的说明作废" would delete the rule lines entirely and fail-closed would turn that into every rule on that camera silently never firing. **Turning off loop suppression when rules are present put the original bug back on the only path that has rules.** Repetition is a decoder property, not a prompt property; with rules the prompt still asks for a free-form description first, so the loop still happens — and when it does it eats the whole token budget before any 規則N: line, leaving every rule unparsed and pushed to False, silently. The previous justification was also only half right: repetition_penalty merely rescales logits and cannot truncate anything, so it is safe to keep on unconditionally; only no_repeat_ngram_size hard-bans, and only that one needs a larger n when rule queries are in the prompt. Also: the note cap now matches miloco's own 500-char limit and marks truncation instead of silently dropping the qualifier users put last; the sidecar reports how many rules produced no parsable verdict, so "the model said no" is distinguishable from "parsing failed" in the logs; in-flight slots leave headroom above the camera count so an on-demand query can't be starved by a realtime window; temp-file cleanup can no longer turn a successful inference into an unhandled 500; video_short_edge gained a lower bound (0 or negative silently disabled the whole payload budget); and the remaining snapshot alias in api.py became a real delegation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… is local Screenshotting the real dashboard surfaced this: the 模型 page carries two 生效中 badges that mean different things. The perception card says 本地 GPU 生效中, and directly below it the model table still marks mimo-v2.5 as 生效中 — while nothing is calling it for perception. A user reading that page has no way to tell which one is actually doing the work, and would reasonably conclude the cloud model is still being billed. The model table now states, when the backend is local, that these models are not serving video perception and how to switch back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tch instead Round 5 showed the STATIC-suppression mechanism from the previous commit was worse than the problem it solved, so it is gone and the shared rule engine is back to byte-identical with upstream. What was wrong with it. `_static_actions_as_prompt` read a `description` field that does not exist on `RuleAction`, so every action fell through to a `did`/`iid`/`value` fallback that drops `params` entirely — a TTS rule's whole message text vanished, and a `call_action` was described to the agent as "set a property to None". The test I wrote for it copy-pasted the production branch into the test body instead of calling `_fire`, against a `SimpleNamespace` carrying that non-existent field; replacing the whole conversion with `pass` left 215 tests passing. The fake schema in the test is exactly why the broken payload shipped. And two losses were structural, not fixable by writing a better prompt: routing through the agent discards `cooldown_minutes`/`idempotent` — the only rate limiter the schema has, which the service layer validates as mandatory for non-idempotent actions, so a TTS rule becomes an agent callback every window — and it discards the action ledger's `source=rule` attribution, breaking the rule→device-change link XiaoMi#406 was built to provide. Rewriting a user's configured automation at runtime, in a way that silently drops its rate limit and its audit trail, to be re-executed by an LLM from a lossy translation, is not a safe default. Switching to the local backend now **refuses** while rules with direct device actions are enabled, and names them. The user decides: disable them, or stay on cloud. No shared-engine surgery, no lost semantics, predictable, testable — and it is what "STATIC 管线先不启用" actually means. Also from round 5: - Note sanitization was defeated two ways: `「规则1: 是 - …` passed the per-line check and *became* a valid verdict line after the quote-stripping ran, and two individually harmless lines joined into one. The invariant now holds on the final rendered text, and the `规则N:` token itself is removed rather than merely displaced — leaving the text in place just means the model can copy it onto its own line. - Rule `query` had no sanitization at all despite being unconstrained free text spliced directly into the condition list, where one newline forges a rule and renumbers the rest. Same treatment. - `unparsed_rules` was computed and logged on the GPU box, then silently dropped at the HTTP boundary by pydantic's default `extra="ignore"`. It now crosses the wire, and miloco warns per device — otherwise "the model emitted garbage" and "the model said no" are the same input to an edge-triggered state machine, and a STATE rule mid-ENTER fires its `on_exit` actions on a person still standing there. - Conflicting duplicate verdicts for one rule resolve to no-verdict instead of last-wins, which could flip a 否 into a 是. - `hmac.compare_digest` raises TypeError on non-ASCII, so one Chinese character in the token turned every auth check into an unhandled 500 while /health still said ok. - `pick_backend` kept `codec` when the frame probe failed, inverting its own documented contract — a box without ffprobe took the codec path and 500'd every window. - The CI job installed torch and 43 nvidia wheels (4.9 GB) because `uv run` in a directory with a pyproject.toml means project mode; `--no-project` gets the same coverage in seconds without evicting the other jobs' caches. - A non-dict sidecar response killed the whole window for every device instead of degrading that one. - The "cloud models are idle" banner never updated after an in-page switch, so it asserted the opposite of the truth in both directions until a reload; and its zh text claimed the models are still called for other things, which is false and absent from the en text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit changed how the local path honours 'no direct device control' — from rewriting rules at runtime to refusing the switch — but left the card saying '规则命中一律交给 agent 决策执行'. That is the same defect as the round-4 blocker: the UI asserting a mechanism the code does not have. The bullet now says what actually happens, and the card lists the enabled rules that will block the switch so the user sees them before clicking rather than in a 400. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uthenticates The health probe never sent credentials. Any deployment with LOCAL_VISION_TOKEN set would therefore green-light the probe and then 401 on every inference window: the card stayed green while perception silently stopped. Only a live run with a token configured exposed it — every test above that layer mocks health_sync. The probe now sends the token, the sidecar reports auth_required/auth_ok from the same comparison the inference path uses, and both the switch endpoint and the engine refuse on a credential mismatch instead of proceeding. Also in this commit: - token input in the card; it is only submitted when non-empty, since an empty string means 'clear the stored credential' (buildSwitchPayload, tested) - generation budget scales with rule count; truncation is reported and logged. Without it, more rules means the trailing verdicts get cut off and fail-closed reads them as 'no match' — silently, and worse the further down the list - on-demand queries relax the repetition guard, which was tuned for periodic captions, not for free-form agent questions - max_pixels now also applies on the frames branch — the branch short segments actually take, where the visual budget was previously unbounded - probe cooldown is keyed on (base_url, token): fixing a wrong address no longer costs the user a 30s wait with the UI already claiming 'local' - switching back to cloud is never blocked, only annotated with cloud_hint; it is the escape hatch from a broken local path Tests: sidecar app.py/video.py had no coverage at all (60 tests now); the admin endpoint had none (11 tests, all five gates mutation-checked); the card's decision logic moved to a pure module so the node-env suite can cover it without pulling jsdom into the repo. An i18n test now fails when a referenced key is missing — the previous commit shipped four raw keys visible on the page. Live: 书房 camera, codec path, 10 windows — median 223 chars / 2063 ms (RTF 0.52), 0% truncation; frames fallback verified separately with max_pixels applied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…not a switch-time check
The refusal added last round only ran at POST /perception-backend. Every route that
reaches the forbidden state afterwards was open: create a rule with actions while
local is active, edit an existing rule to add them, activate a task that bulk-enables
its rules, or edit config.json directly. Nothing on the firing path knew a backend
existed — _select_slot returns ("static", actions) purely on field presence — so the
gas-valve rule the code uses as its own example would have been executed on a verdict
from a vision-only 4B model with no audio corroboration and no identity, while four
surfaces told the user this path performs no device actions.
The gate now sits at the point of execution, which is the only complete one: rules are
driven exclusively by perception (three call sites, all in perception/client.py), so
keying on the perception backend cannot over-block. Refusal is loud — an ERROR naming
the rule plus a rule_action_refused event. Still no rewriting to dynamic: that would
drop cooldown_minutes/idempotent and the source=rule ledger attribution.
Three tests that proved nothing, found by an adversarial audit:
- the frame-budget test asserted only the sample count, so replacing endpoint-inclusive
sampling with floor sampling (which never reaches the last frame — the exact bug the
production comment warns about) left it green. All frames were identical zeros.
- the switch-refusal test patched the helper, called it, and asserted the patch; the
half that ran real code ignored enabled_only, so a *disabled* rule blocking the
switch would have gone unnoticed. There is now an endpoint test using the real lookup.
- the entire local bring-up path (~45 statements: auth refusal, model-loading wait,
construction failure, probe invalidation, cooldown) had zero executed lines while two
tests appeared to cover it — both mocked _init_local_engine away.
Also fixed, each found by review rather than by the suite: probe cooldown never armed
for a rejected credential (~21.6k pointless probes/day); config changes re-probed
synchronously on the API event loop; the explicit restart button was a no-op inside the
cooldown; JSON ints for boolean health fields were dropped, making the auth check
fail-open for third-party sidecars; captions and matched rules carried no time_window,
so every local event lost its 时间 line; clip bytes were never attached, so events were
text-only; per-device sidecar failures were invisible (a camera 503ing every window
showed zero errors); frames branch ignored max_pixels; get_input_config returned None,
rendering 0fps for all three layers; the local button showed green 可达 while the line
below it said the credential was rejected; and duplicate rule verdicts that agreed but
were worded differently were treated as conflicts and fail-closed away.
17 mutations verified caught. Note: the earlier mutation harness restored sources with
shutil.move, which set mtime backwards and left Python using bytecode compiled from the
mutated source — those results were void and have been redone with a cache-safe harness.
Suites: backend 2874, sidecar 71, web 267. Live on 书房: time_window correct in deploy
timezone, codec path 2.5s/window. Clip attachment is test-verified only — no meaningful
event fired during the observation window, so it was not seen end-to-end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…escribing a mechanism that was deleted
Three things this branch got wrong, all found by review rather than by the suite.
**The gate was over-broad.** Last commit's message claimed rules are driven only by
perception (three call sites). There is a fourth: RuleRunner.trigger_rule, reached from
POST /api/rules/{id}/trigger and the CLI — a human or the agent explicitly asking for a
rule to run. That is precisely the actor the design says decides. It was being refused,
and the endpoint translated the None into "Rule not found or disabled", which is simply
false. The gate now applies only to perception-driven fires.
**A refusal left no trace an operator could find.** _fire returned before the RuleLog
write, so the rule's execution history in the UI was empty — the automation stopped and
nothing said why. A refusal is now recorded as RULE_TRIGGER_FAILURE with a reason on
RuleExecuteResult, and it no longer emits the FIRE log line or the rule_fire event for
an execution that did not happen. The blocking-rule list on the model page was rendered
only while still on cloud (rendered under a not-isLocal guard), i.e. hidden in the one state where those rules
are actually being refused; it now shows in both, with wording per state.
**Six places still described the static→dynamic rewriting deleted two commits ago** —
the engine module docstring, capabilities.py, the switch log, the card header comment,
the sidecar README and the knowledge doc all said hits are "一律交 agent 决策执行",
which reads as "your valve rule still runs, just via the agent". It does not run. Also
corrected: video.py asserted as measured fact that a 4s window yields ~4 frames and so
normally falls back to the frames backend — the opposite is true (246 consecutive live
windows all took codec), and a contributor acting on it could conclude the whole
codec-native rationale was dead code.
I also reintroduced synchronous HTTP on the API event loop in the last commit — both
try_reinit(include_failed=True) and the first switch to local probed inline, stalling
camera ingest, SSE and the whole API for up to 3s. Sync probing is now confined to
construction; every other path waits for the threaded tick refresh.
Docs: the sidecar README's env table was broken by a stray blank line I added; four env
vars the code reads were undocumented; the interface section omitted auth_required /
auth_ok / unparsed_rules / truncated, so a third-party sidecar built to the documented
shape would make miloco's credential check fail open — the required fields are now
called out with the consequence of omitting each. The card's health line now says the
sidecar backend shown is the startup value, since the effective one is chosen per request.
Tests: 11 mutations that survived the previous round now fail as they should — the
rejected-credential cooldown, the restart bypass, the no-sync-probe rule, the token
ceiling, the tick ordering, the capability linkage (was a tautology: two equal values,
not a dependency), the timezone (was a shape regex that cannot distinguish any two
zones — the exact 凌晨3点 incident), the sidecar kwarg-rename guard (the double swallowed
**kw), and the manual-trigger exemption. One test I wrote last round was racy: it keyed
sidecar responses by call order while the per-camera encodes finish in thread-pool order,
so which camera got the malformed response drifted. It keys responses by device id now.
Suites: backend 2885, sidecar 73, web 271. Live on 书房: codec path, 1030 ms/window,
time_window correct in deploy timezone, switch log now accurate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…this A maintainer-perspective review found the branch had red CI and three ways it reached users who stay on the cloud backend. Those are the ones that matter most for a feature that is off by default. - **CI was red.** The knowledge-doc table I added is not Prettier-formatted, and .github/workflows/docs.yml checks knowledge/**. Verified both directions against origin/main. - **The rule engine started importing the whole perception package.** A module-level `from miloco.perception.capabilities import ...` in rule/runner.py pulled in cv2, av, numpy and perception.processor — measured +0.36s on `import miloco.rule.runner`, where origin/main has zero top-level perception imports there (its one perception dependency is deliberately function-local). The import is now function-local too; after the fix only `av` remains, and that arrives via miot.client, which runner.py already depended on. - **Cloud users got a permanent orange warning about their own automations.** The blocking-rules box rendered whenever the list was non-empty, so a default install with one ordinary "someone comes in → turn on the light" rule showed a standing warning on the model page. Direct-device rules are the product's most common automation. It now renders only while the local backend is active, which is when those actions are actually being refused; the 400 on a refused switch already lists them by name. - **The shared ABC no longer gets no-op defaults.** Adding them saved five one-liners in the new engine at the cost of the cloud engine's failure mode: a renamed `close()` would silently no-op and leak the identity dispatcher thread instead of raising. LocalVisionEngine implements the five hooks itself. The one edit kept to engine_base.py is the `realtime_perceive` signature, which did not include the `on_early_*` callbacks the caller passes unconditionally — a new implementer copying the abstract signature would TypeError on the first cycle. Also from that review: a fresh httpx.AsyncClient per device per window (4 cameras × 4s windows = a TCP handshake per camera per second, never reaching keep-alive) is now one pooled client released in close(); the two pass-through wrappers in engine/api.py are gone in favour of the shared helpers they delegated to; the orphan uv.lock is removed (nothing consumed it, and it is structurally incomplete since torch is deliberately not a dependency); and the sidecar is now covered by the lint job, which had two import-order errors nobody was running. One real defect fixed alongside: a sidecar stuck loading forever — the load exception is caught deliberately so the process stays up — was re-probed every 4s indefinitely, because only the auth-rejected case armed a cooldown. Loading now gets a short one. Test gap closed, and it was the important one: the device-action gate had never been exercised through a perception-driven fire. Every gate test called `_fire` directly and relied on the parameter default, so marking the perception call site as non-perception — which opens the gas valve on every hit — left all 2885 backend tests green. Two tests now drive it through `update_state`, the entry point the perception engine actually calls, one per backend. Also fixed from the same audit: the tick-ordering test asserted on `inspect.getsource` string positions (green if the call were replaced by anything containing that identifier, red on a harmless refactor) and PipelineProcessor's delegation had no coverage at all — gutting it left the whole "no sync HTTP on the main loop" design dead with 1316 tests passing; the i18n key check missed dynamic `t(cond ? a : b)` usage, so a typo in a new key shipped green while the page rendered the raw identifier; `isReachable`'s test restated the implementation instead of asserting a value, hiding that the card showed a green 可达 badge while the sidecar was still loading and the switch endpoint would refuse; six config fields were never checked for reaching the engine; and `_make_proxy` now asserts field parity with the real object rather than being right by hand. The docs now scope the guarantee honestly: "the local path performs no device actions" constrains the perception layer. Dynamic rules still reach the agent, and the agent can drive devices — the risk is mitigated by an LLM adjudicating first, not eliminated. Suites: backend 2892 (3 consecutive clean runs), sidecar 74, web 271; ruff clean including the sidecar; prettier clean. Live on 书房: 1480 ms median, connection reuse confirmed (7 windows, 2 sockets). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s the docs promised Two fresh lenses this round — a security review and a literal first-run rehearsal of the README by someone who had never seen it. Both found things eight rounds of correctness review had not. **The sidecar could not be installed by following its own README.** `requires-python` said >=3.10 with no upper bound, but codec-video-prep ships Linux-only cp39–cp312 wheels with no sdist and pins numpy<2.0, whose newest matching release also stops at cp312. `python -m venv` on a machine whose `python` is 3.13+ fails outright at `pip install -e .`. The one environment where this ever worked was built by uv on a managed 3.13 with numpy compiled from source — a toolchain the README never mentions. Now pinned to >=3.10,<3.13 with a stated support matrix (Linux only; macOS has no wheels at all, which quietly contradicted "run it on another GPU box"). Also from that rehearsal: `pytest tests/` was documented but pytest was in no dependency group, so the documented command was `command not found` — there is a `dev` extra now. And `pip install -e .` pulls torch regardless, because accelerate requires it; if the reader skips the cu128 line pip silently installs a generic build that is wrong for the hardware the README calls out. The install order is now stated as load-bearing rather than incidental. **A failed model load was indistinguishable from a slow one, forever.** The loader deliberately catches its exception so the process stays up, so a typo'd checkpoint path left /health reporting "loading" indefinitely while miloco told the user "still loading, try again shortly" — advice they could follow until the heat death of the universe. /health now carries `load_error`, both miloco surfaces report it, and `resolve_checkpoint` refuses a path-shaped argument that does not exist instead of passing it to the Hub as a repo id (which produced an error message about repo-id syntax to a user who had typed a directory). **Security findings, none of them blocking.** The README's "no token ⇒ loopback only" rule was a sentence, not a check — while the same document steers deployers toward a separate GPU box, whose obvious `--host 0.0.0.0` yields an unauthenticated inference endpoint carrying home camera footage on the LAN. The service now refuses to start in that configuration. Request bodies were unbounded and both the buffering and the base64 decode happen before the inflight semaphore, on a 40-worker threadpool, so the limiter did not bound memory at all; there is a 64 MiB cap now. `scene_ask` was unsanitized and safe only by an unasserted coincidence (the on-demand path happens to send no rules, so there was no verdict block to suppress) while the on-demand query is agent-authored — it goes through the same sanitizer as camera notes now. Two smaller ones: the probe error was scrubbed to "unreachable" in the admin endpoint but published verbatim (target URL, status, exception class) through the engine-status endpoint, and base_url accepted `?`/`#`, which take over the path the client builds. The prompt-sanitizer docstring claimed more than the code delivers, and I have corrected it rather than the code: it defeats syntactic forgery of a verdict line, not instruction following. Neither does the cloud path, which injects the same agent-writable text into its system prompt with no sanitization at all — this path is safer, not immune, and the docstring now says so. Discoverability: the model page offered a "Local GPU" button whose failure mode was a correct but dead-end "service unreachable" — nothing anywhere told the user what service that is or where to get it. There is a line pointing at services/local-vision/README.md, and the README grew a troubleshooting table for the three failures a first-timer actually hits (stuck loading, silently degraded to the frames backend because ffprobe is missing, token mismatch). Verified: all CI jobs run locally with their exact commands (prettier, markdownlint, workflow-sanity, the sidecar job including its new deps, web install/typecheck/test/build, ruff now covering the sidecar). Backend 2895, sidecar 80, web 271. Live: startup refusal confirmed by actually trying `--host 0.0.0.0` with no token; /health carries the new field; perception at 1432 ms/window; and the first-run hint verified on screen by switching the real deployment to cloud and back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ake failures visible
Two lenses this round: a long-run/fault-recovery soak on the live deployment, and an
upstream-consistency review asking whether this code looks like the repo it is joining.
Both found things nine rounds of correctness, test, security and docs review had not.
**A dead sidecar was invisible to the user.** The cloud path gets a circuit breaker, a
global red banner and a 「立即重试」 button. The local path got nothing: once the engine
was ready it was never probed again, so a sidecar that died mid-run left the status
`ready` forever while every window quietly returned `skipped`. The only signal was
"events stopped appearing", which takes a long time to notice in a home. The engine now
reports sustained failure and the proxy demotes it to the same PREREQ_MISSING state the
cloud prerequisites use — so the existing status ribbon, its message and its restart
button all light up with no new UI. Verified live: killed the sidecar, demotion fired
33 s later, and the overview page showed 「还没准备好 · 本地视觉服务不可达」 with the
restart button.
**And it burned CPU while doing it.** Because the engine stayed up, every window still
ran a libx264 encode per camera before failing to connect — 4 cameras is 4 wasted
encodes every 4 s, forever, with the log filled to match. There is exponential backoff
now (4→8→16 s, capped at 30, cleared by one success). Live: attempts over 45 s dropped
from ~11 to 5.
**Every inference was emitting a warning I had never read.** `target_canvas` is a count
of canvases, not frames: filling 32 of them needs 256 source frames, and a 4 s miloco
window has at most 32. So every single request asked for something impossible and the
model said so, in its own stdout, for the entire evaluation. It is derived from the real
frame count now; the warning is gone and output quality and latency are unchanged.
**A killed process left home-camera footage in /tmp.** Cleanup lives in a `finally`,
which SIGKILL skips. Old segments are swept at startup — but only ones old enough that
they cannot belong to a second sidecar instance sharing the machine.
The consistency review's findings, in order of how much they matter:
- Backend rejections were hardcoded Chinese with no machine-readable code, which the
repo has a written rule against (`OmniHealthBanner.tsx`: "backend message 是硬编码
中文,直接注入会污染英文界面") and an implemented convention for. They carry `code`
now and the card maps it, like `OMNI_CODE_KEY` does. Fixing this exposed that
`apiFetch` stringifies an object `detail` into "[object Object]" — a latent bug the
omni PUT path has had all along; it unwraps `{code, message}` now, so both paths win.
- `GET /perception-backend` did a 3 s health probe, a full rule-table scan and a
`validate_resources` (which mkdirs) on **every** call, and the model page mounts two
components that each call it. The repo separates config reads from explicit probes
(`get_omni_config` is zero-IO). It is a pure read by default now; the card opts in
with `?probe=1` and the model table, which only needs to know which backend is active,
does not.
- `_validated_base_url` duplicated `probe._normalize_base_url` and was weaker (no host
check, so `http:///health` passed). It calls the existing one, and the `?`/`#` rule I
had added moved there, so the omni path gets it too.
- The cross-URL credential wipe compared raw strings, so a trailing slash counted as a
URL change and silently discarded a still-valid token. It uses the same normalized
comparison `_key_by_label` does.
- The pooled `httpx.AsyncClient` had no event-loop guard, in a repo that wrote a comment
block about exactly this failure (`_get_fused_http_client`). It is keyed by loop now,
with the same connection limits.
- The sidecar had no ruff config, so the lint line I added to CI last round was running
ruff's *defaults* — no isort, and `F403` on. Adopting the repo's block immediately
surfaced three import-order violations nobody was checking.
- The no-op defaults I put on `BasePerceptionEngine` are gone (they weakened the cloud
engine's failure mode); the five hooks live on `LocalVisionEngine`. Two pass-through
wrappers in `engine/api.py` are deleted in favour of the shared helpers. The inline
soft-stop reuses `_soft_stop_best_effort`. Two test files I created folded back into
`i18n.test.ts` and `real.test.ts` where the repo keeps those. `perceptionBackend` i18n
moved to its own domain file, per the one-domain-per-file rule.
Mutation-verified: pure-read GET, sustained-failure demotion, the `?`/`#` rule, and the
loop-keyed pool all fail the suite when broken. Suites: backend 2899, sidecar 84, web
274. All CI jobs run locally with their exact commands. Live: kill → backoff → demote →
UI shows it → restart → automatic recovery, no intervention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e contract
Two lenses: a maintainer reading the diff **without the commit messages** (to find what
the code fails to explain about itself), and the multi-camera paths — the concurrency
cap, per-device errors and on-demand fan-out were all written for multiple cameras and
had only ever run with one.
**The capacity warning I added last commit read a field that does not exist.**
`getattr(batch, "window_duration_ms", 0) or 4000.0` — `BatchedSnapshot` has exactly
`snapshots` and `captured_at`; that name exists only as a SQLite column elsewhere. So
the divisor was always the hardcoded 4000, ignoring the user's configured `period_sec`:
set it to 8 and every window warns spuriously, set it to 2 and it never warns when it
should. It reads `get_input_config().period_sec` now — which the same class already knew
how to read, 175 lines above. Verified live: 1331 ms against a 4 s period stays quiet;
a 1 ms period fires immediately.
**The demotion path tore the engine down while violating two invariants the same file
states explicitly.** `stop_to_unconfigured` takes `_engine_lock` ("teardown 必等当前推理
完成 → 杜绝 use-after-close") and awaits `close()` before nulling. The demotion I added
did neither: it could null the engine mid-`on_demand_perceive`, and it dropped the
persistent httpx pool on every demote→rebuild cycle. Same teardown as its neighbour now.
**`rule_hits` was the one untyped field in a contract whose stated purpose is third-party
reimplementation.** `list[dict]`, with three hard requirements the consumer imposes and
no document states: entries positionally aligned with the request's rules, `name`
echoed back (an empty one is *dropped*, not index-matched — the comment even contemplates
"一个只回命中项的第三方边车"), and a `reason` that appears nowhere in the sidecar or the
README. It is a `RuleHit` model now and the README's required-fields table covers it.
The typing earns its keep by normalising: a hit missing `reason` gets `""` rather than
`None` reaching the event text, and engine-internal keys are dropped.
Multi-camera, run against the real sidecar with synthetic frames — **I did not enable the
other two cameras in the user's home**: it was 00:40 and he had scoped the live test to
one room. Four devices concurrent: 4 captions, `device_rule_map` complete, no per-device
errors, rooms labelled correctly; on-demand fan-out labels each room; 8 concurrent
requests against a cap of 5 gave exactly 5 admitted and 3 refused with 503.
That run surfaced a capacity fact worth documenting: the sidecar serialises inference
under one lock, so a window costs roughly the sum of its cameras. At the measured
1.4–2.0 s per camera on real footage, two cameras already approach the 4 s default
period. README now says so, along with the ~32-rule-per-camera budget cliff (past it the
trailing verdicts truncate and fail-closed turns them into silent misses) and the fact
that rules without a camera list broadcast to every camera.
Also from the stranger's read: `perception_executes_device_actions` consulted config
only, which leaves a real window — switching back to cloud writes config first and the
soft-stop that follows is best-effort by design, so config could say `cloud` while the
local engine was still perceiving, and the guard would let device actions through. It
asks the live engine first now, with a test pinning the private attribute chain so a
rename fails loudly instead of silently reopening that window. `scene_ask`, `camera_note`,
`rules` and `query` gained the same size caps `video_b64` has — they sit in the same body,
parsed in the same pre-gate window, and only bounding the video meant the analysis and the
mitigation did not line up. And the JSDoc explaining the auth-rejected-is-not-green rule
was attached to `isReachable` instead of `healthLine`, which is the function that
implements it — two docblocks had stacked and only the second bound.
Mutation-verified: the period lookup, the live-engine preference, the `RuleHit` typing
and the body caps all fail the suite when reverted. Suites: backend 2904, sidecar 86,
web 274. Lint, prettier and every CI job run locally with their exact commands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…last three rounds broke This round wrote no new mechanism. It is a regression audit of rounds 9–11, run as four independent lenses (engine lifecycle, the two-service contract, whether tests were weakened while being "fixed", and which claims later commits made false) with every finding put through an adversarial refutation pass. 35 filed, 31 survived, **none of them blockers** — the first round where that is true. Most were caused by my own recent fixes. **The demotion added two rounds ago flapped forever.** Two defects compounded: - `_consecutive_failures` counted *any* all-device failure, including `encode_failed`, which never touches the network. A PyAV build without libx264 therefore produced 「边车连续 5 窗不可达」 and tore the engine down — pointing the user at the sidecar while the real cause sat in a separate WARNING, and tearing down something a teardown cannot fix. Only sidecar-attributable errors count now. - Worse, the demotion undid itself: `/health` is green, so the next tick's `try_reinit` rebuilt immediately, ran five more failing windows, demoted again — cycling forever, flickering the status ribbon and replacing the httpx pool every cycle. There is a 60 s rebuild cooldown now. Verified live by killing the sidecar: **one** demotion and **one** rebuild over two minutes, where before it would have cycled repeatedly. **Docstrings that state the opposite of the code beneath them.** `capabilities.py` still argued for reading config rather than the live engine — the exact behaviour I changed one commit earlier, for a reason the docstring then contradicted. The `RuleHit` contract text, added specifically to specify the contract for third-party sidecars, described the matching rule backwards and promised a positional-index fallback the consumer does not have (it drops unmatched hits, deliberately, because guessing by position is how 「厨房明火」 ends up carrying 「有人跌倒」's verdict). **`resolve_checkpoint` raised in the wrong place.** I added the fail-fast for a path-shaped checkpoint that does not exist, then called it from `lifespan` — so a typo'd path killed the process before it could serve `/health`, and the user saw "unreachable" with the real reason gone from anywhere they would look. It resolves inside `load()` now, where the existing handler turns it into `load_error`. Which mattered more than it looks, because: **The card still collapsed "load failed" into "still loading".** `load_error` was plumbed end to end two rounds ago and then never reached the surface a user actually watches, which kept advising them to wait for something that will never finish. Tests that were weakened while being fixed, now restored: the demotion tests asserted only the post-conditions and never the two teardown invariants that were the whole point of the commit (they spy on `close()` and assert it happens inside `_engine_lock` now); the codec-canvas tests could not distinguish the derivation from a constant; and the test added to "pin the private attribute chain so a rename fails loudly" never referenced the function whose chain it claimed to pin — it now parses that function's source and checks each hop against the real type. Also: README's `/health` field list, the `LOCAL_VISION_NUM_FRAMES` description (inert on the codec path since round 10), and the documented 413 (pydantic answers 422 first) were all stale; two comments annotated code that a later edit had moved away from them; and the two-clock comment in `realtime_perceive` described both timers backwards. Mutation-verified: failure attribution, the rebuild cooldown, checkpoint resolution in `load()`, and the attribute-chain test all fail the suite when reverted. Suites: backend 2906, sidecar 87, web 276. Lint, prettier and every CI job run locally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… of inheriting the cloud path's The parameters for this path were never researched — they were inherited from the cloud path and then tuned by trial. Reading the model's own code and paper shows why that was wrong in every direction. **The parameters are one constraint chain, not four independent knobs.** From `CodecConfig`: a *canvas* is a mosaic of 16×16 patches selected across `group_size=32` source frames, `images_per_group=4` of them per group, each sized by `max_pixels`. So `canvases = frames / 8`, `frames = window × fps`, and prompt tokens ≈ 223 × canvases (measured). Feeding fewer frames than a canvas budget needs silently downgrades it. What that exposed, in order of how much it cost us: - **`max_frames=32` was throwing away 60% of the frames we already had.** The camera delivers ~78 frames per 4s window (≈20fps); we capped at 32 and got 4 canvases where 10 were available. The cap is 256 now — "take what the window gives". - **Pre-downscaling was pure loss.** `video_short_edge` shrank the frames *before* the model saw them, which destroys the detail its own patch-selection exists to find; `max_pixels` (150000, the reference default ≈ the ViT's native 448²) is where the reduction is supposed to happen. Default is now no downscale. - **The window was inherited from the cloud path**, which uses 4s because it pays per frame. Nothing about that number relates to this model. It has its own now: 12s. - **`target_canvas` was being derived from the frame count** to silence a warning. It is the budget knob, so it is configured: 12. Choosing 12 needed both constraints, and I only had one of them until I measured on the real camera: 12s×20fps = 240 frames feeds up to 30 canvases, but 28 takes 13.8s per window (over budget), 16 takes 11.4s (95% of the window — too tight to absorb a GPU blip), 12 takes ~8s (66%). "Fills the budget" and "finishes inside the window" are separate constraints; a test now asserts the shipped defaults satisfy both. **Per-path separation is now a rule, not a patch.** The last commit split three parameters and left the window shared, which is the same mistake twice. The window follows the *active* backend (`active_window_size_sec`) — the two are mutually exclusive, so switching to local makes it 12s and switching back makes it 4s, with nothing for the user to remember. The model page now shows the parameter group belonging to whichever backend is selected, and the settings drawer's two knobs are labelled cloud-only. Quality on the real camera went from "两个人在用电脑" to "一名戴眼镜的女性坐在电脑前, 穿着深绿色上衣" at the same window occupancy we started with. Suites: backend 2910, sidecar 87, web 276; ruff and prettier clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he VLM
The local path had no identity at all: `identity: False` in the capability
declaration, and 0 of 10,463 windows in a day of real footage produced a
name — every person was "一名男子" / "一位女士". The cloud path answers this
inside its single omni call, so swapping the backend silently dropped the
whole capability.
The obvious port — replicate the cloud scheme against the local model — does
not work. Measured on 7 two-person scenes from real home footage, ground
truth checked by eye, with the cloud prompt shape reproduced faithfully
(member reference strips + full clip + bbox-addressed tracks + JSON out):
per-person correct, gallery order 小亮 first 8/14
per-person correct, gallery order 阳阳 first 0/14
degenerate baselines 4/14 and 10/14
pure local ReID, same scenes and gallery 14/14
29% across both orders, below the 50% a coin flip gets on a two-way choice,
and swapping the order of two names in the roster changes 4 of 7 answers.
The model is not blind — asked only about gender and clothing it is 5/5 —
but cross-image person matching is not something a 4B video model does.
Handing it a blank grey image in place of the query crop still returns a
name, which settles it.
So identity does not go through the model at all:
detect (det_4C.onnx) -> ReID embed -> cosine vs tier_a/*.npy -> roster
Those .npy files have been written by the registration flow all along;
library.py's own comment says they exist so that "未识别 track 跟已注册成员
快速比对" can use them later. This is that later. Nothing new is downloaded
and no model is added — human_body_reid_v2.onnx already ships and already
runs, just for tracking association rather than for naming.
The roster ("小亮[bbox=(357, 242, 467, 785)]", normalised to [0,1000]) rides
the perceive request as a new optional field and renders into the prompt in
the same shape the cloud path uses. That asks the model only to copy a given
name onto a given position, which it does reliably — 7/7 on the same scenes
that produced 8/28 when it had to recognise anyone.
Design points worth keeping:
- Identity is a bypass. Any failure yields an empty roster and the window's
caption and rule verdicts are produced as before. The guard sits at the
engine's call site as well as inside the resolver, because the resolver is
injected and the invariant belongs to the engine.
- A name is never emitted twice in one window; two boxes matching the same
member keep only the higher score. A roster claiming 小亮 is in two places
makes the model write self-contradicting descriptions.
- Below threshold produces no entry rather than a "stranger" entry, so the
model is never nudged into describing a person who is not there.
- 0.70 is measured, not picked: real people score 0.77–0.95 while people
*on the television* — which the detector reports as human at 0.94
confidence — score 0.44–0.67. That cut rejects 8/8 TV false positives with
no false rejections. Note this is the failure mode the cloud path's VLM
never caught: 0/8, even when told explicitly to watch for screens.
- The threshold assumes a current gallery. Against a five-week-old gallery
(different room, different clothes) the same scenes drop to 8/19, because
body ReID is largely clothing. The response is to re-register, not to
lower the threshold, so "people present but nobody recognised" logs the
best similarity seen — throttled, since it is a persistent condition.
Cost is 308ms median per window (detection on 3 sampled frames, CPU), inside
a 12s window, reported per camera in timing.
Capability declarations updated to match; they drive what the UI tells users
they lose by switching backends, and saying "no identity" when there is
identity is the same class of bug in the other direction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… that code cannot fix
A clean-context review found 12 defects; measurement collapses them into four
root causes, and settles the most important question: the wrong names this
shipped to production are **not** fixable in code.
## The one that cannot be fixed here
A 36-day-old gallery does not fail by declining to name people. It names them
confidently and wrongly. On the live study camera it called the man 阳阳 at
0.85 and the woman 小亮 at 0.81 — both far above the 0.70 cut. Measured on the
same 7 two-person scenes, that gallery scores 0/14 per person, and the error is
systematic: all 14 boxes are closer to the same member.
Every in-band rescue was tried and measured:
optimal 1-to-1 assignment still 0/14 — the optimal assignment is wrong
margin rule (top1 − top2) correct 0.000–0.108 vs wrong 0.012–0.141,
fully overlapping and pointing the wrong way
inter-member self-check gap −0.002 fresh vs +0.002 stale, no signal
same-window name collision 84% false positives at scale
So the gallery's freshness is a precondition, not a tuning parameter. This
commit makes that visible instead of pretending otherwise: the library's age is
computed and logged, a stale library warns with the measured consequence, and a
window with people but no matches reports the best similarity it saw. Age comes
from the registration images, never from the .npy — the startup embedding
backfill rewrites every .npy, which would reset the age of a stale library to
zero and destroy the only reliable signal.
## Assignment: correct, but for a different reason, and the order is load-bearing
Per-box argmax lets two people both claim one member; the loser was then deleted
outright by the "one name at most once" dedupe. Replacing it with a 1-to-1
assignment makes uniqueness structural, and it never regressed in measurement.
The order within it is not a detail. Threshold first, then assign:
threshold → assign 96/104
assign → threshold 40/104
Assigning first forces every member to be spent. With one real person and one
television false-positive in frame — this camera's every window — the TV box is
systematically closer to one member, so it takes that name and displaces the
real person onto the other one. The threshold then discards the TV pair and
leaves the human wearing the wrong name.
That also means assignment assumes everyone in frame is enrolled, and the
threshold is the only guard. Its margin is 0.03 (TV crops peak at 0.670). At
0.65 the same set drops to 47/104; at 0.60 all 104 TV boxes get a name.
## Cache: the fingerprint watched files this layer never reads
Change detection keyed on tier_a *image* files, while this layer reads only
meta.json and tier_a/*.npy. Two silent failures followed, both reproduced:
renaming a person never took effect for the life of the process — in a system
that had just called someone by the wrong name — and embeddings produced by
main.py's startup backfill were invisible forever, which under argmax handed
those people's boxes to *other* members.
The fingerprint now hashes the bytes it actually consumes. It deliberately does
not use (mtime, size): on this filesystem st_mtime_ns carries no sub-tick
resolution — two successive writes returned identical timestamps, even across
different files — and 小亮 → 亮亮 happens to be byte-identical in length. The
data is ~3KB per person; hashing it is both cheaper than the reload it guards
and exact, with no granularity question to get wrong later.
Two more behaviours fall out of the same fix. The empty-library short circuit no
longer doubles as a truthiness test, so a fresh install (identity is on by
default) stops rescanning and logging every window — roughly 7k lines/day/camera.
And the "never loaded" sentinel is None rather than (), because () is the
legitimate fingerprint of an empty library: colliding with it meant one transient
read error disabled reloading permanently, in a state indistinguishable from
"nobody is enrolled".
## Concurrency: one shared resolver, no discipline
resolve() runs from concurrent per-device coroutines via to_thread. Lazy init was
check-then-act: six concurrent windows built six Detectors, five discarded, 976MB
peak RSS on a box already hosting the vision sidecar. The gallery was read twice
across a possible reload, so a concurrent refresh between the match and the name
lookup raised KeyError and the fail-open swallowed the entire roster. Per-window
state (best score, miss-log throttle) lived on the instance, so cameras
overwrote each other's diagnostics and only the first of several cameras with a
stale library could ever report it.
Init is now locked, the gallery is snapshotted once per window, and per-window
state is local or keyed by device.
## Also
A member whose embeddings fail to load is now named in the log. Silently
excluding them is not a missing name — argmax gives their box to someone else.
resolve() takes source as an optional second parameter. The resolver is injected;
making it required would TypeError on any substitute implementing resolve(frames),
land in the fail-open, and turn identity off with an empty roster.
Not addressed here, and stated so it is not mistaken for solved: 0.70 was
calibrated against television false-positives, never against people. Unenrolled
humans score a median 0.818 top-1 against a healthy gallery — 33 of 33 stranger
boxes were given a member's name. This design assumes only enrolled members
appear. Fixing it needs face verification or a rejection criterion, not a
threshold tweak.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Xiaomi cameras composite the date and time into the frame before encoding, so
by the time any consumer sees the stream it is pixels. The model reads it, and
reads it wrong.
Measured on 60 visually-confirmed watermarked clips, greedy decoding:
no guard 44/60 (73.3%) of captions report a date or clock
year correct in only 16/44 — 2026 read as 2022, 2020, 2024
clocks inside the window wrong 25% — 13:50 read as 02:50
with guard 0/60 (95% upper bound 6.0%; 0/36 on a second batch)
The damage is not an extra wrong field. A wrong clock propagates into the scene
judgement: at a true 14:04 read as 04:04 the model wrote "the whole scene takes
place on a quiet morning"; at noon read as "4:07 am" it concluded "this suggests
early morning or late at night". In one clip it attributed the camera's own
ticking overlay to the television; in another, after reading "02:26", it
invented a child playing games in the corner that no frame contains.
## The guard is now per camera, and off unless the camera says otherwise
It cannot be unconditional. The sentence suppresses clocks that genuinely exist
in the room: with a digital clock on the far wall, 30 paired clips reported it
8/30 without the guard and 1/30 with it (McNemar p=0.039). Kitchen microwaves,
bedside alarms and wall clocks are ordinary furniture. Narrowing the wording
does not rescue it — scoping the sentence to the top-left corner recovered 1 of
12.
So miloco reads `time-watermark` per camera and tells the sidecar. Unreadable
means off: missing the guard only returns to a pre-existing risk, while adding
it wrongly deletes real information from the description. Third-party cameras,
models without the property, an unbound account and a network blip all land on
the safe side. The value is cached for the process — it is a setting the user
changes every few months, not state, and a MIoT round trip per window would add
a network dependency to always-on perception for nothing.
## It also had a hole
The sentence used to live inside `DEFAULT_SCENE_ASK`, and `build_prompt` starts
with `scene_ask or DEFAULT_SCENE_ASK`. On-demand queries pass the agent's own
question as `scene_ask`, so the whole default — guard included — was replaced.
The scheduled path was protected and the on-demand path ran bare, against the
same watermarked frames. It now appends to whatever question the caller gave.
## What this is not
It is a fallback, not a fix. Of the 73.3 → 0 points, roughly 63 come from
appending *any* "ignore X, don't mention Y" clause — a length-matched sentence
about lens distortion scores 10.0% — and only ~10 points are attributable to
naming the watermark (paired McNemar p=0.031). The effect rides substantially on
perturbing the decode path rather than on instruction following, which means any
future prompt change (rules, camera notes, a roster, a new checkpoint) can push
it back up silently. 0/60 is also not zero: the upper bound is 6.0%, and 300
clips would be needed to claim 1%.
The real fix is upstream: turn the watermark off, or have the device offer OSD
as a per-stream choice so a human reviewing recordings keeps the timestamp while
the perception path does not. Filed separately.
## Rejected: masking the pixels
Tempting and measured, then dropped. Masking is not neutral — drawing a box over
empty ceiling in 12 *unwatermarked* clips changed 0/12 captions relative to
control, where re-running the same file is 6/6 identical; 4 of 12 rewrote the
subject's gender ("a young man ... holding a game controller" became "a woman
... long dark hair ... holding a mouse"). The box also cannot be placed safely:
this one camera has produced both 848x480 and 904x512 frames, and a rectangle
calibrated on the first leaves the last digit of the seconds exposed on the
second — protection that appears enabled and silently is not. And the benefit is
zero: prompt tokens and visual patch counts are identical masked and unmasked.
Automatic detection of the overlay was built and measured too: 100% recall on
120 real watermarked clips with 0 false positives on 240 clean ones, but recall
holds only for an opaque dark plate over a bright background — bottom-right
darkened overlays, inverted text, translucent plates and plates-without-backing
all scored 0/8, and the failure is silent.
## One correction to the record
The premise that the watermark steals codec patch budget is false. It occupies
1.5625% of the patch grid and receives 0.527% of the budget — 0.34x its area
share, systematically under-sampled, because the selector follows encoding cost
and a black plate with thin strokes is cheap. The positive control confirms the
measurement: random noise in the same box takes 11.17%, 7.15x its share. The
real cost is 3.5% of vision tokens on the frames/whole-frame paths and 1.06% of
bitrate — not the main codec path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g a feature this branch adds Pre-publish pass. Nothing here changes behaviour; all of it is the branch contradicting itself in places a reviewer reads first. **CI's ruff job was red.** `ruff check . cli services/local-vision` — the exact command in ci.yml — reported three errors in the identity tests added two commits ago (one unsorted import block, two `for t in ts: t.start()` one-liners). Every round claimed "ruff clean" while running it from a different directory, which resolves a different file set. **Four places still said the local path has no identity.** The ReID commit updated the capability declaration the UI reads, so the card is correct — but `services/local-vision/README.md`'s known-limits list, the knowledge doc's backend comparison table, and the device-action rationale in both `capabilities.py` and the knowledge doc all still asserted 无身份识别. That is the same defect class this branch fixed repeatedly in the other direction: a document stating a property the code does not have. They now say what is actually true, including the part that is not flattering: identity exists but does not come from the model, the 0.70 threshold was calibrated against television false positives and never against people (33 of 33 unenrolled stranger boxes were given a member's name), and a stale gallery fails by naming confidently and wrongly rather than by declining. The device-action rationale needed rewording rather than deletion: "no audio corroboration and no identity" was one of the two reasons a vision-only model should not close a gas valve. Identity now exists, so the sentence stands on its measured failure modes instead. **Two settings docstrings contradicted their own defaults.** `container_fps` defaults to 20 while its description explained why 8 was the right compromise, and `codec_target_canvas` defaults to 12 while its description ended by instructing the reader to write 28. Both are leftovers from the operating-point commit, and both are the kind of thing that makes a reader distrust the numbers around them. Suites unchanged and green: backend 2956, sidecar 100. `ruff check` over all three trees passes; prettier and markdownlint pass over knowledge/ and README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
👋 感谢提交 PR @LeonJoeeee！维护者会尽快 review。 提交前请确认：
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| 刻意宽容:manager 还没建好、感知服务还没起来都是正常状态,不该让这条查询抛错。 | ||
| """ | ||
| try: | ||
| from miloco.manager import get_manager |
|
|
||
| if get_settings().perception.engine_backend != "local": | ||
| return True | ||
| from miloco.perception.local_vision.engine import LocalVisionEngine |
Comment on lines
+22
to
+26
| from miloco.perception.rule_scope import ( | ||
| camera_prompt_map, | ||
| physical_did, | ||
| rules_for_device, | ||
| ) |
Comment on lines
+55
to
+59
| from miloco.perception.rule_scope import ( | ||
| camera_prompt_map, | ||
| physical_did, | ||
| rules_for_device, | ||
| ) |
| 读 KV(进程内缓存)失败时返回空 dict —— **fail-open**:瞬时故障只是少一段 | ||
| 机位指导,不该阻断感知(与语音白名单的 fail-closed 相反,因本项只增益、不涉隐私)。 | ||
| """ | ||
| from miloco.manager import get_manager |
| from miloco.config import get_settings | ||
| from miloco.database.perception_repo import PerceptionLogRepo | ||
| from miloco.perception import omni_probe_registry | ||
| from miloco.perception.capabilities import active_window_size_sec |
| # 实测 +0.36s)拖进规则引擎,而上游的 rule/runner.py 一个感知顶层导入都没有 | ||
| # (它唯一的感知依赖 event_text_builder 同样是函数内导入)。这条依赖箭头 | ||
| # 本来就该是单向的:perception 用 rule,rule 不用 perception。 | ||
| from miloco.perception.capabilities import perception_executes_device_actions |
Contributor
Author
关于
|
[PR #488]: feat(perception+web): 本地视觉感知通路 —— 与云端 API 并列的第二条路径作者: LeonJoeeee 修改方案给感知系统加第二条通路:窗口帧在本地编码成 H.264,交给独立 GPU 边车服务做推理,拿回中文场景描述 + 逐条规则判定。整条通路不需要任何模型厂商 API Key,画面不出本地,token 成本为零。默认仍是云端通路 —— 不主动切换的用户行为完全不变。
历史 review 修复验证第一轮(ci-bot,作者在
第二轮(ci-bot,作者在
第三轮(ci-bot,作者在
第四轮(ci-bot,作者在
第七轮(ci-bot,作者在
问题🔵 建议(可选优化)
try:
from miloco.manager import get_manager
phys = physical_did(did)
data = await get_manager().miot_service.get_device_status(phys, [_OSD_IID])
async def on_demand_perceive(
self, batch: BatchedSnapshot, query: str
) -> OnDemandPerceptionResult | None:
"""主动查询:把问题直接当场景提问送给本地模型。"""
if batch.empty:
return None
if time.monotonic() < self._backoff_until:
return None # 边车退避中,直接告知无结论
snaps = [s for s in batch.snapshots if s.has_video]
if not snaps:
return None
def _client(self) -> httpx.AsyncClient:
loop = asyncio.get_running_loop()
if (
self._async_client is None
or self._async_client_loop is not loop
or self._async_client.is_closed
):
old = self._async_client
if old is not None and not old.is_closed:
try:
loop.create_task(old.aclose())
except RuntimeError:
pass
self._async_client = httpx.AsyncClient(
timeout=self.timeout,
limits=httpx.Limits(max_keepalive_connections=4, max_connections=8),
)
self._async_client_loop = loop
return self._async_client
结论LGTM — 七轮 review 之后工程质量极高:59 文件 10k+ 行(49% 是测试),所有轮次的 🟡 问题均已修复并验证通过(包括本轮验证的第七轮 由 review-pr skill v0.12.0 生成 |
…ese strings, attach the on-demand clip
Three of the four findings from the automated review are real. Fixed here while
`guard` is still unapproved, so nothing has to be re-approved afterwards.
**Two hardcoded Chinese strings reached the UI, in a component whose own comment
forbids exactly that.** `PB_CODE_KEY`'s docstring says "backend message is
hardcoded Chinese; injecting it directly pollutes the English UI", and the
switch-failure path was converted to code lookup for that reason. Two siblings
were missed:
- `blocking_static_rules.join("、")` — an ideographic comma rendered into English
UI. Now `t("perceptionBackend.listSeparator")`.
- `cloud_hint` was a Chinese sentence rendered verbatim by `{state.cloud_hint}`.
It now carries `{code, message, detail?}` and the frontend looks the code up,
matching the pattern already established beside it. `message` stays for logs
and for clients that don't know the code; `detail` carries the validator's
specific missing-file text, which only the backend can produce. An unknown
code falls back to `message` rather than rendering nothing — the contract is
open to newer backends. Ready now yields `null`, not `""`: null and empty
object render differently.
**On-demand queries did not attach the clip they looked at.** The cloud path
does attach it (omni pushes from prompt_builder), so this was a silent
behavioural difference, not a style question — and on-demand is the case that
most needs review material ("was someone at the door just now?"). When the
answer is in doubt there was nothing to check it against, with no error and no
mention in the capability declaration.
The fourth finding — no explicit guard for an unknown backend in `_init_engine`
— is already covered by the pydantic `Literal` at the config layer. Left alone.
Also: ruff was actually failing. Previous commits claimed clean, but were run
from a directory that resolved a different file set than CI's
`ruff check . cli services/local-vision`. Fixed, and the same three files I
touched here were prettier-clean before my edit and are again now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
自动评审的四条我逐条核过代码,三条属实,已修( 🟡
|
…wrong way, plus six smaller ones
The automated review returned LGTM with seven non-blocking findings. All seven
are real; all seven are fixed. One of them was not cosmetic.
## The safety fallback pointed at "allow"
`perception_executes_device_actions()` guards whether the perception layer may
drive devices directly. Its outermost `except` returned `True` — execute — on
the reasoning that an unreadable config should fall back to existing cloud
behaviour. That reasoning is real but the direction is still wrong:
- The costs are asymmetric. Refusing wrongly means one rule does not fire, and
it is recorded as a RULE_TRIGGER_FAILURE — visible, traceable. Allowing
wrongly means a vision-only model closes a gas valve or unlocks a door. The
fallback of a safety invariant can only lean toward refusal.
- The premise mostly does not hold. `get_settings()` is a cached global; if it
raises, the process is broken deeply enough that perception is not running
either. "Config unreadable but automations working normally" is not a state
that exists.
The window is narrow — it needs the engine attribute chain *and* settings to
fail together. But narrow is not a reason to point it the wrong way, and the
attribute chain is five private attributes deep, so any refactor that renames
one link silently disables the first branch. The test that pinned the old
direction is rewritten to pin the new one, with the original concern recorded
rather than deleted.
## A file descriptor leak on a resident path
`sample_frames` released the VideoCapture before each of its two known raises,
but `cv2.cvtColor` and `Image.fromarray` also raise (wrong channel count, odd
dimensions) and those paths leaked. This runs once per window forever; the
symptom would be "after a few hours nothing can be opened", with no visible
connection to this function. Now try/finally.
## Reading the identity library created directories
`refresh_gallery()` built an `IdentityLibrary`, whose `__init__` calls
`_ensure_dirs()`. A mistyped library path therefore got an empty skeleton
quietly created at the wrong place, destroying the most direct diagnostic —
"the directory does not exist" — and leaving the user believing the path was
right. A read must not have side effects; it now walks the directory itself.
## And four smaller ones
- The switch-failure toast appended the backend's Chinese sentence for
`blocking_rules`, in the same component whose comment forbids exactly that.
The payload already carries a `rules` array — pure user data, no prose — so
the frontend now joins that after the localised sentence. `ApiError` grew a
`data` field to carry structured detail beyond `code`/`message`; some errors
need data to be explained, and reaching for `message` is what leaks Chinese.
- "Restart perception" did not clear `_local_rebuild_not_before`, so pressing it
within 60s of a demotion was a silent no-op — the throttle exists for
automatic retry, and should not restrain the user's hand. Same reasoning as
the probe-cache invalidation right above it.
- `real.test.ts` still passed `cloud_hint: ""` where the contract is
`{code,…} | null`. It passed because `""` is falsy, which made the type
contract test worthless.
- `unitCanvas` was empty in English, rendering a bare `()` in the UI.
Left alone: no explicit guard for an unknown backend in `_init_engine` — the
pydantic `Literal` already blocks it at the config layer.
backend 2961 / sidecar 100 / web 279 green; ruff via CI's own invocation, tsc,
and prettier on every touched file all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
第二轮 review 的七条全部属实,已全部修(
|
第三轮 review 的两条 🟡。 1. 重建冷却在配置变更后没被清掉 重建冷却是**上一份配置**挣来的证据(那份配置的引擎推理一直失败),用户把 地址/凭证改对之后它对新配置没有任何依据,却还压着自动重建最多 60 秒。表现 是"我明明改对了,界面还写着持续不可用",而且没有任何提示说明还要再等。 注意它只能从探活这条路解开:_init_local_engine 里那份一模一样的配置检查位于 冷却闸门的**下游**,冷却期内那个分支直接 return,压根走不到。两处重复的配置 检查一并收进 _drop_stale_local_conclusions(),免得它们将来各自漂移。 2. 就绪日志无条件写"无身份识别" 认人是可选项,四条路都会退回 None(配置关着/库是空的/模型缺失/构造抛异常), 写死这半句是双向撒谎:认人在跑时谎报缺失,让人去查一个不存在的故障;真的没 建起来时这行字又和平时一模一样,真正的缺失反而看不出来。改成跟着实际建出来 的对象走,并带上库里认得几个人。 另清理两个死 i18n 键(inputWhy / shortEdgeHint,已被 ...Local/...Cloud 取代)。 验证:两条新回归测试在撤掉对应修复后均失败、恢复后通过;bringup 28 条全绿; sidecar 100;web 279 + tsc;ruff 按 CI 原样调用(cd backend)全过; prettier 对改动文件干净。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
第三轮 review 的 2 条 🟡 已修( 🟡 1 — 配置变更后重建冷却未清已修。补充一点复现时容易被绕过的机制:这个冷却只能从探活那条路解开。
顺手把两处重复的配置检查收进了 回归测试 🟡 2 — 就绪日志无条件写"无身份识别"已修,改成跟着实际建出来的对象走,并带上库里认得几个人。 认人有四条路会退回 两条测试:正向( 🔵 3 — 死 i18n 键已删 🔵 4 —
|
…used import) CodeQL 在 local_vision/engine.py:54 报 unused import。属实:本文件有 from __future__ import annotations,注解全是惰性字符串,而 LocalIdentityResolver 只出现在 identity 参数的注解里,运行时一次都用不到。ruff 不报是因为它把字符串 注解里的引用算作使用——两边看的层面不同,各自都没错。 搬进 if TYPE_CHECKING 两边都满足:运行时不再导入这个名字,类型检查仍解析得到。 render_roster 保持原样,它在 347 行是真的运行时调用。 顺带澄清:identity.py 不反向导入 engine,所以这里不存在真正的循环依赖,只是 一个用不到的名字。 验证:运行时确认模块上已无 LocalIdentityResolver、render_roster 仍在、identity 参数注解仍可读;ruff 按 CI 原样调用全过;local_vision 相关 136 条全绿; 后端全量 2847 passed / 117 failed,与改动前基线逐条一致(117 为本机 401 产物)。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
补一条 CodeQL 的处理说明( CodeQL 在本 PR 上挂了 9 条,两个 Analyze 检查本身是绿的(都是提示级别),但为免评审时逐条去看,先说明清楚。 1 条 unused import —— 已修
已搬进 顺带澄清一点: 8 条 cyclic import —— 未改,说明理由其中 4 条落在 main 上早已存在的文件( 另 3 条落在本 PR 新增的 要彻底消掉需要动 验证
|
同步 main(落后 53 个提交)。此前 PR 处于 CONFLICTING 状态,导致 pull_request
类型的工作流(CI / CodeQL / Docs / OpenGrep)整批不触发——它们跑的是 GitHub 预合成
的合并结果,合不出来就直接跳过,表现为「检查数量变少」而不是「检查变红」。
冲突 5 处,均为双方在同一位置各自追加,一律两边保留:
- web/src/api/real.ts:我的 realGet/SetPerceptionBackend 与 main 的
realUpdateRuleQuery 都追加在文件末尾,共用结尾的 }
- web/tests/real.test.ts:import 列表合并;两个 describe 块拼接
- web/src/i18n/locales/{zh,en}/settings.json:cloudOnlyHint 与 minUrgency* 同层追加
另有 6 个文件双方都改过但自动合并成功(client.py / settings.py / admin/router.py /
types.ts / SettingsDrawer.tsx / api/index.ts),已逐项验证语义未被破坏:
- client.py 里我的 _build_local_identity / _drop_stale_local_conclusions 与
main 的 _filter_suggestions_by_min_urgency 共存
- 本地通路不产建议,main 的紧急度过滤器对空列表正常返回
- 安全不变量仍成立:local 后端下 perception_executes_device_actions() 为 False
格式化说明:real.ts / real.test.ts 的 prettier 告警全部来自 main 既有代码
(逐行核对,30 处与 1 处均不在冲突解决区域内),且 CI 无格式检查步骤,故不做重排,
以免制造与本 PR 无关的大面积 diff。
验证:后端 3029 passed(main 新增 ~182 条全绿),117 failed 与合并前逐文件一致
(本机 401 产物);前端 296 passed + tsc;边车 100;ruff 按 CI 原样调用全过。
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
上一次(fa8a4ea)把 LocalIdentityResolver 搬进 TYPE_CHECKING,没有解决问题—— CodeQL 在合并结果上仍然报同一条 unused import,只是行号从 54 变成 60,而 60 正是 那个 TYPE_CHECKING 块里面。根因先前定位错了。 真正的原因是**注解带着引号**:写成 "LocalIdentityResolver | None" 时,那个名字只 存在于一个字符串字面量里,CodeQL 不解析字符串,于是无论导入放在模块顶层还是 TYPE_CHECKING 块里,它看到的引用数都是零。换句话说,搬导入这个动作从一开始就 救不了它。 本文件有 from __future__ import annotations,那对引号本来就是多余的。去掉之后 名字出现在注解的语法树里,静态分析看得见;而 __future__ 保证运行时依然不会去 解析它——两边仍然各取所需。 风险已核:全仓无 get_type_hints 调用,LocalVisionEngine 也不是 pydantic 模型, 没有任何路径会在运行时解析这个注解。 验证:运行时确认注解仍是未求值的字符串、模块上没有该名字、引擎可正常构造; ruff 按 CI 原样调用全过;local_vision 相关 136 条全绿;后端全量 3029 passed, 117 failed 与合并前逐条一致(本机 401 产物)。CodeQL 本机无 CLI,这一条要等 下一轮 CodeQL 跑完才能确认。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
更正上一条评论的一处错误,外加同步 main。 更正:CodeQL 的 unused import,上次没修成上一条我写「已搬进 真正的原因是注解带着引号。写成 已在 风险已核:全仓无 说明确认边界:我读不到本仓库的 code-scanning alerts(非管理员,API 返回 403),所以无法直接核对该告警是否已关闭。可观察到的是 —— 同步 main(
|
再次同步 main(又落后 85 个提交)。PR 已第二次进入 CONFLICTING —— 该状态会让 pull_request 类型的工作流(CI / CodeQL / Docs / OpenGrep)整批静默不触发,表现为 「检查数量变少」而非「检查变红」,极易被当成还没跑完。 冲突 2 处,均为双方在同一位置各自追加,两边保留: - web/src/lib/types.ts:本 PR 的 PerceptionBackendState 与 main 的 UpgradeCheck / UpgradeStatus 相邻,共用结尾的 } - web/tests/real.test.ts:import 列表合并(realGet/SetPerceptionBackend 与 realEventRefUrl / realEventCropMeta) 本轮 main 动了感知核心(自适应裁剪 XiaoMi#469、说话人检测、追踪服务、相机适配器 XiaoMi#434、 线程削减 XiaoMi#475、web 升级 XiaoMi#424),与本 PR 同区域,故逐项验证语义: - 安全不变量仍成立:local 后端下 perception_executes_device_actions() 为 False - 前两轮 review 的两处 🟡 修复实体仍在(_drop_stale_local_conclusions 里清重建 冷却那行、就绪日志跟随实际认人状态),对应 3 条回归测试全绿 - 本地通路不产 suggestion,main 的紧急度过滤器对空列表正常返回 - camera_adapter 等 main 改动模块可正常导入 验证:后端 3307 passed / 122 failed。122 = 既有 117 + main 新增 5 条 (node_monitor TestProcSeriesEndpoint 那组);逐条核对失败原因,16 条全部是 AuthenticationException: Invalid or missing service token,即本机 401 环境产物, 无一条逻辑失败。前端 345 passed + tsc;边车 100;ruff 按 CI 原样调用全过。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
第七轮 review 的 1 条 🟡 + 1 条新 🔵。 ## 🟡 _library_age_days 只 glob .png,历史 jpg 库过期告警静默失效 失效方向恰好是反的。仓库明确保留了对历史 .jpg/.jpeg 登记图的读取(见 engine/identity/library.py 的目录图注「新写入为 .png 无损;历史库 .jpg/.jpeg 仍可读」、 _backfill 里的 body_*.jpg 扫描,以及 person/router.py 的文件名白名单),而这里只认 .png —— jpg 老库一张图都匹配不上,函数返回 None,调用处那句 `is not None` 落空, 过期告警**永不触发**;日志只写一句「登记距今 未知」,没有任何异常样子。 而被压住的正是这套方案唯一被证实的失效模式:旧库高置信度**认错人**(实测 36 天的 库逐人正确率 0/14,代码侧无解)。越老的库越可能是 jpg 时代留下的,也就越需要这条 告警,偏偏就是它们拿不到。 改成 glob body_* 后按后缀白名单过滤。用白名单而不是"排除已知几种",是为了保住本 函数开头那条约束:.npy 会被 backfill 重写,按它计龄会让 36 天的库显示成"全新"。 ## 🔵 write_temp_video 写失败时泄漏临时文件 mkstemp 一返回文件就已经在盘上了,写入失败时路径还没交给调用方,调用方 finally 里的清理拿不到它,只能等 sweep 一小时后兜底。最现实的触发是磁盘满 —— 而磁盘满会 让每一个窗口都走这条路,一小时攒几百个残留,还都堆在那块已经满了的盘上。 捕 BaseException:KeyboardInterrupt / CancelledError 同样会留下空壳。 验证:两条新测试均已证伪 —— 撤掉对应修复后分别报 「jpg 老库被当成「库龄未知」」与「写失败后把空壳留在磁盘上了」,恢复后通过。 identity 40 条全绿;边车 101 passed;后端全量 3308 passed / 122 failed(122 与本轮 合并后基线逐文件一致,全部是本机 401 环境产物);前端 345 passed + tsc; ruff 按 CI 原样调用全过。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
第七轮的 1 条 🟡 + 1 条新 🔵 已修( 🟡
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
给感知加第二条通路:窗口帧在本地编码成 H.264,交给一个 GPU 边车服务,拿回中文场景描述 + 逐条规则判定。整条通路不需要任何模型厂商 API Key,画面不出本地,token 成本为零。
默认仍是云端通路 —— 不主动切换的用户,行为与今天完全一致。
怎么读这个 PR
59 个文件、9.5k 行,其中 4704 行(49%)是测试。不建议顺着 diff 读。建议按这个顺序看:
backend/miloco/src/miloco/perception/local_vision/engine.py的模块 docstring(开头 ~30 行)backend/miloco/src/miloco/perception/client.py,_init_engine(L289)/_init_local_engine(L357)perception.engine_backend二选一。这里也是"默认不变"这个承诺的落点services/local-vision/README.md的「接口」与「已知限制」两节backend/miloco/src/miloco/perception/local_vision/identity.py(556 行)backend/miloco/src/miloco/perception/capabilities.py+rule/runner.py(L1234 起)backend/miloco/src/miloco/config/settings.py的LocalVisionSettings其余按需:边车实现在
services/local-vision/local_vision/(app.py= HTTP 面,engine.py= 推理,prompts.py= 提示词构建与响应解析);前端在web/src/components/PerceptionBackendCard.tsx+web/src/lib/perceptionBackend.ts(决策逻辑抽成纯模块,便于在 node 环境里测)。如果只想审一处:
identity.py和它的 611 行测试。架构
落在已有的插拔缝上,不动流水线骨架。
perception/engine_base.py早就有BasePerceptionEngine抽象基类(云端PerceptionEngine就是它的实现之一)。本 PR 加一个同接口的LocalVisionEngine,由PerceptionEngineProxy按perception.engine_backend二选一。GPU 推理放独立边车(
services/local-vision/),在 uv workspace 之外。 miloco 的目标硬件是 Mac mini / 树莓派这类 CPU-only 机器,主包绝不能为一个可选功能背上 torch/CUDA。边车可以跑在另一台机器上(那种情况下强制要求访问凭证 —— 未配 token 时服务会拒绝绑定非环回地址,否则等于把家里画面的推理接口无鉴权地挂在局域网上)。不接管模型进程的生命周期:不下载权重、不拉起、不重启。这条边界来自 #144 的教训 —— 1.x 由 miloco 管理本地模型容器,故障面扩散到显卡直通/驱动/容器,最终无人能支持。
契约与模型无关:送视频段 + 提问 + 规则(+ 可选名册)→ 返回描述 + 逐条判定。README 里把每个必填字段和"漏掉它会怎样"逐条列了出来 —— 例如省掉
auth_required/auth_ok,miloco 的凭证检查就会 fail-open。前端:「模型」页新增后端选择卡片,切换即时生效。三种状态(云端生效 / 本地生效 / 边车不可达)都逐一渲染核对过。本地通路生效时,模型表会明说这些云端模型没有在服务视频感知 —— 否则页面上会同时挂着两个「生效中」徽章,用户没法判断到底谁在干活、会不会还在计费。
运行点位是怎么定的
这组参数最初是从云端通路继承来的,然后靠试错微调 —— 每一个方向都是错的。读了模型自己的代码和论文之后才明白为什么:
它们是一条约束链,不是四个独立旋钮。 画布是从
group_size=32帧里选 16×16 patch 拼成的马赛克,每组出images_per_group=4张。于是画布数 = 帧数 / 8,帧数 = 窗口 × 帧率,而prompt token ≈ 223 × 画布数(实测)。喂的帧不够,画布预算就静默降级。window_sizecontainer_fpsmax_framesvideo_short_edgemax_pixels(150000 ≈ ViT 原生 448²)在模型内部做codec_target_canvas选 12 需要两个约束同时成立,而在真机上测之前我只有其中一个:12s×20fps=240 帧最多能喂饱 30 张画布,但实测 28 张要 13.8s(超出窗口)、16 张要 11.4s(占窗口 95%,太贴边,吸收不了一次 GPU 抖动)、12 张约 8s(66%)。"喂得饱"和"跑得完"是两个约束,现在有测试钉住出厂默认值必须同时满足两者。
效果:真机上同一个画面的描述从「两个人在用电脑」变成「一名戴眼镜的女性坐在电脑前,穿着深绿色上衣」,而窗口占用率与起点相当。
分路是一条规则,不是一次补丁。 窗口跟着当前生效的那条通路走(
active_window_size_sec):切到本地自动变 12s,切回云端自动变回 4s,用户不需要记得改回来。模型页只显示当前后端那一组参数。认人:不走视觉大模型
切过去之后,一整天真实素材里 10463 个窗口,0 个产出了名字 —— 每个人都是「一名男子」「一位女士」。云端是在它那一次 omni 调用里顺带把这件事做掉的,所以换后端等于静默地丢掉了整个能力。
最先试的是把云端那套原样搬过来(成员参考图 + 完整片段 + bbox 指人 + JSON 输出),在 7 个真实双人场景上实测:
两种顺序合起来 29%,低于二选一瞎猜;调换名单里两个人的先后,7 个场景有 4 个答案会变。把待识别的那张图换成一张纯灰图,它照样报出一个人名 —— 这一条基本上就定案了。模型本身不瞎(只问性别和衣着是 5/5),但跨图细粒度同人比对不是这个量级的视频模型的能力。
所以认人整条不经过模型:
名册(
"小亮[bbox=(357, 242, 467, 785)]",归一化到 [0,1000])随请求送给边车,渲染进提示词里 —— 模型只需要把给定的名字贴到给定的位置上,这件事它做得很稳(同批场景 7/7,而让它自己认是 8/28)。不新增依赖、不下载权重:
human_body_reid_v2.onnx仓库里本来就有、本来就在跑(用于 DeepSORT 跨帧关联),tier_a/*.npy也是登记流程一直在写的 ——library.py的注释写着它们存下来是为了「后续做『未识别 track 跟已注册成员快速比对』」。这个 PR 就是把那个「后续」补上。成本:每窗中位 308ms(在 3 帧上跑检测,CPU),在 12s 的窗口里,按相机记进 timing。
几条设计要点:
阈值 → 指派96/104,指派 → 阈值40/104。先指派会逼着每个成员都被用掉:这台相机每个窗口都有一个真人 + 一个电视误检,电视框系统性地更靠近某个成员,于是它拿走那个名字,把真人挤到另一个名字上;阈值随后丢掉电视那一对,留下戴着错名字的真人。今天的线上验证:两人同框,12/12 全对、0 次叫反。
时间水印兜底(按相机开关)
米家相机把日期时间合成进画面再编码,所以到了任何消费方手里它就是像素。模型会去读,而且读错:60 段实测,44/60(73.3%) 的描述报出日期或时钟,其中年份只对 16/44。伤害不是多一个错字段 —— 真值 14:04 被读成 04:04 之后,描述写道「整个场景发生在一个安静的早晨」。
挂一句忽略提示能压到 0/60。但不能无条件挂:这句话会压掉屋里真实存在的钟(30 段配对,一个墙上的数字钟从 8/30 降到 1/30,McNemar p=0.039)。所以 miloco 按相机读
time-watermark(prop.2.5)再决定挂不挂,读不到就不挂 —— 漏挂只是退回本来就存在的风险,误挂却会主动删掉画面里的真实信息。第三方相机、没有该属性的机型、未绑定的账号、一次网络抖动,全部落在安全那一侧。同时修掉了一个洞:这句话原先住在
DEFAULT_SCENE_ASK里,而build_prompt是scene_ask or DEFAULT_SCENE_ASK—— 主动查询会把 agent 自己的问题作为scene_ask传进来,于是整个默认值连同这句保护一起被替换掉了。定时通路有保护、主动查询裸奔,对着同一批带水印的画面。现在它是追加到调用方给的问题后面。根治在设备侧,已另开 issue(#487);那个 issue 里也纠正了一个我自己先前信过的说法 —— 水印并不浪费大量 token(它在主 codec 通路上被系统性欠采样:占 patch 网格 1.5625%,只拿到 0.527% 预算)。
已知限制(请当作评审材料的一部分读)
认人
0.70 这个阈值是拿电视误检标定的,从来没拿人标定过。 实测(同日库)真人 0.77–0.95,而电视屏幕里的人(检测器会以 0.94 置信度认为是真人)只有 0.44–0.67,0.70 挡掉 8/8 且 0 误拒 —— 余量只有 0.03(电视框峰值 0.670);调到 0.65 同一批降到 47/104,调到 0.60 则 104 个电视框全部拿到名字。但另一个方向从未被标定:未登记的人对一个健康库 top1 中位 0.818,实测 33 个陌生人框 33 个全被安上了成员名。
也就是说:本方案假定画面里只出现已登记成员。 要真正解决需要人脸校验或专门的拒识判据,不是调阈值能解决的。
身份库过期不是「认不出」,是「高置信度认错」。 一个 36 天前的库在真机上把男的叫成阳阳(0.85)、女的叫成小亮(0.81),都远在 0.70 之上;同样 7 个双人场景逐人 0/14,而且错得系统——14 个框全都更靠近同一个成员。四种代码侧补救全部实测否掉:
所以库的新鲜度是前提条件,不是可调参数。本 PR 能做的只是让它可见:算库龄并记日志、超期告警(带上实测后果)、"画面里有人却一个都没匹配上"时报出看到的最高相似度。库龄取自登记图片而不是
.npy—— 启动时的特征回填会重写每一个.npy,那会把一个陈旧库的年龄清零,毁掉唯一可靠的信号。身份库的保鲜机制在本通路上是缺的:tier_c 入库要靠云端模型做同人校验,本地没有对应物。目前只能靠告警提醒用户重新登记。
一条正面发现,顺带记下:同样 5 张样本,只是分一半给坐姿,认对率从 4/11 变成 11/11(与电视的间隙 +0.090 → +0.145);样本翻倍到 10 张只多 0.03。姿态覆盖决定成败,数量只是锦上添花。
时间水印
通路本身
truncated/unparsed_rules会亮,日志有 WARNING)。注意没有指定相机的规则会广播到每一台,所以这个数是按相机算的。mamba_ssm需要 CUDA ≥12.8 工具链才能产出 sm_120 kernel)。服务会自动熄灯门控并在/health的gate_error里说明,描述与规则判定不受影响。另外门控是在体育解说数据上训练的,家庭场景属分布外 —— 所以event_gate_threshold默认 0(只观测、不据此丢窗口)。codec_id丢掉了,所以在那一层连格式都判断不出来。它需要先把codec_id打通,是另一个改动。这不影响已测得的 codec 收益:那些数字本来就是在重编码后的流上测的。本 PR 刻意不做的
云端常驻 + 本地按需的混合模式。 现在是二选一开关。理由不是做不动,而是:本地通路还没有稳到可以当常驻层 —— 上面那一整节已知限制就是证据,尤其是认人对身份库新鲜度的硬依赖。在还在动的地基上加一层复杂度,不划算。 等这条通路自己站稳了,再谈两条一起跑。
同样刻意不做的还有:运行时把 STATIC 规则改写成 DYNAMIC(会丢掉
cooldown_minutes/idempotent这个 schema 里唯一的限流手段,以及行动台账里source=rule的归属)。带直连设备动作的规则会拒绝切换到本通路并列出它们,由用户决定。验证
ruff check覆盖 backend / cli / 边车三处通过,prettier 与 markdownlint 覆盖knowledge/**与README.md通过,tsc --noEmit通过。本 PR 新增的测试占全部改动的 49%。load()里的 checkpoint 解析、纯读 GET、持续失败降级、loop-keyed 连接池、周期读取、活引擎优先、RuleHit类型化、请求体上限等)。有一轮的变异结果因为 harness 用shutil.move还原源码、把 mtime 改回过去导致 Python 用了旧字节码而作废并重做。/health是绿的,于是下一 tick 立刻重建、再跑 5 个失败窗口、再降级)。多相机路径用合成帧对着真实边车验证:4 台并发拿到 4 条描述、device_rule_map完整、房间标注正确;8 个并发请求打 5 的上限,恰好 5 个进、3 个 503。一点披露
Mage-VL 的模型卡写明 "released for research purposes only and are not intended for product or service deployment"(许可证本身是 Apache-2.0)。本 PR 的主体是通路与契约,Mage-VL 只作为参考实现与实测对象;即使项目对该模型本身有顾虑,通路依然成立 —— 换任何满足契约的本地视觉服务都能工作。