feat: add sensory framework core - #114
Nothing1596 wants to merge 9 commits into
Conversation
|
已按复审建议把 #99 拆成 stacked PR。 Stack 顺序：
说明：当前账号没有向 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fcf2d98858
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| path = Path(ref).expanduser() | ||
| if path.is_file(): | ||
| mime = mimetypes.guess_type(path.name)[0] or "image/png" | ||
| return f"data:{mime};base64,{base64.b64encode(path.read_bytes()).decode('ascii')}" |
There was a problem hiding this comment.
When a remote API or LAN vision sensory provider is enabled, the model controls observe_sensory arguments and can pass a local media_ref/metadata path. This branch treats any readable file path as image input and base64-embeds its bytes into the provider request, so a tool call can exfiltrate arbitrary local files, not just temporary screenshots, before the provider has a chance to reject the payload. Restrict this to trusted temp capture paths and validate that the file is an expected image, or require explicit confirmation for local paths.
Useful? React with 👍 / 👎.
| if mode != SensoryProviderMode.OFF: | ||
| validation_error = _validate_sensory_state(source, state) |
There was a problem hiding this comment.
When the user unchecks the global “启用增强感知工具” box, this loop still validates every source whose per-source mode is not off before building the returned SensorySettings(enabled=...). If a user has a half-filled source, such as Local selected with an empty model, choosing to disable enhanced sensing still gets blocked by _validate_sensory_state, so they cannot save the disabled state without first fixing or turning off each individual source.
Useful? React with 👍 / 👎.
| source_settings = settings.sources[observation.source] | ||
| if not source_settings.context_enabled: |
There was a problem hiding this comment.
When a user turns an individual sensory source off after observations have already been recorded, those stale records can still be injected because this filter only checks context_enabled and confidence. SensorySourceSettings defaults context_enabled to true even for mode=off, so disabling speech/vision/sound stops new provider calls but does not stop old observations from appearing in prompts; check the source mode/enabled flag here before admitting the observation.
Useful? React with 👍 / 👎.
| sources[source] = SensorySourceSettings( | ||
| mode=mode, | ||
| provider_id=provider_id, | ||
| confidence_threshold=float(state.get("confidence_threshold") or 0.5), |
There was a problem hiding this comment.
The settings UI allows the minimum confidence to be set to 0.00, and the settings model accepts it, but this or 0.5 coercion turns that explicit zero back into the default 0.5 when saving. Users who intentionally want to include all observations will see the value silently change on save/reopen; only fall back when the key is missing or invalid, not when it is 0.0.
Useful? React with 👍 / 👎.
| owner.sensory_enabled_check.toggled.connect(owner._sync_sensory_controls) | ||
| owner.sensory_context_enabled_check.toggled.connect(owner._sync_sensory_controls) | ||
| owner.sensory_source_combo.currentIndexChanged.connect(owner._handle_sensory_source_changed) | ||
| owner.sensory_mode_combo.currentIndexChanged.connect(owner._handle_sensory_control_changed) |
There was a problem hiding this comment.
When a new sensory source is switched from the default/off state to “远端 API” or “局域网”, the mode change only captures and syncs the controls; unlike backend changes, it never replaces the existing LM Studio endpoint text. Saving API mode after entering just a model therefore configures http://127.0.0.1:1234/v1, and LAN mode immediately fails validation until the user manually overwrites the stale localhost value, so the selected mode does not produce a usable default.
Useful? React with 👍 / 👎.
完整更新日志与验证结果见 PR Rvosy#137。版本保持 0.9.9-dev，按预发布版本准备。
2 participants
Summary
Validation
Stack
This is PR 1/6. Downstream clean-diff PRs are opened in the fork because this account cannot push intermediate base branches to Rvosy/Sakura.