Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .trellis/spec/backend/quality-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
- idle 收尾要**回溯**到 max(最后输入时刻, 最后媒体活跃时刻),不是收在当前时刻。
- 多命令一次 exec 时用显式分隔符(`---ECHOLOG---`)分段解析,不要靠正则猜哪行是谁的输出(`pmset` 输出里也有裸数字)。
- `lsappinfo info -only name -only bundleid "$(lsappinfo front)"` 无需 TCC 权限;输出形如 `"LSDisplayName"="微信"`。
- 常驻自动任务访问 macOS Keychain 时必须使用禁止授权 UI 的 helper 模式;`KEYCHAIN_AUTH_REQUIRED` 应静默跳过并暂停该凭据的自动重试。只有显式用户操作可使用至少 60 秒的交互式读取;成功后把凭据缓存于进程内并恢复调度,避免按采样周期重复访问 Keychain 或弹系统对话框。列表/页面刷新不得为了展示状态查询 Keychain,进程停止时必须清空缓存。

### 常驻采样器模式

Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,9 @@ launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.echolog.daemon.plist
```

screen-understanding helper 不属于 portable build。本机 smoke 组装使用
`ECHOLOG_MACOS_ADHOC_SMOKE=1 pnpm build:macos-capture`;正式签名使用
`ECHOLOG_MACOS_ADHOC_SMOKE=1 pnpm build:macos-capture && pnpm smoke:macos-helper`;
launchd 安装后使用 `pnpm smoke:launchd-helper` 检查实际 `WorkingDirectory`;
正式签名使用
`ECHOLOG_MACOS_SIGNING_IDENTITY=... pnpm build:macos-release`。启用识图后,截图只在
单次请求内存中存在;数据库只保存结构化理解结果,不保存图片或 API key。

Expand Down
17 changes: 16 additions & 1 deletion README_AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ portable `pnpm build` 不编译 Swift helper。macOS 本机测试可显式执行

```bash
ECHOLOG_MACOS_ADHOC_SMOKE=1 pnpm build:macos-capture
pnpm smoke:macos-helper
```

## 启动后验证
Expand All @@ -55,6 +56,19 @@ node dist/cli/index.js plugins list --json
node dist/cli/index.js plugins doctor --json
```

如果使用 launchd,另跑一次真实运行目录检查:

```bash
pnpm smoke:launchd-helper
```

自动识图只允许 helper 以非交互模式读取 Keychain;需要授权时会静默跳过并暂停
该 Provider 的自动 Keychain 重试,不会周期性弹系统对话框。请在 Web 点击
“立即识别”(或从本机执行 `el screen understanding run --json`),并在 60 秒内
完成一次 Keychain 授权。读取成功后 API key 只进入 daemon 的进程内缓存,自动
调度随即恢复且不再访问 Keychain;daemon 重启会清空缓存,必要时需再次执行一次
显式识别完成授权。

若已把 wrapper 放进 `PATH`,后续使用 `el` 代替 `node dist/cli/index.js`。先读 `el --help` 和目标子命令的 `--help`;它们是参数、枚举、时间格式和示例的权威工具说明。

## CLI 契约
Expand Down Expand Up @@ -98,7 +112,7 @@ screen-understanding 默认关闭。最安全的路径是让人类用户打开 `
2. 在本机 Keychain 保存 API key;
3. 测试截图并由用户授予屏幕录制权限;
4. 选择 Provider、设置周期/预算后启用;
5. 执行一次“立即识别”并检查结构化中文结果。
5. 执行一次“立即识别”并检查结构化中文结果;该显式操作可在周期识别尚未启用时完成 Keychain 授权

Agent 可以读取状态和结果:

Expand All @@ -113,6 +127,7 @@ Provider、Keychain、settings 和 capture test 的完整 HTTP 契约见 [`docs/

- Key 写入/删除、截图测试、立即识别和历史删除只允许 loopback 请求;
- daemon 永远不能调用 `request-permission`,权限必须由交互式用户动作触发;
- Provider 列表和普通页面刷新只能显示内存中已知的密钥状态,不得主动查询或弹出 Keychain;
- settings 更新是带 `expectedVersion` 的全量替换;409 时读取 `currentVersion` 后重新决策,不能盲目覆盖;
- 启用识别前必须选择存在且已有 Keychain 密钥的 Provider;
- 原始截图不得落盘、记录、转发或附在 Agent 回复中,除非用户明确要求处理当前预览且符合其隐私意图。
Expand Down
33 changes: 21 additions & 12 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,9 +352,9 @@ curl -X PUT http://localhost:19827/api/plugins/screen-time/understanding/setting

#### screen-understanding providers 与 Keychain

Provider metadata 存在 PostgreSQL;API Key 只存在当前 macOS 用户的 Keychain,
API 永不返回原文或掩码片段。支持的 provider kind 当前固定为
`openai-compatible`。
Provider metadata 存在 PostgreSQL;API Key 只持久化在当前 macOS 用户的 Keychain,
成功读取后可短暂保存在 daemon 进程内存中。API 永不返回原文或掩码片段;daemon
停止或重启会清空缓存。支持的 provider kind 当前固定为 `openai-compatible`。

```bash
curl http://localhost:19827/api/plugins/screen-time/understanding/providers
Expand Down Expand Up @@ -405,13 +405,15 @@ curl -X DELETE http://localhost:19827/api/plugins/screen-time/understanding/prov
- `expectedVersion`:1–2147483647 的整数;PUT/DELETE 均使用乐观并发。
- `apiKey`:UTF-8 1–4096 bytes,不得有首尾空白、换行或 NUL。

Keychain 状态正常时 `hasApiKey` 为 boolean;helper/Keychain 暂时不可用时为
`null`,metadata 仍可读取和编辑,Web 显示“密钥状态不可用”。
`hasApiKey` 只反映 daemon 内存中已知的状态;daemon 刚启动或状态未知时为 `null`。
`GET /providers` 和普通页面刷新不会为展示状态调用 helper,也不会触发 Keychain UI。
metadata 始终可读取和编辑,Web 对 `null` 显示“密钥状态不可用”。

新建成功为 `201`;更新成功为 `200` 且 version 加一;删除 metadata 成功为
`204`。非法字段/URL/key 返回 `400`;不存在返回 `404`;版本过期、profile
仍被 settings 选择、启用时删除 key,或启用设置却没有可用 key,返回 `409`。
Keychain helper 不可用/失败/超时分别返回脱敏的 `503`/`502`/`504`。
Keychain helper 不可用/失败/超时分别返回脱敏的 `503`/`502`/`504`;需要用户授权
返回 `409 KEYCHAIN_AUTH_REQUIRED`。
Key 的 PUT/DELETE 仅接受 loopback 请求,远端请求返回 `403 PLUGIN_LOCAL_ONLY`。

Key PUT/DELETE 成功响应分别为:
Expand All @@ -435,8 +437,8 @@ Key PUT/DELETE 成功响应分别为:
```

其他稳定冲突 code 包括 `PROVIDER_PROFILE_IN_USE`、`PROVIDER_PROFILE_LIMIT`、
`PROVIDER_KEY_REQUIRED`;平台/helper code 包括 `KEYCHAIN_UNAVAILABLE`、
`KEYCHAIN_OPERATION_FAILED` 和 `PLUGIN_TIMEOUT`。
`PROVIDER_KEY_REQUIRED`;平台/helper code 包括 `KEYCHAIN_AUTH_REQUIRED`、
`KEYCHAIN_UNAVAILABLE`、`KEYCHAIN_OPERATION_FAILED` 和 `PLUGIN_TIMEOUT`。

#### 显式测试截图

Expand Down Expand Up @@ -480,11 +482,18 @@ curl -X POST http://localhost:19827/api/plugins/screen-time/understanding/captur
#### AI 屏幕识别

`POST /api/plugins/screen-time/understanding/run` 是 loopback-only 的显式识别入口,
只接受空 JSON 对象。它要求 settings 中 `enabled=true`、选中了有 Keychain 密钥的
Provider;服务会采集活动显示器、调用 `${baseUrl}/chat/completions`,并只接受包含
只接受空 JSON 对象。它要求选中了有 Keychain 密钥的 Provider;服务会采集活动
显示器、调用 `${baseUrl}/chat/completions`,并只接受包含
`summary`、`activity`、`confidence`、`sensitive`、`apps` 的 JSON 结果。失败不会返回
远端响应正文或 API key。

`enabled` 只控制周期调度;该显式入口在周期识别关闭时仍可执行。它允许 macOS
显示 Keychain 授权 UI,并为用户保留至少 60 秒完成授权。
成功读取后,凭据进入 daemon 的进程内缓存并解除该 Provider 的自动调度阻断。
周期任务只执行禁止 UI 的 Keychain 查询;遇到 `KEYCHAIN_AUTH_REQUIRED` 会静默跳过
本轮并停止重复查询,不会弹窗或把插件降级。后续周期识别直接使用内存缓存,不再
逐轮访问 helper。daemon 重启后如 Keychain 不能无 UI 读取,请再次调用本显式入口。

```bash
curl -X POST http://localhost:19827/api/plugins/screen-time/understanding/run \
-H 'Content-Type: application/json' -d '{}'
Expand Down Expand Up @@ -519,8 +528,8 @@ observation 返回 `404`。它不会删除 request budget ledger。
`dailyRequestBudget` 和可选的 `dailyCostBudgetMicros` 约束执行。一次进程内同时只
允许一个识别任务;临时网络错误按 `maxAttempts` 有界重试。

识别关闭返回 `409 UNDERSTANDING_DISABLED`;未配置 Provider 返回
`409 UNDERSTANDING_PROVIDER_REQUIRED`;请求/成本预算耗尽返回 `429`;模型认证、
未配置 Provider 返回 `409 UNDERSTANDING_PROVIDER_REQUIRED`;请求/成本预算耗尽
返回 `429`;模型认证、
超时、限流、不可达和非法响应分别返回脱敏的 `PROVIDER_AUTH`、
`PROVIDER_TIMEOUT`、`PROVIDER_RATE_LIMITED`、`PROVIDER_UNAVAILABLE` 或
`UNDERSTANDING_RESPONSE_INVALID`。原始截图测试和识别入口均不提供
Expand Down
13 changes: 13 additions & 0 deletions docs/PLUGIN_API.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,19 @@ screen-time packages
`native/macos-capture/build/EchoLogScreenCapture.app/Contents/MacOS/echolog-screen-capture`.
Permission inspection never prompts, and a missing Screen Recording permission
diagnostic does not degrade passive foreground tracking.
`pnpm smoke:macos-helper` verifies the source or release tree runtime path;
`pnpm smoke:launchd-helper` reads the daemon plist `WorkingDirectory` and checks
the helper that launchd will actually load.

Scheduled Keychain status/get operations MUST use the helper's non-interactive
mode, which applies `kSecUseAuthenticationUIFail`. Authorization-required
results MUST be structured and MUST NOT be treated as timeouts or degrade the
plugin. Explicit local user actions MAY use interactive Keychain access with a
timeout of at least 60 seconds. A successful read SHOULD be cached only in the
daemon process; set/delete MUST synchronize that cache and plugin stop MUST
clear it. Provider listing and ordinary page loads MUST NOT query Keychain.
Secrets MUST NOT enter database rows, configuration, argv, logs, errors, smoke
output, or test output.

`GET /api/health` reports Core health. Plugin failures appear in
`GET /api/plugins` and `GET /api/plugins/doctor`. A failed doctor request uses
Expand Down
4 changes: 3 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
"dev": "tsx src/server/app.ts",
"build": "pnpm --filter @echolog/plugin-sdk build && pnpm --filter @echolog/plugin-screen-time build && pnpm --filter @echolog/plugin-tmux-status build && tsup",
"build:macos-capture": "bash scripts/build-macos-capture.sh",
"build:macos-release": "pnpm build && pnpm build:macos-capture",
"build:macos-release": "pnpm build && pnpm build:macos-capture && pnpm smoke:macos-helper",
"smoke:macos-helper": "bash scripts/smoke-macos-helper.sh",
"smoke:launchd-helper": "bash scripts/smoke-macos-helper.sh --launchd",
"package:macos": "bash scripts/package-release.sh --version 0.2.0 --adhoc",
"cli": "tsx src/cli/index.ts",
"migrate": "tsx src/migrate.ts",
Expand Down
20 changes: 18 additions & 2 deletions plugins/screen-time/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,22 +33,38 @@ explicitly:

```bash
ECHOLOG_MACOS_ADHOC_SMOKE=1 pnpm build:macos-capture
pnpm smoke:macos-helper
ECHOLOG_MACOS_SIGNING_IDENTITY='Developer ID Application: ...' pnpm build:macos-release
```

Automatic understanding uses `keychain get --no-auth-ui`. If macOS requires
authorization, the run is skipped without showing UI and further scheduled
Keychain reads for that Provider remain blocked. “立即识别” is an explicit,
interactive operation with a 60-second helper timeout. A successful manual read
caches the credential in daemon memory, clears the block, and lets later
scheduled runs use the cache without invoking the helper again. Set and delete
operations update the same cache; plugin stop or daemon restart clears it.

The artifact is `native/macos-capture/build/EchoLogScreenCapture.app`. The Web
“测试截图” and “立即识别” actions invoke that app identity through macOS
LaunchServices (`/usr/bin/open -W -n ... --args`) and localhost-only routes. The
daemon exclusively pre-creates mode `0600` stdout/stderr files inside one mode
`0700` private temporary directory, validates the PNG, sends it to the selected
OpenAI-compatible vision endpoint, and deletes the directory in `finally`.
Keychain operations execute the inner helper directly; API keys never enter the
database, argv, logs, or API responses. Successful runs persist only structured
Keychain operations execute the inner helper directly. Provider listing and
ordinary page loads report only the daemon's cached key state and never probe
Keychain. API keys never enter the database, argv, logs, or API responses.
Successful runs persist only structured
summary/activity/apps/confidence metadata, with displayed field values required
to be in Simplified Chinese. The one-shot capture process has its own 12-second
hard watchdog, shorter than the daemon's 15-second request timeout.
Never invoke `request-permission` from the daemon.

The `enabled` setting controls periodic scheduling. The loopback-only
“立即识别” action remains available as an explicit one-off run while scheduling
is disabled, so a user can authorize and warm the credential cache before
enabling automatic understanding.

The periodic understanding job checks the database-backed interval every five
seconds. This keeps a configured 120-second interval close to two minutes even
when the setting changes while the daemon is running, instead of rounding it up
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import Foundation
import Darwin
import Security

public let helperVersion = "0.1.0"
public let helperBundleIdentifier = "com.cubeplus1.echolog.screen-capture"
Expand All @@ -9,8 +10,8 @@ public enum HelperCommand: Equatable, Sendable {
case status
case requestPermission
case capture(output: String, maxPixelEdge: Int)
case keychainStatus(service: String, account: String)
case keychainGet(service: String, account: String)
case keychainStatus(service: String, account: String, noAuthUI: Bool)
case keychainGet(service: String, account: String, noAuthUI: Bool)
case keychainSet(service: String, account: String)
case keychainDelete(service: String, account: String)
case version
Expand Down Expand Up @@ -106,16 +107,28 @@ public enum CommandParser {
guard let operation = args.first, ["status", "get", "set", "delete"].contains(operation) else {
throw HelperFailure.invalid("keychain requires status, get, set, or delete")
}
let options = try parseOptions(Array(args.dropFirst()), allowed: ["--service", "--account"])
let rawOptions = Array(args.dropFirst())
let noAuthUICount = rawOptions.filter { $0 == "--no-auth-ui" }.count
guard noAuthUICount <= 1 else {
throw HelperFailure.keychainInvalid("Duplicate option: --no-auth-ui")
}
let noAuthUI = noAuthUICount == 1
if noAuthUI && operation != "status" && operation != "get" {
throw HelperFailure.keychainInvalid("--no-auth-ui is only valid for keychain status or get")
}
let options = try parseOptions(
rawOptions.filter { $0 != "--no-auth-ui" },
allowed: ["--service", "--account"]
)
guard let service = options["--service"], service == screenUnderstandingKeychainService else {
throw HelperFailure.keychainInvalid("--service must use the EchoLog screen-understanding namespace")
}
guard let account = options["--account"], validIdentifier(account) else {
throw HelperFailure.keychainInvalid("--account must be 1-255 printable characters")
}
switch operation {
case "status": return .keychainStatus(service: service, account: account)
case "get": return .keychainGet(service: service, account: account)
case "status": return .keychainStatus(service: service, account: account, noAuthUI: noAuthUI)
case "get": return .keychainGet(service: service, account: account, noAuthUI: noAuthUI)
case "set": return .keychainSet(service: service, account: account)
default: return .keychainDelete(service: service, account: account)
}
Expand Down Expand Up @@ -208,10 +221,13 @@ public struct SecretInput: Decodable, Sendable {
}

public enum KeychainStatusMapping {
public enum Result: Equatable { case present, missing, failure }
public enum Result: Equatable { case present, missing, authRequired, failure }
public static func map(_ status: Int32) -> Result {
if status == 0 { return .present }
if status == -25300 { return .missing }
if status == errSecSuccess { return .present }
if status == errSecItemNotFound { return .missing }
if [errSecInteractionNotAllowed, errSecAuthFailed, errSecUserCanceled].contains(status) {
return .authRequired
}
return .failure
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@ public struct HelperRunner: Sendable {
"bytes": result.bytes,
"capturedAt": result.capturedAt,
])
case let .keychainStatus(service, account):
return try JSONOutput.success(["hasSecret": try keychain.hasSecret(service: service, account: account)])
case let .keychainGet(service, account):
if let secret = try keychain.getSecret(service: service, account: account) {
case let .keychainStatus(service, account, noAuthUI):
return try JSONOutput.success(["hasSecret": try keychain.hasSecret(service: service, account: account, noAuthUI: noAuthUI)])
case let .keychainGet(service, account, noAuthUI):
if let secret = try keychain.getSecret(service: service, account: account, noAuthUI: noAuthUI) {
return try JSONOutput.success(["hasSecret": true, "secret": secret])
}
return try JSONOutput.success(["hasSecret": false])
Expand Down
Loading
Loading