Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions docs/local-backend-run-smoke.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
- 后端:本机全本地 `Qwen3.5-4B-Q4_K_M`(registry 默认项),llama-server `:8090` + Jev 契约服务 `:8092`
- 浏览器:`--backend chrome --headless`(每次 run 独立 profile,可复现)
- 实测日期:2026-09-24
- 原始件(15 个 journal、16 项校准探针、39 步请求日志):`/tmp/jev-local-smoke/`
- 原始件(15 个 journal、16 项校准探针、39 步请求日志)当时在 `/tmp/jev-local-smoke/`,**该目录已随 `/tmp` 清理删除,且从未入库**——下面的数字都来自当时的现场记录,原始 journal 无法再回放
- **`skills/**` 与 `docs/**` 之外一行未改:§6 列出的观测/代码层问题只报告、不改动,本次也没有提交任何 commit**

---
Expand Down Expand Up @@ -54,7 +54,8 @@ node --input-type=module -e "const {createSite}=await import('./tests/fixtures/s

```bash
export TYPESAFE_BASE_URL=http://127.0.0.1:8092 TYPESAFE_API_KEY=local
export JEV_BROWSER_CONFIG=/tmp/jev-local-smoke/config-<name>.json # journalDir + chrome.userDataDir 指向 /tmp
export JEV_BROWSER_CONFIG=/tmp/jev-run/config-<name>.json # 重建:mkdir -p /tmp/jev-run,写一份只改 journalDir 与 chrome.userDataDir 的 config
# (当时的 /tmp/jev-local-smoke/ 已随 /tmp 清理删除,见文首说明)
node skills/jev-browser/bin/jev-browser.mjs run --goal "<goal>" --url <url> \
--backend chrome --headless --max-steps <8|5|6> --json
```
Expand Down Expand Up @@ -198,7 +199,7 @@ node skills/jev-browser/bin/jev-browser.mjs run --goal "<goal>" --url <url> \

按严重度排序(全部有 journal 逐字证据):

1. **p4(登录页)——模型知道该填哪儿、填什么,就是不选「type」。** 同一 state 的完整判读(`/tmp/jev-local-smoke/login.dryrun.json` 重放,确定性复现):
1. **p4(登录页)——模型知道该填哪儿、填什么,就是不选「type」。** 同一 state 的完整判读(当时用 `/tmp/jev-local-smoke/login.dryrun.json` 重放,该文件已随 `/tmp` 清理删除;数字是现场记录,按 §7.3 的 `--dry-run` 流程重放同一登录页可复现):
```
action click 0.430 · stop 0.246 · wait 0.220 · type 0.105 ← 四选一里 type 垫底
type_target e2 (Email) 0.851 ← 正确
Expand Down Expand Up @@ -313,10 +314,10 @@ fetch("http://127.0.0.1:8092/v1/systemone",{method:"POST",headers:{authorization
body:JSON.stringify({model:"local",state:d.state,questions:{goal_done:d.questions.goal_done}})})
.then(r=>r.json()).then(j=>console.log(j.answers.goal_done));' # → noul≈0.969

# 4) 汇总脚本(本次全量数据来自它们;脚本本身在 /tmp,不入库)
node /tmp/jev-local-smoke/probe.mjs # → /tmp/jev-local-smoke/goal-done-probes.json(§4.1 的 16 项)
node /tmp/jev-local-smoke/latency.mjs # → 逐步/逐问延迟画像(§3)
node /tmp/jev-local-smoke/analyze.mjs # → 全部 journal 的逐 step 汇总(§2/§4.2)
# 4) 汇总脚本:probe.mjs / latency.mjs / analyze.mjs 当时在 /tmp/jev-local-smoke/,
# 从未入库,且已随 /tmp 清理删除,无法恢复。它们只是把 journal 聚合成 §3/§4 的表,
# 重跑这轮实测 = 用上面 1)–3) 的命令重建(起 jev-local + fixture,再跑 jev-browser run),
# 产物 journal 落在配置的 journalDir 下,自行汇总即可。
```

注:DDG 在本机 + headless 下会返回**人机验证**(“Unfortunately, bots use DuckDuckGo too. … Select all squares
Expand Down Expand Up @@ -350,7 +351,7 @@ containing a duck: Submit”,见 `r6.dryrun.json` 的 `visible_text`),所

## 9. 终止规则回放:哪条规则能让本地 `run` 正确地报 `success`

离线回放,无新浏览器 run;模型调用只用已保存的 state(§9.4)。脚本 `/tmp/jev-local-smoke/rules.mjs`(纯离线,只读 `steps.jsonl`)。
离线回放,无新浏览器 run;模型调用只用已保存的 state(§9.4)。回放脚本是 `/tmp/jev-local-smoke/rules.mjs`(纯离线,只读 `steps.jsonl`)——**该脚本与它读的 journal 都在 `/tmp/jev-local-smoke/`,已随 `/tmp` 清理删除、从未入库**;§9 的计分表是当时的现场记录,按 §9.1 的口径对一份 journal 重算即可复现。

### 9.1 计分口径(严格)

Expand Down
14 changes: 10 additions & 4 deletions docs/local-gguf-probe.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ llama.cpp 的「首 token 选项标签 logprob」读法,把本 skill 的 `POST

| | 0.6B | 0.8B |
|---|---|---|
| 文件 | `/tmp/gguf/Qwen3-0.6B-Q8_0.gguf` | `./Qwen3.5-0.8B-Q8_0.gguf`(仓库根目录) |
| 文件 | `/tmp/gguf/Qwen3-0.6B-Q8_0.gguf` | `~/.jev-browser/models/Qwen3.5-0.8B-Q8_0.gguf`(模型缓存目录) |
| 字节 | 639,446,688 | 811,843,840 |
| sha256 | `9465e63a22add5354d9bb4b99e90117043c7124007664907259bd16d043bb031` | `0ad885ffd4bb022fc4f0d33a3308fa108ef8613159d3b3a67e23abca056b7a6c` |
| GGUF `general.name` | **Qwen3 0.6B Instruct** ⚠️ | Qwen3.5-0.8B |
Expand All @@ -65,7 +65,12 @@ llama.cpp 的「首 token 选项标签 logprob」读法,把本 skill 的 `POST

> ⚠️ 0.6B 这份量化文件是 **Instruct** 版本(`general.name = "Qwen3 0.6B Instruct"`),
> 不是 JEV-CPU / SemIf 用的 `Qwen3-0.6B-Base`;Base 未测(见 §8)。
> ⚠️ 0.6B 的 GGUF 放在 `/tmp/gguf/`(重启即失);0.8B 放在仓库根目录(未入库,`*.gguf` 未进 `.gitignore`)。
> ⚠️ 0.6B 的 GGUF 放在 `/tmp/gguf/`,**现已不在本机**(`/tmp` 清理掉了,仓库与模型缓存里都没有副本)。
> 重跑 0.6B 这一行前要先重建它:重新下载 Qwen3-0.6B **Instruct** 的 Q8_0 量化
> (`general.name` 必须是 `Qwen3 0.6B Instruct`,不是 `Qwen3-0.6B-Base`),
> 放到任意路径后核对上表的字节数 639,446,688 与 sha256 `9465e63a…bb031`,再把 §2 命令里的 `-m` 指过去。
> 0.8B 放在 `~/.jev-browser/models/`(`jev-local` / `jev-kev` 的模型缓存目录,仍在);
> 仓库内不再保留任何 GGUF 副本(`*.gguf` 已在 `.gitignore` 兜底)。

---

Expand All @@ -76,9 +81,10 @@ llama.cpp 的「首 token 选项标签 logprob」读法,把本 skill 的 `POST
brew install llama.cpp # 实测 0.4.0 / build 10809

# 1) 两个 llama-server(0.6B:8090,0.8B:8091)
/opt/homebrew/bin/llama-server -m /tmp/gguf/Qwen3-0.6B-Q8_0.gguf \
# 0.6B 文件已不在 /tmp/gguf/(见 §1 的重建步骤:重新下载 Instruct Q8_0 并校验 sha256 9465e63a…)
/opt/homebrew/bin/llama-server -m <重建后的 Qwen3-0.6B-Q8_0.gguf 路径> \
--port 8090 --ctx-size 16384 --jinja --threads 8 -np 1 --no-warmup
/opt/homebrew/bin/llama-server -m "$PWD/Qwen3.5-0.8B-Q8_0.gguf" \
/opt/homebrew/bin/llama-server -m ~/.jev-browser/models/Qwen3.5-0.8B-Q8_0.gguf \
--port 8091 --ctx-size 16384 --jinja --threads 8 -np 1 --no-warmup
# --jinja 必须开(适配层用 /apply-template 走模型自带 chat 模板);-np 1 单槽;--no-warmup 跳过启动预热

Expand Down
9 changes: 6 additions & 3 deletions experiments/gguf-provider/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,9 +89,12 @@ default) for that reason; at `-c 8192` llama.cpp hard-fails the request with

- Candidates 2 and 3 (`Qwen3-4B-Instruct-2507`, `gemma-3-4b-it`) never finished downloading — the
link collapsed to ~10–25 KB/s with bursts — so their accuracy **and** their drivability through
the render path (chat template via `/apply-template`) are unexercised. No download is running now;
the retry script is `/tmp/jev-ceiling/fetch-4b.sh` (start with `hub`, then
`bash /tmp/jev-ceiling/run-candidate.sh <slug> 8100`).
the render path (chat template via `/apply-template`) are unexercised. No download is running now.
The retry driver was the scratch script `/tmp/jev-ceiling/fetch-4b.sh`, which is gone with the
`/tmp` cleanup and was never committed; recreate the retry with the tracked tools:
`node experiments/gguf-provider/lab/fetch.mjs <repo>` (resumable download, sha256 gate), then the
per-candidate protocol in `results/local-models-4b.md` §1
(`eval/run.mjs` + `cli.mjs --request` against the llama-server on :8100).
- `ddg-click-target-aapl`'s ground-truth label is debatable: the hand label is `e1` (investing.com,
`in_viewport=false`), while hosted Jev and the 4B both pick `e15` (Yahoo Finance AAPL quote page,
in viewport). Every model is therefore scored down by up to one item on that question.
Expand Down
39 changes: 30 additions & 9 deletions experiments/gguf-provider/results/local-models-4b.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,14 +130,26 @@ MLX 服务),比 GGUF 候选 1 高 3 题,并且与 hosted Jev 天花板同
- 量化:统一 `Q4_K_M`(若两候选差 ≤3 点,再补 `Q8_0`)。
- 天花板:hosted Jev(`api.typesafe.ai`,`jev-latest` → 实测 `jev-1.13.0`),同样 20 条 + 同一步 fixture。

命令(每个候选;`<file>` = `~/.jev-browser/models/` 下的 GGUF):
命令(每个候选;`<file>` = `~/.jev-browser/models/` 下的 GGUF)。
下面这条协议当时包在 `/tmp/jev-ceiling/run-candidate.sh` 里——那是临时脚本,
**已随 `/tmp` 清理删除,仓库内没有副本**;重跑时按下面三步重建它(或直接逐步执行):

```bash
# 1) 起 llama-server(一次只起一个;跑完即停)
/opt/homebrew/bin/llama-server -m ~/.jev-browser/models/<file> \
--port 8100 -c 16384 -ngl 99 --cache-type-k q8_0 --cache-type-v q8_0 -t 8 --jinja -np 1 --no-warmup
bash /tmp/jev-ceiling/run-candidate.sh <slug> 8100 # eval(20+轮转) -> fixture step -> analysis row
# 2) 20 条自明真值 + 标签轮转(--url 指向上面的 llama-server)
node experiments/gguf-provider/eval/run.mjs --url http://127.0.0.1:8100 --json \
> experiments/gguf-provider/results/eval-<slug>.json
node experiments/gguf-provider/eval/run.mjs --url http://127.0.0.1:8100 --rotate --json
# 3) 真实捕获页面的完整一步(fixtures/judge-state.json + judge-questions.json 拼成请求体)
node experiments/gguf-provider/cli.mjs --request experiments/gguf-provider/results/fixture-step-request.json \
--url http://127.0.0.1:8100 > experiments/gguf-provider/results/fixture-step-<slug>.json
```

`run-candidate.sh` 的第三步(汇总行)就是把上面两份产物整理成 `results/analysis-<slug>.json`
的一行,口径见 §4;那部分逻辑也在被删除的临时脚本里,仓库没有留下生成器。

## 2. 候选状态

| # | 候选(HF repo) | 文件 | 量化 | 字节 | sha256(前 16) | 状态 |
Expand Down Expand Up @@ -225,13 +237,19 @@ hf `.cache` 只剩 6 MB 旧分片 —— 928 MB / 202 MB 两个大分片已在

```bash
# 候选 2 与候选 1 只差 1 题(≤3 点)→ 早先 brief 约定补 Q5_K_M/Q8_0 对拍;每个量化 ~2.5-3 GB。
# 驱动脚本 /tmp/jev-ceiling/run-candidate.sh 与 make-table.mjs 已随 /tmp 清理删除(仓库无副本),
# 重跑 = 重建:llama-server + §1 的两步 eval/cli 命令,再用 §4 口径汇总。
/opt/homebrew/bin/llama-server -m ~/.jev-browser/models/<file> --port 8100 -c 16384 \
-ngl 99 --cache-type-k q8_0 --cache-type-v q8_0 -t 8 --jinja -np 1 --no-warmup
bash /tmp/jev-ceiling/run-candidate.sh <slug> 8100 ; hub stop name=eval-4b-8100
node /tmp/jev-ceiling/make-table.mjs qwen35-4b qwen3-4b-instruct-2507 gemma-3-4b-it qwen35-08b-c16384
node experiments/gguf-provider/eval/run.mjs --url http://127.0.0.1:8100 --json
node experiments/gguf-provider/cli.mjs --request experiments/gguf-provider/results/fixture-step-request.json \
--url http://127.0.0.1:8100
# 汇总(make-table.mjs 已删除):按 §4 口径把 eval/fixture 产物整理成 §2 表的一行。
# 当时的调用是:node /tmp/jev-ceiling/make-table.mjs qwen35-4b qwen3-4b-instruct-2507 gemma-3-4b-it qwen35-08b-c16384
```

(上表两行新数据由 `make-table.mjs` 机械复算过,与手写行逐格一致;`eval/**` 代码一行未改。)
(上表两行新数据由 `make-table.mjs` 机械复算过,与手写行逐格一致;`eval/**` 代码一行未改。
该脚本是临时工具,现已不在本机。)

## 4. 备注与判分口径

Expand Down Expand Up @@ -322,14 +340,17 @@ results/ramp-gemma-sustained-2026-09-24.json 本轮:150 MiB 采样窗口
results/throughput-summary-2026-09-24.json 本轮:链路汇总 + 每候选 ETA
```

本轮新增的**仓库内**工具(上一窗口的临时工具仍在 `/tmp/jev-ceiling/`):
本轮新增的**仓库内**工具(上一窗口的临时工具当时在 `/tmp/jev-ceiling/`,现已删除):

```
lab/fetch.mjs 受监督下载器:curl 引擎、hf 分片断点续传、0.3 MB/s 十分钟中止规则、sha256 硬门
lab/throughput.sh 定长分段抓取的吞吐探针
lab/ramp.mjs 采样式速率探针(把 slow start 和稳态区分开 —— 本轮结论的关键工具)
```

临时工具(不在仓库内):`/tmp/jev-ceiling/{ceiling-run,analyze,make-table,gguf-meta}.mjs`、
`/tmp/jev-ceiling/run-candidate.sh`、`/tmp/jev-ceiling/fetch-4b.sh`(后两者本轮仍在使用,
`run-candidate.sh` 就是上面每候选跑的那条协议)。
临时工具(**从未入库,且已随 `/tmp` 清理从本机删除,无法恢复**):
`/tmp/jev-ceiling/{ceiling-run,analyze,make-table,gguf-meta}.mjs`、
`/tmp/jev-ceiling/run-candidate.sh`、`/tmp/jev-ceiling/fetch-4b.sh`。
`run-candidate.sh` 就是 §1 每候选跑的那条协议,等价重建步骤见 §1 的命令块
(`eval/run.mjs` + `cli.mjs --request`,都在仓库内);`fetch-4b.sh` 的下载逻辑
已由仓库内的 `lab/fetch.mjs` 覆盖(断点续传 + sha256 硬门)。
10 changes: 7 additions & 3 deletions experiments/kev-4b/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,10 +329,14 @@ The skill's loopback bar (`thresholds.goalDone` / `goalDoneFinal` = **0.174**, s
goals were replayed against Kev 4B with the same method and the same scoring.

```bash
# Kev 4B already up on :8008 (bin/jev-kev.mjs --row-limit 16384) and the fixture site on :3111
# Kev 4B already up on :8008 (bin/jev-kev.mjs --row-limit 16384) and the fixture site on :3111.
# The original run wrote its journals to /tmp/kev-threshold, which the /tmp cleanup removed;
# the journals were copied into the repo first (results/threshold-replay/journal/, 15 runs),
# so recreate the scratch dir from the tracked copy and point both commands at it:
mkdir -p /tmp/kev-replay && cp -R experiments/kev-4b/results/threshold-replay/. /tmp/kev-replay/
TYPESAFE_BASE_URL=http://127.0.0.1:8008 TYPESAFE_API_KEY=local \
bash experiments/kev-4b/threshold-replay.sh /tmp/kev-threshold
node experiments/kev-4b/threshold-replay.mjs /tmp/kev-threshold
bash experiments/kev-4b/threshold-replay.sh /tmp/kev-replay
node experiments/kev-4b/threshold-replay.mjs /tmp/kev-replay
```

Method, unchanged from §9: one real `chrome --headless` run per goal with the controller's default
Expand Down
Loading