feat(jev-browser): local model backends — GGUF readout, Kev 4B tier, per-backend goal_done profiles - #1
Merged
Conversation
Answers the skill's System One questions from a local llama.cpp server: the question is rendered with labelled options, exactly one token is generated, and the probability mass on each option label at that position is the answer. No API key, no Python, no training, no output parsing — and it speaks the same /v1/systemone contract as hosted Jev, so a run only has to point TYPESAFE_BASE_URL at 127.0.0.1. Measured on 20 construction-graded items (experiments/gguf-provider/RESULTS.md): 0.80 for the shipped Qwen3.5-4B Q4_K_M, against 0.50 for the 0.8B reference entry, 0.55 for "always answer the option listed first" and 0.95 for hosted Jev on the same items — the first local candidate to clear the positional prior. One real browser step (5 questions, 16.1 kB state, 13,729 prompt tokens cold) is 18,298 ms cold / 9,002 ms warm at 3,362 MiB RSS; loopback tokens cost $0. Rotation shows the 4B reads option text where the 0.8B reads position, and its confidence is usable as an abstention signal (>= 0.5 keeps 8/12 answers with 7/8 correct). Models are data, not code: lib/local-models.json carries each entry's file, resolve URL, exact byte size and label, so swapping one is an edit — --list-models prints the registry, --model-name picks per run, --ctx sets the llama.cpp context (16k default: a 100-candidate step renders to ~11.7k tokens). Blobs download into ~/.jev-browser/models and are never vendored. test:local runs the new suites; npm test picks them up through the same glob.
… at $0, chrome needs_user hint
Three defects the first real local-backend run found (docs/local-backend-run-smoke.md §6),
each one silent before:
- ARIA 1.2 puts role="combobox" either on the <input> itself (DuckDuckGo) or on a wrapper
(Wikipedia). The first form was observed as a plain combobox, so `editable` was 0, the
type/type_target/type_value questions were never generated, and runs could only click,
scroll, wait or stop: R3 and R8 clicked the search box and never typed. isComboboxField()
classifies the field as textbox — a combobox element with no input/textarea/contenteditable
inside stays non-editable and is still offered as a click target.
- A loopback baseUrl is a local model that bills nothing, yet runner copied config.pricePerMtok
(0.042/Mtok) into the client: R8 reported costUsd=0.002705 for tokens that are $0, and
budgetUsd counted spend that never happened. isLoopbackBaseUrl()/pricePerMtokFor() now price
127.0.0.1, localhost and ::1 at 0 while still counting tokens.
- A closed chrome session was still announced with the ego wording ("the browser was handed to
you; resume with --space-id …"). In R6 the run was genuinely blocked but handOff was null and
headless had already closed the page, so following the hint found no browser. Without a space
id the summary now says the browser was closed rather than handed over, and points at --keep.
The GGUF readout is the default because it is cheap; this adds the tier for when it is not accurate enough. One command fetches jaredpalmer/kev-4b and its Qwen3.5-4B-Base parent into the HuggingFace cache layout the Kev loader expects, verifies every file against a manifest pinned to one commit (sha256 for LFS blobs, git blob sha1 for the small text files), starts the MLX server on 127.0.0.1:8008 and prints the skill's env line last, on stdout, so it can be eval'd. Why the extra tier earns its 9.34 GB base: 19/20 = 0.95 on the same 20 construction-graded items the GGUF candidates were measured on — tying hosted Jev on that set, and above the readout's 0.80 and the 0.8B's 0.50. 18/20 at the released row limit, where the server refuses the two 55-option click_target questions with HTTP 422; --patch-row-limit exists for exactly that, is never applied implicitly, and prints its diff. Bring-up cost measured end to end (docs/local-kev-bringup.md): clone 32 s, uv sync 58 s, first start to first healthy response ≈ 96 s, adapter 159.7 MB in 58 s, base 9.34 GB over two streams from ModelScope in ~22.5 min. Steady state: ~18 GB idle and ~36 GB of GPU memory under load at the raised limit (heavy swap on a 48 GB machine), mean 2.2 s per item, 12.2 s worst — and $0 per token, since the base URL is loopback. The /v1/models card this server returns names the checkpoint it loaded (run, base), which is what lets a run tell Kev from the readout without being told; the per-backend bar that depends on it lands next. Kev is a trained pointer head, not a first-token logprob readout: on the shared fixture page it reads goal_done 0.0607 where the GGUF 0.8B read 0.6952 — the thresholds are not shared.
goal_done is the question that ends a run, and its value is not the same on the two local backends — 0.85 (the hosted default) is wrong for both. Measured, not guessed: - GGUF readout: over the 15 recorded runs in docs/local-backend-run-smoke.md §9, every run that reached the goal crossed 0.25 and no run that never reached it crossed 0.111, so the bar is the maximin geometric midpoint sqrt(0.111 × 0.273) = 0.174 — 1.57× from each side. At 0.85 only 7/7 reached goals were still recognised, with 3 false stucks. - Kev: the same 15 goals replayed against the checkpoint (46 readings, experiments/kev-4b) show the readout's band does not transfer — at 0.174 Kev calls four not-met pages a success (R5 0.341, p3 0.257, g1 0.195, g4 0.188), each stopping the run before the finishing action. Kev's own sides are 0.341/0.683, giving 0.482 (1.41× each); 0.482 scores 7 correct successes / 0 false / 0 false stucks against 0.85's 5 correct and 2 false stucks. thresholds.profile (auto | hosted | local-readout | kev) chooses the pair, and auto resolves it from the endpoint: a non-loopback baseUrl is hosted, a loopback one is classified by ONE GET /v1/models at run start, because only the endpoint knows which backend it is — a card naming a loaded checkpoint (run/base) is Kev, a name-only card is the GGUF readout. An unreachable or unrecognised loopback endpoint takes the HIGHEST bar and says why: a false stuck stops where you can see it, a false success is silent. A value any config layer set still wins per key (thresholds.configured), and a pinned profile is applied without any network call. The resolution is recorded where a run can be audited: controller writes run.json with the profile, the reason and the applied pair before the first step, so a run that dies later still names the bar it ran under; doctor prints the same as a `goal_done bar` line, with where the value was measured. doctor --home <dir> now loads that home's config, which is the whole point of the flag.
Puts the numbers behind the three local commits on the record, in three layers: the product tier table (README.md, SKILL.md), the operational detail (references/config.md, references/ questions.md), and the raw experiments with their own methodology and datasets (docs/, experiments/). - README/SKILL: what each tier costs and scores. Readout: 2.6 GiB, no Python, no API key, $0, p50 ≈4.3 s per step (one long option list is ~2/3 of it), 0.80 vs hosted Jev's 0.95 on 20 graded items with the 0.8B entry at 0.50. Kev: 19/20 = 0.95 at the raised row limit and 18/20 at the released one, for a venv + MLX + 9.34 GB base + ~18 GB idle / ~36 GB loaded and 2.2 s mean per item. - references/config.md: the profile table with where each bar was measured, the band each was derived from, the per-backend table that shows why 0.174 must not be reused for Kev (4 false successes), and the explicit statement that re-measuring beats trusting either local bar on a model nobody has scored. references/questions.md follows goal_done to its per-backend values. - docs/ and experiments/: the bring-up logs, the first real end-to-end run of a local backend under `run` (15 runs: success 4, stuck 8, needs_user 2, max_steps 1; 39 steps, no retry, no timeout, no 422), the Kev bring-up and threshold replay with its raw per-step readings, and the model-selection research behind the shipped default.
ChenYCL
changed the title
The local backends were only discoverable by reading a doc, and the one number that actually differs between them — the goal_done bar — was printed by doctor alone. This puts the choice in the CLI: `tier list` is one uniform table (what each backend is, what it needs, how to start it, its port, its 20-item score and its bar), `tier status` says what a run would use right now and why, `tier use <name>` prints the export line and start command without touching anything. The tiers and their numbers live in lib/tiers.mjs as data, with `bar` read from THRESHOLD_PROFILES rather than restated, so this table and doctor can never disagree about a number, and the scores are the ones already measured in the repo (hosted 0.95, readout 0.80, Kev 0.95 patched / 0.90 released). Hosted Jev stays the default: DEFAULT_TIER is hosted, a clean HOME resolves to it, and a test asserts no default value points at loopback. doctor's private endpoint classifier is deleted and replaced by the shared probeEndpoint(), so "which backend answers" has one implementation. Its `goal_done bar` line now names the tier beside the values: an unclassified loopback endpoint says `unclassified` and still shows the highest bar rather than asserting an identity nobody established. `tier status` reuses the same resolution, and config problems exit 2 like the launchers, so a script can tell "you asked for something wrong" from "it broke". tests/unit/tier.test.mjs drives the real CLI against a temp HOME: three tiers with hosted marked default, the unclassified fallback when a loopback endpoint is down, a pinned thresholds.profile winning over classification, and `tier use` writing nothing without --persist. Offline throughout (the only endpoint is the mock fixture).
The tier comparison existed twice and in different orders: SKILL.md described the two local tiers with hosted Jev only as a score row to compare against, README.md described them as two experimental paragraphs, and the old `#fully-local-experimental` anchor was the only place either could link to. A reader could not tell that hosted is the default or that the three are one axis. SKILL.md's `## Judging tiers` is now the single list, in weight order — hosted Jev, GGUF readout, Kev 4B — with the `tier` commands above it and a row per fact (needs, disk, memory, start command and port, 20-item score, goal_done bar, known hole). The hosted column is filled in rather than implied, because "the default" is a claim about behaviour: nothing to download, nothing to start, and the bar the other two are measured against. README.md points at that section instead of restating it, and README.zh-CN.md gets the same short block plus the CLI, so both front pages answer "which backend should I use" with one link. references/config.md keeps the threshold mechanics and says plainly that the tier list lives in SKILL.md; references/backends.md now opens by disambiguating the two axes — browser backend vs judging tier — since that is the confusion the word invites. Every `#judging-tiers` link resolves; no reference to the removed anchor survives.
Test-only change; no product code is touched. The online `doctor --json` call added alongside the goal_done bar assertions assumed the command exits 0. It does not on a machine without an optional tool: doctor's exit code covers every check, and `ego-browser cli` is status=fail when the CLI is absent, so on the ubuntu-latest runner the promisified execFile threw and the whole CLI round-trip test failed with "Command failed: ... doctor --json (code 1)". doctor exiting non-zero when a check fails is deliberate and stays: it is what scripts want. The test was the thing making an assumption about the runner. It now takes the failure the same way the `--offline` call three lines above already does, and asserts on stdout, which is printed either way — so it tests the JSON payload instead of the exit code. A real crash still fails it: an empty stdout is not parseable, so JSON.parse throws exactly as before. Verified both ways: the test passes with ego-browser hidden from PATH (the CI condition, reproduced exactly before the change), and `npm test` stays green with it present (86 tests, 81 pass, 0 fail, 5 skipped).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two fully local backends behind the existing
/v1/systemonecontract, plus the per-backendgoal_donebar they need.What is in here
feat(jev-browser): local GGUF readout backend—bin/jev-local.mjs+lib/local.mjs:one token out of a local
llama.cppserver, probability mass on each option label as the answer.No API key, no Python, $0.
fix(jev-browser): typeable ARIA combobox inputs, loopback runs priced at $0, chrome needs_user hint—three silent defects the first real local run exposed:
<input role="combobox">was not typeable(so
typewas never offered), local tokens were billed at the hosted 0.042/Mtok, and a closedchrome session was still announced as "handed to you".
feat(jev-browser): Kev 4B accuracy tier—bin/jev-kev.mjs: fetch + hash-verify thecheckpoint and its base, start the MLX server, print the env line.
feat(jev-browser): per-backend goal_done threshold profiles—thresholds.profile(
auto|hosted|local-readout|kev);autoclassifies a loopback endpoint from oneGET /v1/modelsat run start and falls back to the highest bar when it cannot.docs: measured local-backend results and Kev bring-up— the numbers, the methodology andthe raw datasets.
Measured (20 construction-graded items, same set for every row)
jev-kev.mjs)Bars are measurements, not defaults: 0.174 for the readout (band 0.111 – 0.273, maximin midpoint),
0.482 for Kev (band 0.341 – 0.683) — at the readout's 0.174 Kev calls four not-met pages a success,
which is why the band is per backend.
Verification
npm test: 76 pass / 0 fail / 5 skipped (81 tests, ~73 s). The knownego.test.mjshand-offflake did not fire.
Not in this PR
Qwen3.5-0.8B-Q8_0.ggufprobing blob at the repo root (model blobs are downloaded into~/.jev-browser/models, never vendored).*.logfiles underdocs/+experiments/— already covered by the repo's own*.logignorerule, so they are not force-added.