compute: resolve how GPU work is funded at runtime, and tell the agent honestly - #243
Draft
KB (KB-syntheticsciences) wants to merge 56 commits into
Draft
compute: resolve how GPU work is funded at runtime, and tell the agent honestly#243KB (KB-syntheticsciences) wants to merge 56 commits into
KB (KB-syntheticsciences) wants to merge 56 commits into
Conversation
Throwaway. Answers whether agent-proposes/Atlas-decides holds together, and who enforces an approved deadline. POST /leases and /release are faked because the real ones provision GPUs and bill. GET /options and POST /estimate are free, so [o] probes them for real to verify response shapes. Models the Atlas behaviour read from source but NOT verified at runtime, so each assumption is a toggle rather than a baked-in fact: gate mode (first-hour vs total), server-side deadline present or absent, client alive or dead, release succeeding or failing. Run: cd backend/cli && bun run prototype:guardrail
Rewritten budget-first. The earlier duration-first draft grew a warning event, an extension path, a per-lease expires_at, and client-side early release -- every one an attempt to make a run SUCCEED rather than to stop a bill running away, and each depending on something that does not exist: a completion signal for arbitrary SSH commands, a live agent outliving a long job, or checkpointing. Separating safety (don't overspend) from productivity (don't waste money on a truncated run) collapses the design to two small Atlas changes: 1. Make compute_grants.hard_cap_cents a real running cap. It is debited once for hour one today and never re-checked, so it reads like a spend ceiling and is not one. Re-debit per billing tick; exhaustion releases via the path that already fires when the wallet empties. 2. Accept an optional budget_cents on lease creation and size the grant to it. Optional matters -- the dashboard and atlas compute:up both call that endpoint without it. The agent proposes a dollar budget rather than a duration, because it can judge whether an experiment is worth $30 and cannot predict whether a novel training run converges in 4 hours. Budget exhaustion is then arithmetic the server observes, so no completion signal, extension path, or agent liveness is required. Also records two corrections: atlas compute:up DOES exist (cli commands.mjs:922), so the system prompt is not broken and that acceptance criterion is dropped; and compute:up already takes max_price and dry_run. Accepted cost, stated explicitly: a budget-exhausted job loses its work. Roadmap 56 (checkpointing) is the tracked follow-on. Item 61 splits out.
OpenScience decides how GPU work is paid for from a config value that never checks whether the user has any provider keys. computeBillingMode() returns config.billing?.compute ?? "byok", so a user with zero keys resolves to BYOK -- claiming BYOK with nothing to BYOK with. Replaces it with runtime detection, mirroring what billing.llm already does (its schema says 'unset = auto-detect from the resolved credential', backed by resolveCredentialSource; billing.compute says 'unset = byok', a static default with no detection function). Adds a third state. BillingMode is managed|byok, so 'the user has no keys AND managed compute is unavailable' has nowhere to live and silently resolves to a mode that cannot work. Making 'none' expressible is the point -- it turns a broken instruction into an honest 'connect a provider key'. Records what the prompt currently claims and why each part is false: the published CLI has no compute:up (the Atlas repo does, at the same version number), atlas doctor reports no compute field, and managed compute is off by default behind COMPUTE_RESELL_ENABLED. An agent in managed mode therefore runs an unknown command, cannot check the sanctioned availability signal, and is pointed at the user's own uncapped provider keys as the remedy. Scope is openscience only, no Atlas changes. The budget cap stays parked.
The first draft injected mode guidance into every turn for COMPUTE_AGENTS.
Wrong, for a reason that matters more than token cost: the mode changes
mid-session. A user connects a Modal key in Settings while a session is
running, and a reminder injected at turn 3 is false by turn 12.
Replaces it with a compute_status tool returning {mode, providers,
managed_available, guidance, balance_usd?}, resolved per call. The tool
DESCRIPTION carries the constraint that an injection was doing well
("check before running GPU work") at no extra per-turn cost, since tool
definitions are in every request anyway. So the description constrains and
the result informs; nothing needs injecting.
Resolving on demand also removes the boot-ordering footgun rather than
documenting it: provider keys arrive from three places, two of them injected
at src/index.ts:102 and :106 behind silent .catch handlers, so anything
resolved at startup can be wrong and can regress if someone reorders boot.
By tool-call time every injection has run.
This is the read half of roadmap 51/2 -- compute_status now, compute_submit
when managed compute is real and the budget cap is sound.
Drops the availability-cache-TTL open question, which was an attempt to paper
over staleness that the tool removes outright. Adds two: whether to keep a
one-line prompt pointer in case the agent never calls the tool, and what
timeout compute_status may block for.
…e provider
Folds in the dynamic-loading flow. Three changes.
Resolution point: SkillTool is defined with an async init that already builds
and filters its catalog, and registry.ts:187 calls t.init({agent}) inside
tools() -- so that init runs per request. Resolving there gives per-turn
freshness for free and makes the boot-ordering hazard unreachable by
construction, since every env injection at index.ts:102/:106 has long since
run by the time a turn is served. Both call sites must share one resolver;
two implementations of "which providers are usable" would drift silently.
Filter, do not auto-load. Only usable providers' skills are listed, so the
agent picks the right one because it is the only one offered. Auto-injecting
provider markdown would fight tool/skill.ts, which exists so content is pulled
on demand, and these files are large enough to be costly on turns unrelated to
compute.
A provider is usable only with a key AND a skill. RunPod and Vast have keys
that inject and no skill to load, so a RunPod-only user would otherwise
resolve to byok with an empty toolbox. They now resolve to managed/none with
an honest message. This surfaces roadmap item 5 rather than causing it, and is
added as an open question: write the skills, drop the providers, or label them
as pending.
…uildable The doc said 'approved, ready for implementation planning'. It is neither, and leaving that header on a shared branch invites someone to build it. Two independent reasons it cannot proceed. Managed compute is switched off -- COMPUTE_RESELL_ENABLED defaults to false, so every provider reports funding 'unavailable' and there is no live overspend to guard. And an independent review returned 'Unsound as written' with five verified findings, the worst being that lease_reaper terminates a silent lease at 600s, so a budget lease running SSH work would die having spent ~1.17 dollars and the cap would never bind. Also retracts a retraction. The doc's own 'corrections' section claimed atlas compute:up exists and therefore the system prompt needs no fix. It exists in the Atlas repo but NOT in the published 0.13.2 that the ^0.13.2 pin resolves to -- source and npm disagree at an identical version number. The original finding was right; the correction was the error. The prompt fix moves to compute-mode-detection-design.md. The analysis is kept, not deleted -- it remains the best record of how Atlas compute billing works, and the five findings become the implementation checklist if managed compute is ever switched on.
Establishes ComputeMode, the single shared rule for what makes a GPU provider "usable": a credential in process.env AND a catalogued skill. Tasks 2-4 consume PROVIDERS, SKILLS, and usable() from here. Mutation-proof (Step 5), each applied then reverted: 1. Delete `if (!keyed(spec.env)) continue` in usable() -> fails "a catalogued skill with NO key is not usable" (also collaterally fails 5 other tests, since every catalogued skill now counts as usable regardless of key) 2. Replace `spec.skills.some(...)` branch with unconditional `providers.push(id)` -> fails "a key with NO catalogued skill is not usable" 3. In keyed(), change `group.every(...)` to `group.some(...)` -> fails "modal needs BOTH token vars — id alone is not a key" 4. In keyed(), change `!!process.env[name]` to `name in process.env` -> fails "an empty-string key does not count as set" 5. Hoist the catalog Set to a module-scope memo, computed once -> "a key injected after the first call is seen on the next call" still passes, exactly as expected: env is read fresh every call, so that test can't see a stale skill catalog. It fails two other tests instead, via cross-test cache pollution (an earlier test's catalog leaks into a later one within the same run) — collateral damage, not a defect in this test. Freshness of the skill list itself is Instance.state's job; Task 4 covers per-turn catalog freshness. usable() intentionally does NOT memoize the catalog. All five mutations reproduced the outcomes named in the task brief; no test defects found.
…d 1)
Reviewer found the implementation spec-compliant with no logic defects,
but three test-design gaps in the Task 1 suite. All fixes are test-side;
src/compute/mode.ts is unchanged.
1. Two of modal's three skill names (modal-ml-training,
modal-research-gpu) had zero coverage — every modal test seeded only
modal-serverless-gpu, so the .some() disjunction was never proven to
resolve modal off its 2nd/3rd name. Added a test that seeds each name
alone (plus both Modal env vars) and asserts ["modal"].
2. The SKILLS test compared ComputeMode.SKILLS against
Object.values(PROVIDERS).flatMap(p => p.skills) — the exact expression
SKILLS is implemented as, so it could never catch a misspelled skill
string inside PROVIDERS. Replaced the derived right-hand side with a
hardcoded literal array of the eight required names.
3. No test ever had two or more providers keyed at once, so the
PROVIDERS-declaration-order contract on `providers`/`unusable` (which
Tasks 2-4 rely on) was unguarded. Added one test keying vast, modal,
lambda together (set out of declaration order) asserting
["modal", "lambda", "vast"], and one keying prime + tensorpool
(unusable, no skills seeded) plus lambda (usable) asserting
unusable == ["tensorpool", "prime"] — declaration order, not env-set
order.
Mutation-proof (Step 5 discipline), each applied then reverted:
- spec.skills.some(...) -> catalog.has(spec.skills[0]) (checks only the
first skill name)
-> fails "each of modal's three skill names resolves modal on its own"
(2nd/3rd iterations)
- Corrupt PROVIDERS.modal.skills[1] ("modal-ml-training" ->
"modal-ml-training-x")
-> fails both "each of modal's three skill names..." and "SKILLS
covers every name in PROVIDERS and nothing else" (single mutation
proves both Finding 1 and Finding 2's guards)
- Iterate Object.entries(PROVIDERS).sort(([a], [b]) =>
a.localeCompare(b)) instead of declaration order
-> fails both "providers keyed together return in PROVIDERS
declaration order, not set order" and "unusable providers also
return in PROVIDERS declaration order, not set order"
All three findings' guards held on first mutation; no test needed a
second-order fix.
…n override Adds the managed-availability probe against Atlas's /api/compute/options and the full ComputeMode.resolve() that turns credentialed providers plus that probe into one of byok/managed/none. A failed, unauthenticated, or timed-out (3s) probe always resolves toward none, never managed, so a degraded backend can't promise a capability nobody confirmed. A usable BYOK provider skips the network call entirely. billing.compute now acts as an override that can only narrow the outcome to none, never manufacture a capability that isn't there.
Round-1 review findings on the compute-mode-detection resolver: - No test proved a forced managed override narrows to none when the user also has a usable BYOK provider and the managed probe is unavailable. Added a test that sets both, asserts mode is none and providers still reports the credential. - The cache suite proved same-turn reuse and invalidate(), but never that the cache actually expires — a process-lifetime cache would have passed every existing test. Added a Date.now-stubbed test that crosses the 5s TTL boundary and reads a changed probe result on the other side; tightened the existing within-TTL test to use the same stub so the two bracket the boundary rather than overlap. - Folded the two byte-identical Resolution-construction blocks (managed-override arm, no-override/no-provider fallback) into a single funded() helper.
Nothing is injected into the prompt per turn because compute mode can change mid-session (a user connects a provider key in Settings while a session is running). The agent instead calls this tool before doing any GPU/training/cluster work; its description carries the constraint, its result carries the mode, usable providers, and mode-specific guidance.
Three fixes to test/tool/compute-status.test.ts: - The guidance-distinctness test compared whole `output` strings via a formatting artifact (text after a blank-line separator). A harmless refactor that drops the blank line while keeping three genuinely distinct GUIDANCE strings would have failed it for no reason, and it never actually asserted the guidance differed. Replaced with a contract check: each mode's short, load-bearing guidance phrase must appear in that mode's output and only that mode's output. - The no-skill byok test reused test 1's output-line mutation as its proof of non-vacuity. Retargeted to a distinct mutation this layer owns: deleting `providers` from the returned metadata object. The underlying skill-independence rule is owned and tested directly by ComputeMode in test/compute/mode.ts; this test is a passthrough check by design. - Nothing asserted that the managed balance comes from the same /api/compute/options call that decided availability, not a second round trip. Added call-count tracking (mirrors mode.test.ts) and an explicit assertion of exactly one call.
…b list Test 8 re-derived skill.ts's own offered-skills formula instead of comparing against real tools, and never actually called compute_status despite its name. Split it into two hardcoded-literal scenarios that invoke SkillTool and ComputeStatusTool directly and compare their verdicts, one credentialed and one bare. The ENV scrub list only cleared 5 of the 10 credential vars in ComputeMode.PROVIDERS, so a real PRIME_API_KEY or VAST_API_KEY in a developer's shell could leak into tests asserting an empty catalog. Brought it to the same 10-var list already used in test/compute/mode.test.ts. Also closed a coverage gap: the two "unaffected in every mode" tests never exercised byok, so extended both to a credentialed scenario.
Task 5 removed the atlas-doctor-as-compute-signal fallback from src/session/, but the primary research agent's own Stage 5 prompt still gated managed compute on `atlas doctor --format=json`, which reports CLI auth, nothing about compute. Managed compute is gated server-side on COMPUTE_RESELL_ENABLED (default false), so an authenticated CLI told the agent managed compute worked when it didn't. Stage 5 now calls compute_status directly and routes on its byok/managed/ none verdict. Also fixes a stale `modal` skill reference that no longer resolves under the Task 4 catalog filter (real name: modal-serverless-gpu). Widened compute-prompt.test.ts to scan agent/prompt/*.txt alongside session/**, and replaced the blanket atlas-doctor string check with a paragraph-scoped one so the legitimate atlas-doctor CLI-availability check at research.txt:81-84 stays permitted.
… test Round-1 review found the modal fix incomplete: the skill-index appendix in research.txt still listed bare `modal` at two more spots (the Inference & Deployment and Cloud Compute sections), which is a directory name, not the frontmatter `name` the skill tool resolves on. Corrected both to modal-serverless-gpu. Left the two prose mentions of "Modal" (capitalized, naming the company/platform, not invoking a skill) untouched. The guard test only matched the backtick-wrapped literal, so it never saw the appendix and would have missed a bare-`modal` regression in Stage 5 too. Replaced it with a check that extracts every modal*-shaped token from research.txt and verifies each against ComputeMode.PROVIDERS.modal.skills (the real source of truth), comparing the prompt's tokens against the map rather than the reverse, and renamed it to describe exactly what it checks.
SkillTool.init offered a credentialed provider's GPU skills whenever the provider was usable, even under a managed override where resolve() reports that same providers list for display only. A user with a connected key who switches Settings > Spend > Compute to Managed still saw that provider's skill offered, contradicting compute_status's "not funded here" guidance. It also called the full resolve(), whose no-provider path reaches the authenticated /api/compute/options probe (up to 3s) for a mode value the filter never read — a per-LLM-step network round trip for signed-in users with no GPU keys. Add ComputeMode.offered(): synchronous except for Config, exact in every state without the availability probe, because offered is non-empty iff resolve()'s mode is "byok" and that never depends on the probe. skill.ts now calls offered() instead of resolve(). Prove the equivalence with a matrix test across credential x override x managed-availability, asserting offered() matches resolve()'s byok arm exactly and never touches the network. Also replace the toothless "in managed, no BYOK skill offered" test (zero credentials, so providers was already empty regardless of the filter) with one that credentials a provider under a forced managed override, so it actually exercises the mode/providers disagreement the old filter got wrong.
…ntics billing.compute changed meaning from a static config default to a runtime- resolved override, but the settings API and UI never followed: the GET route coerced unset to "byok" (so the UI showed BYOK as active when nothing had been chosen), the PUT schema had no null case (so there was no way to set it back to auto), and the doc comment still said "compute defaults to byok". A user with no GPU keys and managed available would correctly resolve to managed, see BYOK apparently active in Settings, click Managed, reconsider, and click BYOK to "undo" it — only to persist billing.compute: "byok" and silently lose managed compute with no explanation and no way back except editing the config file by hand. Mirror what llm already does: BillingState/BillingPatch.compute is now nullable, readState() returns null instead of coercing to "byok", and the config.ts schema for billing.compute is nullable too (a persisted null must round-trip through Config's validation, the same way llm's already does). Add an "Auto" card to Billing.tsx's COMPUTE_MODES, matching LLM_MODES' shape and copy conventions, and widen the compute grid to fit the third card. Add settings-billing.test.ts coverage for the unset-round-trips-as-unset and PUT-null-sets-auto cases, plus a beforeEach reset of Config's in-process config cache — a read-only GET test would otherwise observe a previous test's in-memory state after that test's own afterEach had already deleted the file underneath it.
…b list Two assertions that couldn't fail: compute-prompt.test.ts checked config.ts as a whole for "auto-detect", but billing.llm's description (untouched by this branch) already contains that word, so the assertion passed regardless of what billing.compute's own description said. Scope it to the compute field's description specifically. compute-status.test.ts scrubbed 4 of the 10 credential vars ComputeMode. PROVIDERS actually checks, unlike the sibling skill-compute-filter.test.ts which scrubs all ten after a prior review caught the same gap there. An ambient PRIME_API_KEY, TENSORPOOL_KEY, VAST_API_KEY or LAMBDA_LABS_API_KEY in a developer's shell produced false failures in three tests. Derive the scrub list from ComputeMode.PROVIDERS instead of hand-typing it, so it can't drift from the sibling file's list again; verified programmatically that the derived list matches PROVIDERS exactly and that an ambient PRIME_API_KEY no longer leaks through.
Ran tooling/repo/generate.ts (openapi.json -> hey-api client codegen).
The generated openapi.json, sdk.gen.ts, and types.gen.ts still described
compute as 'managed'|'byok' non-nullable and claimed spend was billed "via
the bundled atlas CLI" with "Unset = byok" — both false since this branch
deleted the atlas CLI dependency and made compute an override with an auto
(null) state. Regeneration picks up config.ts's now-nullable schema and
corrected description everywhere it's duplicated in the spec (Config,
SettingsBillingGetResponse x2, SettingsBillingUpdateData).
The generator's repo-wide `bunx prettier --write .` pass (tooling/repo/
format.ts) also reformatted two unrelated files under pre-existing
formatting drift (backend/cli/src/compute/PROTOTYPE-guardrail-{model,repl}.ts,
whitespace/line-wrap only) — reverted before this commit, not part of it.
… proof Re-review round 2, three closing items: 1. (Minor, the only real defect) test/compute/mode.test.ts's offered() matrix test called resolve() first, which can warm the 5s availability cache; a subsequent offered() implementation that itself called available() would be served from that warm cache and record no fetch, passing the "no network call" assertion regardless. Proved by swapping offered()'s body for the equivalent-but-probe-hitting `resolve()`-then-filter form and confirming it passed 28/28 before this fix. Add ComputeMode.invalidate() right after resetting `calls`, so a probe-hitting implementation has nowhere to hide; reran the same swap and confirmed it now fails with the recorded /api/compute/options URL, then reverted the swap. 2. mode.ts's doc comment and the matrix test's title both stated offered() is non-empty "iff" mode is byok. False in one direction: a runpod- or vast-only credential resolves to byok with offered() empty, since those providers carry skills: []. Reworded to the two one-way implications the code actually guarantees. 3. "zero I/O, safe to call every LLM step" overclaimed. Config.get() is Instance-memoized but its first read per instance can fetch a well-known config URL (config.ts:82-105) when that auth type is configured. Reworded to say precisely what changed: the per-step Atlas compute-availability round trip is eliminated; Config.get()'s own (at most once per instance) cost is unrelated and unchanged.
… checklist Managed compute is switched ON in production (resell_enabled: true, four operator providers, 292 options), so the banner's load-bearing reason for parking - no live overspend to guard - is false. That was read from a default in config.py and contradicted by the deployed service: the fourth time this investigation drew a wrong conclusion from source, and the first made by the correction to the third. Folds in the design proposed this session and the edge case it raised: - change 0, the prerequisite: the lease reaper terminates any lease with no telemetry ~10 min after creation, and create_lease mints no runner token, so a user lease cannot prove liveness. A $30 budget dies having spent $1.17 and no budget can bind. Scope heartbeat staleness to leases that have a runner token. - change 3: a rolling window cap, because a per-lease cap does not bound sequential leases. - change 4: attach a persistent volume so exhaustion costs the compute rather than the work. Atlas already has a volumes API that leases do not use; RunPod gets volumeInGb, which dies with the pod. - change 5: budget extension, admitted only because its absence changes nothing about enforcement. - RunPod as the managed default: the only provider leaving no account-level key artifact. Records the Vast and Prime key leaks found alongside. Headline acceptance criterion is now the property the last attempt's tests and criteria both omitted: a budget of $B at $R/h lasts about B/R hours. The prompt half of the original problem is resolved and marked so.
The banner said 'third time' while the lesson section says four. Four is right: CLI contents, whether the prompt was broken, whether managed compute was reachable, and the parked banner's own claim that reselling was off.
Ran the CLI from source: compute:up --dry-run --gpu h100 fails with HTTP 400 Unknown SKU because Vast's offer ids churn between the options fetch and the estimate call. Vast supplies 204 of 292 live options so it is nearly always the cheapest pick, making the default path fail while --provider lambda and --provider runpod succeed. No retry exists. Server-side selection fixes this and the duplicate-resolver problem at the same time.
The blanket 'read from source, not verified at runtime' caveat is now misleading - it would have readers distrust the live checks against production. Separates what was confirmed against thesis-synsc from what still needs confirming, since the doc's own history is four wrong conclusions drawn from source.
The mode-detection spec was shipped but still asserted a 'key AND skill'
rule that was overruled during implementation, a skill-name table that
matched nothing, and that managed compute was unavailable. The guardrails
spec was revived but overlapped it heavily. Two documents disagreed with
each other and with production.
compute-management-design.md is now the single spec:
Part 1 - mode detection, shipped, with the corrections folded in and a
table of what was verified against the running binary and backend
Part 2 - guardrails, to build, changes 0-5 with change 0 (the reaper
killing user leases at ten minutes) called out as a live-defect
prerequisite rather than a checklist item
Opens with production reality verified against thesis-synsc, because the
predecessors' shared failure was reasoning from source about a deployed
system - four times, the last made by a correction to the third.
Both predecessors keep their content as historical record behind a
superseded banner; the mode-detection banner names its two wrong claims
explicitly so nobody builds from them.
…sign
The three compute specs cross-referenced each other, two of them as
superseded, and the surviving one carried claims that source disagrees
with. Replace all three with a single document.
New in Part B, decided this round:
- Atlas resolves the SKU from {gpu, count, max_hourly_cents} and leases
atomically, instead of the agent picking from 292 options. Fixes the
Vast offer-ID race by construction rather than by client retry.
- Three tools (launch/list/release), not one with an action parameter,
so the permission rule can ask on launch and allow on list.
- The agent never holds key material: the one-time private key goes to
~/.config/openscience/compute/<lease_id>.pem at 0600 and the tool
returns key_path. Keeps it out of the transcript and compaction.
- Atlas is the truth for what is running; the .pem is the only local
state, because it is the only value that cannot be re-fetched.
- compute_launch prompts by default, silenced only by explicit config.
UX, not enforcement.
Every Part B claim now carries a file:line citation, verified against
the Atlas checkout at 7b0e9b6. That check corrected a predecessor
finding: the acquire-time grant debit IS rolled back (lease_manager.py
:77 and :209), just not before the first tick. The double-count trap
is real; the reasoning printed for it was not.
Self-review checked the endpoints instead of assuming them, and the
draft was wrong three times:
- The SSH private key is not one-time and not unrecoverable. Atlas
stores it encrypted on the lease row and GET /leases/{id}/connection
decrypts it for the owner (routes/compute.py:450-508), added so the
Compute tab could re-offer it after a reload. The .pem is therefore a
cache, not a record — OpenScience keeps no durable compute state at
all, and a deleted key no longer strands a paid box.
- ssh_port was missing from the flow. RunPod NATs SSH to a high port and
Vast uses an ssh-proxy port, so the connect string the spec showed
would time out on the two providers that matter most.
- GET /leases returns every lease for the user, not the running ones
(compute_repo.py:446). compute_list has to filter by status.
Also softens the Atlas CLI defect: printing the key without saving it is
a usability gap, not data loss, since /connection re-serves it.
…ise GPUs Adds what a comparable multi-provider aggregator does that this design did not, plus two gaps its page exposed. Change 10 — measure boot and availability, rank on it. Cheapest per hour inverts: boot is billed wall-clock, and published 7-day distributions put RunPod near a 59s median with a tight spread against Vast's ~1m9s median with a tail past six minutes. Cheapest-first sends most launches to Vast. The dataset costs nothing: compute_leases.ready_at already exists (migrations.py:555) and update_lease_status already stamps it (compute_repo.py:425-427) — nothing populates it only because nothing flips a GPU lease to ready, which change 0(a) fixes. Ranking becomes cheapest above a floor, degrading to pure price with no history. That same tail makes PROVISION_TIMEOUT_SECONDS=600 a hazard rather than a formality: one global constant across a 6x spread will reap legitimate Vast provisions. Now per-provider, derived from measured p99. The client's readiness poll must outlive the server's timeout and defer to its verdict — a shorter client bound kills launches that were about to succeed. Change 11 — pin the image. Nothing said what is on the box, and the answer varies: RunPod pins a CUDA devel image, Vast launches pytorch/pytorch:latest (vast_provider.py:220), Lambda and Prime set none. A floating tag means the same experiment run a month apart gets a different toolchain with no record. For a reproducible-research tool that is a correctness bug. Defines a minimum environment contract, pins per provider, records the resolved image on the lease. Canonical GPU map replaces substring matching, at SXM/PCIe/NVL granularity — the three H100 variants differ in throughput and price, so "h100" is not a usable resolver input. Unmappable options are excluded from ranking rather than guessed at.
…laims A second adversarial pass found more than the first. Every finding below was re-verified independently against source before this rewrite. SECURITY, live in production today. VastProvider.acquire posts each lease's public key to the shared operator ACCOUNT (vast_provider.py :206-211), and the module docstring states the purpose: "registers the public key on the Vast account so new instances pick it up" (:10-16). Managed leases all run on one operator credential (:78). So one user's private key opens another user's box. The spec called this a leak into the operator account — hygiene — and proposed delete-on-release, which cannot fix it: concurrently live leases have their keys on the account by construction. Now change 9(a), blocking, and the fix is to drop the account POST in favour of the per-instance attach already at :244-248. Change 9 was also unimplementable as written. Prime returns the pod name where the key id belongs (prime_intellect_provider.py:274) and that name is identical across all of a user's leases; Vast keeps no identifier; there is no column to store one; and "on failed launch" has no hook outside each provider's acquire. Change 10's "free dataset" was wrong three ways, all from correct citations. ready_at is stamped at poll time (compute_repo.py:408, :426-428) under a 60s sweep (config.py:68), against a signal that is a 10s difference in medians. The sample is censored at the timeout it was meant to derive. And no availability series exists: nothing ever writes 'failed' to compute_leases. Cheapest-first ranked on the wrong column. price_cents_per_hour is the raw provider rate on both funding paths; what the user pays is price_cents_per_hour_display, zero on BYOK (routes/compute.py:161-162). The resolver would have preferred a billed offer over a free one. Criterion 22 mandated shipping a regression: change 1 alone IS the ~23h regression, since the default-grant fix lives in change 2. They now land together. Also: catalog cache and orphan-lease reap promoted to changes 12 and 13 (both were prerequisites with no ticket); image pinning is impossible on Lambda and Prime, so non-compliant providers are excluded from ranking the way volume_id already narrows it; change 8's release_pending status would have kept billing and holding a concurrency slot; the readiness poll had no status vocabulary to poll on and no endpoint returning the timeout it bounds on; the fan-out is 5 requests not 10; global/index.ts :46 is the cache dir, not config — the cited path would have written a private key under a cache path.
Four tasks in the atlas repo, scoped to what every other compute change waits on: the cross-tenant SSH key on Vast, and the two reaper branches that kill a user lease ten minutes after it is created. Deliberately excludes budgets, the resolver, volumes and the OpenScience tools. Until a lease survives, none of them can be observed to work.
BYOK leases cannot be promoted so they still die at 600s; Vast's release claims terminated regardless of what happened; Prime leaks an account SSH key per lease and stores the pod name where the key id belongs; Lambda's booting status reads as unknown. Volumes and the catalog cache are excluded: both are full spec changes needing design decisions, not defects.
/api/compute/options reports a managed provider whenever reselling is on and an operator key exists, independent of wallet balance. Live testing against a deployed backend with a zero wallet showed compute_status telling the agent to run GPU work through managed compute while also forbidding the only fallback (the user's own keys) -- every lease acquire in that state returns HTTP 402 insufficient_cli_credit. Same defect class Part A removed from mode resolution, one layer up in the guidance text. Make the managed guidance a function of balance: unaffordable (balance exactly 0) now says the wallet is empty, launches will be refused, and to top up or connect a provider key. The resolved mode is untouched -- managed capability genuinely exists, only the funds don't, and that distinction needs different user-facing advice than "none" would give.
package.json publishes "src", so both PROTOTYPE-guardrail-*.ts files (~492 lines) shipped to npm despite their own header saying "throwaway, not wired into the product, delete or lift, don't ship". The repl also read ATLAS_TOKEN and made live authenticated fetches, and both files were typechecked on every build. Nothing imports them; e12e486 keeps them in history if the state machine is ever lifted.
…nmeasured probe Resolution gains `origin` — "environment", "config:byok" or "config:managed". Without it a caller cannot tell "managed because the environment says so", where connecting a provider key flips the next call to byok, from "managed because billing.compute pins it", where connecting a key changes neither resolve() (funded() never reads providers) nor offered() (empty under a managed override). Those two states need opposite advice, and the guidance layer has been giving the first state's advice to both. The override VALUE is carried rather than a forced/not-forced bit because "none" is reachable from both overrides. `managed` becomes optional. Both byok arms returned `managed: false` without probing — the skip is deliberate (a byok user never pays for the round trip), so the answer was never measured. Absent now means "not checked".
… not have Three strings claimed something untrue. Managed with a positive balance said "Run GPU work through managed compute" and "Do not use the user's own provider keys". There is no managed launch mechanism here: ComputeTools is [ComputeStatusTool], the only /api/compute call in the product is mode.ts's read-only /options probe, and compute_launch/list/ release are Part B, unbuilt. A signed-in keyless user with a funded wallet — the default path — was told to do the impossible and forbidden the only fallback. It now says managed is funded, that OpenScience cannot launch it, and that connecting a provider key is the working path. `none` offered a top-up. No path to none is balance-related — probe() returns managed:false only for no session, non-2xx, a network/parse failure, or no provider with funding "managed", and Atlas reports managed regardless of balance — so topping up could never move a user out of none. Both remedies assumed the user could act. Under an explicit billing.compute override they cannot, so the advice now switches on the new `origin` and names the setting instead. Availability is printed tri-state. The byok arms never probe, so "managed available: no" was an unmeasured assertion; it now reads "not checked" and metadata.managed_available is absent rather than false.
Stage 5 gated skill loading on compute_status and then, two lines later, told the agent unconditionally to load `modal-research-gpu` and `modal-serverless-gpu` — precisely the names ComputeMode.SKILLS hides when Modal is not credentialed. The unconditional lines won. They are folded into the byok branch, and the `managed` branch no longer says to run the work through managed compute, which the client cannot do. The guard test is written over ComputeMode.SKILLS and every COMPUTE_AGENTS prompt rather than over research.txt, so a sibling prompt cannot reintroduce it.
research.txt and biology.txt both said "then restart openscience", which contradicts this branch's central claim and the test that asserts it (compute-status.test.ts, "a credential connected between two calls changes the answer, no restart"). The Compute and Credentials panels call applyComputeEnv/applyCredentialEnv on save, and a key added in the hosted dashboard lands via refreshIfStale's background sync on the next message. Both lines now also point GPU provider keys at Settings ▸ Compute, the surface compute_status's own guidance names. The bans (compute:up, restart) now run over every COMPUTE_AGENTS prompt instead of research.txt alone. Widening them caught a third file the review had not listed: ml.txt's "in Settings → Credentials and restart". Its skill-by-directory names (`modal`, `tinker`, `tensorpool`) are a separate known defect and are untouched.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
KB (KB-syntheticsciences)
marked this pull request as draft
August 1, 2026 12:35
bubblewrapArgs mounts the whole fs read-only and deliberately refuses $HOME as a writable root (tooBroadToConfine), so any tool that writes to ~/.cache on startup — zsh's compdump/history lock, pip, npm, uv — fails with "Read-only file system" inside the sandbox. Mount a tmpfs over the resolved XDG cache dir (XDG_CACHE_HOME, else ~/.cache) after the root ro-bind and before the policy.writable binds: writes succeed so the tools stop erroring, but nothing persists to the real home, so the containment tooBroadToConfine enforces is untouched.
Changes 1 and 2, which must land together: making the cap real turns the existing rate x 24h grant sizing into a live ceiling, and the one-hour debit taken at acquire would kill a no-budget lease an hour early. Central decision recorded: the grant update is a set-to-total, not an increment, mirroring the tick's own cumulative charge. That gets replay safety for free, supersedes the acquire debit rather than double-counting it, and preserves the full plan TTL.
…wrong The spec prescribed "the billing tick must re-debit the grant". An increment cannot mirror a tick that charges cumulatively, and following that literally rebuilds the double-count it warned about: measured by mutation, a $10 budget at $6.99/h dies at 26.0 minutes instead of ~1.4 hours, with grant.spent_cents reading 300 where wall-clock is 180. The correction — a set-to-total — is now recorded above the original text rather than replacing it, since the reasoning is the useful part. The premise was confirmed in production before the fix (a 34c/hr lease with hard_cap 816 and spent frozen at 34) and the property confirmed after it (release at 5160s against a theoretical 5150s). The "confirm the money path against pytest" caveat is discharged for change 1; it still stands for the resolver, volumes and the rolling cap. The spawn-path known defect is closed, and was worse than recorded: 500c flat for every SKU, killing a 4-hour A100 at 1.38h and refusing an H100 at acquire outright.
06-compute-integrations.md carried a 'Correction to the initial audit' asserting that atlas compute:up is a real command in the published 0.13.2 and that this repo pins @synsci/atlas@^0.5.12. Both are false. package.json:123 pins ^0.13.2, and npm pack @synsci/atlas@latest resolves to 0.13.2 with zero files containing compute:up or compute:lease. The commands live only in the atlas repo -- 3e1d1ca removed them, 0.13.1 and 0.13.2 shipped without them, 205bbc0 re-added them with no version bump -- so source and artifact disagree at an identical version. It is a release problem, not a pinning problem. The retraction is kept in place rather than deleted: a source-read conclusion about a published artifact, checked against neither, is the useful part of the record. Path C was also overtaken. aa9b314 merged an SSH/Slurm/PBS job dispatcher to main on 2026-07-29, two days after the 52845c3 audit baseline, so the 'store-only dead-end, no dispatch' verdict is wrong for SSH hosts and still right for model endpoints. The same commit falsifies ROADMAP notes on 2, 3, 51, 52, 58 and finding 1; those notes are corrected and dated, and no status mark is re-graded, which would need a fresh audit and a recount. ROADMAP 5, 55 and 103 gain what the Atlas guardrail work changed and, more importantly, what it does not: it sits on unmerged draft PRs and OpenScience still has no tool that can launch a lease, so nothing meets this document's own bar for DONE.
… premise All three ran to completion on atlas feat/compute-lease-prerequisites, which is still an unmerged draft PR. Their bodies are kept as written -- they are the record of what was planned -- and each gains a banner stating where it ran, what came out, and what execution proved the plan wrong about. The banners carry what the plans could not know. Promotion needed a reachability gate, justified by RunPod returning desiredStatus=RUNNING with no address, and the first version of that gate broke Modal. A destroyed Vast instance read as provisioning forever. A review finding demanding a 404 was overturned by a live probe: Vast never 404s. Deploying exposed a migration race across all 22 ADD COLUMN sites. A catalog-stability heuristic with perfect retrospective separation failed its first prospective test and is retracted. The budget-cap plan also gets an inline correction it needs to stop misleading a future implementer: Task 3 asserts the effective-balance lookup is already in scope in create_lease. It is not. create_lease never reads the balance -- the wallet check is in lease_manager.acquire_lease, and the adjacent compute_estimate is what has it in scope. Clamping required a new conditional lookup.
… live testing measured Changes 0, 1, 2, 8 and 9 were listed as shipped in the Part B heading, but only 1 and 2 carried a banner -- so sections 0, 8 and 9 still read as unbuilt work with instructions to go and do them. Each now says what shipped and, more usefully, what execution forced that the section never specified: promotion needs a coordinate gate because RunPod reports RUNNING with no address, and that gate needs an ssh_key_name clause because Modal has no SSH at all. Change 8 is deliberately NOT marked shipped. The honesty half is done across all three providers and release_lease now acts on the verdict, but an unconfirmed teardown leaves the row matching both list_active_leases and count_active_managed_gpu_leases -- so the user who asked to release keeps being charged and keeps burning a concurrency slot, which is verbatim the failure the section says to decide against. Criterion 16 is one-third met. Change 9 is the one item here proven adversarially rather than inferred: two boxes on one Vast operator account, each key refused against the other's. Measured facts folded in where they change a decision. Vast offer ids churn ~50% between consecutive catalog fetches, and a stability heuristic with perfect retrospective separation failed prospectively -- retracted, retry is unavoidable. RunPod's no-capacity 500 is mapped to the same 400 as Vast's stale offer, and the two need opposite responses, so the resolver must discriminate on the message. RunPod's own catalog query asks for a price list, not an inventory. The third-party '>6 minute Vast tail' did not reproduce; successful boots ran 34-50s and the real failure mode is 'never boots', which a longer timeout cannot fix. Ranking by reliability2 beat a 0.98 threshold. The bounds table is rewritten: five of seven rows had changed, and change 5 is now the sharpest gap, since every remaining bound is per-lease.
Changes 12 and 3, planned together because of one load-bearing interaction the spec does not cover: a cache cannot make offers fresher. Vast churns ~50% between consecutive fetches, so a cached catalog is half-stale within seconds exactly as an uncached one is by the time a caller acts. The cache is a cost fix; the resolver's retry must bypass it or it re-reads the same dead offers forever. Scoped by what live testing measured rather than what the spec assumed: both failure modes arrive as 400 and need opposite responses, and the stock/reliability signals are deliberately left out because the one heuristic we tried was falsified prospectively.
…e plan The resolver reads no provider error message: it excludes already-refused (provider, sku) pairs and re-resolves against a fresh catalog, which gets Vast and RunPod right with one rule and cannot rot when a provider rewrites its prose. The plan prescribed message matching; the code is right. Ranking on the display rate alone was degenerate within BYOK, where every row displays 0 -- the order collapsed to alphabetical and leased a $9.00/h box over a $3.00/h one. Key is now (display, raw, provider, sku). Also records what the canonical GPU map does not cover: 37% of live Vast rows and 44% of RunPod's, missing B300, MI300X, GH200 and the workstation line, with Prime Intellect unmappable only because list_options drops the offer's socket.
Three real leases through the real route. The path works end to end: resolved in 5.1s, ready with SSH coordinates at t+30s, /connection 200, released and confirmed, one grant, nothing left running. Two findings that change what we can claim. Vast caps every query at 64 offers and ignores limit entirely, so 'cheapest' is cheapest of a narrow window rather than of Vast. And the 201 names the offer id but never the GPU, so a caller that asked for an RTX-3090 cannot tell from the response that it got one. Also records a theory that was tested and refuted: the churn is NOT mostly an artifact of our cheapest-per-(gpu_name, count) dedup. 35 of 42 dropped rows were gone from Vast's raw response too. The retry's premise stands.
… live A lease carries gpu_model (canonical id, NULL rather than a guess), gpu_name (the provider's own string) and gpu_count, on all three launch paths. Verified against real Vast hardware. Also records the retry firing in production conditions for the first time: RunPod's no-capacity refusal, the offer excluded, a fresh re-resolve, and an honest 503 naming what was tried.
…ets tools Four defects found by live probing plus one decision already taken. The rate limiter meters reads from the launch budget, so an agent rate-limits itself out on a single launch-and-wait. A Vast 429 empties the catalog rather than degrading. Prime Intellect is unreachable for a one-line reason. An unconfirmed teardown keeps billing the user, and both docstrings claim a reaper retry that does not exist. Also records that Vast has no pagination -- offset and from both 400 -- so widening past the 64-offer window is a design change, not a parameter.
Measured: the cheap query is 3% usable, the premium query 98%. The difference is that premium filters by name and cheap does not -- it sorts by price, and Vast's cheapest inventory is mid-tier consumer cards, of which the map holds exactly three. Task 5 now derives the query from the taxonomy instead of maintaining a second list beside it. Task 3 stays datacenter-only. Widening to the mid-tier was reconsidered and rejected: all of them are 16GB or less, the map already covers 3090/4090/5090, and VRAM is the binding constraint for this product's work.
That measurement compared two multi-name query windows, which is only sound if a window is the deterministic cheapest-64 of its filter. A single-card query found 45 offers priced below the shared window's own ceiling that the window did not contain -- impossible under that model. But it does not generalise either: a narrow num_gpus in [1] query run twice 1.5s apart returned identical results, 64/64 shared ids. So the window is a sample for some query shapes and not others, and neither 'the window is a sample' nor '36% of offers die per 40s' is established. The retry is unaffected -- a SKU that fails to launch must be replaced whatever the cause. What is not established is WHY a SKU goes stale, which any future staleness or stability heuristic would need measured first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The client half of compute funding. Resolves
byok | managed | nonefrom the actual environment and reports it through a tool the agent pulls from.Pairs with synthetic-sciences/atlas#208, which makes a managed lease survivable. Neither depends on the other to merge.
What was broken
computeBillingMode()returnedconfig.billing?.compute ?? "byok"and never looked at the environment. A user with zero GPU credentials resolved tobyok— claiming BYOK with nothing to BYOK with — and "no compute is available" had no representation at all. Meanwhile the prompts pointed the agent atatlas compute:up, which is absent from the published CLI, and atatlas doctor, which reports no compute field.What this does
managedwould reproduce the original bug of promising an unconfirmed capability.billing.computebecomes a nullable override that may narrow tononebut can never manufacture a capability. Nullable end-to-end, including an "Auto" card, because without it a user who picked BYOK with no keys was trapped innonewith no way back.compute_statustool. Nothing is injected per turn: the mode changes mid-session, so a reminder written at turn 3 is false by turn 12. The tool description carries the constraint; the result carries the specifics.byok. This is a listing filter, not a gate — a hidden skill is still loadable by exact name and the agent still hasbash.Honesty fixes found by live testing
Two survived until the client was pointed at a real backend:
At a zero wallet the tool told the agent to spend it. The backend reports providers as managed regardless of balance, so
compute_statusemittedmanagedwith$0.00and said "run GPU work billed to Credits" while every lease attempt returned 402.With a funded wallet it told the agent to do something this client cannot do. There is no managed launch path here — the only
/api/computecall in the product is the read-only probe — yet the guidance said "run GPU work through managed compute" and forbade the BYOK fallback, leaving the agent with nothing. The Compute settings panel says the same thing from the product side: managed accelerators are coming later.Both now say what is actually true, and the mode is left alone in each case: an empty wallet is missing funds, not a missing capability, and the two need different advice.
Also
research.txthad unconditionalLoad: modal-*directives sitting directly under the conditionalcompute_statusblock that governs themTesting
bun test test/tool/ test/compute/ test/session/compute-prompt.test.ts→ 175 pass / 0 fail.Verified against a real deployment: the probe resolves
managedwith a live balance, and refusal at a zero wallet returns a structured 402.Known gap
A wallet of one cent still gets the funded guidance, though acquiring requires a full hour of the SKU's rate up front. Closing that honestly means threading the cheapest available rate through the options endpoint so the tool can compare against a real number rather than zero.