feat(loading): bounded priority-ordered boot fetch + body-texture atlas#505
Merged
Conversation
The load-order grill surfaced work that is adjacent to the boot scheduler but deliberately outside it. Capturing each so the spec can stay scoped to the delta it actually needs: - scale-gated asset demand: ~68 MB of the ~101.7 MB boot fetch renders nothing at the Earth boot view, because surveyDeepZoom fades glade, sdss, milliquas and mcpm to zero below 0.002 Mpc. Worth more than the scheduler itself, so it gets its own design. - filaments + flow lack SCALE_FADE_BANDS rows, which blocks their participation in the gate above. - famous_stars_meta.json fetches at React mount outside ASSET_WIRING, though it is only needed when a star InfoCard opens. - jupiter/saturn 404 on the large texture tier: the registry declares a ceiling for files that do not exist on disk. - dead files in public/data/. - body-texture store consolidation: four renderers each hand-roll map storage and placeholder fallback. Also narrows the existing asset-loading audit item: its "what loads, when, and how big" half is discharged by the grill transcript, leaving the eviction policy and the debug UI sweep.
A cold boot fetches ~101.7 MB at default settings with no ordering and no concurrency limit, so every asset starves every other and the first view takes far longer than it should. The loading system was designed when the app had no stars and no textured planets. The design is three pieces: - a bounded-concurrency priority queue at the AssetSlot fetch layer, reusing PriorityQueue, N=2, never preempting. The bound IS the mechanism: under HTTP/2 every request fires at once and shares bandwidth, so reordering without a limit changes nothing. - one static priority integer per ASSET_WIRING row, payload size folded in by the author. No positional input at all. - a single sRGB atlas of 512x256 surface tiles for the 13 registry bodies, cropped into the existing per-body GPU textures at upload. The atlas is a transport format, not a sampling format, so no shader, binding or uniform changes. The atlas is what lets the scheduler stay simple: a universal low-res set means no body is ever untextured, which removes the need to predict where the camera is heading and deletes the destination-tracking the design originally carried. Includes the 20-question grill transcript with the rejected alternatives, and a Ground preparation section recording the refactor-ground checkpoint: four prep refactors (P1-P4) sequenced before the feature commits, all on one PR.
Sequences the spec into commit-sized tasks. Part 1 carries the four prep refactors and the bounded queue; part 2 the body-texture atlas and the closing entanglement-radar and verification passes. Prep commits land before feature commits, all on one PR. The five correctness landmines are explicit task steps rather than prose, because each one passes every test while being wrong: - popHighestPriority pops the LARGEST priority, so the enqueue site negates; backwards, boot fetches in exactly reverse order. - the drop trigger is a new idle && !demand edge, not release(), which cannot see a queued-but-unstarted slot. - rank 6 needs distinct integers or ties fetch large-before-small. - copyExternalImageToTexture's origin is in unflipped source coordinates while setMap uploads with flipY, so a wrong assumption mirrors the planet and samples the wrong tile row. - load()'s promise must resolve on every terminal path or an abandoned load pins its queue slot. Writing the plan surfaced two places the spec under-estimated. The reevaluateDemand tests build a cast EngineState with no subsystems bag, so the new queue lookup throws into the per-row catch and takes every assertion down with it. And earthRenderer's setMap destroys the cell it replaces, so P4b needs a real committed/placeholder split rather than just accepting a bitmap, or a late atlas tile would destroy Blue Marble.
The frame's flat-vs-textured split inferred "this body has a texture" from the bodyTextures slot's current() being non-null. That proxy holds only while every bound texture has a slot behind it; a shared body-atlas tile breaks it, and the naive fix (atlasReady || slotReady) would be a second branch on one discriminant. Residency is a rendering fact, so hasMap(body, kind) on TexturedBodyRenderer answers it from what is actually bound. Behaviour is unchanged today: a committed slot is exactly a body with a committed map. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the five per-kind `let` cells and `setMap`'s five-branch retirement ladder into two per-kind maps: `placeholders` (the 1x1s, seeded at construction and freed only at teardown) and `committed` (the real map once uploaded). `buildBindGroup` resolves each map binding as committed-over-placeholder, so a map arriving out of order can no longer destroy the texture already on screen — `setMap` retires only a prior COMMITTED texture of the same kind. Binding numbers, the placeholder texels, and the layout entries now come from one `KIND_CFG` table (mirroring texturedBodyRenderer), which also removes the parallel five-branch texture-label ternary. Behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a REQUIRED `priority: number` to `AssetWiringRow` (lower is fetched first, payload size folded in by the author) and sets the concrete integer on every row in `ASSET_WIRING`. `pointRow` takes the rank as a parameter because the eight galaxy catalogs do not share one; `bodyTextureRow` and `starCatalogRow` each carry a constant. The six bulk-survey ranks 60-65 are distinct on purpose: `popHighestPriority` breaks ties by first-encountered, and the registry array's order would then fetch GLADE before Milliquas. The two stub-row test files gain `priority: 0` so the required field compiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ommit Makes the already-async work visible: load() now returns a Promise<void> that resolves after commit completes or after any terminal early exit (abort, give-up, superseded race-check), and never rejects. Every return inside runLoad is a plain return in one async function, so returning it satisfies every terminal path — this is the seam the boot-time load queue needs to bound in-flight concurrency correctly. forceReload() and the direct .load() call sites outside the queue's scope (tier transition, companion assets, debug volume) mark the call `void` now that it returns a promise they don't await. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Boot load priority prep (P1): the queue's concurrency bound was a module- global constant, so every consumer was forced to the same limit. The asset queue wants 2 (a few big one-shot boot fetches shouldn't flood the connection pool), while the thumbnail queue wants 4 (many small streaming fetches as the camera moves). Make the limit a constructor argument defaulting to MAX_CONCURRENT_FETCHES so galaxyAtlasSubsystem stays on the current default while future callers can choose their own bound. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dyRenderer The bind group resolved an uncommitted sphere map as `res.maps.get(kind) ?? placeholderMaps.get(kind)!`. A per-BODY stand-in (an atlas tile) would make that a three-term chain restated at every rebuild site, with eviction having to know which term to fall back to. Split the placeholder into two layers instead: the shared 1x1 per-kind textures stay the default, and `BodyResources.placeholders` holds an optional per-body override. `placeholderFor(bodyId, kind)` picks between them, so the chain stays two-term and `clearMap` lands on whatever the resolver gives at rebuild time. `hasMap` now reports the union of the two layers: a body drawn from an override is textured. `destroy` frees the override textures alongside the committed maps. No override is seeded yet, so behaviour is unchanged; the seeding entry point lands with the body-texture atlas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sceneBodyPartition now asks state.gpu.texturedBodyRenderer.hasMap(id, 'surface') for residency instead of the bodyTextures asset slot. Update the fake texturedBodyRenderer/planetRenderer fixtures in texturedBodiesLayer and planetsLayer's tests to express residency via hasMap, dropping the now-dead assetSlots.bodyTextures seeding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds drop(key) and destroy() to PriorityQueue. drop() removes a PENDING entry only, never touching inFlight — responses aren't resumable and the spec's "never preempt" decision is explicit. destroy() clears every pending entry and resolves outstanding drain() callers so teardown can't hang, while letting in-flight tasks run out on their own. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds ASSET_QUEUE_CONCURRENCY (N=2, distinct from the thumbnail queue's MAX_CONCURRENT_FETCHES) and constructs state.subsystems.assetQueue eagerly alongside scheduler/fades, so a future evaluateRows can enqueue boot fetches before the GPU init IIFE finishes. Wires its destroy() into engine.destroy(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spec test 1 for PriorityQueue ("runs at most the constructed limit
simultaneously") was lost in a git stash collision between two
concurrent agents working task P1 of the boot-load-priority plan; the
plan's checkbox was ticked despite the test never landing. Re-adds it
per the plan's P1 section: 6 tasks on new PriorityQueue(2), gated on
promises, asserting maxInFlight is exactly 2 (not <=, since a <=
assertion would also pass for a queue that serialises everything).
Also confirms the P1 repair item survived: the pre-existing
MAX_CONCURRENT_FETCHES test already constructs its queue explicitly
as new PriorityQueue(MAX_CONCURRENT_FETCHES).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The body atlas is a transport format: tiles are cropped out of the one decoded bitmap into each body's existing texture at upload. That crop needs a rect in unflipped source coordinates, and with the ring dropped every cell is the same 512x256, so the rect derives from the tile index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One 2048x1024 sRGB WebP holding a 512x256 surface tile per registry body, composited from the small tiers the same run writes, plus the generated layout the runtime looks a tile index up in. Membership derives from textureBuildEntries() filtered to `surface`, so the registry stays the one enumeration of the textured set and the atlas cannot go stale behind a re-curated body. A body with no source on disk keeps its index and leaves a mid-grey cell, so a --dev fetch never shifts every later body onto its neighbour's face. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`evaluateRows` no longer calls `slot.load()` directly. A demanded idle row is enqueued onto `state.subsystems.assetQueue` with `priority: -row.priority` (the queue pops the LARGEST priority, the wiring rank table reads lower-first), and the loop grows a third edge: an idle row whose demand went false DROPS its pending entry, which the `ready`-gated evict edge structurally cannot do. Both the idle guard and `req(state.tier)` are re-evaluated inside the enqueued closure, so a slot claimed by a direct `.load()` in the gap is left alone and a tier change while an entry sits pending yields the request for the tier in force when it runs. Test repairs: six state fixtures across five files gain a per-call `assetQueue`; the cases whose assertions count demanded rows rather than concurrency now drain the queue first. `demandTable`'s `firedKeys` re-runs the loop to a fixpoint, since `famousMeta` demands on the Famous row having started and the queue defers that start past the pass that enqueued it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`setPlaceholderMap(bodyId, kind, atlas, rect)` crops one tile out of the low-resolution all-bodies atlas into the body's per-(body, kind) placeholder override layer, so a body reached before its hi-res map lands shows its own surface rather than flat grey. The atlas is a transport format, not a sampling format: the tile is cropped at upload into an ordinary per-body texture, so nothing about the shader, the bind-group layout, the sampler or the UVs changes, and the atlas bitmap is never bound. Writing the override layer (never `maps`) is what makes arrival order irrelevant — a committed hi-res map shadows the tile by construction, with no slot-state peek in any commit path. The crop uses `copyExternalImageToTexture`'s source `origin`, which the spec pins to unflipped top-left source coordinates while `flipY: true` flips the selected region alone; the comment at the copy states that convention so the next reader does not re-derive it. Still needs a visual pass (a wrong assumption here is invisible to the mock device). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`setPlaceholderMap(kind, atlas, rect)` crops one tile out of the low-resolution all-bodies atlas into that kind's placeholder slot, replacing the 1x1 mid-blue, so an Earth reached before the multi-megabyte Blue Marble lands shows a recognisable low-res Earth rather than a plain blue ball. The tile lands in `placeholders`, never `committed`, and each setter frees only its own layer: `setMap` retires a prior committed map, `setPlaceholderMap` a prior placeholder. A tile arriving after the hi-res map therefore cannot clobber it, and neither commit path has to ask which one landed first. Earth has no `clearMap` and never evicts, so that out-of-order protection is the only thing this site buys from the two-layer split. Format comes from `isLinearTextureKind` exactly as `setMap` does, so the tile and the map that later shadows it can never disagree on sRGB-vs-linear. Only `'surface'` has an atlas tile; the other four kinds keep their 1x1s. The crop follows `texturedBodyRenderer`'s convention (8c27868): the source `origin`, which the spec pins to unflipped top-left coordinates while `flipY: true` flips the selected region alone. Still needs a visual pass — the mock device rasterises nothing, so a mirrored or wrong-row crop is green in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…3 placeholders Adds the `'bodyTextureAtlas'` asset: a singleton sidecar key alongside `famousMeta` / `constellations` / `flow`, with a named `EngineAssetSlots` field, its own fetcher and slot factory, and an `ASSET_WIRING` row at priority 0 — the head of the rank table, because the whole point of the asset is to arrive before any hi-res texture. Everything the previous tasks landed was inert until this: the atlas is now actually fetched (~160 KB, unconditionally at boot, deliberately NOT proximity-gated) and its one decoded bitmap seeds every textured body's placeholder layer, so a body reached before its own multi-MB map lands shows its own surface instead of a flat albedo sphere. The fan-out derives its membership from `BODY_ATLAS_LAYOUT` (generated from `BODY_TEXTURE_REGISTRY`, the only enumeration of the textured-body set), and routes Earth to `earthRenderer` and the other twelve to `texturedBodyRenderer` — the same split `commitBodyTexture` already performs. Both sinks write the PLACEHOLDER layer, never the committed one, so an atlas arriving after a hi-res map cannot clobber it and no commit path has to ask which landed first. Renderers cannot be late either: initGpu constructs both a phase before wireSlots mints this slot, and `resourcesFor` mints a body's GPU resources lazily, so seeding a body that has never been drawn just creates them early. The null-guards are the usual destroy-race posture, not an ordering workaround. Error posture is silent-optional-asset: a 404 or decode failure warns and leaves every renderer on the 1x1 it drew before this feature existed. No new test — the wiring is compile-checked end to end and the result is pixels. The one test edit repairs the ASSET_WIRING membership pin. Still needs the visual pass: a cold load should show every visible body textured from the first frame it is drawable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two findings from the body-texture visual pass, neither caused by the load-priority work: - a thin fully-transparent ring between a body's surface and its atmosphere shell (seen on Mars). Filed as needs-investigation with a geometric hypothesis and a falsifiable first step, because a different cause points at a different fix. - Mars reads over-saturated. Filed as the general question rather than a Mars fudge: the `sss` sources are enhanced rather than colorimetric and no target appearance is recorded anywhere, so a design has to pick a reference before "calibrated" means anything. Notes that any correction must also hit the atlas tiles, or a body's placeholder and its hi-res map will shift hue mid-load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`body-atlas.webp` was written twice: a string literal in `bodyAtlasFetcher` and `BODY_ATLAS_FILENAME` in the emitter. A mismatch degrades as a silent 404 — the atlas never arrives, every body falls back to grey, and neither a test nor the compiler can see it. The emitter now copies its constant into `bodyAtlas.generated.ts` beside the layout and the grid, the two other facts both ends must agree on, and the fetcher builds its URL from that. Filename and bytes are written by the same call, so they cannot drift — the guarantee `bodyTextureFilename` already gives the per-body tiers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The row-major index→pixel derivation was spelled twice: `atlasTileRect` for the runtime crop, and an inline `left`/`top` computation in the emitter. A tool that packed column-major while the runtime read row-major would hand every body a neighbour's face, with the atlas otherwise valid and nothing to catch it — the same failure class as the filename, in the same file. The emitter now calls `atlasTileRect`, which it can: it already imports `bodyTextureFilename` across the src/tools boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…witch `isTextureResident` stopped meaning "the `bodyTextures` slot holds a committed bitmap" when residency moved onto `texturedBodyRenderer.hasMap`; the docblock still said it did, which is now the exact inference the switch exists to forbid. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The entanglement-radar sweep over the boot load-priority branch found three knots that are real but wider than this feature: - `ASSET_QUEUE_CONCURRENCY` is not the system-wide bound it reads as — five call sites still load outside the queue. - "famousMeta rides Famous" is authored three times, in three mechanisms none of which implies the others. - The `bodyId === 'earth'` commit routing is now branched in two places; appended to the body-texture store consolidation item, whose sink table would subsume it. Ticks 4.1 in the part-2 plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The asset-loading section rendered slots in engine-mint order with only terminal state, so reading the boot fetch order off it meant knowing ASSET_WIRING by heart — and a large asset that was dequeued first still looked "late" because only its completion was visible. Rows now sort by the authored ASSET_WIRING rank (lower fetches first) and carry that rank in its own column, plus a `start→done` timing column in seconds from the first fetch start. Start order and completion order are therefore separately readable. The rank map is derived from ASSET_WIRING via slotFor, not restated: a static slot-name → rank table would be the same numbers in a second place. Slots gain `startedAtMs()` — a load-attempt-scoped cell beside `lastRequest()`, since `reduceLoadState` is clock-free by design. AssetLoadingTitle and SlotRow move to their own files with their own CSS modules, sharing the five LoadState colours through a composed `loadStateColors.module.css`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
evaluateRows enqueued its loads row by row, and PriorityQueue.enqueue starts a task the instant a slot is free. So the first N demanded rows WALKED started immediately in ASSET_WIRING array order, before any better-ranked row further down the table was evaluated; priority only governed slots that freed later. That let rank-60 SDSS, second in the array, hold one of the two boot pipes for 22 s ahead of everything the Earth opening view actually draws. Add PriorityQueue.enqueueMany: admit the whole batch into pending, then tryStart once, so the first N started are the N best-ranked entries. evaluateRows now collects demanded rows and submits them in one call. The galaxy-thumbnail queue keeps eager per-frame enqueue untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
skymap | e084081 | Commit Preview URL Branch Preview URL |
Jul 24 2026, 08:31 PM |
Part 1 (prep + bounded asset queue) is 39/39, shipped in PR #505. Part 2 (body-texture atlas) stays in plans/ until its 9 manual visual/perf attestations are checked against a cold local load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Boots at Earth but fetched ~101.7 MB of catalogs with no ordering and no
concurrency bound, so the first view took far longer than it should. This
introduces a bounded, priority-ordered fetch scheduler plus a ≤1 MB body-texture
atlas that gives every body a low-res placeholder while its hi-res map loads.
What lands
reevaluateDemandno longerfires
slot.load()per demanded row; it collects the whole demand batch andsubmits it through
state.subsystems.assetQueue(ASSET_QUEUE_CONCURRENCY = 2).The queue starts at most N fetches, ordered by each row's authored
priority,and re-runs safely on every state change via its dedup semantics.
priorityperASSET_WIRINGrow — one integer per asset, lowerfetched first. Body atlas 0 → hi-res body textures 10 → famous/meta 20s →
structure/constellations 30s → 2MRS 40 → stars 50 → the megaparsec catalogs
60s → singleton volumes 70–82 → pgc aliases 90.
build-texturesemits a 2048×1024 / ~161 KB WebP with13 tiles; a dedicated slot fetches it first and fans each tile out as a cropped
placeholder (via
copyExternalImageToTextureorigin, a transport-only crop —no shader/BGL/UV changes). Two-layer resolution (
committed ?? placeholder)guarantees a late atlas tile never clobbers a landed hi-res map.
starts (
enqueueMany), so the first N starts are the N best-ranked demandedrows, not the first N walked. Without it, array-position rows (e.g.
sdss-pointsat rank 60) grabbed a boot pipe at t=0 ahead of
starCatalog:gaiaStars.Known gaps (captured in docs/backlog, deliberately out of scope)
makeRunTierTransitioncallsslot.load()directly per source, so a zoom-driven tier change reverts tounbounded, array-ordered fetching. Body-texture tier changes DO go through the
queue (via
staleTierEvict→ release → re-enqueue).enabledonly, sothey load at boot even though they aren't visible from the Earth home view.
A scale-band demand gate (extending the
bodyTextureRowproximity pattern) isthe deeper lever and will be designed with the powers-of-ten scale-ladder work.
transfer at half speed while sharing with lower-ranked heavies. Left at N=2 for
now; a weight-aware bound is the eventual answer.
Verification
npm run typecheckclean (both tsconfigs)npm test— 867 files / 5026 tests passing(batch ranking) and the queue's per-instance bound.
Manual throttled cold-cache waterfall verification and the three visual
placeholder checks (Mars/Jupiter/Earth tile orientation) are pending a local
build.
🤖 Generated with Claude Code