feat: coordinate heavy local accelerator jobs

.changelog/next/added-issue-4174.md

@@ -0,0 +1 @@
+- Local image, video, 3D, and LoRA jobs now coordinate access to the machine accelerator to prevent overlapping GPU workloads.

server/lib/README.md

@@ -27,6 +27,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
 | `sharedSchemas.js` | Cross-domain Zod fragments that `validation.js` and the per-domain `*Validation.js` files both need, kept in a leaf module so the domain files never import back through `validation.js` (its hoisted `export * from` lines would TDZ). `grokVideoDurationSchema` (grok clip-length union), `cloudModelIdString(message)` (cloud-CLI model-id charset/bounds), `recordRenderPinFields` (the `imageMode`/`imageModelId` per-record render pin pair), and `isSafeSubdirFilter(v)` (relative path with no wildcard, `..` segment, or leading `/`), and `csvIdsParam({ max, maxIdLength, truncate })` (the `?ids=a,b,c` batch-by-id query param: trims, drops empties, reads all-blank as absent, and either 400s or silently slices an over-cap batch). |
 | `agentOutputMarkers.js` | Status lines PortOS itself appends to an agent's output buffer: `SENTINEL_COMPLETION_MARKER` (the line `ingestDoneSentinel` writes just before the agent's `.agent-done` summary) plus `isAgentLifecycleLine(line)`/`stripLifecycleLines(lines)`, which match PortOS's telemetry on its actual message shapes rather than on a leading emoji (an agent's own summary may well start a line with `✅`). Pure — shared by the emitter (`services/agentTuiSpawning.js`) and by readers that must show only the agent's own words (notably the generated PR description). |
 | `agentSentinel.js` | The `.agent-done` completion sentinel: `DONE_SENTINEL_NAME`, `doneSentinelName(agentId)` → the per-instance filename `.agent-done-<agentId>` (worktree-less agents share one workspace, so a shared name lets concurrent runs clobber — and finalize on — each other's signal), `doneSentinelPath(workspacePath, agentId)` → the single path every producer and consumer resolves, + pure `parseSentinelPayload(contents)` → `{ summary, payload }`. Back-compat — a plain-markdown sentinel yields `payload: null`; a JSON object yields its structured `payload` for a programmatic-I/O task type's `processTaskOutput` hook. `salvageSentinelPayload(contents)` (async) is the lenient second tier — runs `jsonExtract` over a fenced/prose-trailed/control-char-corrupted envelope so a less-capable model's near-valid output still surfaces its `payload` instead of being dropped. `extractSentinelPayloadFromTranscript(transcript, isPayload)` (async) is the third tier for programmatic-I/O types ONLY — scans the ANSI-stripped PTY transcript, newest balanced JSON block first, for a payload the model PRINTED instead of writing, and adopts it only if the owning hook's shape predicate accepts it (#3640). |
+| `heavyJobClaim.js` | Cross-process, machine-wide accelerator claim: `claimHeavyLocalJob({ kind, id, timeoutMs? })` either acquires an ephemeral `data/` lock or reports its active holder; dead-PID claims are reclaimed, and a spawned child can take over the recorded PID across a server restart. |
 | `agentValidation.js` | Social-bot agent schemas (personality, Moltbook/Moltworld accounts, automation schedules, agent tools + Moltworld payloads) and CoS Feature Agent definitions. |
 | `quotaBurnConfig.js` | Quota-burn plan shape: the provider families, the burn-job type alphabet + catalog the config page renders its form from, `QUOTA_BURN_BOUNDS` (the one bounds table the normalizer clamps to, the Zod schemas reject against, and the catalog descriptors publish as min/max), and total normalization (`normalizeQuotaBurnConfig`). Owns the dispatch-cap sentinel too (`QUOTA_BURN_UNLIMITED_DISPATCHES` / `isUnlimitedDispatchCap`) — -1 means the window is not counted, and is the default. Also owns the queued burn task's description shape (`burnTaskDescription` / `quotaBurnFamilyOfDescription`), shared with migration 225, and the `run once` vocabulary (`quotaBurnJobKey` / `jobIsSpent`, plus the two family predicates `familyIsConfigured` and `familyHasRunnableJobs`). Pure — no storage, no provider I/O. |
 | `quotaBurnPresets.js` | `QUOTA_BURN_PROMPT_PRESETS` — ready-made single-focus audit prompts for `agent-prompt` burn jobs (UX, a11y, mobile, failure paths, perf, test gaps, dead code, data safety, docs, security), each filing GitHub issues and changing no code. Templates: picking one COPIES its prompt into the job, so editing them never rewrites a configured job. `findQuotaBurnPreset(id)`. |
@@ -166,6 +167,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
 | `imageClean.js` | `cleanImageBuffer(buf, { metadata, denoise })` (composable opt-in pipeline: lossless metadata/C2PA strip + optional median/sharpen denoise) · `stripPngMetadataChunks` / `stripPngC2PAChunk` (lossless PNG-chunk removers) · `compositeIgnoreZone(base, original, mask, { feather })` (preserve-region compositing: restore original pixels into a feathered mask over a diffused result) · `autoCleanGeneratedImage` (in-place clean for post-generation hook). HTTP route in `routes/imageClean.js` wraps `cleanImageBuffer` and appends a CPU light diffusion pass (`applyLightRegen` from `services/imageGen/regen.js`) for the `diffusion=light` SynthID-disruption step. |
 | `imageWatermark.js` | `removeCornerWatermark` (erases the visible Gemini/Nano-Banana bottom-right ✦ via dependency-free harmonic/Laplace inpaint) + pure helpers `resolveWatermarkRegion` / `inpaintRegion`. Distinct from SynthID regen — this targets the *visible* corner logo. |
 | `localImageFilename.js` | `localImageFilename(urlOrPath)` resolves a stored image reference to the bare gallery-image filename under `data/images/` (or null for empty/external-URL/non-image-path) — the unit the peer-sync asset pipeline hashes + transfers. Single source of truth for the authors/artists/albums/Creative-Director filename resolvers (`headshotImageFilename`/`portraitImageFilename`/`coverImageFilename`/`startingImageFilename` are thin wrappers). Also exports `assetBasename(pathOrName)`, the shared strip-querystring→basename primitive (reused by moodBoard's `imageUrlToAppAsset`). |
+| `localMemory.js` | Shared best-effort local-model unload + available-memory report (`prepareLocalMemory`) for GPU-heavy local work; only loopback Ollama / LM Studio backends are evicted. |
 | `threejsModel.js` | Validated declarative procedural-model schema plus deterministic standalone Three.js factory export for the Three.js Models workspace. Also the cross-section gate: `evaluateThreejsFlatness(spec)` counts distinct vertex planes per axis — relative to the mesh's own size, with a rotation-invariant zero-volume check behind it — and treats an `extrude` with no bevel thickness as the two-plane slab it is to report when the majority of identity-priority features are built only from flat parts — a model that reads right head-on and like cardboard when orbited. `buildThreejsFlatnessFeedback(flatness)` turns that into default refinement feedback; `listSpecNames(names)` is the shared capped name-list formatter both gates use in finding messages. Also the material-plausibility gate: `evaluateThreejsMaterialPlausibility(spec)` keys a bounded per-family prior table (metal, wood, plastic, glass, fabric, ceramic, rubber, stone, leather, paper) off tokens in each material's id and reports channels whose values the named substance does not support — metalness 0.9 oak, transmission 1.0 steel — skipping any id that names no family or two, and any channel the material's `type` never forwards to Three.js. Advisory only: it never clamps, because a stylized model may legitimately break every prior. `buildThreejsMaterialFeedback(plausibility)` turns that into default refinement feedback. |
 | `threejsModelCoverage.js` | `evaluateThreejsPartCoverage(spec, { family })` — structural assembly gate over an already-validated Three.js spec: flags promised features fused onto the same part set, geometry claimed by no detail, and details nothing was built for (folded minor relief stays a `note`). With a subject family it also warns on a required component the spec never mentions at all — the one check that can fault a spec for what it failed to *promise*. `buildThreejsCoverageFeedback(coverage)` turns the error findings (plus any family gap) into default refinement feedback. |
 | `threejsModelFamilies.js` | Curated subject-family checklists for Three.js generation. `THREEJS_MODEL_FAMILY_OPTIONS` / `THREEJS_MODEL_FAMILY_IDS` drive the picker and route validation; `buildThreejsFamilyChecklist(id)` returns the prompt block spliced into generation (`''` for the default `general`, so an un-narrowed subject keeps the unchanged general-purpose prompt); `findMissingFamilyComponents(spec, id)` reports which required components the spec never mentions. |

server/lib/heavyJobClaim.js

@@ -0,0 +1,180 @@
+/**
+ * Machine-wide claim for a local accelerator job.
+ *
+ * The claim is deliberately a file below the install's data root rather than an
+ * in-process flag: PortOS worktrees and restarted server processes share the
+ * same machine, and must therefore see the same owner. A dead process is
+ * reclaimed on the next acquisition, so a crash cannot wedge local rendering.
+ */
+
+import { randomUUID } from 'crypto';
+import { mkdir, readFile, unlink, writeFile } from 'fs/promises';
+import { existsSync, readFileSync, unlinkSync } from 'fs';
+import { dirname, join } from 'path';
+import { PATHS } from './fileUtils.js';
+
+export const HEAVY_LOCAL_JOB_CLAIM_PATH = join(PATHS.data, 'heavy-local-job.claim.json');
+export const HEAVY_LOCAL_JOB_STALE_MS = 24 * 60 * 60 * 1000;
+
+const heldClaims = new Map();
+let exitCleanupInstalled = false;
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+
+export function isPidAlive(pid) {
+  if (!Number.isInteger(pid) || pid <= 0) return false;
+  try {
+    process.kill(pid, 0);
+    return true;
+  } catch (err) {
+    return err?.code === 'EPERM';
+  }
+}
+
+const parseHolder = (raw) => {
+  try {
+    const holder = JSON.parse(raw);
+    if (!holder || typeof holder.kind !== 'string' || typeof holder.id !== 'string') return null;
+    return holder;
+  } catch {
+    return null;
+  }
+};
+
+const readHolder = async (claimPath) => {
+  const raw = await readFile(claimPath, 'utf8').catch((err) => (err?.code === 'ENOENT' ? null : Promise.reject(err)));
+  return raw === null ? null : parseHolder(raw);
+};
+
+const cleanupHeldClaims = () => {
+  for (const [claimPath, token] of heldClaims) {
+    try {
+      if (existsSync(claimPath) && parseHolder(readFileSync(claimPath, 'utf8'))?.token === token) unlinkSync(claimPath);
+    } catch {
+      // Process-exit cleanup is best-effort. A stale PID is reclaimed later.
+    }
+  }
+};
+
+const installExitCleanup = () => {
+  if (exitCleanupInstalled) return;
+  exitCleanupInstalled = true;
+  process.once('exit', cleanupHeldClaims);
+};
+
+const holderMessage = (holder, { stale = false } = {}) => {
+  const started = Number.isFinite(holder?.startedAt) ? `, started ${new Date(holder.startedAt).toISOString()}` : '';
+  const suffix = stale ? ' The recorded claim is older than the safety ceiling.' : '';
+  return `Local accelerator is in use by ${holder?.kind || 'another job'} ${holder?.id || 'unknown'}${started}.${suffix}`;
+};
+
+const publicHolder = ({ kind, id, pid, startedAt } = {}) => ({ kind, id, pid, startedAt });
+
+/**
+ * Claim the machine-local accelerator.
+ *
+ * A zero timeout refuses immediately, suitable for an interactive Generate
+ * action. Background callers may provide a bounded timeout to wait for a
+ * currently-running job without silently waiting forever.
+ */
+export async function claimHeavyLocalJob({
+  kind,
+  id,
+  timeoutMs = 0,
+  claimPath = HEAVY_LOCAL_JOB_CLAIM_PATH,
+  pid = process.pid,
+  now = () => Date.now(),
+  wait = sleep,
+  pidIsAlive = isPidAlive,
+} = {}) {
+  if (typeof kind !== 'string' || !kind.trim() || typeof id !== 'string' || !id.trim()) {
+    throw new Error('A heavy local job claim requires non-empty kind and id.');
+  }
+  const deadline = now() + Math.max(0, timeoutMs);
+  const token = randomUUID();
+  const ours = { kind, id, pid, startedAt: now(), token };
+  await mkdir(dirname(claimPath), { recursive: true });
+
+  for (;;) {
+    try {
+      await writeFile(claimPath, JSON.stringify(ours), { flag: 'wx' });
+      heldClaims.set(claimPath, token);
+      installExitCleanup();
+      let released = false;
+      return {
+        ok: true,
+        holder: publicHolder(ours),
+        // Detached media children outlive a server restart. Transfer the PID
+        // recorded on disk once one exists so process-exit cleanup does not
+        // release a claim while that child is still consuming the accelerator.
+        async handoffTo(childPid) {
+          if (!Number.isInteger(childPid) || childPid <= 0) return;
+          const current = await readHolder(claimPath);
+          if (current?.token !== token) return;
+          ours.pid = childPid;
+          await writeFile(claimPath, JSON.stringify(ours));
+          heldClaims.delete(claimPath);
+        },
+        async release() {
+          if (released) return;
+          released = true;
+          heldClaims.delete(claimPath);
+          const current = await readHolder(claimPath);
+          if (current?.token === token) await unlink(claimPath).catch((err) => {
+            if (err?.code !== 'ENOENT') throw err;
+          });
+        },
+      };
+    } catch (err) {
+      if (err?.code !== 'EEXIST') throw err;
+    }
+
+    const holder = await readHolder(claimPath);
+    if (!holder || !pidIsAlive(holder.pid)) {
+      await unlink(claimPath).catch((err) => {
+        if (err?.code !== 'ENOENT') throw err;
+      });
+      continue;
+    }
+    const stale = now() - holder.startedAt > HEAVY_LOCAL_JOB_STALE_MS;
+    if (now() >= deadline) {
+      return { ok: false, holder: publicHolder(holder), stale, message: holderMessage(holder, { stale }), release: async () => {} };
+    }
+    await wait(Math.min(250, Math.max(1, deadline - now())));
+  }
+}
+
+/**
+ * Adopt a claim already recorded on disk, for a process that did not
+ * acquire it — a restarted server re-attaching to a detached child that
+ * survived the crash (#1332). The surviving child already holds the
+ * machine-wide accelerator claim (transferred to it via `handoffTo` before
+ * the restart); calling `claimHeavyLocalJob` again here would see that live
+ * claim as a COMPETING job and refuse it, failing every restart-survived run
+ * outright. This never contends for the lock — it only recognizes a claim
+ * that already names this exact job and PID, so it's safe to call blind.
+ *
+ * Returns `null` when no on-disk claim matches `kind`/`id`/`pid` (e.g. an
+ * in-flight run that predates this claim file, or a genuinely different
+ * holder), so the caller can fall back to `claimHeavyLocalJob`.
+ */
+export async function adoptHeavyLocalJob({
+  kind, id, pid, claimPath = HEAVY_LOCAL_JOB_CLAIM_PATH,
+} = {}) {
+  const holder = await readHolder(claimPath);
+  if (holder?.kind !== kind || holder.id !== id || holder.pid !== pid || typeof holder.token !== 'string') return null;
+  let released = false;
+  return {
+    ok: true,
+    holder: publicHolder(holder),
+    async handoffTo() {},
+    async release() {
+      if (released) return;
+      released = true;
+      const current = await readHolder(claimPath);
+      if (current?.token === holder.token) await unlink(claimPath).catch((err) => {
+        if (err?.code !== 'ENOENT') throw err;
+      });
+    },
+  };
+}