From fd8ce72b9ab4ef172228943ffb25b5f12ff3eb7b Mon Sep 17 00:00:00 2001 From: Emmanuel Boudrant Date: Tue, 7 Jul 2026 07:51:40 -0700 Subject: [PATCH] feat: load test results from uploaded archive files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a multi-file picker beside the Scan-from-URL input that accepts .zip/.tar/.tar.gz archives, merges their build/ outputs into one overlay + scan, and aborts with a conflict dialog when two archives contain the same path with differing content (identical duplicates merge silently). - src/localArchive.js: sniff/extract (unzip|tar) → stage → hash → merge + conflict-detect → overlay /build via fs.cpSync. Reuses remoteFetch path-safety + compat helpers. - src/scanJobs.js: upload registry, startScanFromUploads/runUploads/ doOverlayAndScan, kind-aware confirmScanJob/finishJob, getJob conflicts. - src/handler.js: POST /api/uploads (raw streamed, byte-capped) and POST /api/projects/:id/scan-from-uploads. - Picker lives inside the hidden url-form → project-card screenshot baselines unchanged. - tests/node/localArchive.test.js: 11 cases (helpers + e2e merge/dup/conflict/ needs_confirmation/size-cap/validation). Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/rules/scanning.md | 13 + CHANGES/2026-07-07_feat-scan-from-files.md | 68 +++++ docs/index.html | 2 +- src/handler.js | 58 +++- src/localArchive.js | 190 ++++++++++++ src/remoteFetch.js | 2 + src/scanJobs.js | 148 +++++++++- static/css/app.css | 8 + static/js/home.js | 95 +++++- tests/node/localArchive.test.js | 324 +++++++++++++++++++++ 10 files changed, 897 insertions(+), 11 deletions(-) create mode 100644 CHANGES/2026-07-07_feat-scan-from-files.md create mode 100644 src/localArchive.js create mode 100644 tests/node/localArchive.test.js diff --git a/.claude/rules/scanning.md b/.claude/rules/scanning.md index 025bf66..1aee46e 100644 --- a/.claude/rules/scanning.md +++ b/.claude/rules/scanning.md @@ -25,3 +25,16 @@ When a Watch tick or manual Re-scan re-runs the scanner, `updateScanModule` (in ## Result-source per profile Each profile declares `result_source: 'junit' | 'files'`. JUnit-driven profiles trust the XML's pass/fail summary (e.g. Paparazzi, Compose Screenshot Testing). File-only profiles use mtime clustering alone (legacy fallback for tools without JUnit). Adding a new tool means picking the right `result_source` — don't assume one or the other. + +## Overlaying CI artifacts: URL vs uploaded files + +Two ways to pull a CI run's `build/` outputs onto a project before scanning, both landing in the same `runScan` after overlay: + +- **Scan from URL** (`src/remoteFetch.js`, `scanJobs.startScanFromUrl`): download one `.tar`/`.tar.gz`, extract selected `/build` dirs straight from the tar. +- **Scan from uploaded files** (`src/localArchive.js`, `scanJobs.startScanFromUploads`): accept several `.zip`/`.tar`/`.tar.gz` files (raw-streamed via `POST /api/uploads` → temp file → opaque id), fully extract each to a stage dir, then **merge**. + +**Uploads merge + conflict rule:** the same archive-internal path appearing in two archives is fine **iff** the bytes are identical (sha256) — merged silently. Differing bytes is a **conflict**: the job aborts with `status:'failed'` + a `conflicts:[{path, archives}]` list and **writes nothing to the project**. Don't "resolve" conflicts by last-wins; the whole point is that we can't know which delta is authoritative. + +**Why full extraction for uploads (not tar member-selection like the URL flow):** we need file contents on disk to hash for conflict detection, it unifies zip+tar behind one pipeline, and it sidesteps the `[bracket]`-glob problem in Paparazzi parameterized names. Overlay copies only `/build` subtrees via `fs.cpSync` — never `src/`/goldens. Format is chosen by **magic-byte sniff**, not extension. Zip extraction shells out to `unzip` (macOS + Linux CI have it; Windows does not — zip-on-Windows is out of scope). + +The `needs_confirmation` park/resume and per-terminal-state temp cleanup are shared with the URL flow via the kind-aware `confirmScanJob` / `finishJob` in `src/scanJobs.js`. diff --git a/CHANGES/2026-07-07_feat-scan-from-files.md b/CHANGES/2026-07-07_feat-scan-from-files.md new file mode 100644 index 0000000..8c36c1a --- /dev/null +++ b/CHANGES/2026-07-07_feat-scan-from-files.md @@ -0,0 +1,68 @@ +# feat: load test results from uploaded archive files (multi-file, conflict-aware) + +**Date:** 2026-07-07 +**Type:** Feature + +## Intent +Until now the only way to overlay a CI run's screenshot-test `build/` outputs onto a +project was **Scan from URL** (download one `.tar`/`.tar.gz`). Users who have result +archives on disk, or who produce several sharded CI archives, had no path in. This adds a +**file picker** next to the URL input that accepts **multiple** `.zip`/`.tar`/`.tar.gz` +files, **merges** them into a single overlay + scan, and — when two archives contain the +same file with **different content** — **aborts** with an error dialog listing the +conflicts. Identical-content duplicates are merged silently. + +### Prompts summary +1. "For the load test result from zip file, we currently propose a URL input, I also want a + file picker that supports multiple files; if a duplicate file is found in multiple zip + files it should show an error dialog if the duplicate file name's content is different." +2. Confirmed: accept zip + tar/tar.gz; merge into one scan; abort-all on any conflict. + +## Changes + +### `src/localArchive.js` (new) +- Sibling of `remoteFetch.js` for local uploads. `sniffFormat` (magic bytes), `extractArchive` + (`unzip`/`tar`), `collectBuildMembers` (walk + `isUnsafeMember` guard + symlink skip), + `hashFile` (sha256), `mergeStages` (cross-archive conflict detection), `overlayBuildDirs` + (`fs.cpSync` of `/build` subtrees only), and temp/stage cleanup. Reuses + `remoteFetch`'s `isBuildMember`/`isUnsafeMember`/`moduleRootOf`/`checkCompat`/`MAX_DOWNLOAD_BYTES`. + +### `src/remoteFetch.js` +- Export `byteCapTransform` so the raw-upload route can reuse the streaming byte cap. + +### `src/scanJobs.js` +- Upload registry (`registerUpload`/`takeUploads`) with TTL reaping of unconsumed uploads. +- `startScanFromUploads` → `runUploads` pipeline: extract all → merge/conflict-detect → + abort on conflict (project untouched) → `checkCompat` → park `needs_confirmation` or + `doOverlayAndScan` → `runScan`. +- `finishJob` now frees stage dirs + raw uploads and carries a `conflicts` payload; `getJob` + surfaces `conflicts`. `confirmScanFromUrl` generalized to kind-aware `confirmScanJob`. + +### `src/handler.js` +- `POST /api/uploads` (raw stream → temp file, byte-capped, filename/extension validated) and + `POST /api/projects/:id/scan-from-uploads` (`{uploadIds}` → job). Confirm route now calls + `confirmScanJob`. + +### `static/js/home.js`, `static/css/app.css` +- File picker added **inside the existing hidden Scan-from-URL form** (keeps project-card + screenshot baselines unchanged). `_scanFromFiles` uploads each file then starts the merge + scan; `_showConflictDialog` lists conflicting paths on a conflict abort. + +### Tests & docs +- `tests/node/localArchive.test.js` (11 cases: pure helpers + e2e merge / identical-dup / + conflict-abort-project-untouched / zip+tar / needs_confirmation / size-cap 413 / validation). +- `docs/index.html` and `.claude/rules/scanning.md` document the upload + conflict behavior. + +## Files modified + +| File | Change | +|------|--------| +| `src/localArchive.js` | New module: extract/merge/conflict-detect/overlay uploaded archives | +| `src/remoteFetch.js` | Export `byteCapTransform` | +| `src/scanJobs.js` | Upload registry, `startScanFromUploads`/`runUploads`/`doOverlayAndScan`, kind-aware `confirmScanJob`/`finishJob`, `getJob` conflicts | +| `src/handler.js` | `POST /api/uploads`, `POST /api/projects/:id/scan-from-uploads`, confirm route rename | +| `static/js/home.js` | File picker in the URL form, `_scanFromFiles`, conflict dialog | +| `static/css/app.css` | `.url-form-or` divider | +| `tests/node/localArchive.test.js` | New test suite | +| `docs/index.html` | Document uploaded-archive workflow | +| `.claude/rules/scanning.md` | Document URL-vs-uploads overlay + conflict rule | diff --git a/docs/index.html b/docs/index.html index 14355ba..7388b96 100644 --- a/docs/index.html +++ b/docs/index.html @@ -693,7 +693,7 @@

Live watch mode

Review CI results remotely

-

Paste a CI run's result tarball URL (e.g. a Jenkins artifact) and Papa Stud.io downloads it, overlays the failures onto your local project, and scans — reviewing the remote run against your local goldens.

+

Paste a CI run's result tarball URL (e.g. a Jenkins artifact) — or upload one or more .zip/.tar/.tar.gz archives from disk. Papa Stud.io overlays the failures onto your local project and scans against your local goldens. Multiple archives are merged into one scan; if two disagree on a file's content, the scan is aborted and the conflicts are listed.

diff --git a/src/handler.js b/src/handler.js index 0182694..d4fd53b 100644 --- a/src/handler.js +++ b/src/handler.js @@ -15,8 +15,11 @@ const updateCheck = require('./updateCheck'); const apng = require('./apng'); const imageSlice = require('./imageSlice'); const remoteFetch = require('./remoteFetch'); +const localArchive = require('./localArchive'); +const { pipeline } = require('stream/promises'); const STATIC_DIR = path.resolve(__dirname, '..', 'static'); +const UPLOAD_EXT_RE = /\.(zip|tar|tar\.gz|tgz)$/i; const INDEX_HTML = path.join(STATIC_DIR, 'index.html'); function createRouter() { @@ -288,10 +291,59 @@ function createRouter() { res.status(202).json({ jobId }); }); - // Resume a scan-from-url job that parked because the tarball's module layout - // didn't line up with the project (compat mismatch). + // Stream one uploaded archive to a temp file and return an opaque id the + // client later names in scan-from-uploads. Body is the raw file bytes + // (Content-Type is ignored; express.json() leaves non-JSON bodies intact, so + // the request stream reaches us here unread). We pipe straight to disk — never + // buffering the (up to 2 GB) body in memory — and enforce the same size cap as + // a scan-from-url download. + // + // codeql[js/missing-rate-limiting]: by design — single-user local app, server + // binds 127.0.0.1 (no remote attacker). See `.claude/rules/dev_workflow.md`. + router.post('/api/uploads', async (req, res) => { + const raw = req.query.filename; + const filename = typeof raw === 'string' ? path.basename(decodeURIComponent(raw)) : ''; + if (!filename || !UPLOAD_EXT_RE.test(filename)) { + return res.status(400).json({ error: 'filename with .zip/.tar/.tar.gz extension required' }); + } + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'papastud-upload-')); + const dest = path.join(dir, 'archive'); + try { + await pipeline(req, remoteFetch.byteCapTransform(localArchive.MAX_UPLOAD_BYTES), fs.createWriteStream(dest)); + } catch (e) { + localArchive.cleanupUpload(dest); + const tooLarge = /too large/.test(e.message || ''); + return res.status(tooLarge ? 413 : 400).json({ error: e.message || 'upload failed' }); + } + const size = fs.statSync(dest).size; + const uploadId = scanJobs.registerUpload({ path: dest, filename, dir }); + res.status(201).json({ uploadId, filename, size }); + }); + + // Merge several uploaded archives (by upload id) and overlay their build/ + // artifacts onto a project, then scan — like scan-from-url but for local + // files. May park at `needs_confirmation`, or fail with a `conflicts` list + // when the archives disagree on a file's content. + router.post('/api/projects/:id/scan-from-uploads', (req, res) => { + const project = projects.getProject(req.params.id); + if (!project) return res.status(404).json({ error: 'project not found' }); + const uploadIds = req.body && req.body.uploadIds; + if (!Array.isArray(uploadIds) || uploadIds.length === 0) { + return res.status(400).json({ error: 'uploadIds required' }); + } + if (uploadIds.length > localArchive.MAX_UPLOAD_FILES) { + return res.status(400).json({ error: `too many files (max ${localArchive.MAX_UPLOAD_FILES})` }); + } + const uploadList = scanJobs.takeUploads(uploadIds); + if (!uploadList) return res.status(400).json({ error: 'unknown or expired upload id' }); + const jobId = scanJobs.startScanFromUploads(project, uploadList); + res.status(202).json({ jobId }); + }); + + // Resume a scan job (URL or uploads) that parked because the archive's module + // layout didn't line up with the project (compat mismatch). router.post('/api/scan-jobs/:id/confirm', (req, res) => { - if (scanJobs.confirmScanFromUrl(req.params.id)) { + if (scanJobs.confirmScanJob(req.params.id)) { res.json({ ok: true }); } else { res.status(404).json({ error: 'job not awaiting confirmation' }); diff --git a/src/localArchive.js b/src/localArchive.js new file mode 100644 index 0000000..30b169d --- /dev/null +++ b/src/localArchive.js @@ -0,0 +1,190 @@ +/** + * Merge locally-uploaded screenshot-test result archives and overlay their + * build outputs onto a project directory. + * + * This is the file-upload sibling of `remoteFetch.js` (scan-from-URL). Where + * that module downloads a single `.tar` and extracts selected `build/` dirs + * straight out of the tar, this one accepts *several* archives (`.zip` or + * `.tar`/`.tar.gz`), extracts each fully into its own staging dir, and then: + * + * 1. collects the safe `build/` members of every stage, + * 2. hashes them to detect cross-archive conflicts (same path, different + * bytes) — which abort the whole operation, and + * 3. copies the `/build` subtrees into the project (identical + * duplicates across archives are harmless overwrites). + * + * Full extraction (rather than tar member-selection) is deliberate: we need + * file contents on disk to hash for conflict detection, it unifies zip + tar + * handling behind one pipeline, and it sidesteps the `[bracket]` glob problem + * that Paparazzi parameterized snapshot names cause with `tar -T` member lists. + * + * No new npm deps: we shell out to the system `unzip` (zip) and `tar` (tar, + * gzip auto-detected). Both are present on macOS and Linux CI runners. + */ + +'use strict'; + +const fs = require('fs'); +const path = require('path'); +const crypto = require('crypto'); +const { spawnSync } = require('child_process'); +const remoteFetch = require('./remoteFetch'); + +// Same 2 GB ceiling as a scan-from-url download, applied per uploaded file. +const MAX_UPLOAD_BYTES = remoteFetch.MAX_DOWNLOAD_BYTES; + +// Cap on how many archives a single merge can combine. Well past any real +// sharded-CI run; a backstop against a pathological request. +const MAX_UPLOAD_FILES = 32; + +const SPAWN_OPTS = { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 }; + +// Detect the archive format from its magic bytes rather than the filename. +// Zip local-file/central-dir/end-of-central-dir headers all start `PK`. +// Everything else is treated as tar (system `tar` auto-detects gzip, so plain +// `.tar` and `.tar.gz` both extract; a genuinely bad file surfaces as a real +// `tar -xf` error rather than being guessed here). +function sniffFormat(filePath) { + const fd = fs.openSync(filePath, 'r'); + try { + const buf = Buffer.alloc(4); + const n = fs.readSync(fd, buf, 0, 4, 0); + if (n >= 2 && buf[0] === 0x50 && buf[1] === 0x4b) return 'zip'; // "PK" + return 'tar'; + } finally { + fs.closeSync(fd); + } +} + +// Extract one archive fully into stageDir (created if needed). Throws with the +// tail of the extractor's stderr on failure. +function extractArchive(filePath, stageDir) { + fs.mkdirSync(stageDir, { recursive: true }); + const format = sniffFormat(filePath); + const r = format === 'zip' + ? spawnSync('unzip', ['-o', '-q', filePath, '-d', stageDir], SPAWN_OPTS) + : spawnSync('tar', ['-xf', filePath, '-C', stageDir], SPAWN_OPTS); + if (r.status !== 0) { + const detail = (r.stderr || '').slice(-300) || `${format} extract failed`; + throw new Error(`could not extract archive: ${detail}`); + } +} + +// Walk a staged archive and return its safe `build/` members as +// { relPath (POSIX), absPath }. Directories and symlinks are skipped; +// a member whose path escapes the stage root throws (defence-in-depth against +// zip-slip on top of the extractor's own guards). +function collectBuildMembers(stageDir) { + const members = []; + const walk = (dir) => { + for (const dirent of fs.readdirSync(dir, { withFileTypes: true })) { + const abs = path.join(dir, dirent.name); + if (dirent.isSymbolicLink()) continue; + if (dirent.isDirectory()) { walk(abs); continue; } + if (!dirent.isFile()) continue; + const relPath = path.relative(stageDir, abs).split(path.sep).join('/'); + if (!remoteFetch.isBuildMember(relPath)) continue; + if (remoteFetch.isUnsafeMember(relPath)) { + throw new Error(`unsafe path in archive: ${relPath}`); + } + members.push({ relPath, absPath: abs }); + } + }; + walk(stageDir); + return members; +} + +function hashFile(absPath) { + return crypto.createHash('sha256').update(fs.readFileSync(absPath)).digest('hex'); +} + +// Merge the build members of several staged archives, detecting conflicts. +// `stages` is [{ name, dir }] (name = original filename, for reporting). +// +// Returns { members, moduleRoots, conflicts }: +// - members: one winning { relPath, absPath } per unique path. +// - moduleRoots: distinct module roots across all members (sorted). +// - conflicts: [{ path, archives }] — a path present in >1 archive with +// differing bytes. Empty when the merge is clean. Identical-content +// duplicates are merged silently (no conflict). +function mergeStages(stages) { + const seen = new Map(); // relPath -> { hash, absPath, archives:Set } + const conflicts = new Map(); // relPath -> Set of archive names + for (const stage of stages) { + for (const { relPath, absPath } of collectBuildMembers(stage.dir)) { + const prior = seen.get(relPath); + if (!prior) { + seen.set(relPath, { hash: hashFile(absPath), absPath, archives: new Set([stage.name]) }); + continue; + } + const hash = hashFile(absPath); + if (hash === prior.hash) { + prior.archives.add(stage.name); // identical duplicate — fine + continue; + } + let names = conflicts.get(relPath); + if (!names) { names = new Set(prior.archives); conflicts.set(relPath, names); } + names.add(stage.name); + } + } + const members = [...seen.entries()].map(([relPath, v]) => ({ relPath, absPath: v.absPath })); + const roots = new Set(members.map(m => remoteFetch.moduleRootOf(m.relPath))); + return { + members, + moduleRoots: [...roots].sort(), + conflicts: [...conflicts.entries()] + .map(([p, names]) => ({ path: p, archives: [...names].sort() })) + .sort((a, b) => a.path.localeCompare(b.path)), + }; +} + +// Copy the `/build` subtrees from every stage into projectRoot. +// Only `build/` dirs are ever written, so source and goldens are never +// clobbered. When no conflict was detected, cross-stage overwrites of a shared +// path are byte-identical and therefore safe. Returns the number of build dirs +// copied. +function overlayBuildDirs(stages, moduleRoots, projectRoot) { + let copied = 0; + for (const stage of stages) { + for (const root of moduleRoots) { + const src = root ? path.join(stage.dir, root, 'build') : path.join(stage.dir, 'build'); + let isDir = false; + try { isDir = fs.statSync(src).isDirectory(); } catch {} + if (!isDir) continue; + const dest = root ? path.join(projectRoot, root, 'build') : path.join(projectRoot, 'build'); + fs.cpSync(src, dest, { recursive: true, force: true, dereference: false }); + copied++; + } + } + return copied; +} + +function cleanupStage(dir) { + if (!dir) return; + try { fs.rmSync(dir, { recursive: true, force: true }); } catch {} +} + +function cleanupStages(stages) { + if (!stages) return; + for (const s of stages) cleanupStage(s.dir); +} + +// Remove a raw upload's temp dir (each upload lives in its own mkdtemp dir). +function cleanupUpload(uploadPath) { + if (!uploadPath) return; + try { fs.rmSync(path.dirname(uploadPath), { recursive: true, force: true }); } catch {} +} + +module.exports = { + MAX_UPLOAD_BYTES, + MAX_UPLOAD_FILES, + sniffFormat, + extractArchive, + collectBuildMembers, + hashFile, + mergeStages, + overlayBuildDirs, + cleanupStage, + cleanupStages, + cleanupUpload, +}; diff --git a/src/remoteFetch.js b/src/remoteFetch.js index 0955be6..47531ee 100644 --- a/src/remoteFetch.js +++ b/src/remoteFetch.js @@ -210,4 +210,6 @@ module.exports = { isBuildMember, moduleRootOf, buildDirsForModules, + // shared with localArchive.js (raw upload byte cap) + byteCapTransform, }; diff --git a/src/scanJobs.js b/src/scanJobs.js index 4df9346..e20f443 100644 --- a/src/scanJobs.js +++ b/src/scanJobs.js @@ -5,12 +5,14 @@ const crypto = require('crypto'); const fs = require('fs'); +const os = require('os'); const path = require('path'); const projects = require('./projects'); const { scanProjectIncrementalSync, processSingleModule } = require('./scanner'); const { createWatcher } = require('./watcher'); const { getStrategy } = require('./strategies'); const remoteFetch = require('./remoteFetch'); +const localArchive = require('./localArchive'); function projectCacheDir(projectId) { return path.join(projects.getDataDir(), 'cache', 'xcresult', projectId); @@ -28,6 +30,35 @@ const jobs = new Map(); const watchers = new Map(); // scanId -> watcher const JOB_TTL = 300_000; // 5 minutes in ms +// --- Upload registry (raw file uploads awaiting a scan-from-uploads call) --- +// +// A raw upload streams to a temp file and gets an opaque id; the client later +// names those ids in scan-from-uploads. Kept here (not in handler.js) so the +// existing cleanupOldJobs interval can also reap uploads that were streamed but +// never consumed. Entries are removed on takeUploads (ownership passes to the +// job) or TTL-reaped if abandoned. +const uploads = new Map(); // uploadId -> { path, filename, dir, created_at } + +function registerUpload({ path: uploadPath, filename, dir }) { + const uploadId = crypto.randomBytes(8).toString('hex'); + uploads.set(uploadId, { path: uploadPath, filename, dir, created_at: Date.now() }); + return uploadId; +} + +// Resolve ids to their temp files and remove them from the registry so the +// caller (a scan job) owns cleanup. Returns null if any id is unknown, without +// consuming the others (they will TTL-reap). +function takeUploads(ids) { + const out = []; + for (const id of ids) { + const u = uploads.get(id); + if (!u) return null; + out.push({ uploadId: id, path: u.path, filename: u.filename }); + } + for (const id of ids) uploads.delete(id); + return out; +} + // Build and register a job with the shared progress shape. `extra` carries // status-specific fields (e.g. the fetch job's tarball bookkeeping). function makeJob(project, status, extra = {}) { @@ -58,11 +89,17 @@ function makeJob(project, status, extra = {}) { // Move a job to a terminal state, releasing any downloaded tarball exactly // once. Every terminal transition (cancel, timeout, failure) goes through here // so the temp file can't be leaked. -function finishJob(job, status, error = null) { +function finishJob(job, status, error = null, conflicts = null) { remoteFetch.cleanupTemp(job._tarFile); job._tarFile = null; + if (job._stageDirs) { localArchive.cleanupStages(job._stageDirs); job._stageDirs = null; } + if (job._uploadPaths) { + for (const u of job._uploadPaths) localArchive.cleanupUpload(u.path); + job._uploadPaths = null; + } job.status = status; if (error !== null) job.error = error; + if (conflicts !== null) job.conflicts = conflicts; job._finished_at = Date.now(); } @@ -92,6 +129,7 @@ function getJob(jobId) { scanId: job.scan_id, error: job.error, compat: job.compat || null, + conflicts: job.conflicts || null, }; } @@ -110,6 +148,13 @@ function cancelJob(jobId) { function cleanupOldJobs() { const now = Date.now(); + // Reap raw uploads that were streamed but never consumed by a scan. + for (const [id, u] of uploads) { + if (now - u.created_at > JOB_TTL) { + localArchive.cleanupUpload(u.path); + uploads.delete(id); + } + } for (const [id, job] of jobs) { // Drop a tarball left parked on an un-confirmed fetch job that's gone stale. if (job.status === 'needs_confirmation' && job._parked_at && now - job._parked_at > JOB_TTL) { @@ -230,14 +275,23 @@ async function runFetch(jobId, project, url) { } } -// Resume a job parked on a compat-mismatch confirmation. -function confirmScanFromUrl(jobId) { +// Resume a job parked on a compat-mismatch confirmation. Kind-aware: the URL +// flow re-extracts from its tarball, the uploads flow overlays its stage dirs. +// Both mirror their happy path's confirm semantics — overlay *all* module roots +// present in the archive(s), since the user confirmed writing despite the +// mismatch (creating new /build dirs in the project as needed). +function confirmScanJob(jobId) { const job = jobs.get(jobId); if (!job || job.status !== 'needs_confirmation') return false; job.status = 'extracting'; // Defer the (blocking) extract off the request so /confirm returns immediately. - const { _project, _tarFile, _modules } = job; - setImmediate(() => doExtractAndScan(jobId, _project, _tarFile, _modules)); + if (job._kind === 'uploads') { + const { _project, _stageDirs, compat } = job; + setImmediate(() => doOverlayAndScan(jobId, _project, _stageDirs, compat.modules)); + } else { + const { _project, _tarFile, _modules } = job; + setImmediate(() => doExtractAndScan(jobId, _project, _tarFile, _modules)); + } return true; } @@ -257,6 +311,87 @@ function doExtractAndScan(jobId, project, tarFile, moduleRoots) { runScan(jobId, project); } +// --- Scan from uploaded files (merge several local archives, overlay, scan) --- + +// Kick off an extract + merge + conflict-check + overlay + scan against an +// existing project. `uploads` is [{ uploadId, path, filename }]; ownership of +// the temp files passes to the job (freed on any terminal transition). +function startScanFromUploads(project, uploadList) { + cleanupOldJobs(); + log(`[uploads] scan-from-uploads for "${project.name}" <- ${uploadList.length} file(s)`); + const job = makeJob(project, 'extracting', { + _kind: 'uploads', + _project: project, + _uploadPaths: uploadList.map(u => ({ path: u.path, filename: u.filename })), + _stageDirs: null, + _modules: null, + _parked_at: null, + conflicts: null, + }); + setImmediate(() => runUploads(job.id, project, uploadList)); + return job.id; +} + +function runUploads(jobId, project, uploadList) { + const job = jobs.get(jobId); + if (!job) return; + try { + job.status = 'extracting'; + // 1. Extract every archive fully into its own stage dir. + const stageDirs = []; + job._stageDirs = stageDirs; + for (const u of uploadList) { + if (job._cancelFn()) return finishJob(job, 'cancelled'); + const stageDir = fs.mkdtempSync(path.join(os.tmpdir(), 'papastud-stage-')); + localArchive.extractArchive(u.path, stageDir); + stageDirs.push({ name: u.filename, dir: stageDir }); + } + if (job._cancelFn()) return finishJob(job, 'cancelled'); + + // 2. Merge + detect cross-archive content conflicts. + const { members, conflicts } = localArchive.mergeStages(stageDirs); + if (conflicts.length) { + // Abort before touching the project — the deltas disagree and we can't + // know which is authoritative. + return finishJob(job, 'failed', 'conflicting files across archives', conflicts); + } + if (!members.length) throw new Error('no build/ artifacts found in archives'); + + // 3. Which modules line up with this project's layout? checkCompat works on + // member-name strings, so hand it the relative paths. + const compat = remoteFetch.checkCompat(members.map(m => m.relPath), project.path); + job._modules = compat.matched; + if (!compat.compatible) { + job.compat = compat; + job.status = 'needs_confirmation'; + job._parked_at = Date.now(); + return; + } + doOverlayAndScan(jobId, project, stageDirs, compat.matched); + } catch (e) { + finishJob(job, 'failed', e.message); + } +} + +function doOverlayAndScan(jobId, project, stageDirs, moduleRoots) { + const job = jobs.get(jobId); + if (!job) { localArchive.cleanupStages(stageDirs); return; } + try { + localArchive.overlayBuildDirs(stageDirs, moduleRoots, project.path); + } catch (e) { + finishJob(job, 'failed', e.message); + return; + } + // Free the stage dirs and raw uploads now that build outputs are overlaid. + localArchive.cleanupStages(stageDirs); + job._stageDirs = null; + for (const u of job._uploadPaths || []) localArchive.cleanupUpload(u.path); + job._uploadPaths = null; + // Hand off to the normal scan flow against the now-overlaid project dir. + job.status = 'discovering'; + runScan(jobId, project); +} + // --- Watcher management --- // Whether Watch mode is meaningful for a scan's strategy. Returns null when the @@ -333,7 +468,8 @@ function isWatching(scanId) { } module.exports = { - startScan, startScanFromUrl, confirmScanFromUrl, + startScan, startScanFromUrl, confirmScanJob, + startScanFromUploads, registerUpload, takeUploads, getJob, cancelJob, startWatching, stopWatching, stopAllWatching, isWatching, watchSupported, }; diff --git a/static/css/app.css b/static/css/app.css index 6567fbd..369e3a8 100644 --- a/static/css/app.css +++ b/static/css/app.css @@ -240,6 +240,14 @@ select.input { appearance: auto; } } .url-form-hint { font-size: 11px; color: var(--text-dim); margin-top: var(--space-sm); } .url-form-hint code { font-family: monospace; } +.url-form-or { + display: flex; align-items: center; gap: var(--space-sm); + font-size: 11px; color: var(--text-dim); text-transform: uppercase; letter-spacing: 0.04em; + margin: var(--space-md) 0 var(--space-sm); +} +.url-form-or::before, .url-form-or::after { + content: ''; flex: 1; height: 1px; background: var(--border); +} /* Keep the card-action icon button grouped with its siblings (the global .btn-icon:first-of-type auto-margin is meant for the header, not here). */ .card-actions .btn-icon:first-of-type { margin-left: 0; } diff --git a/static/js/home.js b/static/js/home.js index 82c1daa..5fec5bc 100644 --- a/static/js/home.js +++ b/static/js/home.js @@ -127,6 +127,12 @@ function _renderProjects(projectsList) {
Downloads the tarball and overlays its build/ outputs onto this project, then scans against your local goldens.
+
or upload archive files
+
+ + +
+
Select one or more .zip / .tar / .tar.gz archives. They're merged and overlaid onto this project. If two archives contain the same file with different content, the scan is aborted and the conflicts are listed.