media-stack panel: provisioning can deliver mediaStack.php and userMediaStackPanel.php from mismatched versions, causing PHP fatal error on every panel load

[MagnaCapax]

Origin: Customer support ticket. Suggested Fix reflects ticket agent investigation -- do not derive fix from title or symptom alone.

Summary

A freshly-provisioned seedbox account received www/mediaStack.php and www/userMediaStackPanel.php -- a coupled pair of per-user media-stack panel files -- from mismatched versions, causing a PHP fatal error on every load of the Media Stack panel. Related to #586 (skel-sync/propagation gap on EXISTING homes during update.php) but distinct: this instance occurred at initial PROVISIONING, on a brand-new account, not on a home that predated a later update.

Symptom

PHP Fatal error:  Uncaught Error: Call to undefined function pmssMediaStackPanelHomePath() in www/mediaStack.php:155
Stack trace:
#0 www/mediaStack.php(25): pmssMediaStackPanelStartHandle(...)
#1 {main}

www/mediaStack.php called pmssMediaStackPanelStartHandle(), which internally called pmssMediaStackPanelHomePath(). That function is defined in the current /etc/skel/www/userMediaStackPanel.php, but the affected account's own userMediaStackPanel.php was still on the pre-refactor version -- old PMSS_MEDIA_STACK_MEMORY_MINIMUM_BYTES/PMSS_MEDIA_STACK_MEMORY_UNLIMITED_BYTES constants, calling pmssCustomerHomePath() directly instead of the newer helper. Every attempt to load the panel crashed; the account holder could never reach the install button.

Root Cause

Both files land in a new account's home via the same skel-copy step during provisioning. /etc/skel/www/userMediaStackPanel.php on the host already had the current, correct content (verified: grep -c 'function pmssMediaStackPanelHomePath' on skel returned non-zero) at the time this account was created. So the account's own copy did not come from a stale HOST skel -- the account-level delivery of this one file lagged behind the account-level delivery of its coupled sibling (mediaStack.php), even though both were written within the same few-minute provisioning window on the same account.

ADR 0022 already establishes that guiv-delivered customer-panel files "MUST be self-contained within the guiv delivery set" specifically to prevent one file calling a function only a newer sibling defines. This instance is exactly the failure ADR 0022 exists to prevent, occurring at provisioning rather than at a later update.php convergence pass.

I have not traced the exact provisioning code path that produced the mismatched pair -- only the observed inconsistency and its resolution.

Reproduction

Not reliably reproducible on demand. Detection after the fact: for any account, grep www/mediaStack.php for functions it calls that are undefined in that SAME account's www/userMediaStackPanel.php (or the reverse).

Environment

PMSS version at incident time: git/main (skel dated within ~24h of commit 3fbb7efa "refactor(compression): media-stack panel -- remove redundant adapters"), Debian 12 (bookworm).

Suggested Fix

A provisioning-time (and/or step2 per-user sync) consistency check across the coupled media-stack panel file pair: after writing the file set for a (re)provisioned or updated account, verify no function called by mediaStack.php is undefined given that SAME account's userMediaStackPanel.php (a cheap php -l + symbol-presence check would catch this class immediately). Until then, php /scripts/update.php git/main is a full accidental workaround -- it re-delivers both files together from current skel.

Related

Refs #586 (documents the sibling defect class on EXISTING homes across five+ prior instances, later marked should-not-implement because its original root-cause hypothesis about updateUserFile() was shown incorrect -- filing separately here with an accurate root cause for the provisioning-time variant rather than reopening/appending to that issue).

---

_{Tier-3 declared at filing: --why no-known-fix — GH#631 tier gate. Declared fix size: unknown lines vs 40-line body — size gate (operator directive 2026-07-29). Owner: etc/skel/www/userMediaStackPanel.php in MagnaCapax/PMSS — owner gate.}

[MagnaCapax]

Persona Verdict

Strategic perspective layer: seven outside-viewpoint business/engineering lenses applied to
the issue above. The technical facts, citations, and verification are established in the
investigation and adversarial review comments on this issue; this section is not a
re-verification of those facts, it is judgment applied on top of them.

Issue: #953 — media-stack panel: provisioning can deliver mediaStack.php and userMediaStackPanel.php from mismatched versions, causing PHP fatal error on every panel load
Reviewed: 2026-09-25

Merit Check (Shared Grounding)

No prior adversarial-review comment exists on this issue yet (first pass across all three lanes), so Tier A is regenerated here.

Concrete present demand: a brand-new account holder "could never reach the install button" — the media-stack panel fatals on every load from the moment the account is created.
Function (one sentence): ensure the two coupled panel files a single account receives at delivery time (mediaStack.php + userMediaStackPanel.php) always come from the same version pair, so a shared-helper call never targets a function the sibling doesn't define.
Success metric (30 days, independent of code's own existence): zero Call to undefined function fatals logged against www/mediaStack.php for newly-provisioned accounts, cross-checked against media-stack-panel-tagged ticket volume — not "the check exists," but "the fatal stops occurring."
Catastrophic if implemented (naive reading — a hard-blocking provisioning-time check): a php -l + symbol-presence gate that FAILS the provisioning pipeline on mismatch → if the gate itself has a bug, or reads skel mid-refresh at the exact moment skel is being updated (skel/www saw 65 commits in the last 60 days, confirmed via git log --oneline --since="60 days ago" -- etc/skel/www/ in the PMSS dev tree) → the gate can false-positive during any of those ~1/day refreshes → the entire provisioning pipeline fails, not just delivery of the one file pair. A guard meant to catch a rare panel bug becomes a systemic provisioning-pipeline outage.
Catastrophic if NOT implemented: the same defect class recurs a fourth time (see Munger below for the count) at a frequency nobody has measured, and each recurrence can silently break the panel delivery step with no independent detection mechanism — this issue documents one instance that got filed; nothing in the provisioning pipeline itself currently surfaces the failure without an external report.
Asymmetry verdict signal: favors a narrow, non-blocking fix over the issue's literal suggested implementation. A logging/self-healing guard (re-copy from skel on mismatch, or degrade gracefully) has the recurring-defect upside with none of the fleet-wide-outage downside; a hard-blocking provisioning gate inverts that trade.
Deletion alternative: mediaStack.php already contains function_exists() guards in two other places in the same file (verified: grep -c "function_exists(" etc/skel/www/mediaStack.php → 2) — the exact defensive pattern welcome.php already uses per #586 for the same class of stale-sibling risk. Wrapping the specific failing call (pmssMediaStackPanelHomePath() at mediaStack.php:155 per the stack trace) in the same guard is a ~10-line, same-file, same-convention change that ships today and converts a hard fatal into graceful degradation — no new mechanism, no new test infrastructure, nothing to maintain. Not rejected — this is the recommended immediate stopgap; it does not by itself explain or prevent WHY the pair diverges.
Cost: build — the function_exists() guard: hours, not days. The general fix (see Musk/Huang) was already scoped in ADR 0022 in June and never built — unscoped here, but known to span provisioning + update.php step2 + the guiv heal. maintenance/yr — near-zero for the guard; a CI test (if built) is a one-time addition with near-zero ongoing cost. blast — every newly-provisioned account going forward, gated by an unmeasured race-window frequency. undo — trivial (git revert), no data migration.
Dependency blast (Hyrum): userMediaStackPanel.php is required by BOTH mediaStack.php and welcome.php (verified: grep -ln "userMediaStackPanel.php" etc/skel/www/*.php); three independent delivery paths consume the same skel source tree — initial provisioning skel-copy, update.php step2's per-user sync (scripts/lib/update/users/filesystem.php:212,235), and the guiv heal. A fix must not break any of the three, and none of the three today share a single "deliver this bundle atomically" primitive.
Musk Step 1 (dumbest requirement, named and challenged): the requirement being defended is "a version mismatch between these two files must fatal the whole panel" — nothing in the product spec requires that; it's simply what happens when a call isn't guarded. Challenged: is a loud fatal sometimes preferable to a silently-degraded feature, so the bug gets noticed and fixed? Weak — welcome.php's existing placeholder-on-function_exists()-false pattern (from #586) is already PMSS's chosen convention for this exact failure shape, so "let it fatal loudly" is inconsistent with how the codebase already handles the sibling case.

Tier B lenses applied:

• Chesterton's Fence + Callers (triggered — reconsidering a coupling): the split between mediaStack.php and userMediaStackPanel.php was reviewed in ADR 0022 itself; Option A (inline everything, remove the coupling) was explicitly rejected for DRY reasons because the shared-library pattern serves 13+ skel/www files (verified via grep -ln "require_once __DIR__" etc/skel/www/*.php, including chained dependencies like panelSessionPage.php → panelSessionAuth.php → panelSessionStore.php). Merging files is not a live option; the fence stands for a real reason.
• Gall's Law / Second-System (triggered — issue proposes new provisioning-time infrastructure): ship the narrow function_exists() guard first (hours), evolve toward the general dependency-closure CI gate ADR 0022 already promised in June second. Do not attempt "verify every coupled pair fleet-wide, blocking" as a first cut — that is the second system, and PMSS has direct precedent for what happens when a delivery-pipeline change ships without incremental verification (ADR 0022's own postmortem describes a June 3 outage from the .scriptsInc.php refactor breaking the panel-rendering path on an affected host).
• Bezos Type-1/Type-2 classification of the fix itself: a non-blocking guard or self-heal is Type 2 (reversible, ship fast, git revert if wrong). A hard-blocking provisioning-time gate is functionally Type 1 — if it misfires it stops the provisioning pipeline platform-wide — and should be canaried, not shipped straight to full rollout.
• Goodhart's Law (mild trigger — proposed success metric): "zero fatals" as the sole metric could be gamed by demoting the fatal to a silently-swallowed warning without ever fixing why the pair diverges — track BOTH "fatal count" and "mismatch-detected-and-healed count" so a drop in the first isn't mistaken for the defect class disappearing.

Persona Verdicts

Musk (first principles, deletion): The dumbest requirement being defended is that provisioning needs a NEW bespoke consistency checker at all — that's an add, not a delete, and it only catches this ONE file pair. The actual defect is that skel content isn't delivered as an atomic unit: with 65 commits touching etc/skel/www/ in 60 days and three independent code paths (provisioning skel-copy, update.php step2, the guiv heal) each separately reading from that same moving source tree, a torn read is structurally inevitable somewhere, sometime — this is just the first one that got witnessed and reported. Delete the redundant delivery paths' independent assumptions about skel being stable mid-read; converge on one atomic bundle-delivery primitive (e.g. a versioned skel snapshot swapped by symlink) that all three consumers use, and the bespoke per-pair checker becomes unnecessary. 10x simpler version: one atomic read, zero new checking code.
Challenged by: grep -ln "userMediaStackPanel.php" etc/skel/www/*.php to test whether "just merge the two files" (an even more radical delete) was viable — found welcome.php also requires the same shared file, and ADR 0022 already rejected inlining for DRY reasons across a 13-file pattern. Revised to: delete the delivery-path redundancy, not the file split.

Hormozi (ROI, customer LTV): This fatals on the very FIRST panel render of a brand-new account — structurally the highest-acquisition-cost point in the entire funnel, before any product value has been demonstrated. The reported instance is the one that surfaced as a filed issue; the economics that actually matter are the unmeasured denominator — a broken first-touch render is a mechanism with a naturally low self-reporting rate, unlike a mid-lifecycle bug that an invested account holder is more likely to escalate. The fix (a function_exists() guard, hours of work) is cheap; the asymmetric downside sits at the exact point in the funnel where acquisition cost is fully sunk and no retention value has accrued yet. High-leverage, not busy work — but only the guard, not the full CI-gate rebuild, which is a separate lower-urgency investment.
Challenged by: git log --oneline --since="60 days ago" -- etc/skel/www/ to check whether this is a rare one-off (low urgency) or a live, recurring exposure window — found 65 commits in 60 days, i.e. skel/www changes roughly daily. Revised from "maybe a coincidence, low urgency" to "an open, actively-refreshed race surface — urgency holds."

Buffett (moat, compounding): Reliability at the panel layer IS part of PM's moat — "it just works" is the promise self-hosted/managed seedbox competitors don't reliably keep. This is the THIRD time this exact defect class has surfaced since June (the fleet-wide .scriptsInc.php outage that produced ADR 0022, then #586 on existing homes, now #953 at provisioning) and ADR 0022's own explicit follow-up — "implement the dependency-closure CI test" — was never built (verified: no dependency-closure/guiv-delivery test exists anywhere under scripts/lib/tests/, confirmed by rg -li "dependency.closure|dependencyClosure" returning only the ADR document itself). That's compounding NEGATIVELY — unpaid architectural debt from a promise made in an accepted ADR, not a new discovery. Margin of safety says: ship the cheap function_exists() guard today as a stopgap (low cost, no new failure mode), but the real fix is finally paying down the June debt — not filing a fourth issue in six months when this recurs again.
Challenged by: re-read #586's disposition directly (tools/gh/view-issue 586) to check whether "recurring defect class" is accurate or whether #586 was closed because it was a false alarm — confirmed #586 carries the should-not-implement label specifically because its ORIGINAL root-cause hypothesis (updateUserFile() silently failing) was shown incorrect, not because the underlying symptom class was fake. Held: the pattern (coupled panel files, divergent delivery) is real across all three incidents even though each had a different specific mechanism.

Munger (inversion, avoiding stupidity): Invert: how do we guarantee a fourth occurrence? Keep treating each instance as a fresh, novel investigation that earns its own new issue with "an accurate root cause" (this issue's own framing, explicitly distinguishing itself from reopening #586) instead of ever landing the general fix ADR 0022 already specified. The incentive misalignment: a review pipeline that rewards "found a new, more accurate root cause for this recurrence" produces good forensic writeups and zero convergence on the actual defect class — three postmortems, three distinct root-cause narratives, zero dependency-closure test. Lollapalooza effect in play: (shared-library coupling across files) + (three uncoordinated delivery paths reading a constantly-changing source tree) + (no CI gate) is exactly the combination that guarantees recurrence — no single factor alone would.
Challenged by: checked whether the "no CI gate exists" claim might itself be stale — re-ran find scripts/lib/tests -iname "*guiv*" -o -iname "*skel*" -o -iname "*delivery*" against the current PMSS tree; found five skeleton-adjacent tests (IndexSkeletonFrameDataTest, SkeletonWebLocalAssetTest, SkeletonInstallerPermissionsTest, FilemanagerSkeletonContractTest, skeletonPathTest) but none test cross-file dependency closure or coupled-pair consistency. Held: the specific ADR 0022 follow-up remains unbuilt.

Bezos (customer, Type 1/2): Working backwards from the intended flow: the first render of a sold feature is a fatal error with no fallback path forward — the issue's own text states the install control was unreachable on the affected account. That is functionally a Type 1 (irreversible) event for the first-impression moment even though the underlying code change is trivially reversible for PM — a broken first render of a purchased feature can't be un-rendered. The fix belongs at the provisioning boundary, before the account is ever handed over, not as a post-hoc remediation that depends on someone noticing, reporting, and waiting. Day 1 test: does the fix increase organizational agility? A non-blocking self-heal guard does (ship fast, low risk); a hard-blocking provisioning gate does NOT (it adds a fleet-wide failure mode to account creation itself and needs careful, deliberate rollout — Type 1 risk on PM's own side too).
Challenged by: re-read the issue's own text for whether a fast per-account remediation already exists, rather than assuming one needs to be invented — the issue itself states "php /scripts/update.php git/main is a full accidental workaround" for an affected account. Held and sharpened: that workaround still depends on a human noticing and running it; a self-serve or automatic remediation (not requiring anyone to notice and wait) is the real customer-obsessed target, since the manual fix already exists and still relies on someone being the detection mechanism.

Jobs (simplicity, focus, no): Say no to inventing a new provisioning-time verification subsystem before finishing what the codebase already started. mediaStack.php already contains function_exists() guards in two other spots in the same file (verified via grep -c) — the file already knows how to degrade gracefully; it simply didn't apply that same discipline to this one call. The simplest fix is the one consistent with the file's own existing craftsmanship: guard the call, ship a placeholder or safe fallback, don't fatal the whole panel over one function. Whether you'd be proud of this test: does the product feel like a designed, resilient system, or a stack of accidents? Right now the worst case is "panel doesn't load at all"; the fix should make the worst case "panel shows a degraded state," matching what welcome.php already does for its own coupled-file risk.
Challenged by: checked whether the graceful-degradation convention is genuinely established codebase-wide or a one-off in welcome.php — grep -c "function_exists(" etc/skel/www/*.php shows the pattern used in 15 different skel/www files (2 to 67 occurrences each, scriptsInc.php itself uses it 67 times). Held with higher confidence: this is a real, established house style, not an isolated precedent being cherry-picked to fit the argument.

Huang (infrastructure, platform): The underlying gap is that PMSS has THREE separate point-solution delivery mechanisms for what should be one platform primitive — "deliver a versioned, internally-consistent file bundle to a home directory" — with provisioning's initial skel-copy, update.php step2's per-user sync loop, and the guiv heal each independently reimplementing "how do I get files onto disk" and none of them guaranteeing atomicity against the other two or against a moving skel source. This isn't isolated to the media-stack pair: grep -ln "require_once __DIR__" across skel/www turns up 13+ files with cross-file dependencies, including a four-deep chain (panelSessionPage → panelSessionAuth → panelSession → panelSessionStore) — any one of those chains can suffer the identical mismatched-delivery failure the next time skel changes mid-copy. Building the atomic bundle-delivery primitive once, and making all three call sites consume it, is a platform investment; patching media-stack specifically is a point-solution that leaves the other 12+ coupled chains exposed to the same class of bug.
Challenged by: verified the chain depth claim directly with grep -n "require_once __DIR__" etc/skel/www/panelSession*.php output above rather than asserting genericly — confirmed real four-file chain, not a hypothetical example. Held: the platform-scope argument is grounded in an actual, deeper dependency chain than the two-file pair this issue reports.

Anti-Pattern Hunt

REGEX_FOR_SEMANTICS: PASS — no regex/keyword rule is being used to infer meaning, intent, or classification anywhere in this issue or its suggested fix.
PREMATURE_OPTIMIZATION: PASS — the suggested fix is a targeted consistency check tied to a demonstrated failure, not a cap/limit/cooldown layered on an unproven base feature; the closer risk (a blocking check shipped before understanding the actual race) is captured in the Merit Check catastrophic-if-implemented chain, not this class.
FALSE_COMPLETION_CLAIM: PASS — the issue is an open bug report; no completion, fix, or verification is claimed anywhere in it.
Operator-handoff pattern: PASS — the issue includes a concrete, owned Suggested Fix section rather than punting the decision to the operator with no recommendation.
FABRICATED_AUTHORITY_OR_PATH: PASS — spot-checked the cited artifacts directly rather than trusting the issue text: scripts/lib/update/users/filesystem.php genuinely lists both files at lines 212 and 235; ADR 0022 exists at the cited path with content matching the quoted invariant; userMediaStackPanel.php genuinely defines pmssMediaStackPanelHomePath-adjacent functions and is required by mediaStack.php. No invented paths or quotes found.
HEAD_OF_LINE_BLOCKING: PASS (not yet applicable) — this is the first review pass on a freshly-filed issue; no repeated re-selection without progress has occurred yet. Given the defect CLASS's third-recurrence history, this is worth watching across future passes if this specific issue itself stalls.
READINESS_EVAL_CITES_STALE_BLOCKER: PASS (not yet applicable) — no readiness/NO-GO evaluation exists yet on this issue to check for stale-blocker citation.

Synthesis

Consensus (4+ personas): Musk, Jobs, Huang, and Buffett converge on the same structural point from different angles: the correct response is not the issue's literal proposed mechanism (a new bespoke provisioning-time checker) but fixing the underlying non-atomic, multi-path delivery of a shared file bundle — because the same coupling pattern recurs across 13+ files (Huang), the split itself is architecturally justified and shouldn't be undone (Musk, citing ADR 0022's own rejection of inlining), the codebase already has an established graceful-degradation convention that just wasn't applied here (Jobs), and this is unpaid debt from a June ADR follow-up that was never built (Buffett, Munger). That's five of seven converging on "finish/extend the existing pattern, don't build something new" — genuine high-confidence signal.

Divergence: Bezos and Hormozi push urgency (Day-0 signup churn, invisible in ticket volume, fix it now) while Buffett and Munger frame this as a slower architectural-debt problem where the immediate stopgap (the function_exists() guard) is cheap and sufficient near-term, and the "real" fix (the CI gate) is a deliberate, unhurried build — not something to rush given the same rush is what caused the June 3 fleet outage ADR 0022 postmortems. The tension is real: ship a narrow guard today (all seven agree on that much) vs. how much urgency to attach to the general dependency-closure work this issue's Suggested Fix is really asking for.

Blind spots: None of the seven lenses directly interrogates WHETHER the race actually happens the way hypothesized (concurrent skel refresh during provisioning's copy step) versus some other provisioning-code-path bug entirely — the issue's own author explicitly states "I have not traced the exact provisioning code path that produced the mismatched pair." That's an engineering-forensics gap no business/strategy lens fills; only the investigation lane's source-level trace can close it.

Lingering unknowns: (1) The exact provisioning code path that produced the mismatch is untraced — is it genuinely a concurrent-skel-refresh race, or a different bug in the provisioning script itself? (2) What is the actual incidence rate across the fleet — is #953 a one-off or the first-detected instance of an ongoing, silent trickle? (3) Is /etc/skel delivered to production hosts atomically (e.g. a versioned-directory symlink swap) or file-by-file — this repo's etc/skel/ reflects source content, not the live host delivery mechanism, and verifying host-side atomicity would require SSH access outside this review's scope.

Bottom line: Ship the cheap, in-file function_exists() guard now (all seven agree it is low-cost and consistent with existing PMSS convention) as an immediate stopgap, and build the ADR 0022 dependency-closure CI gate next for this defect class — that gate was already scoped and accepted in June, so building it now closes a specific, named, three-months-open gap rather than waiting for a fourth recurrence to justify it.

— Väinämöinen

---

_{🤖 automated-run · runner=dev-persona-20260925065704-3778542 · type=dev-persona · session=c3fbb81b-73c7-44ae-8561-7e769271d5c8 · 2026-09-25T05:21Z}