Skip to content

AtmosphereVisualEvidenceTest: NightClear/NightOvercast goldens drift (RMSE 13.08 / 9.40 vs threshold 8) — permanently red #754

Description

@drsnuggles8

Describe the bug
AtmosphereVisualEvidenceTest.DayNightWeatherMatrixRendersAndHoldsContracts fails on every run, in both the slot-based and heap-bindless configurations, on two of its matrix cells:

AtmosphereVisualEvidenceTest.cpp(408): error: Expected: (rmse) <= (kGoldenRmseThreshold),
    actual: 13.077706847047567 vs 8
    Capture 'NightClear' drifted from its golden (RMSE 13.077706847047567)

AtmosphereVisualEvidenceTest.cpp(408): error: Expected: (rmse) <= (kGoldenRmseThreshold),
    actual: 9.4036103562132709 vs 8
    Capture 'NightOvercast' drifted from its golden (RMSE 9.4036103562132709)

It is deterministic, not flaky. The two RMSE values are identical to the last digit across runs taken days apart, on different builds, in both configurations. This is a fixed offset between what the renderer produces and what the committed goldens hold — not noise, and not a threshold that is marginally too tight.

The failure is confined to NIGHT, and that is the diagnostic lead. NightClear and NightOvercast fail; NightStorm passes, as do all of Dawn, Dusk and Noon. Whatever changed is specific to the night sky path (star field, moon, night-side scattering) and appears to be suppressed by heavy overcast — which is presumably why NightStorm survives.

Root cause (unknown — two candidates, and they have opposite fixes)

  1. The goldens are stale. They were last written by 91e1e10 (World: atmosphere & sky — weather director, astronomical time-of-day, volumetric clouds #633, atmosphere/weather director), and a later change altered night rendering without rebasing them. Fix: confirm the current output is correct, then rebase.
  2. Night rendering regressed after those goldens were captured. Fix: find and fix the regression — rebasing would silently bless the bug.

Do not rebase without looking at the images first. RMSE 13 on a night sky is not subtle; per CLAUDE.md's rendering-verification rule, open Atmosphere_NightClear.png against its golden and decide which of the two it is. A blind OLOENGINE_GOLDEN_REBASE=1 here would convert a possible rendering regression into a permanent, invisible baseline.

To Reproduce

build\OloEngine\tests\Debug\OloEngine-Tests.exe --gtest_filter=AtmosphereVisualEvidenceTest.*

Requires a GL 4.6 context (SKIPs headless). Fails identically every time.

Expected behavior
Either the goldens match the renderer, or they are rebased deliberately after visual confirmation.

Why it matters beyond one red test
This is the only consistently failing test in the suite, so "N passed, 1 failed" has become the expected result — precisely the state in which a new regression is invisible. During #691 Phase 3 this test was repeatedly the known-good baseline for "did my change break anything?", and every such comparison was one unrelated red away from ambiguity.

Not caused by #691. The bindless branch touches neither AtmosphereVisualEvidenceTest.cpp nor AtmosphereSky.glsl/StarNestSky.glsl; it fails identically with the heap disabled (the shipped configuration); and the goldens' last-touching commit is on master.

Screenshots / logs
Goldens: OloEditor/assets/tests/visual/Atmosphere_Night{Clear,Overcast,Storm}.png.

Score

capability: 1
craft: 3
stability: 3
decay: 3
effort: 1
confidence: 0.8
learning: 1
fun: 2
kano: table-stakes
blocked_by: []
blocks: []

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrobustnessProduction hardening / shipping robustness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions