Include all subtests in HTML report with outcome styling - #60
Merged
Conversation
Show visual comparison panels (baseline, rendered, FLIP heatmap) for every subtest outcome, not just failures. Panels use green borders for passes, red for failures, and grey for skips, with a colored badge label. Image cells use black backgrounds for transparent renders and print-color-adjust for PDF fidelity. Generate FLIP heatmaps conditionally: only when --html is active (for the report) or when FLIP exceeds the threshold (for debugging). Expand all report rows by default via render_collapsed.
There was a problem hiding this comment.
Pull request overview
Expands HTML render reports to show outcome-styled visual comparisons for all subtests.
Changes:
- Generates FLIP heatmaps for HTML reports or failed comparisons.
- Adds passed, failed, and skipped report styling.
- Expands report rows by default.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
contrib/tests/test_render.py |
Adds HTML-aware heatmap generation. |
contrib/tests/conftest.py |
Builds outcome-styled visual report panels. |
contrib/tests/pyproject.toml |
Expands HTML report rows by default. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+350
to
+354
| valid_elem_name = mx.createValidName(subtest_name) | ||
| rendered_files = list(output_path.glob(f"{valid_elem_name}_*.png")) | ||
| rendered_files = [ | ||
| f for f in rendered_files if not f.name.endswith("_diff.png") | ||
| ] |
Comment on lines
276
to
+279
| def pytest_runtest_logreport(report): | ||
| """Append visual comparisons for failed subtests to the HTML report.""" | ||
| if type(report).__name__ == "SubtestReport" and report.failed: | ||
| try: | ||
| from pytest_html import extras | ||
| except ImportError: | ||
| return | ||
|
|
||
| funcargs = _node_funcargs.get(report.nodeid) | ||
| if not funcargs: | ||
| return | ||
|
|
||
| from test_render import RenderEnvironment, RenderTestCase | ||
|
|
||
| case = funcargs.get("case") | ||
| if not isinstance(case, RenderTestCase): | ||
| return | ||
|
|
||
| env = None | ||
| for arg_val in funcargs.values(): | ||
| if isinstance(arg_val, RenderEnvironment): | ||
| env = arg_val | ||
| break | ||
| if not env: | ||
| return | ||
|
|
||
| context = getattr(report, "context", None) | ||
| subtest_name = context.msg if context else None | ||
| if not subtest_name: | ||
| return | ||
|
|
||
| output_path = env.get_output_path(case) | ||
|
|
||
| if not output_path or not output_path.exists(): | ||
| return | ||
|
|
||
| # Find the rendered file | ||
| import MaterialX as mx | ||
| valid_elem_name = mx.createValidName(subtest_name) | ||
| rendered_files = list(output_path.glob(f"{valid_elem_name}_*.png")) | ||
| rendered_files = [ | ||
| f for f in rendered_files if not f.name.endswith("_diff.png") | ||
| ] | ||
|
|
||
| if not rendered_files: | ||
| return | ||
|
|
||
| rendered_file = rendered_files[0] | ||
|
|
||
| # Derive ref image from the environment's image_ref_env_subpath | ||
| image_ref_dir = env.get_image_ref_dir(output_path) | ||
| if image_ref_dir: | ||
| baseline_file = image_ref_dir / rendered_file.name | ||
| if not baseline_file.exists(): | ||
| baseline_file = None | ||
| else: | ||
| baseline_file = None | ||
| heatmap_file = rendered_file.parent / f"{rendered_file.stem}_diff.png" | ||
|
|
||
| # Determine HTML report directory to compute relative paths for images | ||
| import os | ||
| try: | ||
| htmlpath_str = ( | ||
| _pytest_config.getoption("htmlpath") | ||
| if _pytest_config | ||
| else None | ||
| ) | ||
| except ValueError: | ||
| htmlpath_str = None | ||
| html_dir = ( | ||
| Path(htmlpath_str).parent.resolve() if htmlpath_str else None | ||
| ) | ||
|
|
||
| try: | ||
| is_self_contained = ( | ||
| _pytest_config.getoption("self_contained_html") | ||
| if _pytest_config | ||
| else False | ||
| ) | ||
| except ValueError: | ||
| is_self_contained = False | ||
|
|
||
| def get_image_src(path: Path) -> str: | ||
| if not path or not path.exists(): | ||
| return "" | ||
| if is_self_contained: | ||
| import base64 | ||
| try: | ||
| with open(path, "rb") as f: | ||
| encoded = base64.b64encode(f.read()).decode("utf-8") | ||
| return f"data:image/png;base64,{encoded}" | ||
| except Exception: | ||
| pass | ||
| elif html_dir: | ||
| try: | ||
| return os.path.relpath( | ||
| path.resolve(), html_dir, | ||
| ).replace("\\", "/") | ||
| except ValueError: | ||
| return path.resolve().as_uri() | ||
| return path.resolve().as_uri() | ||
|
|
||
| rendered_src = get_image_src(rendered_file) | ||
| baseline_src = get_image_src(baseline_file) | ||
| heatmap_src = get_image_src(heatmap_file) | ||
|
|
||
| if not rendered_src: | ||
| return | ||
|
|
||
| if baseline_src: | ||
| baseline_img_tag = ( | ||
| f'<img src="{baseline_src}" style="max-width: 100%; ' | ||
| f'height: auto; border: 1px solid #ccc; border-radius: 4px;" />' | ||
| ) | ||
| else: | ||
| baseline_img_tag = ( | ||
| '<div style="padding: 50px 10px; background: #eee; ' | ||
| 'border: 1px dashed #ccc; border-radius: 4px; color: #666; ' | ||
| 'font-size: 12px;">Baseline image missing</div>' | ||
| ) | ||
|
|
||
| if heatmap_src: | ||
| heatmap_img_tag = ( | ||
| f'<img src="{heatmap_src}" style="max-width: 100%; ' | ||
| f'height: auto; border: 1px solid #ccc; border-radius: 4px;" />' | ||
| ) | ||
| else: | ||
| heatmap_img_tag = ( | ||
| '<div style="padding: 50px 10px; background: #eee; ' | ||
| 'border: 1px dashed #ccc; border-radius: 4px; color: #666; ' | ||
| 'font-size: 12px;">No heatmap (comparison passed or skipped)</div>' | ||
| ) | ||
|
|
||
| html_content = f""" | ||
| <div style="margin-top: 15px; padding: 15px; border: 1px solid #e74c3c; border-radius: 6px; background: #fdf2f2; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;"> | ||
| <h4 style="margin: 0 0 12px 0; color: #c0392b; font-size: 14px;">Visual Comparison for {subtest_name}</h4> | ||
| <div style="display: flex; gap: 15px; flex-wrap: wrap;"> | ||
| <div style="flex: 1; min-width: 220px; text-align: center;"> | ||
| <div style="font-weight: bold; margin-bottom: 6px; font-size: 12px; color: #555;">Baseline (Reference)</div> | ||
| {baseline_img_tag} | ||
| </div> | ||
| <div style="flex: 1; min-width: 220px; text-align: center;"> | ||
| <div style="font-weight: bold; margin-bottom: 6px; font-size: 12px; color: #555;">Rendered (Current)</div> | ||
| <img src="{rendered_src}" style="max-width: 100%; height: auto; border: 1px solid #ccc; border-radius: 4px;" /> | ||
| </div> | ||
| <div style="flex: 1; min-width: 220px; text-align: center;"> | ||
| <div style="font-weight: bold; margin-bottom: 6px; font-size: 12px; color: #555;">FLIP Heatmap</div> | ||
| {heatmap_img_tag} | ||
| </div> | ||
| </div> | ||
| </div> | ||
| """ | ||
| _subtest_html_extras[report.nodeid].append(extras.html(html_content)) | ||
| """Append visual comparisons for every subtest to the HTML report.""" | ||
| if type(report).__name__ == "SubtestReport": | ||
| _append_visual_extras(report) |
Preserve the original semantics where NaN FLIP values fail the comparison instead of silently passing. Update README to document all-outcome visual panels and conditional heatmap generation.
ppenenko
force-pushed
the
metashade/html-report-all-tests
branch
from
August 24, 2026 21:10
8caa084 to
449c318
Compare
Missing baselines should surface as test failures, not silent skips that hide the problem in CI and reports. Signed-off-by: Pavlo Penenko <pavlo.penenko@autodesk.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
Suppressed comments (2)
contrib/tests/conftest.py:346
- With
--no-render --html, this hook can find PNGs left in the persistent developer output tree and label them as the current render even though this run produced no image. Short-circuit no-render runs (or track files produced by the current subtest) before scanning the output directory.
output_path = env.get_output_path(case)
if not output_path or not output_path.exists():
contrib/tests/conftest.py:355
- This still omits skipped subtests on a clean output tree: both skip branches in
test_render.py:600-607run beforerender_element, so no PNG exists and this return prevents the promised grey panel/badge. If an old PNG does exist, it is instead presented as the current render. Emit an outcome-only/placeholder panel for skipped subtests without relying on a filesystem artifact from an earlier run.
rendered_files = list(output_path.glob(f"{valid_elem_name}_*.png"))
rendered_files = [
f for f in rendered_files if not f.name.endswith("_diff.png")
]
if not rendered_files:
Comment on lines
+493
to
+497
| Uses NVIDIA FLIP to compute a perceptual difference metric. | ||
| Saves a magma heatmap (``*_diff.png``) next to the rendered image | ||
| when *save_heatmap* is ``True`` (for the HTML report) or when the | ||
| mean FLIP error exceeds *threshold*. Skips gracefully when | ||
| reference images are missing. |
Comment on lines
+155
to
+159
| generating an HTML report (`--html`), all subtests (passed, failed, | ||
| and skipped) include side-by-side panels showing the reference image, | ||
| rendered image, and FLIP heatmap, styled by outcome. Heatmaps are | ||
| generated whenever `--html` is active or when FLIP exceeds the | ||
| threshold. |
The docstring for _compare_render and the README still said missing references are skipped; they now fail. Signed-off-by: Pavlo Penenko <pavlo.penenko@autodesk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--htmlis active or when FLIP exceeds thresholdrender_collapsed = ""in pyproject.toml)print-color-adjustfor PDF fidelityCommits
pytest.fail()replacespytest.skip()for missing baselinesTest plan
pytest test_render_metashade.py --no-render— all shader tests pass--html, no heatmaps generated for passing tests