Expand regression testing for supported demos

[jarcherNV]

Summary

Add reusable benchmark tools for realtime demos that can run locally first, then later power CI smoke tests and scheduled regression tracking.

The goal is to make realtime performance and quality validation repeatable, rather than relying on manual demo launches, copied logs, and ad hoc visual inspection for every acceleration or refactor PR.

Motivation

Recent realtime infrastructure work required repeatedly launching demos by hand, capturing logs, comparing latency numbers, and visually checking generated output. That process is slow, subjective, and hard to repeat consistently.

A shared benchmark harness would also help when bringing up future models and demos, because it would provide a common way to compare acceleration features, measure startup and latency impact, and inspect output quality.

Proposal

Build one benchmark tool that supports multiple execution modes:

• Local developer mode: run selected demo scenarios, save outputs, collect metrics, and generate an HTML report for manual review.
• CI smoke mode: run a short, controlled subset that catches crashes, blank output, severe latency regressions, and obvious functional failures.
• Scheduled benchmark mode: run longer benchmark suites for trend tracking, stricter performance analysis, and automated quality evaluation.

The first implementation should focus on the local developer mode. CI gating can be added after the metrics, report format, and baselines have had real use.

Metrics To Capture

The harness should collect structured metrics such as:

• startup and model load time
• warmup/prewarm time
• first-frame or first-chunk latency
• steady-state chunk latency
• per-stage timing where available, such as encode, diffuse, decode, finalize, raster, presentation, and transport
• GPU memory allocated, reserved, and peak usage
• dropped frames, stalls, late chunks, or other realtime health indicators
• branch, commit, GPU, CUDA/runtime, model config, scenario, and command metadata

Quality Reporting

Where available, the harness should support automated quality evaluation, such as WorldLens metrics for:

• overall visual quality
• temporal coherence
• visual consistency
• prompt or scene adherence
• other model/demo-specific quality signals

These should initially be reported and tracked rather than used as strict PR gates, until the expected variance is understood.

Artifacts

Each run should produce useful artifacts for both humans and automation:

• saved output videos, preferably MP4
• structured JSON/CSV metrics
• logs for each scenario
• an HTML report with timing summaries, charts, memory usage, quality scores, environment metadata, and links/previews for generated videos

CI Follow-Up

After the local harness is stable, add CI integration in stages:

1. Add a short smoke suite that verifies selected realtime demos start, generate valid output, and do not regress severely.
2. Add scheduled benchmark jobs for longer performance and quality tracking.
3. Promote specific metrics to hard PR gates only after baselines and variance are well understood.

Acceptance Criteria

• A developer can run one command locally to benchmark one or more realtime demo scenarios.
• The harness emits structured machine-readable metrics.
• The harness generates an HTML report with saved output videos for visual inspection.
• Startup, latency, memory, and stage timing metrics are captured.
• Automated quality evaluators can be plugged in when available.
• The output format is suitable for later CI smoke tests and scheduled regression tracking.

[jarcherNV]

I don't think this would replace the current benchmark harness or quality testing in the CI, instead I think this would first just be a script you can run locally, and then we would hook that up to the existing CI stuff for expanded regression testing there.