Add frame pacing against the headset display clock

[monofunc]

tl;dr Adds a server-side pacer that phase-locks frame release to the headset's vsync grid.

Results

Wi-Fi

• Stale frames: 1220 (10.3/s) to 5 (0.06%)
• Skipped frames: 128 to ~0
• Encode callback average: 14.9 ms to 8.7 ms

ADB

• Stale frames: 249 to 53 (0.63%)
• Skipped frames: 1046 to ~0
• Encode callback average: 18.5 ms to 8.9 ms

Tested on Quest 3 + macOS 15 with default settings.

TODO

• Refactor architecture for visionOS (currently falls back to a slightly better replacement for the original throttle)
• Test under link hiccups

[demonixis]

Those numbers are incredible :O I'll start review that tomorrow and merge it as soon as you're ready, thanks!

[dingyifei]

Will need a small rebase if #23 is merged first, this one supersedes #20's "PR8 absolute-grid WaitFrame pacing, generic half (~−40)" and require some hard coded stuff to be changed on "PR6"

[monofunc]

@demonixis @dingyifei did you have a chance to try it? Just want to know it works fine for you. :D

[dingyifei]

It seems to work on my end after some tweaking and I think my game got smoother. The latency issues with my setup require some changes to be made for the frame pacing to work correctly. I tested with modified ALVR server and client so there are some confounding variables there.

I'm in the process of moving encoding out of Rosetta, which will allow me to use H265 and reduce pipeline latency for gaming.

---

Tried the frame pacer on a higher-latency streaming path and hit a case where the closed loop never engages. Sharing a root cause, a fix that works, and numbers, in case it's useful upstream.

Setup

Embedded server_core streaming to a standalone Quest 3 over WiFi (H.264, ~72 Hz), with the encoder running under Rosetta. End-to-end motion-to-photon is ~100–120 ms; median breakdown from the client GRAPH stats:

stage	median
encoder	~15–23 ms
network	~5–8 ms
decoder	~26–29 ms
decoder_queue	~30 ms
vsync_queue	~20 ms
total	~100–120 ms

So the display lag the pacer must model (predicted-display minus release tick) is ~55–70 ms = 4–5 display periods, versus the few-ms lag on a low-latency USB path.

Symptom

In closed-loop mode the pacer never leaves the warming state:

• poseOffset rails at 55.56 ms (= 4 * period, the MaxDisplayOffsetPeriods clamp)
• floor = 0.00 ms (warming) indefinitely; render lead stays at the 20 ms startup default
• unattributedFrames climbs ~700/interval — every feedback packet comes back with slack invalid

Grid lock itself is fine (skippedTicks=0, wake precision ~0.03–0.05 ms) — only the adaptive layer is dead, so it silently degrades to a fixed-lead pacer.

Root cause

Three constants (FramePacer.h / OnDisplayLag) assume the pipeline fits in a couple periods, and they form a dependency cycle that can't bootstrap otherwise:

1. Pose-offset clamp MaxDisplayOffsetPeriods = 4 — servo needs ~55–70 ms but caps at 4*period ≈ 55.6 ms, rails at the edge.
2. Settle gate |displayLag| < period/4 (~3.5 ms) — decoder time swings 26→48 ms, so residual jitters past a quarter period and the streak keeps resetting.
3. Render-lead cap + stall gate MaxRenderLeadPeriods = 3 (~42 ms), StallGatePeriods = 2 (~28 ms) — required lead is ~65 ms; startup lead is 20 ms so first samples are ~-45 ms slack, beyond the 2-period stall gate, discarded as "stalls" → lead never bumps up → floor never gets a non-negative sample → stays warming → settled target stays 0 → server stamps target=0 → client marks every slack invalid. Circular.

What worked

Widening the three envelopes so a deep pipeline can bootstrap, without changing shallow-path behavior (wider bounds are strictly permissive):

• MaxDisplayOffsetPeriods 4 → 10
• settle gate period/4 → ~period/2
• MaxRenderLeadPeriods 3 → 8, StallGatePeriods tracked to MaxRenderLeadPeriods

Result on the same hardware: floor warming → learned at ~52–72 ms, unattributedFrames → 0, lead settles ~66–77 ms, skippedTicks stays 0.

Suggestion

1. Expose these as tunables (or derive from a declared pipeline-latency budget) — depth is a transport/decode property, not a universal constant.
2. Or derive the envelopes from the observed pipeline so the pacer self-scales.

---

Great news:

After I move encode from rosetta to native and switching to HEVC, encode delay and decode queue delay are greatly reduced

┌───────────────┬────────────┬────────────────┬────────────────┐
│     Stage     │  HEVC now  │ H.264 baseline │       Δ        │
├───────────────┼────────────┼────────────────┼────────────────┤
│ Total M2P     │ 93.5ms p50 │ ~114ms         │ -20ms          │
├───────────────┼────────────┼────────────────┼────────────────┤
│ Quest decoder │ 16.6ms     │ ~28-32ms       │ roughly halved │
├───────────────┼────────────┼────────────────┼────────────────┤
│ Decoder queue │ 1.1ms      │ ~30ms          │ collapsed      │
├───────────────┼────────────┼────────────────┼────────────────┤
│ Network       │ 2.7ms      │ ~6.4ms (WiFi)  │ USB helping    │
├───────────────┼────────────┼────────────────┼────────────────┤
│ Encoder       │ 37.6ms     │ ~35-40ms       │ unchanged      │
└───────────────┴────────────┴────────────────┴────────────────┘

I think it is approaching usable level M2P (USB helping is because I've been using Wifi -> Router -> Wifi).

Given most setups have m2p at 50ms~80ms, it is really close to that without fine tuning

Btw the HEVC data produced didn't use your pacer, but a naive implementation, so your pacer implementation should result in even better perf!