design: reconciliation policy for reported vs measured device profile fields (post-#55)

[drunkcoding]

Context

#55 (now closed) added server-side measurement of five device-profile fields (measured_lat_ns, measured_ul_bw_bps, measured_dl_bw_bps, measured_flops, measured_flops_verified) via the new M1→M2→M3 probe pipeline. These are written to the device-tracker side-by-side with the seven device-reported legacy fields (flops, memory, ul_bw, dl_bw, ul_lat, dl_lat, uuid) via DeviceTracker::UpdateMeasuredProfile; the legacy fields are explicitly never overwritten.

This was deliberate. #55 carved out reconciliation as "out of scope — separate issue once data is flowing" so the trust-model fix could land without prejudging downstream consumers.

The question

Once measurement is enabled in production and we've seen real measured-vs-reported skews, what value should downstream schedulers and analytics consume when they ask for "the device's FLOPS / bandwidth / latency"?

Three obvious policies, each with tradeoffs:

Policy	Pro	Con
min(reported, measured)	Conservative; resists devices over-stating capability	Discards real measurement signal if device under-reports honestly
measured (when present, else reported)	Uses the most defensible number	Single probe is a point estimate; can be noisy
Weighted blend (e.g. EMA over multiple measurement rounds)	Smooths noise; tracks drift	Needs storage, decay-constant tuning, repeated probing (which #55 explicitly does not do)

Inputs needed before deciding

1. Real data. Run Device measurement: probe wire protocol + server coordinator (#45 steps 3-8) #55's --measurement flag in a representative deployment for a few hours / days and look at the distribution of measured / reported ratios per field per device. The decision is data-driven, not a priori.
2. Consumer audit. Identify every reader of DeviceProfileData::flops / bandwidth / latency in the scheduler, dispatch gate, and analytics. Each may want a different policy (analytics probably wants raw both; scheduler probably wants min).
3. Re-probe cadence. Decide whether measurement is one-shot at registration (current behavior) or periodic. The policy choice changes a lot if re-probing happens.

Suggested next step

Don't pick a policy until #55 has been running in at least one real deployment for long enough to populate logs/perf_server.log with measured-vs-reported deltas. Then write a short data-driven proposal and ship the chosen policy as a separate, single-purpose PR. Until then, both fields remain available side-by-side and consumers explicitly choose.

Related

• Device measurement: probe wire protocol + server coordinator (#45 steps 3-8) #55 (closed)
• Device measurement: trust model needs revisit #45 (closed)
• csrc/backend/device_tracker.cpp — UpdateMeasuredProfile and the DeviceProfileData getters that consumers go through

[drunkcoding]

Data-collection plumbing for this decision is now in place on dev (commits 2e36b20, 92c7dca, 1bce883). The issue stays open — no reconciliation policy has been chosen; this is the prerequisite observability the "Suggested next step" called for.

What landed (policy-neutral):

• Every measured-profile update now emits one PROFILE_DELTA row to logs/perf_server.log with reported vs measured vs ratio per field (flops/ul_bw/dl_bw), via a new LogProfileDelta in device_tracker.cpp wired into UpdateMeasuredProfile.
• Row-format logic is a pure FormatProfileDeltaRow() seam (csrc/backend/profile_delta_format.h), unit-tested in tests/cpp/unit/zerocopy/test_profile_delta_log.cpp.
• docs/deployment.md documents the schema + analysis workflow.

Confirmed unchanged: the vtime calculator (proxy_svr.cpp:452-461) still reads the device-reported legacy fields verbatim — no scheduling behavior changed. With the shipped default (probes off) no rows are emitted.

Audit finding worth recording for the eventual policy: the measured_* fields were previously write-only for decision-making (stored/serialized/logged but never read by any scheduler). The single consumer of perf numbers is that one vtime block, so a future reconciliation helper has exactly one injection point.

Unit caveat the data exposes: reported latency is in microseconds (ul_lat/dl_lat), measured latency is in nanoseconds (measured_lat_ns) — a 1000x mismatch. The log deliberately emits no latency ratio column, only raw columns side by side, so analysis normalizes rather than inheriting the error. Whoever picks the policy must reconcile this unit difference first.

Next step (unchanged from the issue): run a representative deployment with --measurement, then grep '^PROFILE_DELTA' logs/perf_server.log and study the per-field ratio distribution before choosing min / measured / blend.