Skip to content

Commit a951a19

Browse files
committed
fix: publish the cost window's own load, and drop the aborted run's orphan row
AUDITING THE FIRST COST-CARRYING SNAPSHOT FOUND A NUMBER NOBODY COULD CHECK. one-api published cpu_us_per_request 13,207 and cost_core_utilisation 2.56%. Those two are consistent only if the cost window carried about 7.75 rps - but the sweep's rung at the SAME concurrency carried 35. Both readings are defensible and they mean opposite things: either the gateway is genuinely cheap, or that window simply did less work than the sweep did. Nothing in the artifact could separate them, because `cpu_us_per_request` is CPU divided by a request count that was published NOWHERE. A number that re-derives from nothing is the thing this project refuses to ship. `cost_window_ok` and `cost_window_rps` now publish the window's own completed count and rate, so the cost divides by something a reader can see. Worth recording, because it nearly passed: my first check of these figures was `cpu_us_per_request x rps_per_cpu_second == 1,000,000`, and it "passed". It is TAUTOLOGICAL - the two are the same quantity inverted, so the product is a million by construction whatever the data says. A check that cannot fail told me the numbers were sound while the real question went unasked. ALSO REVERTED: the aborted run's one-api row (525f9c5). Killing a run mid-flight left one gateway published at engine 26c7436 while the other thirteen sat at dd58497 - a mixed board, and a row measured by a run that was stopped before it could be trusted. R3 caught it immediately ("the board is rendering from a stale/mis-selected run"), which is the freshness guard doing exactly its job. An aborted run must leave nothing behind. 514 engine tests, clippy/fmt clean, 206 site tests, check-consistency clean, auditor fixtures pass.
1 parent 525f9c5 commit a951a19

6 files changed

Lines changed: 30 additions & 17673 deletions

File tree

audit-every-metric.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@
3838
# (`procsample::cost` returns HarnessError rather than subtracting into a negative) - so if one
3939
# ever reaches the artifact, the refusal has been bypassed and this is the second line of defence.
4040
"cpu_us_per_request", "rps_per_cpu_second", "cost_window_conc", "cost_threads",
41-
"cost_core_utilisation",
41+
"cost_core_utilisation", "cost_window_ok", "cost_window_rps",
4242
"cost_nonvol_ctxt_per_request", "cost_majflt",
4343
}
4444
# THERE IS NO p50<=p99 CHECK HERE, AND THE ONE THAT USED TO BE WAS UNSOUND.

engine/src/metric.rs

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1764,6 +1764,8 @@ impl Metric for Cost {
17641764
"cpu_us_per_request",
17651765
"rps_per_cpu_second",
17661766
"cost_window_conc",
1767+
"cost_window_ok",
1768+
"cost_window_rps",
17671769
"cost_core_utilisation",
17681770
"cost_threads",
17691771
"cost_nonvol_ctxt_per_request",
@@ -1825,6 +1827,16 @@ impl Metric for Cost {
18251827
"cost_window_conc",
18261828
Measurement::Measured(f64::from(COST_WINDOW_CONCURRENCY)),
18271829
),
1830+
// THE WINDOW'S OWN LOAD, published so the cost is CHECKABLE.
1831+
//
1832+
// Without these, `cpu_us_per_request` is unverifiable: it can only be re-derived from the
1833+
// request count it was divided by, and that count lived nowhere. Auditing the first
1834+
// cost-carrying snapshot, I could not tell whether a low utilisation meant the gateway
1835+
// was cheap or the window had simply carried less load than the sweep's rung at the same
1836+
// concurrency - two opposite readings of the same number, with nothing in the artifact to
1837+
// separate them. Every published number must re-derive from what is published beside it.
1838+
("cost_window_ok", Measurement::Measured(stats.ok as f64)),
1839+
("cost_window_rps", Measurement::Measured(stats.rps())),
18281840
// THE READING THAT SAYS WHETHER THE PEAK IS A CEILING. At ~1.0 the gateway had filled the
18291841
// cores it was given and the throughput number is a wall; well below it, the limit is
18301842
// somewhere else and the peak means something else.

engine/src/record.rs

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -427,6 +427,16 @@ pub struct CellPerf {
427427
pub cost_window_conc: Measurement<i64>,
428428
/// Threads in the gateway's tree during the cost window. Not a rate: it is the shape of the
429429
/// concurrency model, thread-per-connection against async, on evidence rather than on a claim.
430+
/// Requests the cost window actually completed, and the rate it carried.
431+
///
432+
/// PUBLISHED SO THE COST IS CHECKABLE. `cpu_us_per_request` is CPU divided by this count, and
433+
/// without it the figure re-derives from nothing. It also separates two opposite readings of a
434+
/// low utilisation - a genuinely cheap gateway, or a window that simply carried less load than
435+
/// the sweep did at the same concurrency - which are indistinguishable without knowing the load.
436+
#[serde(default = "measurement_default")]
437+
pub cost_window_ok: Measurement<f64>,
438+
#[serde(default = "measurement_default")]
439+
pub cost_window_rps: Measurement<f64>,
430440
/// Utilisation of the cores the gateway was PINNED to, across the cost window. 1.0 = every
431441
/// pinned core fully busy.
432442
///
@@ -870,6 +880,8 @@ mod tests {
870880
cpu_us_per_request: Measurement::Measured(37.5),
871881
rps_per_cpu_second: Measurement::Measured(26_666.0),
872882
cost_window_conc: Measurement::Measured(8),
883+
cost_window_ok: Measurement::Measured(2048.0),
884+
cost_window_rps: Measurement::Measured(341.0),
873885
cost_core_utilisation: Measurement::Measured(0.97),
874886
cost_threads: Measurement::Measured(9.0),
875887
cost_nonvol_ctxt_per_request: Measurement::Measured(0.25),

engine/src/suite.rs

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,8 @@ fn judge_cost(
210210
out.cpu_us_per_request = f("cpu_us_per_request");
211211
out.rps_per_cpu_second = f("rps_per_cpu_second");
212212
out.cost_window_conc = as_i64(metrics.get("cost_window_conc"));
213+
out.cost_window_ok = f("cost_window_ok");
214+
out.cost_window_rps = f("cost_window_rps");
213215
out.cost_core_utilisation = f("cost_core_utilisation");
214216
out.cost_threads = f("cost_threads");
215217
out.cost_nonvol_ctxt_per_request = f("cost_nonvol_ctxt_per_request");
@@ -1335,6 +1337,8 @@ fn withhold_refuted_perf(p: CellPerf, why: &str) -> CellPerf {
13351337
cpu_us_per_request: withheld_f(),
13361338
rps_per_cpu_second: withheld_f(),
13371339
cost_window_conc: withheld(),
1340+
cost_window_ok: withheld_f(),
1341+
cost_window_rps: withheld_f(),
13381342
cost_core_utilisation: withheld_f(),
13391343
cost_threads: withheld_f(),
13401344
cost_nonvol_ctxt_per_request: withheld_f(),

0 commit comments

Comments
 (0)