Summary
The upload benchmark currently rewards a server for NOT ingesting the request body. A server can answer the ~60-byte request head with a 200 (reading Content-Length from the header), then close/RST the connection without ever reading the 20 MB body — and it scores as the fastest upload server, because it does the least work. Neither the throughput scoring nor validate.sh detects this.
I found this while optimizing an engine's upload path (vanilla): its epoll backend had a latent bug that made it answer from the head and close, and it topped the upload chart precisely because of that bug. Any engine — accidentally or deliberately — can do the same. This is a harness-robustness gap, not a single-engine problem.
Evidence (local, 8 conns, 16 MiB bodies, default GC)
| Server behavior |
Req/s |
Client socket errors |
Body actually ingested? |
Server peak RSS |
| Answer from head + close (the "cheat") |
27,691 |
278,891 writes |
❌ no (RST after head) |
11 MiB |
| Correctly stream + ingest the body |
615 |
0 |
✅ yes |
6 MiB |
The "cheat" is 45× faster on the leaderboard while transferring almost none of the body — and a load generator records ~278k write failures that the harness never looks at.
Root cause in the harness
1. Throughput = 2xx/sec, with no error accounting.
scripts/lib/tools/gcannon.sh → gcannon_parse() computes rps from the 2xx count only:
rps = 2xx_count / duration
A server that answers every request from the head with a 200 and discards the body maximizes 2xx/sec. Client-side write/connection-reset errors (which a non-ingesting server causes en masse) are never parsed, so they don't penalize the score.
2. The /upload correctness contract is satisfiable without reading the body.
scripts/validate.sh (the upload block, ~L767–805) asserts the response equals the declared Content-Length:
ACTUAL_LARGE=$( { dd if=/dev/urandom bs=1024 count=$upload_bs | \
curl -s --max-time 60 -X POST --data-binary @- ".../upload"; } || true )
[ "$ACTUAL_LARGE" = "$upload_size" ] && PASSBut the byte count the engine returns comes from the Content-Length header, which is in the head — so a server that never reads the body still echoes the right number and passes. The || true also swallows a curl broken-pipe error (exit 55) caused by the early close, and %{size_upload} is not checked, so a partial/aborted upload still validates.
Net: the cheat is invisible to both the score and the correctness gate.
Suggested improvements (any subset helps)
-
Make the /upload response depend on the body content, not its declared length. Strongest fix: require the handler to return a checksum/digest (e.g. CRC32/FNV/SHA-1) or the last N bytes of the received body. A server that discards the body cannot compute it, so it must actually ingest. validate.sh computes the same digest client-side and compares.
-
Validate full-body transfer in validate.sh. Drop || true, check curl's exit code, and assert curl -w '%{size_upload}' equals the intended body size. An early close → non-zero exit / short size_upload → fail.
-
Parse and penalize load-generator errors. Have gcannon_parse() extract connection-reset / write-error / non-2xx counters and fail (or zero) the result when they exceed a small threshold. A correct upload server produces ~0; a head-and-close server produces ~1 per request.
-
Rank upload on body throughput, not request rate. Report MB/s of accepted request body (bytes the server actually drained), not 2xx/sec. This naturally rewards efficient ingestion and gives a non-ingesting server a near-zero score. gcannon already tracks bytes sent/accepted.
-
(Optional) Slow-body probe. Send the body in timed fragments and assert the response arrives only after the last fragment — catches "respond-before-drain" servers that desync real clients even when they do eventually read the body.
Context / fix on the engine side
The companion fix in the vanilla engine makes both backends drain-then-respond (stream the body off the socket, then answer) so they ingest the body correctly and keep-alive — io_uring upload throughput +58% and peak RSS −41% in the same local test. PR: enghitalo/vanilla#61
Happy to send a PR for any of the harness improvements above (1+2 are small and high-value).
Summary
The
uploadbenchmark currently rewards a server for NOT ingesting the request body. A server can answer the ~60-byte request head with a200(readingContent-Lengthfrom the header), then close/RST the connection without ever reading the 20 MB body — and it scores as the fastest upload server, because it does the least work. Neither the throughput scoring norvalidate.shdetects this.I found this while optimizing an engine's upload path (
vanilla): its epoll backend had a latent bug that made it answer from the head and close, and it topped theuploadchart precisely because of that bug. Any engine — accidentally or deliberately — can do the same. This is a harness-robustness gap, not a single-engine problem.Evidence (local, 8 conns, 16 MiB bodies, default GC)
The "cheat" is 45× faster on the leaderboard while transferring almost none of the body — and a load generator records ~278k write failures that the harness never looks at.
Root cause in the harness
1. Throughput = 2xx/sec, with no error accounting.
scripts/lib/tools/gcannon.sh→gcannon_parse()computesrpsfrom the 2xx count only:A server that answers every request from the head with a
200and discards the body maximizes 2xx/sec. Client-side write/connection-reset errors (which a non-ingesting server causes en masse) are never parsed, so they don't penalize the score.2. The
/uploadcorrectness contract is satisfiable without reading the body.scripts/validate.sh(theuploadblock, ~L767–805) asserts the response equals the declaredContent-Length:But the byte count the engine returns comes from the
Content-Lengthheader, which is in the head — so a server that never reads the body still echoes the right number and passes. The|| truealso swallows acurlbroken-pipe error (exit 55) caused by the early close, and%{size_upload}is not checked, so a partial/aborted upload still validates.Net: the cheat is invisible to both the score and the correctness gate.
Suggested improvements (any subset helps)
Make the
/uploadresponse depend on the body content, not its declared length. Strongest fix: require the handler to return a checksum/digest (e.g. CRC32/FNV/SHA-1) or the last N bytes of the received body. A server that discards the body cannot compute it, so it must actually ingest.validate.shcomputes the same digest client-side and compares.Validate full-body transfer in
validate.sh. Drop|| true, checkcurl's exit code, and assertcurl -w '%{size_upload}'equals the intended body size. An early close → non-zero exit / shortsize_upload→ fail.Parse and penalize load-generator errors. Have
gcannon_parse()extract connection-reset / write-error / non-2xx counters and fail (or zero) the result when they exceed a small threshold. A correct upload server produces ~0; a head-and-close server produces ~1 per request.Rank upload on body throughput, not request rate. Report MB/s of accepted request body (bytes the server actually drained), not 2xx/sec. This naturally rewards efficient ingestion and gives a non-ingesting server a near-zero score. gcannon already tracks bytes sent/accepted.
(Optional) Slow-body probe. Send the body in timed fragments and assert the response arrives only after the last fragment — catches "respond-before-drain" servers that desync real clients even when they do eventually read the body.
Context / fix on the engine side
The companion fix in the
vanillaengine makes both backends drain-then-respond (stream the body off the socket, then answer) so they ingest the body correctly and keep-alive — io_uring upload throughput +58% and peak RSS −41% in the same local test. PR: enghitalo/vanilla#61Happy to send a PR for any of the harness improvements above (1+2 are small and high-value).