diff --git a/README.md b/README.md index e980ee48..ad8431dd 100644 --- a/README.md +++ b/README.md @@ -122,15 +122,16 @@ Run accuracy evaluation with Pass@1 scoring using pre-defined benchmarks: ## Documentation -| Guide | Description | -| -------------------------------------------------------------- | ------------------------------------- | -| [CLI Quick Reference](docs/CLI_QUICK_REFERENCE.md) | Command-line interface guide | -| [CLI Design](docs/CLI_DESIGN.md) | CLI architecture and design decisions | -| [Local Testing](docs/LOCAL_TESTING.md) | Test with the echo server | -| [Client Performance Tuning](docs/CLIENT_PERFORMANCE_TUNING.md) | Endpoint client optimization | -| [Performance Architecture](docs/PERF_ARCHITECTURE.md) | Performance architecture deep dive | -| [Development Guide](docs/DEVELOPMENT.md) | Development setup and workflow | -| [CONTRIBUTING.md](CONTRIBUTING.md) | How to contribute | +| Guide | Description | +| -------------------------------------------------------------- | ----------------------------------------- | +| [CLI Quick Reference](docs/CLI_QUICK_REFERENCE.md) | Command-line interface guide | +| [CLI Design](docs/CLI_DESIGN.md) | CLI architecture and design decisions | +| [Run Output Structure](docs/OUTPUT_STRUCTURE.md) | What a run writes to its report directory | +| [Local Testing](docs/LOCAL_TESTING.md) | Test with the echo server | +| [Client Performance Tuning](docs/CLIENT_PERFORMANCE_TUNING.md) | Endpoint client optimization | +| [Performance Architecture](docs/PERF_ARCHITECTURE.md) | Performance architecture deep dive | +| [Development Guide](docs/DEVELOPMENT.md) | Development setup and workflow | +| [CONTRIBUTING.md](CONTRIBUTING.md) | How to contribute | ## Contributing diff --git a/docs/CLI_QUICK_REFERENCE.md b/docs/CLI_QUICK_REFERENCE.md index 0c511a85..8868a36c 100644 --- a/docs/CLI_QUICK_REFERENCE.md +++ b/docs/CLI_QUICK_REFERENCE.md @@ -103,10 +103,10 @@ Flag names shown as `--full.dotted.path --alias`. Both forms work. - `--client.max-connections --max-connections` - Max TCP connections (-1=unlimited) - `--endpoint-config.api-key --api-key` - API authentication - `--endpoint-config.api-type --api-type` - API type: openai/sglang (default: openai) -- `--report-dir` - Report output directory - Note: applies to CLI-driven `benchmark offline` / `benchmark online`; `benchmark from-config` - does not expose a CLI override for `report_dir`. Set it in the YAML only if you need to control - the output location; otherwise a default report directory is used. +- `--report-dir` - Report output directory. Available on `benchmark offline` / `benchmark online` + and on `benchmark from-config` (where it overrides the YAML `report_dir`). Unset, the run writes + to a timestamped default directory. See [Run Output Structure](OUTPUT_STRUCTURE.md) for what + lands there. - `--timeout` - Whole-run watchdog (`settings.timeouts.run_timeout_s`; unset = off) - Other timeout flags: `--timeouts.service-ready-timeout-s`, `--timeouts.warmup-drain-timeout-s`, `--timeouts.performance-drain-timeout-s`, `--timeouts.accuracy-drain-timeout-s`, `--timeouts.metrics-drain-timeout-s`, `--timeouts.interrupted-teardown-grace-s` - `--enable-cpu-affinity / --no-cpu-affinity` - NUMA-aware CPU pinning (default: true) @@ -180,6 +180,9 @@ Report directories contain a sanitized `config.yaml`: credentials and other secret values are replaced with ``. Restore those values before reusing that file as benchmark input. +A combined (`both`) run writes performance and accuracy artifacts into one report directory — +see [Run Output Structure](OUTPUT_STRUCTURE.md) for the full layout. + ## Load Patterns **max_throughput** - Offline mode diff --git a/docs/OUTPUT_STRUCTURE.md b/docs/OUTPUT_STRUCTURE.md new file mode 100644 index 00000000..7aee0629 --- /dev/null +++ b/docs/OUTPUT_STRUCTURE.md @@ -0,0 +1,101 @@ +# Run Output Structure + +What a benchmark run writes to its report directory. The focus is the **combined run** +(`--mode both`) — one process that measures performance _and_ scores accuracy into a single +directory. + +For the shape of the perf report itself see [metrics/report_design.md](metrics/report_design.md); +for the compliance-audit subtree see [compliance_audit_plan.md](compliance_audit_plan.md). + +## Where the output goes + +Point a run at a directory of your choosing with the top-level `report_dir` field in the +benchmark config file: + +```yaml +report_dir: results/gptoss_120b_offline +``` + +Every benchmark subcommand also takes `--report-dir` on the command line, including +`benchmark from-config`, where it overrides the config file's value. If neither is set, the run +writes to a timestamped directory under the system temp dir: `$TMPDIR/reports_`. + +| Set via | Result | +| ---------------------------------- | ---------------------------- | +| `report_dir:` in the config file | as given | +| `--report-dir` on the command line | as given; wins over the file | +| neither | timestamped temp directory | + +The directory is resolved once, before any work starts, created during setup, and `config.yaml` is +written into it immediately — so it exists even if the run fails early. Artifacts accumulate there +as the run progresses and are **salvaged on interrupt**: a Ctrl-C'd or timed-out run still leaves +`events.jsonl`, `report.txt`, and a `result_summary.json` marked `state: "interrupted"`, +`complete: false`. + +## Layout of a combined run + +``` +/ +├── config.yaml # resolved config actually used, secrets redacted +├── report.txt # human-readable report (perf + accuracy headline) +├── events.jsonl # per-sample event log, all non-warmup phases +├── sample_idx_map.json # {phase/dataset name: {sample_uuid: sample_index}} +├── performance/ +│ └── result_summary.json # machine-readable perf report (performance phase only) +├── accuracy/ +│ └── accuracy_results.json # per-dataset scores + weighted average +├── metrics/ +│ ├── .ready # aggregator startup marker (touched once, at startup) +│ └── final_snapshot.json # terminal metrics snapshot; the source of the perf report +└── profiling.json # only when settings.profiling.engine is set +``` + +Scorer-specific and audit artifacts add to this tree; see [Scorer-specific +artifacts](#scorer-specific-artifacts) and [Compliance audit subtree](#compliance-audit-subtree). + +## Scorer-specific artifacts + +Some scorers write their own files into `report_dir` in addition to the standard set: + +| Path | Produced by | +| ------------------------------------------------------------ | ---------------------------------------------------- | +| `scores.json` | agentic inline scorer (per-turn / per-domain detail) | +| `per_entry_scores.json` | BFCL v4 multi-turn CLI | +| `swe_bench_results.json`, `swe_bench_runs//` | SWE-bench scorer | +| `vbench_videos/`, `vbench_results/`, `vbench_subprocess.log` | VBench (video-gen) scorer | +| `deepseek_eval/`, `deepseek_eval_subprocess.log` | legacy MLPerf DeepSeek-R1 scorer | + +## Compliance audit subtree + +When the config carries an `audit:` block, the audit runs after the main benchmark and shares the +same `report_dir`, nesting each of its phases as a complete run directory of its own: + +``` +/ +├── … # the main run's artifacts, as above +└── audit/ + ├── audit_.json # e.g. audit_output_caching_test.json + ├── verify_.txt # "Performance check pass: true/false" + ├── reference/ # full run dir for the reference phase + └── output_caching/ # full run dir for the audit phase +``` + +## Optional plots + +Plots are not produced by a run. Generate them afterwards from a finished report directory: + +```bash +python scripts/plot_results.py # writes /plots/*.png +``` + +It reads `accuracy/accuracy_results.json`, `scores.json`, and +`performance/result_summary.json`, and skips any distribution the run did not record. + +## Downstream consumers + +| Tool | Reads | +| ------------------------------------------------ | ---------------------------------------------------------------------------------- | +| `scripts/check_compliance.py` | `config.yaml`, `accuracy/accuracy_results.json` / `scores.json` | +| `scripts/publish_submission.py` | `performance/result_summary.json`, `accuracy/accuracy_results.json`, `config.yaml` | +| `scripts/plot_results.py` | as above | +| `scripts/early_stopping_estimate_from_events.py` | `events.jsonl` |