From 9d3ff33fbeb88be5b98af5ee6582460d7d38bb44 Mon Sep 17 00:00:00 2001 From: anandhu-eng Date: Wed, 9 Sep 2026 20:46:12 +0530 Subject: [PATCH] docs: document output folder structure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add docs/OUTPUT_STRUCTURE.md describing what a run writes to its report directory: where the directory comes from, the layout of a combined perf + accuracy run, the scorer-specific and compliance-audit files that can appear alongside it, and which tools read which artifact. Link it from the README documentation table and from the CLI quick reference, and correct the CLI quick reference note that said `benchmark from-config` has no `--report-dir` override — it does. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 19 +++---- docs/CLI_QUICK_REFERENCE.md | 11 ++-- docs/OUTPUT_STRUCTURE.md | 101 ++++++++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+), 13 deletions(-) create mode 100644 docs/OUTPUT_STRUCTURE.md diff --git a/README.md b/README.md index e980ee486..ad8431dd1 100644 --- a/README.md +++ b/README.md @@ -122,15 +122,16 @@ Run accuracy evaluation with Pass@1 scoring using pre-defined benchmarks: ## Documentation -| Guide | Description | -| -------------------------------------------------------------- | ------------------------------------- | -| [CLI Quick Reference](docs/CLI_QUICK_REFERENCE.md) | Command-line interface guide | -| [CLI Design](docs/CLI_DESIGN.md) | CLI architecture and design decisions | -| [Local Testing](docs/LOCAL_TESTING.md) | Test with the echo server | -| [Client Performance Tuning](docs/CLIENT_PERFORMANCE_TUNING.md) | Endpoint client optimization | -| [Performance Architecture](docs/PERF_ARCHITECTURE.md) | Performance architecture deep dive | -| [Development Guide](docs/DEVELOPMENT.md) | Development setup and workflow | -| [CONTRIBUTING.md](CONTRIBUTING.md) | How to contribute | +| Guide | Description | +| -------------------------------------------------------------- | ----------------------------------------- | +| [CLI Quick Reference](docs/CLI_QUICK_REFERENCE.md) | Command-line interface guide | +| [CLI Design](docs/CLI_DESIGN.md) | CLI architecture and design decisions | +| [Run Output Structure](docs/OUTPUT_STRUCTURE.md) | What a run writes to its report directory | +| [Local Testing](docs/LOCAL_TESTING.md) | Test with the echo server | +| [Client Performance Tuning](docs/CLIENT_PERFORMANCE_TUNING.md) | Endpoint client optimization | +| [Performance Architecture](docs/PERF_ARCHITECTURE.md) | Performance architecture deep dive | +| [Development Guide](docs/DEVELOPMENT.md) | Development setup and workflow | +| [CONTRIBUTING.md](CONTRIBUTING.md) | How to contribute | ## Contributing diff --git a/docs/CLI_QUICK_REFERENCE.md b/docs/CLI_QUICK_REFERENCE.md index 0c511a850..8868a36c3 100644 --- a/docs/CLI_QUICK_REFERENCE.md +++ b/docs/CLI_QUICK_REFERENCE.md @@ -103,10 +103,10 @@ Flag names shown as `--full.dotted.path --alias`. Both forms work. - `--client.max-connections --max-connections` - Max TCP connections (-1=unlimited) - `--endpoint-config.api-key --api-key` - API authentication - `--endpoint-config.api-type --api-type` - API type: openai/sglang (default: openai) -- `--report-dir` - Report output directory - Note: applies to CLI-driven `benchmark offline` / `benchmark online`; `benchmark from-config` - does not expose a CLI override for `report_dir`. Set it in the YAML only if you need to control - the output location; otherwise a default report directory is used. +- `--report-dir` - Report output directory. Available on `benchmark offline` / `benchmark online` + and on `benchmark from-config` (where it overrides the YAML `report_dir`). Unset, the run writes + to a timestamped default directory. See [Run Output Structure](OUTPUT_STRUCTURE.md) for what + lands there. - `--timeout` - Whole-run watchdog (`settings.timeouts.run_timeout_s`; unset = off) - Other timeout flags: `--timeouts.service-ready-timeout-s`, `--timeouts.warmup-drain-timeout-s`, `--timeouts.performance-drain-timeout-s`, `--timeouts.accuracy-drain-timeout-s`, `--timeouts.metrics-drain-timeout-s`, `--timeouts.interrupted-teardown-grace-s` - `--enable-cpu-affinity / --no-cpu-affinity` - NUMA-aware CPU pinning (default: true) @@ -180,6 +180,9 @@ Report directories contain a sanitized `config.yaml`: credentials and other secret values are replaced with ``. Restore those values before reusing that file as benchmark input. +A combined (`both`) run writes performance and accuracy artifacts into one report directory — +see [Run Output Structure](OUTPUT_STRUCTURE.md) for the full layout. + ## Load Patterns **max_throughput** - Offline mode diff --git a/docs/OUTPUT_STRUCTURE.md b/docs/OUTPUT_STRUCTURE.md new file mode 100644 index 000000000..7aee0629b --- /dev/null +++ b/docs/OUTPUT_STRUCTURE.md @@ -0,0 +1,101 @@ +# Run Output Structure + +What a benchmark run writes to its report directory. The focus is the **combined run** +(`--mode both`) — one process that measures performance _and_ scores accuracy into a single +directory. + +For the shape of the perf report itself see [metrics/report_design.md](metrics/report_design.md); +for the compliance-audit subtree see [compliance_audit_plan.md](compliance_audit_plan.md). + +## Where the output goes + +Point a run at a directory of your choosing with the top-level `report_dir` field in the +benchmark config file: + +```yaml +report_dir: results/gptoss_120b_offline +``` + +Every benchmark subcommand also takes `--report-dir` on the command line, including +`benchmark from-config`, where it overrides the config file's value. If neither is set, the run +writes to a timestamped directory under the system temp dir: `$TMPDIR/reports_`. + +| Set via | Result | +| ---------------------------------- | ---------------------------- | +| `report_dir:` in the config file | as given | +| `--report-dir` on the command line | as given; wins over the file | +| neither | timestamped temp directory | + +The directory is resolved once, before any work starts, created during setup, and `config.yaml` is +written into it immediately — so it exists even if the run fails early. Artifacts accumulate there +as the run progresses and are **salvaged on interrupt**: a Ctrl-C'd or timed-out run still leaves +`events.jsonl`, `report.txt`, and a `result_summary.json` marked `state: "interrupted"`, +`complete: false`. + +## Layout of a combined run + +``` +/ +├── config.yaml # resolved config actually used, secrets redacted +├── report.txt # human-readable report (perf + accuracy headline) +├── events.jsonl # per-sample event log, all non-warmup phases +├── sample_idx_map.json # {phase/dataset name: {sample_uuid: sample_index}} +├── performance/ +│ └── result_summary.json # machine-readable perf report (performance phase only) +├── accuracy/ +│ └── accuracy_results.json # per-dataset scores + weighted average +├── metrics/ +│ ├── .ready # aggregator startup marker (touched once, at startup) +│ └── final_snapshot.json # terminal metrics snapshot; the source of the perf report +└── profiling.json # only when settings.profiling.engine is set +``` + +Scorer-specific and audit artifacts add to this tree; see [Scorer-specific +artifacts](#scorer-specific-artifacts) and [Compliance audit subtree](#compliance-audit-subtree). + +## Scorer-specific artifacts + +Some scorers write their own files into `report_dir` in addition to the standard set: + +| Path | Produced by | +| ------------------------------------------------------------ | ---------------------------------------------------- | +| `scores.json` | agentic inline scorer (per-turn / per-domain detail) | +| `per_entry_scores.json` | BFCL v4 multi-turn CLI | +| `swe_bench_results.json`, `swe_bench_runs//` | SWE-bench scorer | +| `vbench_videos/`, `vbench_results/`, `vbench_subprocess.log` | VBench (video-gen) scorer | +| `deepseek_eval/`, `deepseek_eval_subprocess.log` | legacy MLPerf DeepSeek-R1 scorer | + +## Compliance audit subtree + +When the config carries an `audit:` block, the audit runs after the main benchmark and shares the +same `report_dir`, nesting each of its phases as a complete run directory of its own: + +``` +/ +├── … # the main run's artifacts, as above +└── audit/ + ├── audit_.json # e.g. audit_output_caching_test.json + ├── verify_.txt # "Performance check pass: true/false" + ├── reference/ # full run dir for the reference phase + └── output_caching/ # full run dir for the audit phase +``` + +## Optional plots + +Plots are not produced by a run. Generate them afterwards from a finished report directory: + +```bash +python scripts/plot_results.py # writes /plots/*.png +``` + +It reads `accuracy/accuracy_results.json`, `scores.json`, and +`performance/result_summary.json`, and skips any distribution the run did not record. + +## Downstream consumers + +| Tool | Reads | +| ------------------------------------------------ | ---------------------------------------------------------------------------------- | +| `scripts/check_compliance.py` | `config.yaml`, `accuracy/accuracy_results.json` / `scores.json` | +| `scripts/publish_submission.py` | `performance/result_summary.json`, `accuracy/accuracy_results.json`, `config.yaml` | +| `scripts/plot_results.py` | as above | +| `scripts/early_stopping_estimate_from_events.py` | `events.jsonl` |