fuzzamoto-bench: Add benchmarking suite for fuzzamoto-libafl

[str8outtaheap]

Summary

Add a headless benchmarking pipeline for fuzzamoto-libafl campaigns with CLI orchestration, stats collection, and comparison tools.

CLI Commands

• benchmark run --suite <yaml> --output <dir>: Run the fuzzer sequentially for runs iterations of duration seconds each, collecting periodic stats snapshots.
• benchmark compare --baseline <dir> --candidate <dir> [--suite]: Compare two run directories (default) or two suite roots (--suite) using the JSON summaries.

YAML Configuration

duration: 600           # seconds per run
runs: 3                 # number of runs
cores: "0-3"            # core assignment
timeout_ms: 1000        # per-execution timeout
share_dir: /path/to/share
corpus_seed: /path/to/corpus
fuzzer_path: target/release/fuzzamoto-libafl
bench_snapshot_secs: 30 # stats collection interval

Outputs

Per-run

• run_XX/out/bench/bench-cpu_*.csv - raw per-core samples (emitted by the fuzzer)
• run_XX/stats.csv - merged samples (cpu,elapsed_s,execs,execs_per_sec,coverage_pct,corpus_size,crashes)
• run_XX/summary.json - run summary (final_elapsed_s,total_execs,mean_execs_per_sec,max_coverage_pct,final_corpus_size,metadata)
• run_XX/report.md - human-readable run summary

Suite-level

• suite_summary.json - aggregated metrics (runs,coverage_mean,corpus_mean)

Fuzzer Integration

• BenchStatsStage collects periodic samples during fuzzing (enabled with --features bench) and writes bench-cpu_*.csv.

Quick Start

# Build
cargo build --release -p fuzzamoto-cli -p fuzzamoto-libafl --features bench

# Run benchmark suite
./target/release/fuzzamoto-cli benchmark run \
  --suite your_own.yaml \
  --output /tmp/bench-out

# Compare two runs
./target/release/fuzzamoto-cli benchmark compare \
  --baseline /tmp/bench-out/run_00 \
  --candidate /tmp/bench-out/run_01

# Compare two suites
./target/release/fuzzamoto-cli benchmark compare \
  --baseline /tmp/bench-out \
  --candidate /tmp/bench-out2 \
  --suite

[dergoegge]

Can you open a PR with only the fuzzamoto-libafl changes? We can get that merged already and then worry about the cli additions later.

[str8outtaheap]

Can you open a PR with only the fuzzamoto-libafl changes? We can get that merged already and then worry about the cli additions later.

libafl-only changes are now in #104