v0.6.3

• macOS version gate: run and models refuse to start on macOS older than 27.0 (or on non-macOS hosts), exiting with code 2 and naming both the detected version and the latest supported macOS (27.0+).
• Doctor diagnostics: fm-bench doctor still runs on unsupported hosts so it can report the macOS mismatch, the latest supported version, and a failed fm probe instead of hard-exiting first.
• Supported platforms doc: new docs/supported-platforms.md describing the fixed macOS requirement and which commands are gated.
• Package metadata: package.json now declares "os": ["darwin"] so npm marks the package as macOS-only.
• Pack integrity: npm run check:pack (wired into prepack) walks runtime imports and fails if any required file is missing from the published tarball.
• Verified on macOS 27.0 build 26A5378n (Apple M5 Pro) with the installed fm CLI: doctor, quick/standard benchmarks, validate/export/history, and CI-style SLO runs all passed.

Full Changelog: v0.6.2...v0.6.3

v0.6.2

• macOS 27 beta 3 compatibility: verified on macOS 27.0 beta 3 build 26A5378j with the installed fm CLI and Xcode 26.6 toolchain.
• Compare accuracy: fm-bench compare now warns when two reports were captured on different macOS build versions, even when both report the same product version such as 27.0.
• Compare metadata: compare headers now show the macOS product version and build, making beta-to-beta Foundation Models benchmark changes easier to audit.

Full Changelog: v0.6.1...v0.6.2

v0.6.1

• Release workflow: GitHub release bodies are generated from CHANGELOG.md (not empty auto-notes). Re-running Release on an existing tag updates release notes.

Full Changelog: v0.6.0...v0.6.1

v0.6.0

• Report schema v1: JSON reports include schemaVersion, reportId, and a suite block (profile, prompt count, environment fingerprint) for shareable, comparable benchmarks.
• Richer environment in every run: hardware model, CPU, memory, thermal/power snapshot, and a short fm --help digest (matches what doctor already probes).
• validate command: verify one or more report JSON files before sharing or CI ingestion.
• export command and --export-html: standalone HTML reports with embedded JSON for humans and automation.
• compare improvements: compatibility warnings for mismatched suites, hardware, or macOS; --strict exits 2 when suites differ; metadata shows tags and hardware.
• history improvements: sort by startedAt, show tag/note/profile column.
• --output-dir: optional tag suffix in filenames; pair JSON + HTML with --export-html.
• --out: .html extension writes a shareable HTML report.
• Documentation: docs/report-format.md, updated README and methodology cross-links.
• Tests for schema, compare compatibility, and HTML export.

Full Changelog: v0.5.3...v0.6.0

v0.5.3

• Hardened Release workflow: skip npm publish when the version is already on the registry (safe to re-run after partial failures), verify package.json version matches the git tag, and skip duplicate GitHub releases.
• CI now runs npm run publish:dry-run, uses concurrency groups to cancel superseded runs, and documents release steps in docs/releasing.md.
• Version workflow pushes explicitly to main with concurrency protection.
• Added Dependabot for GitHub Actions and a CI status badge on the README.
• Normalized package.json bin path for npm publish (npm pkg fix).

Full Changelog: v0.5.2...v0.5.3

v0.5.2

Fixes

• doctor works on macOS 27. pmset -g therm no longer prints CPU_Scheduler_Limit when idle, and pmset -g batt now reports AC power on a separate line. The thermal and battery checks were silently dropped or misclassified. Added dedicated parsers (parseThermalOutput, parseBatteryOutput in src/system.js) that recognize both the legacy wording and the macOS 27 form; the macOS 27 idle thermal state is now reported explicitly as no thermal pressure.

Verified

• macOS 27.0 (Build 26A5368g), Apple M5 Pro, Node v24.16.0
• npm test → 33/33 pass
• npm run lint → clean
• fm-bench doctor, models, legend, and a live benchmark run all work end-to-end on macOS 27

Full Changelog: v0.5.1...v0.5.2

What's Changed

• fix(doctor): support macOS 27 pmset therm/batt output by @devinoldenburg in #1

New Contributors

• @devinoldenburg made their first contribution in #1

Full Changelog: v0.5.1...v0.5.2

v0.5.1

What's New in 0.5.1

Documentation update — no code changes.

• Repositioned README as the GeekBench for Apple Foundation Models
• New title and tagline lead the page
• Quick Start moved before Install so readers see real output immediately
• Commands section replaced with a scannable table
• Benchmark Options replaced with Common Recipes (copy-paste examples grouped by use case) and four focused option tables: Workload, Quality Gates, Output, and Display
• Table Legend + Live Progress + Terminal Colors merged into a single Colors and Legend section
• Requirements and Development sections tightened

v0.5.0

What's New in 0.5.0

New Commands

fm-bench compare <before.json> <after.json>
Diff two saved JSON reports side-by-side. Shows absolute values and percent change (green/yellow/red) for TTFT P50/P95, E2E P50/P95/P99, TPOT, tokens/s, RPS, success rate, and CV. Use --json for structured diff output.

fm-bench history [dir]
Scan a directory for fm-bench JSON reports and print a chronological trend table. Pairs with --output-dir to build a persistent benchmark archive.

New Prompt Profiles

Profile	Prompts	Focus
reasoning	5	Multi-step math, logic sequences, causal chains, Fermi estimation, code debugging
coding	5	Code review, refactoring, algorithms, code explanation, system design
creative	5	Product announcements, error message rewrites, analogies, commit messages, doc writing

New Flags

• --retry <n> — Retry failed fm respond calls up to n times with exponential backoff (500ms → 4s cap). Reduces noise from transient model-busy errors.
• --ci — Disables color and progress, exits with code 1 if any run fails or any SLO budget is violated. Prints fm-bench ci: PASS or fm-bench ci: FAIL — <reasons> to stderr.
• --tag <name> (repeatable) — Label runs for later identification. Tags appear in the JSON payload and table report header.
• --note <text> — Freeform annotation attached to the JSON payload and report header.
• --output-dir <dir> — Auto-save a timestamped JSON report (fm-bench_<timestamp>_<model>.json) to a directory on every run.
• --histogram — Print an ASCII bar chart of E2E latency distribution (up to 20 auto-sized buckets) after the report.

Enhanced doctor Command

Now also reports:

• Hardware model (hw.model) from sysctl
• CPU brand string
• Total memory in GB
• Thermal throttle limit via pmset -g therm (warns when below 100%)
• Battery percentage and charging state via pmset -g batt

Quick Example

# Track performance over time
fm-bench --profile coding --runs 5 --output-dir reports/ --tag baseline

# After a macOS update:
fm-bench --profile coding --runs 5 --output-dir reports/ --tag post-update

# See the full history trend
fm-bench history reports/

# Compare the two specific runs
fm-bench compare reports/fm-bench_..._baseline.json reports/fm-bench_..._post-update.json

# Use in CI
fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5
echo $?  # 0 = pass, 1 = fail

v0.5.0

Full Changelog: v0.4.4...v0.5.0

v0.4.4

What changed

• Fixed benchmark result tables so long NOTE and DESCRIPTION cells wrap instead of ending in ....
• Fixed fm-bench models so model descriptions and quota/unavailable reasons also wrap cleanly.
• Kept numeric benchmark columns compact while shrinking wrapped prose columns to respect --width.

This directly fixes unavailable pcc output like Error: PCC inference is not available in this context. so the full reason is visible in the terminal.

Full Changelog: v0.4.3...v0.4.4