feat(sim): fall/crash detection benchmark — fixed scenario suite + recall/false-alarm metrics

[nandal]

Background

To answer "is model B actually better than model A?" we need a fixed evaluation suite and honest metrics, not vibes. This becomes the gate every model version — and every future OTA push — must pass. (Design discussion #58; epics #28 / #37.)

The task

• A frozen set of labelled scenarios (synthetic from the generator issue + held-out samples from the dataset-adapter issue).
• Compute and report: recall (don't miss real events), false-alarm rate / precision (the metric that prevents alarm fatigue), per-class confusion, and latency-to-detect.
• A simple report output (CLI / Markdown) so a PR can show before → after.
• Structure it so it can later run in CI as a regression gate.

Acceptance criteria

• Given a model + the suite, prints recall, false-alarm rate, and a per-class breakdown.
• The suite is versioned/frozen — changing it is a deliberate, reviewed act.
• Documented: how to plug in a model and run it.

Guardrails / notes

• This is the structural enforcement of "false alarm > no alarm — but keep false alarms cheap": false-alarm rate is a first-class metric, not an afterthought.
• Report the train/test distribution honestly — a model validated only on synthetic data must say so (promise the mechanism, never the outcome).

Why this is a good first issue

Bounded scope, high leverage — it's the yardstick everything else is measured against. Part of the on-device acute-detector epic.

[wrokin]

🤖 wrokin triage

Create a fixed, versioned benchmark suite for fall/crash detection models, reporting recall, false-alarm rate, per-class metrics, and latency, designed to run as a CI regression gate.

Suggested priority: medium
Labels applied: enhancement, good first issue
Possible duplicate: no

_{Model: deepseek/deepseek-v4-pro · your key, your model (BYOK)}