feat: add serial factory test runner#69
Conversation
Adds tools/factory_test/run_factory_test.py — a pyserial-based tool that opens a USB serial connection to a freshly flashed board, reads firmware telemetry for a configurable window, and evaluates six pass/fail checks: serial open, self-test PASS, 5+ sample packets, ADC raw in range, no FAULT_EVENT seen, heap_min above floor. Regex patterns match the actual firmware telemetry format from src/telemetry_format.c. Exits 0 on PASS, 1 on FAIL, and writes a JSON result file with per-check status and named failure reason. Includes three sample result JSON artifacts covering pass, ADC out-of-range failure, and heap-below-floor failure cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ddfb42349
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if fault == "NONE": | ||
| result["checks"]["fault_none"] = True | ||
|
|
||
| m = HEALTH_RE.search(line) | ||
| if m: | ||
| heap_min = int(m.group("heap_min")) | ||
| result["min_heap_seen"] = heap_min | ||
| if heap_min >= args.heap_floor: | ||
| result["checks"]["heap_above_floor"] = True | ||
|
|
||
| if FAULT_EVENT_RE.search(line): | ||
| result["checks"]["fault_none"] = False |
There was a problem hiding this comment.
After a FAULT_EVENT is seen, fault_none is set to False, but later sample lines with fault=NONE set it back to True, allowing a board that faulted during the test window to pass if enough clean samples arrive afterward. The protocol says this check should fail when any fault event appears, so this state needs to be sticky once tripped.
Useful? React with 👍 / 👎.
| if args.adc_min <= adc_raw <= args.adc_max: | ||
| result["checks"]["adc_sane"] = True |
There was a problem hiding this comment.
The ADC check currently becomes True after the first in-range sample and never flips back, so a later out-of-range reading can be ignored and still produce PASS. Because this test is intended to validate ADC sanity over the whole capture window, any observed out-of-range adc_raw should keep adc_sane failed.
Useful? React with 👍 / 👎.
| if heap_min >= args.heap_floor: | ||
| result["checks"]["heap_above_floor"] = True |
There was a problem hiding this comment.
heap_above_floor is set to True once a single heap_min reading is above the threshold and is never cleared, so later low-memory readings below --heap-floor can be missed. This can incorrectly PASS boards that dip under the required floor during the run; the check should be based on the minimum heap_min observed across the window.
Useful? React with 👍 / 👎.
1 participant
Summary
tools/factory_test/run_factory_test.py— pyserial-based factory test tool that reads live firmware UART telemetry and evaluates six checks: serial open,event=SELF_TESTresult=PASS, ≥5 sample packets, ADC raw in[--adc-min, --adc-max], noevent=FAULT_EVENT, andheap_min ≥ --heap-floorsrc/telemetry_format.c; exits 0 on PASS / 1 on FAIL with a namedfailure_reasonfield in the JSON outputdocs/factory_test_protocol.mdcovering the check definitions, tool usage, pass/fail JSON schema, and CI compatibility notessample_pass.json,sample_fail_adc.json(ADC out of range),sample_fail_watchdog.json(heap below floor)Test plan
python tools/factory_test/run_factory_test.py --helpexits 0