Where: faircode/cli.py's benchmark subcommand - the run_benchmark(...) call has no exception handling, unlike every other failure path in the same block (the top-level import at try/except, the missing-manifests check, and - per a related fix - the write_report matplotlib import).
The gap: verified three distinct ways to crash faircode benchmark with a raw Python traceback instead of a clean CLI error:
- Manifest with a YAML syntax error - unhandled
yaml.parser.ParserError.
- Manifest missing a required key (e.g. no
target section):
$ faircode benchmark bad_manifest/audit.yaml
...
File "faircode/manifest.py", line 141, in from_dict
target = TargetSpec(**data["target"])
KeyError: 'target'
- Manifest pointing at a 0-row dataset - unhandled
ValueError: With n_samples=0, ... from deep inside sklearn's train_test_split, raised via faircode/benchmark.py.
All three propagate uncaught all the way to main().
Why it matters: faircode benchmark already handles the "missing optional dependency" and "no manifests found" failure modes cleanly - a malformed manifest or degenerate dataset are just as easy for a real user to hit (a typo'd YAML key, a dataset that got filtered down to nothing upstream) and get none of that same care.
Suggested fix: wrap the run_benchmark(...) call in a try/except catching yaml.YAMLError, KeyError, and ValueError, printing error: <manifest path>: <message> and returning exit code 2, matching this file's existing style for other anticipated failures.
Where:
faircode/cli.py'sbenchmarksubcommand - therun_benchmark(...)call has no exception handling, unlike every other failure path in the same block (the top-level import at try/except, the missing-manifests check, and - per a related fix - thewrite_reportmatplotlib import).The gap: verified three distinct ways to crash
faircode benchmarkwith a raw Python traceback instead of a clean CLI error:yaml.parser.ParserError.targetsection):ValueError: With n_samples=0, ...from deep inside sklearn'strain_test_split, raised viafaircode/benchmark.py.All three propagate uncaught all the way to
main().Why it matters:
faircode benchmarkalready handles the "missing optional dependency" and "no manifests found" failure modes cleanly - a malformed manifest or degenerate dataset are just as easy for a real user to hit (a typo'd YAML key, a dataset that got filtered down to nothing upstream) and get none of that same care.Suggested fix: wrap the
run_benchmark(...)call in a try/except catchingyaml.YAMLError,KeyError, andValueError, printingerror: <manifest path>: <message>and returning exit code 2, matching this file's existing style for other anticipated failures.