feat(benchmarks): add 25-defect dataset, manifest loader, and Pass@k evaluation harness - #5

Merged

Abdurrafay19 merged 18 commits into

masterfrom

bench/defect-evaluation

Sep 11, 2026

Commits on Sep 5, 2026

feat(benchmarks): add manifest schema and first defect category (mutable_default_arguments, 5/25)
Abdurrafay19
committed
0984dd1
test(benchmarks): cover manifest loading against real dataset files
Abdurrafay19
committed
3cb5f99
feat(benchmarks): add manifest file with mutable default arguments defects
Abdurrafay19
committed
29fbac5
feat(benchmarks): expand manifest with additional defect categories and descriptions
Abdurrafay19
committed
ec609b5
feat(benchmarks): complete 25-defect dataset across 5 categories (unhandled_nonetype, off_by_one_slicing, float_precision, resource_leaks)
Abdurrafay19
committed
c298f49

Commits on Sep 10, 2026

chore: add initial configuration for pytest, ruff, and mypy in pyproject.toml
Abdurrafay19
committed
4b23ae9
fix(ci): update Ruff and Mypy commands to include specific benchmark files
Abdurrafay19
committed
63f331c
feat(benchmarks): add comprehensive test cases for various defect categories including mutable defaults, float precision, off-by-one errors, and resource leaks
Abdurrafay19
committed
ff17a67
feat(evaluate): implement defect evaluation framework with grading against oracle tests
Abdurrafay19
committed
d8044d6
feat(manifest): add unique ID validation for defect records in DefectManifest
Abdurrafay19
committed
61bb654
fix(profiler): remove type ignore for pynvml import to enable type checking
Abdurrafay19
committed
97c9e76
fix(manual_profiler_test): ensure main() is called with a newline at the end of the file
Abdurrafay19
committed
09108ab
fix(manual_reflection_test): ensure main() is called with a newline at the end of the file
Abdurrafay19
committed
2fe11c5
fix(manual_smoke_test): ensure main() is called with a newline at the end of the file
Abdurrafay19
committed
69a8e13
test(evaluate): add tests for defect evaluation and grading logic
Abdurrafay19
committed
a9a73b8
test(graph): add tests for build_structured_llm and improve audit prompt coverage
Abdurrafay19
committed
2dd714e
test(manifest): add validation for duplicate defect IDs in DefectManifest
Abdurrafay19
committed
93f569c
test(profiler): mock pynvml.nvmlInit to ensure consistent no-GPU fallback behavior
Abdurrafay19
committed
3baa6e7

Navigation Menu

feat(benchmarks): add 25-defect dataset, manifest loader, and Pass@k evaluation harness - #5

feat(benchmarks): add 25-defect dataset, manifest loader, and Pass@k evaluation harness#5
Abdurrafay19 merged 18 commits into
masterfrom
bench/defect-evaluation

Commits on Sep 5, 2026

feat(benchmarks): add manifest schema and first defect category (mutable_default_arguments, 5/25)

test(benchmarks): cover manifest loading against real dataset files

feat(benchmarks): add manifest file with mutable default arguments defects

feat(benchmarks): expand manifest with additional defect categories and descriptions

feat(benchmarks): complete 25-defect dataset across 5 categories (unhandled_nonetype, off_by_one_slicing, float_precision, resource_leaks)

Commits on Sep 10, 2026

chore: add initial configuration for pytest, ruff, and mypy in pyproject.toml

fix(ci): update Ruff and Mypy commands to include specific benchmark files

feat(benchmarks): add comprehensive test cases for various defect categories including mutable defaults, float precision, off-by-one errors, and resource leaks

feat(evaluate): implement defect evaluation framework with grading against oracle tests

feat(manifest): add unique ID validation for defect records in DefectManifest

fix(profiler): remove type ignore for pynvml import to enable type checking

fix(manual_profiler_test): ensure main() is called with a newline at the end of the file

fix(manual_reflection_test): ensure main() is called with a newline at the end of the file

fix(manual_smoke_test): ensure main() is called with a newline at the end of the file

test(evaluate): add tests for defect evaluation and grading logic

test(graph): add tests for build_structured_llm and improve audit prompt coverage

test(manifest): add validation for duplicate defect IDs in DefectManifest

test(profiler): mock pynvml.nvmlInit to ensure consistent no-GPU fallback behavior