v0.5.1

What's Changed

• Fix broken code fences in README by @Copilot in #104
• Refactor humaneval_infilling.py to load multiple subsets from the dataset and remove TODO comment by @Ki-Seki in #105
• bump version to 0.5.1 in pyproject.toml and uv.lock by @Ki-Seki in #106

Full Changelog: v0.5.0...v0.5.1

v0.5.0

What's Changed

• Add SCIERC benchmark by @Ki-Seki in #97
• Update to v0.5.0 by @Ki-Seki in #99

Full Changelog: v0.4.0...v0.5.0

v0.4.0

What's Changed

• Update vllm version from 0.14.0 to 0.18.1 by @Ki-Seki in #86
• Add throughput (tokens/s) and TTFT metrics to ppl and match evaluators by @Copilot in #87
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #84
• Add HumanEval Infilling as a code generation + unit-test eval task by @Copilot in #89
• chore: bump version to 0.4.0 in pyproject.toml and uv.lock by @Ki-Seki in #94

Full Changelog: v0.3.0...v0.4.0

v0.3.0

What's Changed

• chore: ignore eval.log.* files by @Duguce in #76
• feat: add upper bound to auto reason_budget by @Duguce in #78
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #77
• Add README documentation by @Copilot in #79
• refactor: remove rich dependency by @Copilot in #81
• Unify error handling across all evaluators by @Copilot in #82
• chore: bump version to 0.3.0 by @Ki-Seki in #83

Full Changelog: v0.2.7...v0.3.0

v0.2.7

What's Changed

• fix (cv): enhance error logging in CVEvaluator and add top_p parameter by @Ki-Seki in #72
• refactor(logging): add logger configuration for gimkit by @Ki-Seki in #73
• chore: bump version to 0.2.7 in pyproject.toml and uv.lock by @Ki-Seki in #74

Full Changelog: v0.2.6...v0.2.7

v0.2.6

What's Changed

• build: update gimkit by @Ki-Seki in #48
• refactor: change counter tokenizer & ppl model by @Ki-Seki in #51
• fix: exclude errors when calculating averages in the match task by @Ki-Seki in #49
• fix: make answer shuffling deterministic by removing python hash by @Duguce in #56
• refactor: replace CTP with windowed normalized perplexity metrics by @Ki-Seki in #52
• refactor: move counter tokenizer argument to the evaluator section by @Ki-Seki in #58
• feat: add golden truth evaluation option and update dataset column selection by @Ki-Seki in #61
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #60
• refactor: ensure results are sorted by filename before writing to CSV by @Ki-Seki in #62
• feat: add cv-parse dataset creation scripts by @Ki-Seki in #59
• feat: add CV evaluation functionality by @Ki-Seki in #63
• feat: gimbench-cv-parse dataset final pipeline by @Ki-Seki in #64
• fix: src/gimbench/cv by @Ki-Seki in #65
• feat(mcqa): improve prompt and eval robustness by @Duguce in #69
• chore: bump version to 0.2.6 by @Ki-Seki in #70

New Contributors

• @Duguce made their first contribution in #56

Full Changelog: v0.2.5...v0.2.6

v0.2.5

What's Changed

• fix: sanitize secret arguments in execution command by @Ki-Seki in #37
• refactor: remove old scripts by @Ki-Seki in #40
• fix: improve error message for context window size in CTP evaluation by @Ki-Seki in #39
• build: update deps by @Ki-Seki in #41
• feat: add max_model_len argument for model context window by @Ki-Seki in #42
• chore: bump to v0.2.5 by @Ki-Seki in #43

Full Changelog: v0.2.4...v0.2.5

v0.2.4

What's Changed

• ci: pre-commit autoupdate by @pre-commit-ci[bot] in #22
• fix: update .gitignore to remove eval results by @Ki-Seki in #24
• refactor: uniformly use SimpleGIM by @Ki-Seki in #26
• feat: add Normalized CTP by @Ki-Seki in #32
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #28
• feat: add auto-budget option by @Ki-Seki in #33
• feat: display progress information in log by @Ki-Seki in #34
• chore: bump version to 0.2.4 by @Ki-Seki in #35

Full Changelog: v0.2.3...v0.2.4

v0.2.3

What's Changed

• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #19
• fix: improve git information retrieval and update eval_env structure by @Ki-Seki in #20
• chore: bump version to 0.2.3 by @Ki-Seki in #21

New Contributors

• @pre-commit-ci[bot] made their first contribution in #19

Full Changelog: v0.2.2...v0.2.3

v0.2.2

What's Changed

• docs: Add MIT License to the project by @Ki-Seki in #16
• fix: update gimkit dependency to version 0.1.0a8 by @Ki-Seki in #17
• chore: bump version to 0.2.2 by @Ki-Seki in #18

Full Changelog: v0.2.1...v0.2.2