SelfCorrect-Agent

A reproducible framework for measuring when LLM self-correction fixes wrong answers—and when it breaks correct ones.

SelfCorrect compares four correction strategies on a fixed 100-question subset of GSM8K. It tracks answer transitions—not only final accuracy—so a method gets credit for fixing an error and is penalized when it damages a correct answer.

中文简介：这是一个可复现的大模型自纠错评估项目，重点比较“改对”和“改错”， 而不是只展示最终准确率。项目在深圳大数据项目工作区中完成。

Why this project

“Ask the model to reflect” is not automatically an improvement. A useful self-correction system must answer three questions:

1. Did the final answer become more accurate?
2. How many initially wrong answers were fixed?
3. How many initially correct answers were broken?

This repository makes those transitions visible and reproducible.

Methods

baseline      Question -> Answer
self_refine   Question -> Answer -> Self feedback -> Revision
reflexion     Question -> Answer -> Specific lesson -> Memory -> Retry
critic        Question -> Answer -> Calculator feedback -> Revision

All agents expose the same interface:

answer, trace = agent.solve(question)

Reference answers never enter the agent prompt and are not used for stopping. They are only passed to the grader after solving, preventing oracle leakage.

Results

Method	Rounds	Initial	Final	Fixed	Broken	Net corrections
Baseline	0	98%	98%	0	0	0
Self-refine	1	98%	98%	0	0	0
Self-refine	2	98%	99%	1	0	+1
Self-refine	3	98%	98%	0	0	0
Reflexion	2	96%	96%	2	2	0
Tool critic	1	98%	97%	0	1	-1

The key observation is not that one strategy always wins. Two-round self-refinement produced a small net gain in this run, while reflexion fixed and broke the same number of answers and the tool critic broke one correct answer. Because initial generations can differ across methods, transition metrics should be interpreted within each method.

Quick start

Requirements: Python 3.10+ and a DeepSeek API key.

git clone https://github.com/sclm3010/SelfCorrect-Agent.git
cd SelfCorrect-Agent
python -m pip install -r requirements.txt
Copy-Item .env.example .env

Add your key to .env:

DEEPSEEK_API_KEY=your_api_key

Run a single problem:

python main.py solve "Tom has 5 apples and buys 3 bags with 4 apples each. How many apples?" --method baseline
python main.py solve "the same problem" --method self_refine --rounds 2
python main.py solve "the same problem" --method reflexion --rounds 2
python main.py solve "the same problem" --method critic --rounds 1

Run a small evaluation before launching the full matrix:

python main.py eval --method self_refine --rounds 2 --limit 5
python eval/run_matrix.py
python main.py report

Cached responses make interrupted evaluations resumable.

Metrics

• accuracy: final-round accuracy.
• initial_accuracy: accuracy before correction.
• fixed: initially wrong, finally correct.
• broken: initially correct, finally wrong.
• net_corrections = fixed - broken.
• round_accuracy: accuracy after each revision round.

Repository structure

SelfCorrect-Agent/
  agents/                 # Baseline, self-refine, reflexion, critic
  tools/                  # Calculator and isolated code execution
  eval/                   # Dataset, grader, metrics, reports, results
  docs/                   # Research notes, proposal, experiment plan
  tests/                  # Offline unit tests
  scripts/generate_demo.py
  main.py
  llm_client.py

Tests

Tests do not call a remote API:

python -m unittest discover -s tests -v

The suite covers agent interfaces, transition metrics, number extraction, calculator validation, and isolated code execution.

Reproducibility and safety

• Temperature is fixed at 0; question order and Python seed use 42.
• Cache keys include model, temperature, and complete prompt.
• Remote provider updates can still cause uncached generations to drift.
• tools/code_exec.py uses a temporary directory, subprocess, and timeout, but it is not an operating-system security sandbox. Do not execute untrusted code.
• The calculator checks arithmetic expressions, not semantic understanding.

Research artifacts

• Evaluation report
• Baseline failure review
• Paper notes
• Research proposal
• Experiment plan
• Three-minute demo script

The benchmark subset originates from GSM8K.