+
+ 
InfiCoder-Eval: Systematically Evaluating Question-Answering
- for Code Large Language Models
+ InfiAgent-Eval: Evaluating Agents on Data Analysis Tasks
@@ -109,24 +125,16 @@ InfiCoder-Eval: Systematically Evaluati
-
-
-
-
-
-
- Report (draft version)
-
-
+
-
+
- Inference Repo
+ Project Repo
-
+
@@ -156,9 +164,9 @@ InfiCoder-Eval: Systematically Evaluati
- InfiCoder-Eval is a systematic benchmark and evaluation framework for the free-form question-answering ability of code language models.
+ InfiAgent is a project to build and evalute agents. We start from data analysis and build a benchmark InfiAgent-Eval.
- 
+ 
@@ -172,28 +180,7 @@
Overview
- Large language models for code (code LLMs)
- have made huge progress. Evaluation benchmarks
- for code LLMs, such as HumanEval, DS-1000,
- and MBPP, predominantly focus on code generation. But they are insufficient to evaluate code
- LLMs’ multifaceted ability. To fill this gap, we
- propose InfiCoder-Eval, a large-scale free-form
- question-answering (QA) benchmark for code.
- InfiCoder-Eval comprises 270 carefully picked
- high-quality StackOverflow questions, covering
- 18 programming languages. To tackle the evaluation challenge, InfiCoder-Eval includes an evaluation framework integrating four types of model-free metrics, and domain experts design the concrete criteria for each question. As confirmed
- with human experiments, InfiCoder-Eval evaluation aligns with humans better than model-based evaluation and runs much faster at the
- same time. We conduct a systematic evaluation with InfiCoder-Eval for more than 30 code
- LLMs, leading to several interesting findings. For
- example, though open-source code LLMs show
- competitive performance with proprietary models in code generation (e.g., HumanEval), they
- still have a large gap compared to proprietary
- ones in InfiCoder-Eval and even the best proprietary LLM (GPT4) is still far from perfect (best
- open-source model Deepseek-Coder 33B Instruct
- achieves 50.34% and GPT4 achieves 58.89%).
- Furthermore, our detailed analysis reveals several
- weaknesses of current code LLMs. Benchmark,
- evaluation tools, and detailed results are all publicly available.
+ The advent of Large Language Models (LLMs) has spurred the development of LLM-augmented Autonomous Agents (LAAs). These agents are capable of generating and executing code through ongoing interactions between their core LLM and the code execution environment. In this project, we introduce Infinite Agent (InfiAgent), a LAA focused data analysis and code writing. Our agent is fine-tuned on multiple open-sourced LLMs including Llama2, chatGLM3, and Code Llama. The fine-tuning process employs a unique pipeline for Supervised Fine-Tuning (SFT) data collection, involving the creation and optimization of ReAct conversations using GPT. Furthermore, we have developed a GPT-enabled automatic evaluation benchmark question set (InfiAgent-Eval), which covers various data analysis aspects such as visualization, correlation analysis, and data transformation, providing a comprehensive means for quantitatively assessing LAAs' performance across diverse tasks. Our preliminary results suggest that Infinite Agent could significantly advance the field of autonomous code generation and execution, with potential implications in areas such as software development, data science, and automated problem-solving. This page is dedicated to elucidating the intricacies of the InfiAgent-Eval framework, encompassing aspects such as dataset construction, evaluation metrics, analytical assessment, leaderboard organization, and the procedural nuances of pipeline onboarding.
@@ -208,28 +195,42 @@ Overview
- Statistics and Examples
+ Dataset Construction
-
- InfiCoder-Eval comprises 270 carefully picked high-quality Stack Overflow questions, covering 18 programming languages.
-
- 
-
- We recruited five domain experts to create the benchmark and annotate the correctness evaluation criteria.
- Specifically, the InfiCoder-Eval framework integrates four types of model-free metrics for evaluating the correctness: keywords matching, blank filling, unit testing, and dialogue similarity.
-
+ We build data analysis query and response given existing csv files. Here is the construction pipeline.
+ 
+
+ Our dataset contains 400 questions with 72 csv files. Here're some examples of our generated questions.
+ 
+ We classify csv files used in the evaluation dataset into 9 categories based on their domains:
- 
+
+ - Finance and Economics
+ - Health and Medical
+ - Demographics and Social Science
+ - Marketing and Consumer Behavior
+ - Energy and Environmental Monitoring
+ - Transportation, Logistics, and Tourism
+ - Culture, Entertainment, and Media
+ - Scientific Research and Technology
+ - Other Categories
+
+
+ Here's the pie chart for the file categorization:
+
+ 
+
+ We also check the statistics on the data analysis concepts involved by each question.
+
+ 
+
+ We manually check 50 questions generated by our methods and found:
+
+ 98% questions are meaningful and only in one case, the question become meaningless becase of the wrong regulation expression matching. We also found one case that the question itself is partly unreasonable considering the csv file it based on and in the stage of answer gathering and filtering, we successfully filter it out because ADA didn't return the same answer for 3 times.
-
- Below is the question type, metric type, and length statistics.
-
-
- 
-
@@ -243,10 +244,29 @@ Statistics and Examples
- Comparison
+ Metrics
- Existing benchmarks weigh heavily on code generation, unit-test-based evaluation, and a limited set of programming languages. InfiCoder-Eval processes a much higher diversity to reflect real-world code LLMs’ usage scenarios and is far from saturation.
- 
+
+
+
+ For closed-form questions, we have following metrics:
+
+Proportional Accuracy by Subquestions (PASQ):
+$$
+\text{PSAQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{M_i} \sum_{j=1}^{M_i} I_{ij} \right)
+$$
+Here, $N$ is the total number of questions, $M_i$ is the number of subquestions for the i-th question, and $I_{ij}$ is the indicator function for the j-th subquestion of the i-th question.
+Accuracy by Questions (ABQ):
+$$
+\text{ABQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \prod_{j=1}^{M_i} I_{ij} \right)
+$$
+In this expression, the product
+$\prod_{j=1}^{M_i} I_{ij}$ equals 1 if all subquestions of the \(i\)-th question are answered correctly, and 0 otherwise.
+Uniform Accuracy by Subquestions (UASQ):
+$$
+\text{UASQ} = \frac{1}{\sum_{i=1}^{N} M_i} \sum_{i=1}^{N} \sum_{j=1}^{M_i} I_{ij}
+$$
+Here, the total accuracy is the sum of the values of the indicator function across all subquestions, normalized by the total number of subquestions in the dataset.
@@ -254,49 +274,21 @@ Comparison
-
-
+
- Prompts and Evaluation Protocol
+ Evaluation
- Each question contains a system prompt and content prompt.
- For questions whose responses are mainly in natural language, the system prompt is
-
-
- You are a professional assistant for programmers. By default, questions and answers are in Markdown format. You are chatting with programmers, so please answer as briefly as possible.
-
-
-
- For other questions, the system prompt is
-
-
- You are a professional assistant for programmers. By default, questions and answers are in Markdown format.
-
-
-
- We then format the system prompt and content prompt following each model's default instruction template.
- If no instruction template specified, we use the prompt format
-
- {system prompt}\n{content prompt}
-
-
- We adopt best@10 as the main evaluation metric, where 10 responses are sampled and evaluated for each question and the best score per question is recorded and summed up.
- Throughout the evaluation, we set sampling temperature T to be 0.2 and top p cut-off threshold to be 0.9.
- We leave the exploration of other hyperparameters as the future work.
-
- For score computation, we treat each question equally with one point each.
- Given 270 questions in the benchmark, the full score is 270, and we by default report the percentage score (achieved score divided by the full score which is 270).
- The one point for each question can be further decomposed into a few scoring points within each question.
- For example, a question may contain four keywords with weights 2, 1, 1, and 1 each.
- Then, matching each keyword can contribute to 0.4, 0.2, 0.2, and 0.2 points respectively to the final score.
-
+
+
+ For closed-form questions, we prompt LLM with question description and constraints at first time. Considering that most models hardly follow the format requirements, we add a reformat step after the models respond with gpt-3.5-turbo-16k which formats the responses with the format requirements. Here's a figure illustrating this process:
+ 
-
+
@@ -305,20 +297,26 @@ Prompts and Evaluation Protocol
Leaderboard
+
+
+ In this section, we furnish a comprehensive evaluation of both close-source LLMs such as GPT-4 and GPT-3.5, as well as widely-utilized open-source LLMs. It is observed that the close-source LLMs demonstrate a proficient capacity to adhere to agent directives and produce logically coherent responses.
+ Regrettably, the open-source LLMs enumerated below are currently incapable of adhering to agent directives or delivering substantively meaningful responses. In response to this limitation, we have developed a Supervised Fine-Tuning (SFT) dataset aimed at refining these models. For these open-source LLMs, the table below delineates the enhancements observed post-finetuning. Please anticipate the imminent release of both the SFT dataset and its comprehensive details.
+ 
+
-
-
+
+
@@ -327,18 +325,22 @@ Leaderboard
-
Notice: we set the max tokens to generate=1024 (since GPT4 generates 662 tokens without the constraint, we provide some wiggle room by setting to 1024 tokens)
+
+
+ Notice: We set temperature=0.2, top_p=1.0 and frequency_penalty=0.0 for all the models.
+
+
-
+
@@ -346,13 +348,13 @@ Leaderboard
Rank
Model Name
# Params. (in B)
- Full Set Score
- Full Set Std
- Dev Set Score
- Dev Set Std
+ Proportional Accuracy by Subquestions
+ Accuracy by questions
+ Uniform Accuracy by subquestions
+
@@ -363,756 +365,179 @@ Leaderboard
GPT-4 (0613)
- /
+ /
- 58.89%
-
- 0.74%
-
- 61.28%
+
+ 65.26%
+ 66.05%
+ 59.75%
+
2
- deepseek-coder-33b-instruct
+ GPT-3.5 (turbo-0613)
- 33
-
- 50.34%
+ /
+
+ 55.35%
+ 52.21%
+ 47.25%
+
3
- GPT-3.5-turbo (0613)
-
-
- /
+ CodeLlama-7B
- 46.70%
- 0.70%
+ 7
- 46.16%
+
+ 47.59%
+ 44.67%
+ 39.59%
+
4
- WizardCoder-Python-34B-V1.0
-
+ CodeLlama-7B-python
- 34
- 45.16%
+ 7
+
+ 47.03%
+ 40.78%
+ 40.86%
+
5
- CodeLlama-34B-Instruct
-
+ Llama2-7B
- 34
- 43.71%
+ 7
+
+ 37.53%
+ 34.01%
+ 32.49%
+
6
- deepseek-coder-6.7b-instruct
-
-
- 6.7
-
- 42.97%
-
- 0.22%
-
- 42.28%
-
- 0.24%
-
- 43.66%
-
- 0.26%
-
-
-
-
- 7
-
- WizardCoder-Python-13B-V1.0
-
-
- 13
-
- 41.22%
-
- 0.75%
-
- 41.46%
-
- 1.13%
-
- 40.97%
-
- 0.42%
-
-
-
-
- 8
-
- WizardCoder-Python-7B-V1.0
-
-
- 7
-
- 40.30%
-
- 1.15%
-
- 43.70%
-
- 2.09%
-
- 36.90%
-
- 0.22%
-
-
-
-
- 9
-
- CodeLlama-34B
-
+ WizardCoder-Python-7B-V1.0
- 34
- 39.75%
+ 7
+
+ 33.59%
+ 30.48%
+ 28.75%
+
- 31
-
- phi-1.5
-
-
- 1.5
-
- 16.63%
-
- 0.03%
+ 7
- 14.10%
+ ChatGLM3
- 0.84%
- 19.16%
+ 6
- 0.87%
+
+ 18.10%
+ 15.19%
+ 14.84%
+
@@ -1126,62 +551,64 @@ Leaderboard
-
+
- Try the Benchmark!
+ Pipeline Onboarding
- Step 0: Setup
+ Setup
- - Convert or save your model weights in Hugging Face Transformers format.
- - Clone the two repositories: Inference Repo and Evaluation Repo.
- No requirement on the local directory paths.
- - Set global environment variable:
- export INFERENCE_REPO_PATH=[evaluation repo]/batched_prompts/suite_v2.0.0_dev.csv
+ - We have to initialize the environment using the following code:
+ conda create -n agent python==3.9.12
+pip3 install -r requirements.txt
+ - We also build a Python code sandbox in our pipeline based on docker, you can use the following code to build your docker image:
+ docker build -t myimg .
- Step 1: Generate Response for Your Model
+ Example 1: Demo Usage
- - Set the working directory to Inference Repo.
- The inference repo is forked and slightly modified from bigcode-evaluation-harness framework. We leverage its function for inference.
- - Determine the prompt format to use, which corresponds to task name.
- We support these format for now: code-ffqa-v2 (the default one, system + '\n' + content), code-ffqa-v2-endn (system + '\n' + content + '\n'), code-ffqa-v2-deepseek-chat (deepseek-coder-instruct format), code-ffqa-v2-baichuan2 (baichuan2 models format), code-ffqa-v2-zypher (zypher-7b-beta format), code-ffqa-v2-octo (octopack model format), code-ffqa-v2-wizard (wizard-python model format), code-ffqa-v2-phi (phi-1.5 model format), and code-ffqa-v2-inficoder (our InfiCoder model format).
- Feel free to contribute by adding your model format, which is easy - just slightly modify bigcode_eval/tasks/code_ffqa_v200.py a bit.
- - Run batch inference to generate responses for question prompts.
- accelerate launch [inference repo dir]/main.py --model [your model path / hugging face hub path] --tasks [determined task name above] --batch_size [batch_size] --precision bf16 --n_samples 30 --do_sample True --temperature 0.2 --top_p 0.9 --save_generations --save_references --trust_remote_code --generation_only --max_new_tokens 1024 --save_generations_path [output raw response file path].json --eos='[EOS string]'
- This command will output two files in your working directory: [output raw response file path].json which stores responses and references.json which stores case names as the index.
- - Export responses and case names to evaluation-capatible csv file.
- python3 [inference repo dir]/ffqa_processor.py [output raw response file path].json references.json [response csv file].csv
- This command will join the two output files above into one csv file [response csv file].csv which can be processed by the evaluation framework below.
+ - You can easily use the following command to start a demo using APIs.
+ # initialize poetry
+poetry init
+# Supported LLM: OPEN_AI, AZURE_OPEN_AI
+# api_key is required for API-based models
+bash run_demo.sh --llm AZURE_OPEN_AI --api_key 123
+ - After running the above code, an interactive frontend interface will be displayed.
+ 
+ - You can enter prompts in the dialogue box, and if you need to upload a file, you can select the file for upload in the "browse files" section.
+ - Click the "run code interpreter" button, and the backend will execute our pipeline. The agent will generate code and execute it in the sandbox, and the results will be returned on the interactive page upon completion.
- Step 2: Evaluation (Dev Set)
+ Example 2: Running With Local Models
- - Setup the evaluation framework: Evaluation Repo.
+ - Our local LLM service is developed on vLLM. First, you can start a vLLM model serving by running this command:
+ # Take llama-2-7b as an example
+python3 src/activities/vllm_api_server.py --model "meta-llama/Llama-2-7b-hf" --served_model_name "meta-llama/Llama-2-7b-hf"
At this point, we only support Linux environment.
- Run pip3 install -r requirements.txt, then ./setup.sh (time costly, usually 1-2 hours) which installs necessary compilers and packages for multi-lingual execution environment.
- - Check the evaluation environment.
- Run python3 env_check.py to check and fix the environment incompatibility according to the console output. If the console output is "You're good to go.", then we can proceed.
- - Unpack the csv output.
- Unpack the csv output file from the previous inference step into a directory where each response is stored in a separate txt.
- python3 adaptors/csv_response_unpacker.py [response csv file].csv [response save dir]
- We recommend to save the responses in a directory in responses/, i.e., let [response save dir]=responses/.... The above script will create the [response save dir] directory if it does not exist.
- - Run main evaluation.
- python3 grader_main.py suite_v2.0.0_dev.yaml [response save dir]
- The evaluation takes around 15 min - 45 min.
- When it finishes, there are two output files: results/suite_v2.0.0_dev_[response save dir base name].txt (short summary) and results/suite_v2.0.0_dev_[response save dir base name].yaml (all details).
- You can also customized the output paths by --result_summary_path and --result_detail_path arguments respectively.
- - Get statistics and print the results.
- python3 print_result_stat.py [result detail path] [summary txt path]
- In console output and [summary txt path], a nice table will be printed, including the overall score and percentage and the sub-scores for each question type, metric type, and programming language.
+ - You can try this command if the serving is successfully starting:
+ curl http://localhost:8000/v1/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "meta-llama/Llama-2-7b-hf",
+ "prompt": "San Francisco is a",
+ "max_tokens": 7,
+ "temperature": 0
+ }'
+ - Then you can run the demo following this command:
+ bash run_demo.sh --llm "meta-llama/Llama-2-7b-hf"
- Step 3: Evaluate (Test Set)
-
- Available upon request (email us).
-
+
+ Example 3: Running Without Front-end
+
+ Our demo is designed to default to the use of the front-end. If you prefer not to use the front-end and instead perform command-line operations or process large amounts of data, you can refer to the following commands:
+ # Run with API.
+python3 ./src/activities/eval.py --llm AZURE_OPEN_AI --api_key 123
+# Run with local model.Take llama-2-7b as an example.
+python3 ./src/activities/eval.py --llm "meta-llama/Llama-2-7b-hf"
+
@@ -1203,11 +630,11 @@ Acknowledgement
BibTeX
@misc{li2023inficodereval,
- author = {InfiCoderTeam},
- title = {InfiCoder-Eval: Systematically Evaluating Question-Answering for Code Large Language Models},
+ author = {InfiAgent Team},
+ title = {InfiAgent: Building and Evaluating Agents on Data Analysis},
year = {2023},
publisher = {Github Pages},
- howpublished = "\url{https://github.com/infi-coder/inficoder-eval}"
+ howpublished = "\url{https://infiagent.github.io/}"
}
@@ -1217,12 +644,12 @@ BibTeX
InfiCoder-Eval: Systematically Evaluati
-
-
-
-
-
-
- Report (draft version)
-
-
+
-
+
- Inference Repo
+ Project Repo
-
+
@@ -156,9 +164,9 @@ InfiCoder-Eval: Systematically Evaluati
- InfiCoder-Eval is a systematic benchmark and evaluation framework for the free-form question-answering ability of code language models.
+ InfiAgent is a project to build and evalute agents. We start from data analysis and build a benchmark InfiAgent-Eval.
-
+
@@ -172,28 +180,7 @@
Overview
- Large language models for code (code LLMs)
- have made huge progress. Evaluation benchmarks
- for code LLMs, such as HumanEval, DS-1000,
- and MBPP, predominantly focus on code generation. But they are insufficient to evaluate code
- LLMs’ multifaceted ability. To fill this gap, we
- propose InfiCoder-Eval, a large-scale free-form
- question-answering (QA) benchmark for code.
- InfiCoder-Eval comprises 270 carefully picked
- high-quality StackOverflow questions, covering
- 18 programming languages. To tackle the evaluation challenge, InfiCoder-Eval includes an evaluation framework integrating four types of model-free metrics, and domain experts design the concrete criteria for each question. As confirmed
- with human experiments, InfiCoder-Eval evaluation aligns with humans better than model-based evaluation and runs much faster at the
- same time. We conduct a systematic evaluation with InfiCoder-Eval for more than 30 code
- LLMs, leading to several interesting findings. For
- example, though open-source code LLMs show
- competitive performance with proprietary models in code generation (e.g., HumanEval), they
- still have a large gap compared to proprietary
- ones in InfiCoder-Eval and even the best proprietary LLM (GPT4) is still far from perfect (best
- open-source model Deepseek-Coder 33B Instruct
- achieves 50.34% and GPT4 achieves 58.89%).
- Furthermore, our detailed analysis reveals several
- weaknesses of current code LLMs. Benchmark,
- evaluation tools, and detailed results are all publicly available.
+ The advent of Large Language Models (LLMs) has spurred the development of LLM-augmented Autonomous Agents (LAAs). These agents are capable of generating and executing code through ongoing interactions between their core LLM and the code execution environment. In this project, we introduce Infinite Agent (InfiAgent), a LAA focused data analysis and code writing. Our agent is fine-tuned on multiple open-sourced LLMs including Llama2, chatGLM3, and Code Llama. The fine-tuning process employs a unique pipeline for Supervised Fine-Tuning (SFT) data collection, involving the creation and optimization of ReAct conversations using GPT. Furthermore, we have developed a GPT-enabled automatic evaluation benchmark question set (InfiAgent-Eval), which covers various data analysis aspects such as visualization, correlation analysis, and data transformation, providing a comprehensive means for quantitatively assessing LAAs' performance across diverse tasks. Our preliminary results suggest that Infinite Agent could significantly advance the field of autonomous code generation and execution, with potential implications in areas such as software development, data science, and automated problem-solving. This page is dedicated to elucidating the intricacies of the InfiAgent-Eval framework, encompassing aspects such as dataset construction, evaluation metrics, analytical assessment, leaderboard organization, and the procedural nuances of pipeline onboarding.
@@ -208,28 +195,42 @@ Overview
- Statistics and Examples
+ Dataset Construction
-
- InfiCoder-Eval comprises 270 carefully picked high-quality Stack Overflow questions, covering 18 programming languages.
-
-
-
- We recruited five domain experts to create the benchmark and annotate the correctness evaluation criteria.
- Specifically, the InfiCoder-Eval framework integrates four types of model-free metrics for evaluating the correctness: keywords matching, blank filling, unit testing, and dialogue similarity.
-
+ We build data analysis query and response given existing csv files. Here is the construction pipeline.
+
+
+ Our dataset contains 400 questions with 72 csv files. Here're some examples of our generated questions.
+
+ We classify csv files used in the evaluation dataset into 9 categories based on their domains:
-
+
+ - Finance and Economics
+ - Health and Medical
+ - Demographics and Social Science
+ - Marketing and Consumer Behavior
+ - Energy and Environmental Monitoring
+ - Transportation, Logistics, and Tourism
+ - Culture, Entertainment, and Media
+ - Scientific Research and Technology
+ - Other Categories
+
+
+ Here's the pie chart for the file categorization:
+
+
+
+ We also check the statistics on the data analysis concepts involved by each question.
+
+
+
+ We manually check 50 questions generated by our methods and found:
+
+ 98% questions are meaningful and only in one case, the question become meaningless becase of the wrong regulation expression matching. We also found one case that the question itself is partly unreasonable considering the csv file it based on and in the stage of answer gathering and filtering, we successfully filter it out because ADA didn't return the same answer for 3 times.
-
- Below is the question type, metric type, and length statistics.
-
-
-
-
@@ -243,10 +244,29 @@ Statistics and Examples
- Comparison
+ Metrics
- Existing benchmarks weigh heavily on code generation, unit-test-based evaluation, and a limited set of programming languages. InfiCoder-Eval processes a much higher diversity to reflect real-world code LLMs’ usage scenarios and is far from saturation.
-
+
+
+
+ For closed-form questions, we have following metrics:
+
+Proportional Accuracy by Subquestions (PASQ):
+$$
+\text{PSAQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{M_i} \sum_{j=1}^{M_i} I_{ij} \right)
+$$
+Here, $N$ is the total number of questions, $M_i$ is the number of subquestions for the i-th question, and $I_{ij}$ is the indicator function for the j-th subquestion of the i-th question.
+Accuracy by Questions (ABQ):
+$$
+\text{ABQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \prod_{j=1}^{M_i} I_{ij} \right)
+$$
+In this expression, the product
+$\prod_{j=1}^{M_i} I_{ij}$ equals 1 if all subquestions of the \(i\)-th question are answered correctly, and 0 otherwise.
+Uniform Accuracy by Subquestions (UASQ):
+$$
+\text{UASQ} = \frac{1}{\sum_{i=1}^{N} M_i} \sum_{i=1}^{N} \sum_{j=1}^{M_i} I_{ij}
+$$
+Here, the total accuracy is the sum of the values of the indicator function across all subquestions, normalized by the total number of subquestions in the dataset.
@@ -254,49 +274,21 @@ Comparison
-
-
+
- Prompts and Evaluation Protocol
+ Evaluation
- Each question contains a system prompt and content prompt.
- For questions whose responses are mainly in natural language, the system prompt is
-
-
- You are a professional assistant for programmers. By default, questions and answers are in Markdown format. You are chatting with programmers, so please answer as briefly as possible.
-
-
-
- For other questions, the system prompt is
-
-
- You are a professional assistant for programmers. By default, questions and answers are in Markdown format.
-
-
-
- We then format the system prompt and content prompt following each model's default instruction template.
- If no instruction template specified, we use the prompt format
-
- {system prompt}\n{content prompt}
-
-
- We adopt best@10 as the main evaluation metric, where 10 responses are sampled and evaluated for each question and the best score per question is recorded and summed up.
- Throughout the evaluation, we set sampling temperature T to be 0.2 and top p cut-off threshold to be 0.9.
- We leave the exploration of other hyperparameters as the future work.
-
- For score computation, we treat each question equally with one point each.
- Given 270 questions in the benchmark, the full score is 270, and we by default report the percentage score (achieved score divided by the full score which is 270).
- The one point for each question can be further decomposed into a few scoring points within each question.
- For example, a question may contain four keywords with weights 2, 1, 1, and 1 each.
- Then, matching each keyword can contribute to 0.4, 0.2, 0.2, and 0.2 points respectively to the final score.
-
+
+
+ For closed-form questions, we prompt LLM with question description and constraints at first time. Considering that most models hardly follow the format requirements, we add a reformat step after the models respond with gpt-3.5-turbo-16k which formats the responses with the format requirements. Here's a figure illustrating this process:
+
-
+
@@ -305,20 +297,26 @@ Prompts and Evaluation Protocol
Leaderboard
+
+
+ In this section, we furnish a comprehensive evaluation of both close-source LLMs such as GPT-4 and GPT-3.5, as well as widely-utilized open-source LLMs. It is observed that the close-source LLMs demonstrate a proficient capacity to adhere to agent directives and produce logically coherent responses.
+ Regrettably, the open-source LLMs enumerated below are currently incapable of adhering to agent directives or delivering substantively meaningful responses. In response to this limitation, we have developed a Supervised Fine-Tuning (SFT) dataset aimed at refining these models. For these open-source LLMs, the table below delineates the enhancements observed post-finetuning. Please anticipate the imminent release of both the SFT dataset and its comprehensive details.
+
+
-
-
+
+
@@ -327,18 +325,22 @@ Leaderboard
-
Notice: we set the max tokens to generate=1024 (since GPT4 generates 662 tokens without the constraint, we provide some wiggle room by setting to 1024 tokens)
+
+
+ Notice: We set temperature=0.2, top_p=1.0 and frequency_penalty=0.0 for all the models.
+
+
-
+
@@ -346,13 +348,13 @@ Leaderboard
Rank
Model Name
# Params. (in B)
- Full Set Score
- Full Set Std
- Dev Set Score
- Dev Set Std
+ Proportional Accuracy by Subquestions
+ Accuracy by questions
+ Uniform Accuracy by subquestions
+
@@ -363,756 +365,179 @@ Leaderboard
GPT-4 (0613)
- /
+ /
- 58.89%
-
- 0.74%
-
- 61.28%
+
+ 65.26%
+ 66.05%
+ 59.75%
+
2
- deepseek-coder-33b-instruct
+ GPT-3.5 (turbo-0613)
- 33
-
- 50.34%
+ /
+
+ 55.35%
+ 52.21%
+ 47.25%
+
3
- GPT-3.5-turbo (0613)
-
-
- /
+ CodeLlama-7B
- 46.70%
- 0.70%
+ 7
- 46.16%
+
+ 47.59%
+ 44.67%
+ 39.59%
+
4
- WizardCoder-Python-34B-V1.0
-
+ CodeLlama-7B-python
- 34
- 45.16%
+ 7
+
+ 47.03%
+ 40.78%
+ 40.86%
+
5
- CodeLlama-34B-Instruct
-
+ Llama2-7B
- 34
- 43.71%
+ 7
+
+ 37.53%
+ 34.01%
+ 32.49%
+
6
- deepseek-coder-6.7b-instruct
-
-
- 6.7
-
- 42.97%
-
- 0.22%
-
- 42.28%
-
- 0.24%
-
- 43.66%
-
- 0.26%
-
-
-
-
- 7
-
- WizardCoder-Python-13B-V1.0
-
-
- 13
-
- 41.22%
-
- 0.75%
-
- 41.46%
-
- 1.13%
-
- 40.97%
-
- 0.42%
-
-
-
-
- 8
-
- WizardCoder-Python-7B-V1.0
-
-
- 7
-
- 40.30%
-
- 1.15%
-
- 43.70%
-
- 2.09%
-
- 36.90%
-
- 0.22%
-
-
-
-
- 9
-
- CodeLlama-34B
-
+ WizardCoder-Python-7B-V1.0
- 34
- 39.75%
+ 7
+
+ 33.59%
+ 30.48%
+ 28.75%
+
- 31
-
- phi-1.5
-
-
- 1.5
-
- 16.63%
-
- 0.03%
+ 7
- 14.10%
+ ChatGLM3
- 0.84%
- 19.16%
+ 6
- 0.87%
+
+ 18.10%
+ 15.19%
+ 14.84%
+
@@ -1126,62 +551,64 @@ Leaderboard
-
+
- Try the Benchmark!
+ Pipeline Onboarding
- Step 0: Setup
+ Setup
- - Convert or save your model weights in Hugging Face Transformers format.
- - Clone the two repositories: Inference Repo and Evaluation Repo.
- No requirement on the local directory paths.
- - Set global environment variable:
- export INFERENCE_REPO_PATH=[evaluation repo]/batched_prompts/suite_v2.0.0_dev.csv
+ - We have to initialize the environment using the following code:
+ conda create -n agent python==3.9.12
+pip3 install -r requirements.txt
+ - We also build a Python code sandbox in our pipeline based on docker, you can use the following code to build your docker image:
+ docker build -t myimg .
- Step 1: Generate Response for Your Model
+ Example 1: Demo Usage
- - Set the working directory to Inference Repo.
- The inference repo is forked and slightly modified from bigcode-evaluation-harness framework. We leverage its function for inference.
- - Determine the prompt format to use, which corresponds to task name.
- We support these format for now: code-ffqa-v2 (the default one, system + '\n' + content), code-ffqa-v2-endn (system + '\n' + content + '\n'), code-ffqa-v2-deepseek-chat (deepseek-coder-instruct format), code-ffqa-v2-baichuan2 (baichuan2 models format), code-ffqa-v2-zypher (zypher-7b-beta format), code-ffqa-v2-octo (octopack model format), code-ffqa-v2-wizard (wizard-python model format), code-ffqa-v2-phi (phi-1.5 model format), and code-ffqa-v2-inficoder (our InfiCoder model format).
- Feel free to contribute by adding your model format, which is easy - just slightly modify bigcode_eval/tasks/code_ffqa_v200.py a bit.
- - Run batch inference to generate responses for question prompts.
- accelerate launch [inference repo dir]/main.py --model [your model path / hugging face hub path] --tasks [determined task name above] --batch_size [batch_size] --precision bf16 --n_samples 30 --do_sample True --temperature 0.2 --top_p 0.9 --save_generations --save_references --trust_remote_code --generation_only --max_new_tokens 1024 --save_generations_path [output raw response file path].json --eos='[EOS string]'
- This command will output two files in your working directory: [output raw response file path].json which stores responses and references.json which stores case names as the index.
- - Export responses and case names to evaluation-capatible csv file.
- python3 [inference repo dir]/ffqa_processor.py [output raw response file path].json references.json [response csv file].csv
- This command will join the two output files above into one csv file [response csv file].csv which can be processed by the evaluation framework below.
+ - You can easily use the following command to start a demo using APIs.
+ # initialize poetry
+poetry init
+# Supported LLM: OPEN_AI, AZURE_OPEN_AI
+# api_key is required for API-based models
+bash run_demo.sh --llm AZURE_OPEN_AI --api_key 123
+ - After running the above code, an interactive frontend interface will be displayed.
+
+ - You can enter prompts in the dialogue box, and if you need to upload a file, you can select the file for upload in the "browse files" section.
+ - Click the "run code interpreter" button, and the backend will execute our pipeline. The agent will generate code and execute it in the sandbox, and the results will be returned on the interactive page upon completion.
- Step 2: Evaluation (Dev Set)
+ Example 2: Running With Local Models
- - Setup the evaluation framework: Evaluation Repo.
+ - Our local LLM service is developed on vLLM. First, you can start a vLLM model serving by running this command:
+ # Take llama-2-7b as an example
+python3 src/activities/vllm_api_server.py --model "meta-llama/Llama-2-7b-hf" --served_model_name "meta-llama/Llama-2-7b-hf"
At this point, we only support Linux environment.
- Run pip3 install -r requirements.txt, then ./setup.sh (time costly, usually 1-2 hours) which installs necessary compilers and packages for multi-lingual execution environment.
- - Check the evaluation environment.
- Run python3 env_check.py to check and fix the environment incompatibility according to the console output. If the console output is "You're good to go.", then we can proceed.
- - Unpack the csv output.
- Unpack the csv output file from the previous inference step into a directory where each response is stored in a separate txt.
- python3 adaptors/csv_response_unpacker.py [response csv file].csv [response save dir]
- We recommend to save the responses in a directory in responses/, i.e., let [response save dir]=responses/.... The above script will create the [response save dir] directory if it does not exist.
- - Run main evaluation.
- python3 grader_main.py suite_v2.0.0_dev.yaml [response save dir]
- The evaluation takes around 15 min - 45 min.
- When it finishes, there are two output files: results/suite_v2.0.0_dev_[response save dir base name].txt (short summary) and results/suite_v2.0.0_dev_[response save dir base name].yaml (all details).
- You can also customized the output paths by --result_summary_path and --result_detail_path arguments respectively.
- - Get statistics and print the results.
- python3 print_result_stat.py [result detail path] [summary txt path]
- In console output and [summary txt path], a nice table will be printed, including the overall score and percentage and the sub-scores for each question type, metric type, and programming language.
+ - You can try this command if the serving is successfully starting:
+ curl http://localhost:8000/v1/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "meta-llama/Llama-2-7b-hf",
+ "prompt": "San Francisco is a",
+ "max_tokens": 7,
+ "temperature": 0
+ }'
+ - Then you can run the demo following this command:
+ bash run_demo.sh --llm "meta-llama/Llama-2-7b-hf"
- Step 3: Evaluate (Test Set)
-
- Available upon request (email us).
-
+
+ Example 3: Running Without Front-end
+
+ Our demo is designed to default to the use of the front-end. If you prefer not to use the front-end and instead perform command-line operations or process large amounts of data, you can refer to the following commands:
+ # Run with API.
+python3 ./src/activities/eval.py --llm AZURE_OPEN_AI --api_key 123
+# Run with local model.Take llama-2-7b as an example.
+python3 ./src/activities/eval.py --llm "meta-llama/Llama-2-7b-hf"
+
@@ -1203,11 +630,11 @@ Acknowledgement
BibTeX
@misc{li2023inficodereval,
- author = {InfiCoderTeam},
- title = {InfiCoder-Eval: Systematically Evaluating Question-Answering for Code Large Language Models},
+ author = {InfiAgent Team},
+ title = {InfiAgent: Building and Evaluating Agents on Data Analysis},
year = {2023},
publisher = {Github Pages},
- howpublished = "\url{https://github.com/infi-coder/inficoder-eval}"
+ howpublished = "\url{https://infiagent.github.io/}"
}
@@ -1217,12 +644,12 @@ BibTeX
InfiCoder-Eval: Systematically Evaluati
- InfiCoder-Eval is a systematic benchmark and evaluation framework for the free-form question-answering ability of code language models.
+ InfiAgent is a project to build and evalute agents. We start from data analysis and build a benchmark InfiAgent-Eval.
-
+
@@ -172,28 +180,7 @@
Overview
- Large language models for code (code LLMs)
- have made huge progress. Evaluation benchmarks
- for code LLMs, such as HumanEval, DS-1000,
- and MBPP, predominantly focus on code generation. But they are insufficient to evaluate code
- LLMs’ multifaceted ability. To fill this gap, we
- propose InfiCoder-Eval, a large-scale free-form
- question-answering (QA) benchmark for code.
- InfiCoder-Eval comprises 270 carefully picked
- high-quality StackOverflow questions, covering
- 18 programming languages. To tackle the evaluation challenge, InfiCoder-Eval includes an evaluation framework integrating four types of model-free metrics, and domain experts design the concrete criteria for each question. As confirmed
- with human experiments, InfiCoder-Eval evaluation aligns with humans better than model-based evaluation and runs much faster at the
- same time. We conduct a systematic evaluation with InfiCoder-Eval for more than 30 code
- LLMs, leading to several interesting findings. For
- example, though open-source code LLMs show
- competitive performance with proprietary models in code generation (e.g., HumanEval), they
- still have a large gap compared to proprietary
- ones in InfiCoder-Eval and even the best proprietary LLM (GPT4) is still far from perfect (best
- open-source model Deepseek-Coder 33B Instruct
- achieves 50.34% and GPT4 achieves 58.89%).
- Furthermore, our detailed analysis reveals several
- weaknesses of current code LLMs. Benchmark,
- evaluation tools, and detailed results are all publicly available.
+ The advent of Large Language Models (LLMs) has spurred the development of LLM-augmented Autonomous Agents (LAAs). These agents are capable of generating and executing code through ongoing interactions between their core LLM and the code execution environment. In this project, we introduce Infinite Agent (InfiAgent), a LAA focused data analysis and code writing. Our agent is fine-tuned on multiple open-sourced LLMs including Llama2, chatGLM3, and Code Llama. The fine-tuning process employs a unique pipeline for Supervised Fine-Tuning (SFT) data collection, involving the creation and optimization of ReAct conversations using GPT. Furthermore, we have developed a GPT-enabled automatic evaluation benchmark question set (InfiAgent-Eval), which covers various data analysis aspects such as visualization, correlation analysis, and data transformation, providing a comprehensive means for quantitatively assessing LAAs' performance across diverse tasks. Our preliminary results suggest that Infinite Agent could significantly advance the field of autonomous code generation and execution, with potential implications in areas such as software development, data science, and automated problem-solving. This page is dedicated to elucidating the intricacies of the InfiAgent-Eval framework, encompassing aspects such as dataset construction, evaluation metrics, analytical assessment, leaderboard organization, and the procedural nuances of pipeline onboarding.
- InfiCoder-Eval is a systematic benchmark and evaluation framework for the free-form question-answering ability of code language models. + InfiAgent is a project to build and evalute agents. We start from data analysis and build a benchmark InfiAgent-Eval.
-
+
Overview
- Large language models for code (code LLMs) - have made huge progress. Evaluation benchmarks - for code LLMs, such as HumanEval, DS-1000, - and MBPP, predominantly focus on code generation. But they are insufficient to evaluate code - LLMs’ multifaceted ability. To fill this gap, we - propose InfiCoder-Eval, a large-scale free-form - question-answering (QA) benchmark for code. - InfiCoder-Eval comprises 270 carefully picked - high-quality StackOverflow questions, covering - 18 programming languages. To tackle the evaluation challenge, InfiCoder-Eval includes an evaluation framework integrating four types of model-free metrics, and domain experts design the concrete criteria for each question. As confirmed - with human experiments, InfiCoder-Eval evaluation aligns with humans better than model-based evaluation and runs much faster at the - same time. We conduct a systematic evaluation with InfiCoder-Eval for more than 30 code - LLMs, leading to several interesting findings. For - example, though open-source code LLMs show - competitive performance with proprietary models in code generation (e.g., HumanEval), they - still have a large gap compared to proprietary - ones in InfiCoder-Eval and even the best proprietary LLM (GPT4) is still far from perfect (best - open-source model Deepseek-Coder 33B Instruct - achieves 50.34% and GPT4 achieves 58.89%). - Furthermore, our detailed analysis reveals several - weaknesses of current code LLMs. Benchmark, - evaluation tools, and detailed results are all publicly available. + The advent of Large Language Models (LLMs) has spurred the development of LLM-augmented Autonomous Agents (LAAs). These agents are capable of generating and executing code through ongoing interactions between their core LLM and the code execution environment. In this project, we introduce Infinite Agent (InfiAgent), a LAA focused data analysis and code writing. Our agent is fine-tuned on multiple open-sourced LLMs including Llama2, chatGLM3, and Code Llama. The fine-tuning process employs a unique pipeline for Supervised Fine-Tuning (SFT) data collection, involving the creation and optimization of ReAct conversations using GPT. Furthermore, we have developed a GPT-enabled automatic evaluation benchmark question set (InfiAgent-Eval), which covers various data analysis aspects such as visualization, correlation analysis, and data transformation, providing a comprehensive means for quantitatively assessing LAAs' performance across diverse tasks. Our preliminary results suggest that Infinite Agent could significantly advance the field of autonomous code generation and execution, with potential implications in areas such as software development, data science, and automated problem-solving. This page is dedicated to elucidating the intricacies of the InfiAgent-Eval framework, encompassing aspects such as dataset construction, evaluation metrics, analytical assessment, leaderboard organization, and the procedural nuances of pipeline onboarding.
Overview
Statistics and Examples
+Dataset Construction
- InfiCoder-Eval comprises 270 carefully picked high-quality Stack Overflow questions, covering 18 programming languages. -
-
- - We recruited five domain experts to create the benchmark and annotate the correctness evaluation criteria. - Specifically, the InfiCoder-Eval framework integrates four types of model-free metrics for evaluating the correctness: keywords matching, blank filling, unit testing, and dialogue similarity. -
+ We build data analysis query and response given existing csv files. Here is the construction pipeline. +
+
+ Our dataset contains 400 questions with 72 csv files. Here're some examples of our generated questions.
+
+ We classify csv files used in the evaluation dataset into 9 categories based on their domains:
-
+ -
+
- Finance and Economics +
- Health and Medical +
- Demographics and Social Science +
- Marketing and Consumer Behavior +
- Energy and Environmental Monitoring +
- Transportation, Logistics, and Tourism +
- Culture, Entertainment, and Media +
- Scientific Research and Technology +
- Other Categories +
Here's the pie chart for the file categorization:
+ +
+
+ We also check the statistics on the data analysis concepts involved by each question.
+ +
+
+ We manually check 50 questions generated by our methods and found:
+
+ 98% questions are meaningful and only in one case, the question become meaningless becase of the wrong regulation expression matching. We also found one case that the question itself is partly unreasonable considering the csv file it based on and in the stage of answer gathering and filtering, we successfully filter it out because ADA didn't return the same answer for 3 times.
- - Below is the question type, metric type, and length statistics. -
-
- Statistics and Examples
Comparison
+Metrics
Existing benchmarks weigh heavily on code generation, unit-test-based evaluation, and a limited set of programming languages. InfiCoder-Eval processes a much higher diversity to reflect real-world code LLMs’ usage scenarios and is far from saturation.
-
+
+
+
+ For closed-form questions, we have following metrics:
+
+Proportional Accuracy by Subquestions (PASQ):
+$$
+\text{PSAQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{M_i} \sum_{j=1}^{M_i} I_{ij} \right)
+$$
+Here, $N$ is the total number of questions, $M_i$ is the number of subquestions for the i-th question, and $I_{ij}$ is the indicator function for the j-th subquestion of the i-th question.
+Accuracy by Questions (ABQ):
+$$
+\text{ABQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \prod_{j=1}^{M_i} I_{ij} \right)
+$$
+In this expression, the product
+$\prod_{j=1}^{M_i} I_{ij}$ equals 1 if all subquestions of the \(i\)-th question are answered correctly, and 0 otherwise.
+Uniform Accuracy by Subquestions (UASQ):
+$$
+\text{UASQ} = \frac{1}{\sum_{i=1}^{N} M_i} \sum_{i=1}^{N} \sum_{j=1}^{M_i} I_{ij}
+$$
+Here, the total accuracy is the sum of the values of the indicator function across all subquestions, normalized by the total number of subquestions in the dataset.
Comparison
Prompts and Evaluation Protocol
+Evaluation
You are a professional assistant for programmers. By default, questions and answers are in Markdown format. You are chatting with programmers, so please answer as briefly as possible.
-
- You are a professional assistant for programmers. By default, questions and answers are in Markdown format.
-
- {system prompt}\n{content prompt}
-
- We adopt best@10 as the main evaluation metric, where 10 responses are sampled and evaluated for each question and the best score per question is recorded and summed up. - Throughout the evaluation, we set sampling temperature T to be 0.2 and top p cut-off threshold to be 0.9. - We leave the exploration of other hyperparameters as the future work. -
-For score computation, we treat each question equally with one point each. - Given 270 questions in the benchmark, the full score is 270, and we by default report the percentage score (achieved score divided by the full score which is 270). - The one point for each question can be further decomposed into a few scoring points within each question. - For example, a question may contain four keywords with weights 2, 1, 1, and 1 each. - Then, matching each keyword can contribute to 0.4, 0.2, 0.2, and 0.2 points respectively to the final score. -
+ + +For closed-form questions, we prompt LLM with question description and constraints at first time. Considering that most models hardly follow the format requirements, we add a reformat step after the models respond with gpt-3.5-turbo-16k which formats the responses with the format requirements. Here's a figure illustrating this process:
+
Prompts and Evaluation Protocol
Leaderboard
+ +In this section, we furnish a comprehensive evaluation of both close-source LLMs such as GPT-4 and GPT-3.5, as well as widely-utilized open-source LLMs. It is observed that the close-source LLMs demonstrate a proficient capacity to adhere to agent directives and produce logically coherent responses. + Regrettably, the open-source LLMs enumerated below are currently incapable of adhering to agent directives or delivering substantively meaningful responses. In response to this limitation, we have developed a Supervised Fine-Tuning (SFT) dataset aimed at refining these models. For these open-source LLMs, the table below delineates the enhancements observed post-finetuning. Please anticipate the imminent release of both the SFT dataset and its comprehensive details.
+
+ Leaderboard
-
+
| Rank | Model Name | # Params. (in B) | -Full Set Score | -Full Set Std | -Dev Set Score | -Dev Set Std | +Proportional Accuracy by Subquestions | +Accuracy by questions | +Uniform Accuracy by subquestions | + @@ -363,756 +365,179 @@GPT-4 (0613) | -/ | +/ | -58.89% | - -0.74% | - -61.28% | + +65.26% | +66.05% | +59.75% | +
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | -deepseek-coder-33b-instruct | +GPT-3.5 (turbo-0613) | -33 | - -50.34% | +/ | + +55.35% | +52.21% | +47.25% | +|||||||
| 3 | -GPT-3.5-turbo (0613) | - - -/ | +CodeLlama-7B | -46.70% | -0.70% | +7 | -46.16% | + +47.59% | +44.67% | +39.59% | +|||||
| 4 | -WizardCoder-Python-34B-V1.0 | - +CodeLlama-7B-python | -34 | -45.16% | +7 | + +47.03% | +40.78% | +40.86% | +|||||||
| 5 | -CodeLlama-34B-Instruct | - +Llama2-7B | -34 | -43.71% | +7 | + +37.53% | +34.01% | +32.49% | +|||||||
| 6 | -deepseek-coder-6.7b-instruct | - - -6.7 | - -42.97% | - -0.22% | - -42.28% | - -0.24% | - -43.66% | - -0.26% | - -- | ||||||
| 7 | - -WizardCoder-Python-13B-V1.0 | - - -13 | - -41.22% | - -0.75% | - -41.46% | - -1.13% | - -40.97% | - -0.42% | - -- | ||||||
| 8 | - -WizardCoder-Python-7B-V1.0 | - - -7 | - -40.30% | - -1.15% | - -43.70% | - -2.09% | - -36.90% | - -0.22% | - -- | ||||||
| 9 | - -CodeLlama-34B | - +WizardCoder-Python-7B-V1.0 | -34 | -39.75% | +7 | + +33.59% | +30.48% | +28.75% | +|||||||
| 31 | - -phi-1.5 | - - -1.5 | - -16.63% | - -0.03% | +7 | -14.10% | +ChatGLM3 | -0.84% | -19.16% | +6 | -0.87% | + +18.10% | +15.19% | +14.84% | +