[Design update] Update Design of Judge Model and GEdit Bench in design doc#214
[Design update] Update Design of Judge Model and GEdit Bench in design doc#214SJTUyh wants to merge 4 commits intoAISBench:masterfrom
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! 此拉取请求主要更新了AISBench的设计文档，详细阐述了新增的裁判模型评估能力和GEdit基准测试支持。这些更新旨在扩展平台的评估范围，使其能够更全面地衡量大型语言模型和多模态模型的性能，特别是在文本正确性判断和图片编辑质量评估方面。文档的更新涵盖了从特性需求到系统架构、数据流、实现原理及兼容性分析等多个方面，为理解和使用这些新功能提供了全面的指导。 Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
| 4. 查看评估结果，与OpenCompass结果对比 | ||
|
|
||
| #### 场景2：多模态模型全面评估 | ||
| #### 场景2：多模态生成类模型全面评估 |
There was a problem hiding this comment.
章节标题似乎与内容不符。标题修改为“多模态生成类模型全面评估”，但该场景下的子场景（如视觉问答、视频理解、OCR等）均为理解类任务，而非生成类任务。这可能会让读者感到困惑。建议将标题改回“多模态模型全面评估”或修改为“多模态理解类模型全面评估”以保持一致性。
| #### 场景2：多模态生成类模型全面评估 | |
| #### 场景2：多模态理解类模型全面评估 |
| - 目前多模态生成类模型的开源服务化框架（vllm omni）还不成熟，当前工具只适配了基于昇腾NPU的Qwen-Image-Edit模型后端 | ||
| - GEdit评估需要两阶段裁判模型推理：SC（语义一致性）评估和PQ（感知质量）评估 | ||
|
|
||
| ### 2.3.1硬件限制 |
There was a problem hiding this comment.
此处的章节标题编号 2.3.1 与前面的章节 2.3.1 系统位置及周边接口 重复。这似乎是一个文档结构或编号错误，建议修正为正确的序号（例如 2.3.7），以保证文档的结构清晰和一致性。
| - 预期结果：使用环境变量的API Key（优先级更高） | ||
|
|
||
| # 6.Use Case三至六实现 | ||
| # 6.Use Case三至七实现 |
There was a problem hiding this comment.
章节标题为“Use Case三至七实现”，但接下来的内容实际包含了对“Use Case八”的实现说明。为了保持标题与内容的一致性，建议将标题修改为“Use Case三至八实现”。
| # 6.Use Case三至七实现 | |
| # 6.Use Case三至八实现 |
|
|
||
| - `EncodingTimeCalculator.calculate()`: 计算encoding阶段耗时 | ||
| - `TraceReplayCalculator.replay()`: 流量负载复现 | ||
| - `TraceReplayCalculator.replay()`: 流量负载复 |
There was a problem hiding this comment.
此处有拼写错误，“流量负载复”应为“流量负载复现”。
| - `TraceReplayCalculator.replay()`: 流量负载复 | |
| - `TraceReplayCalculator.replay()`: 流量负载复现 |
| - GEdit评估依赖LMM裁判模型进行SC和PQ评估 | ||
| - 裁判模型需要具备图片理解能力（如Qwen2.5-VL-7B） | ||
| - 裁判模型推理与被测模型推理解耦，可部署在不同硬件平台 | ||
| - 裁判模型推理结果通过judger（裁判）标识，与常规评估结果区分现 |
There was a problem hiding this comment.
此处有拼写错误，“区分现”应为“区分”。
| - 裁判模型推理结果通过judger（裁判）标识，与常规评估结果区分现 | |
| - 裁判模型推理结果通过judger（裁判）标识，与常规评估结果区分 |
1 participant
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。
Update Design of Judge Model and GEdit Bench in design doc
📝 Modification / 修改内容
Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。
Update Design of Judge Model and GEdit Bench in design doc
📐 Associated Test Results / 关联测试结果
Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。
NA
Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
🌟 Use cases (Optional) / 使用案例（可选）
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build