feat(ComputerSystems): add CacheReplacementPolicyOptimization benchmark

[y-ji24]

This benchmark is a test task for an agent that automatically evaluates and merges engineering tasks.

The task is to optimize an online cache replacement policy using a real-world Meta kvcache workload published through CacheLib. The goal is to improve cache hit rate under a fixed cache capacity, with CacheLib’s LRU policy serving as the baseline.

The agent must produce a valid policy that performs better than the baseline on a frozen held-out trace from the same workload family. The evaluator enforces online, causal decisions and checks the result against the offline-optimal hit-rate bound, enabling a reproducible assessment of the agent’s engineering solution.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark task named CacheReplacementPolicyOptimization under the ComputerSystems category. The task challenges agents to implement an online cache eviction policy to optimize hit rates based on real-world Meta production KV-cache traces.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: Added the new task to the "ComputerSystems" section in the task registry.
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/README.md: Provides a high-level overview, layout description, and local execution instructions.
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/Task.md: Detailed task description, objective, submission contract, and scoring metrics.
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/baseline/solution.py: Implements a standard LRU (Least Recently Used) policy as the baseline.
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/docs/PROVENANCE.md: An exceptionally detailed document tracing the origin of every constant, trace file, and metric to real Meta/CacheLib artifacts.
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/verification/: Contains the evaluator and reference cache models (e.g., evaluator.py, reference_cache.py).
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/traces/: Includes real-world KV-cache trace snippets (CSV format).

2. AI Content Analysis

• Estimated AI Component: 5%
• Reasoning & Evidence: The vast majority of the content exhibits high domain-specific nuance and academic rigor that is untypical for generic AI generation. The PROVENANCE.md file cites specific SOSP/NSDI papers (e.g., SIEVE NSDI'24, S3-FIFO SOSP'23) and links to specific AWS S3 buckets and CacheLib source code lines (e.g., KVReplayGenerator.h line 82). The code structure follows a strict, custom evaluation protocol (FE-BATCH-JSON-STDIO-V1). While some boilerplate in solution.py (like the OrderedDict LRU implementation) is standard, the overall integration and documentation are clearly the work of a domain expert.

3. Engineering & Economic Assessment

• Engineering Reality Check: Production-grade. Unlike "toy" cache problems, this uses real production traces from Meta. It addresses critical systems engineering constraints: fixed DRAM budget, online decision-making (preventing look-ahead cheating), and comparison against the theoretical offline optimum (Belady’s algorithm). It handles edge cases like DELETE operations and op_count expansion.
• Economic Value: High. In large-scale distributed systems (like Meta's), even a 1-2% increase in cache hit rate translates to massive reductions in backend QPS, network bandwidth consumption, and tail latency, leading to significant infrastructure cost savings.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: CacheReplacementPolicyOptimization
  • Execution & Dependencies: The README.md provides a clear bash snippet for local verification. Dependencies are minimal (Python standard library), ensuring high portability.
• Documentation Quality: Excellent. The documentation is comprehensive. PROVENANCE.md is a benchmark for transparency, explaining why certain data was rejected or scaled. No spelling or grammatical errors were detected in the provided diff.
• Organizational Structure: Logical and Modular. The separation of baseline, docs, traces, and verification follows best practices for benchmark suites.

5. Security & Privacy Check

• Sensitive Files: Clean. The .gitignore and .gitattributes are correctly configured to handle CSV traces and prevent __pycache__ leaks. No API keys or .env files were found.
• Absolute Paths: None detected. The code uses relative paths and pathlib for file handling.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 在 ComputerSystems 类别下引入了一个名为 CacheReplacementPolicyOptimization 的新工程基准任务。该任务要求智能体实现一种在线缓存淘汰策略，以在真实的 Meta 生产级 KV 缓存追踪数据（traces）上优化命中率。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: 在任务注册表的“ComputerSystems”部分添加了新任务。
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/README.md: 提供了高层概述、目录结构说明和本地运行指南。
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/Task.md: 详细的任务描述、目标、提交协议和评分指标。
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/baseline/solution.py: 实现了标准的 LRU（最近最少使用）策略作为基准线。
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/docs/PROVENANCE.md: 一份极其详尽的文档，追溯了每个常量、追踪文件和指标在 Meta/CacheLib 真实工件中的来源。
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/verification/: 包含评估器和参考缓存模型（如 evaluator.py, reference_cache.py）。
  • benchmarks/ComputerSystems/CacheReplacementPolicyOptimization/traces/: 包含真实的 KV 缓存追踪数据片段（CSV 格式）。

2. AI 成分分析

• 预估 AI 含量: 5%
• 判断依据与证据: 绝大部分内容表现出极高的领域特定细微差别和学术严谨性，这在通用的 AI 生成内容中并不常见。PROVENANCE.md 引用了具体的 SOSP/NSDI 论文（如 SIEVE NSDI'24, S3-FIFO SOSP'23），并链接到了具体的 AWS S3 存储桶和 CacheLib 源代码行号（如 KVReplayGenerator.h 第 82 行）。代码结构遵循严格的自定义评估协议（FE-BATCH-JSON-STDIO-V1）。虽然 solution.py 中的一些模板代码（如使用 OrderedDict 实现 LRU）比较常见，但整体集成和文档显然是领域专家的手笔。

3. 工程与经济评估

• 工程现实检验: 生产级别。 与“玩具级”缓存问题不同，本项目使用来自 Meta 的真实生产追踪数据。它解决了关键的系统工程约束：固定的 DRAM 预算、在线决策（防止预知未来的作弊行为）以及与理论离线最优解（Belady 算法）的对比。它还处理了 DELETE 操作和 op_count 展开等边缘情况。
• 经济价值: 高。 在 Meta 这样的大规模分布式系统中，即使缓存命中率提高 1-2%，也会转化为后端 QPS、网络带宽消耗和尾部延迟的巨大降幅，从而显著降低基础设施成本。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: CacheReplacementPolicyOptimization
  • 运行与依赖: README.md 提供了一个清晰的 bash 代码段用于本地验证。依赖项极少（仅 Python 标准库），确保了高移植性。
• 文档质量: 优秀。 文档非常全面。PROVENANCE.md 是透明度的典范，解释了为什么某些数据被拒绝或缩放。在提供的 diff 中未检测到拼写或语法错误。
• 组织结构: 逻辑清晰且模块化。 baseline、docs、traces 和 verification 的划分符合基准测试套件的最佳实践。

5. 安全与隐私检查

• 敏感文件: 未发现异常。 .gitignore 和 .gitattributes 配置正确，处理了 CSV 追踪数据并防止了 __pycache__ 泄露。未发现 API 密钥或 .env 文件。
• 绝对路径: 未检测到。 代码使用相对路径和 pathlib 进行文件处理。