Add dynamic edge service replica placement benchmark

[bingxinli0607]

Summary

This adds EdgeServiceReplicaPlacement, a CPU-only unified benchmark under ComputerSystems. A candidate implements a dynamic replica-placement and routing policy; an independent deterministic simulator evaluates it across workload, failure, and link conditions.

I placed this benchmark under ComputerSystems based on the current repository taxonomy; I am happy to move it if maintainers prefer another category.

Engineering problem

The benchmark models capacity headroom, failure-domain placement, traffic steering, cold starts, recovery, compute usage, and cross-region traffic in a deliberately small edge topology. It is a reduced-order engineering benchmark, not a production cloud or Kubernetes model.

Candidate policy

Candidates implement decide(observation: dict) -> dict inside the EVOLVE-BLOCK. The observation contains current and recent workload, live nodes, active/pending replicas, RTT, and link limits. The action specifies desired replica counts and route fractions. Candidates cannot submit their own metrics or score.

Simulator and scenarios

• 3 regions, 6 nodes, and 3 independent services
• 24 five-minute control periods per scenario
• 5 scenario families with 2 deterministic variants each
• Normal diurnal demand, regional bursts, node failure plus burst, link degradation, and post-failure migration
• One-period replica cold start, capacity-aware routing, reduced-order P95/P99 latency, SLO violations, compute cost, cross-region traffic, and recovery steps

The action changes pending/active state and later costs, so this is not a one-shot placement or static combinatorial optimization problem.

Verification

The candidate runs in a separate worker process with bounded JSON-lines I/O, per-decision and per-scenario timeouts, stderr draining, process-tree cleanup, and strict schema/type/range checks.

The evaluator independently recomputes every metric and rejects non-finite values, bool-as-int counts, unknown or duplicate IDs, over-capacity placements, illegal route sums, and routes to pending or failed nodes.

This process boundary is reliability isolation, not an OS-level security sandbox.

The verification suite contains 36 tests. A separate three-period tiny oracle enumerates all 723 feasible trajectories and exactly matches production raw metrics, normalized components, and final score.

Scoring

Each scenario reports availability, request-weighted P95/P99 latency, P99 SLO violation, compute cost, cross-region traffic/cost, and recovery.

Dimensionless capped loss components are combined as 100 * exp(-loss); the final score combines the mean and P20 across ten scenarios.

Tail latency is normalized continuously by the median configured service P99 SLO. Compute and bandwidth retain feedback through three reference-budget units.

All weights, references, caps, parameter provenance, and sensitivity results are documented.

Relative weight perturbations of -5%, -2%, +2%, and +5% found no severe numerical fragility or obvious high-score extreme-policy exploit.

Baselines and optimization headroom

Policy	Score
initial/reasonable	73.9434
weak static	74.0766
handwritten strong	74.4628
evolved best (10-iteration sanity check)	74.6663
SLA-first overprovision	73.1402
fixed full capacity	70.2510
local routing only	70.0369
ignore failures	62.1667
zero replicas	11.7809
cost-first underprovision	11.1039
minimum replicas	9.5307
aggressive cross-region	7.6351

The labels do not impose a required ordering. Weak slightly exceeds the initial policy because it spends no cross-region bandwidth while accepting worse reliability/SLA metrics.

The handwritten strong policy is 0.5195 points above initial. The evolved best is 0.7229 points above initial.

Agent optimization sanity check

A 10-iteration OpenEvolve run with an OpenAI-compatible LLM backend was used as a generative-optimization sanity check.

• Initial score: 73.9434
• Handwritten strong calibration policy: 74.4628
• Best evolved score: 74.6663
• Parser success: 10/10
• Evaluator-valid candidates: 4/10
• Evaluator-invalid candidates: 6/10
• Best valid candidate: iteration 6

The best candidate improved availability, tail latency, SLA violations, compute cost, and cross-region cost rather than exploiting the scoring function.

This is a limited sanity check, not a broad claim about model performance. The benchmark itself does not depend on the provider used for the experiment, and no provider-specific compatibility code or prompt is included in this task.

Validation

• pytest: 36 passed
• Direct evaluator: valid on 10/10 scenarios, score 73.9434
• frontier_eval/run_eval.sh: valid, score 73.9434
• Unified zero-iteration: valid, score 73.9434
• Deterministic replay: 3 byte-identical metrics/artifacts runs
• Tiny oracle: 723 trajectories with exact raw/component/score match
• Metadata/readonly audit: exit 0; 22 task readonly entries, 0 missing
• Scoring/extreme/sensitivity analysis: 11 policies evaluated

Runtime and dependencies

The benchmark is CPU-only and its evaluator uses the Python standard library.

No GPU, cluster, external service, or downloaded dataset is required.

Known simplifications

• Workload, RTT, service rate, response size, failure, SLO, and abstract cost parameters are documented synthetic calibration assumptions or engineering simplifications, not industry-validated measurements.
• The latency relation is monotone and reduced-order, not an empirical packet or queueing fit.
• Services are independent.
• The benchmark intentionally omits Kubernetes, service DAGs, databases, packet-level networking, large traces, GPUs, and RL training.
• Public deterministic scenarios can be overfit; only one variant per family is returned in feedback artifacts.
• A Python subprocess is not a hostile-code security sandbox.

Reproduction

python -m pytest benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification -q

python benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/evaluator.py \
  benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/scripts/init.py

python benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/tiny_oracle.py

python benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/calibration/analyze_scoring.py

python -m frontier_eval \
  task=unified \
  task.benchmark=ComputerSystems/EdgeServiceReplicaPlacement \
  algorithm=openevolve \
  algorithm.iterations=0

python scripts/ops/audit_unified_metadata_readonly.py

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new benchmark task named EdgeServiceReplicaPlacement under the ComputerSystems category. It is a simulation-based engineering problem focused on dynamic edge-service replica placement and traffic routing under various stress conditions (workload bursts, node failures, and link degradation).
• Modified File Structure & Modifications:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: Updated the task registry table to include EdgeServiceReplicaPlacement under the ComputerSystems section.
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/README.md & README_zh-CN.md: Provided high-level task descriptions, verification commands, and environment workarounds.
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/Task.md & Task_zh-CN.md: Defined the technical interface (decide function), observation/action schemas, hard-invalid conditions, and the multi-objective scoring function.
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/baseline/result_log.txt: Documented the calibration process, baseline performance, and sensitivity analysis results.
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/calibration/analyze_scoring.py: A comprehensive script for reproducing score pipelines, testing extreme policies (e.g., zero replicas, local-only), and performing weight sensitivity checks.

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The code and documentation exhibit high domain-specific nuance (e.g., P99 SLO normalization, RTT-based routing, and specific edge computing constraints like "one-period cold start"). The analyze_scoring.py script contains complex heuristic logic for different policy modes (sla_first, cost_first, etc.) that reflects intentional engineering design rather than generic AI boilerplate. AI likely assisted in generating the dual-language documentation templates and standard Python boilerplate (e.g., argparse setup and OrderedDict usage), but the core logic is highly specialized.

3. Engineering & Economic Assessment

• Engineering Reality Check: This is a production-grade simulation. Unlike "toy" examples, it accounts for realistic distributed systems challenges:
  • Statefulness: Cold starts for new replicas.
  • Network Realities: Cross-region RTT and bandwidth limits.
  • Failure Modes: Simultaneous node failures and workload bursts.
  • Isolation: Uses a process boundary for agent execution to prevent state leakage.
• Economic Value: High. The task directly models the trade-offs between infrastructure cost (compute/bandwidth) and user experience (latency/availability). Solutions that optimize these metrics have direct applications in reducing cloud egress costs and improving SLA compliance for global edge networks.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: ComputerSystems/EdgeServiceReplicaPlacement
  • Execution & Dependencies: The README.md clearly documents the execution commands for both direct validation (python verification/evaluator.py) and unified evaluation via frontier_eval. It also notes environment-specific requirements like PYTHONUTF8=1 for Windows.
• Documentation Quality: Excellent. The documentation is comprehensive, providing both high-level summaries and deep technical specifications. The inclusion of a scoring-calibration-report and tiny_oracle analysis demonstrates a high level of rigor. No significant grammatical errors or formatting inconsistencies were detected.
• Organizational Structure: Logical and Modular. The separation of baseline, calibration, and verification logic follows standard software engineering best practices for benchmark development.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were found.
• Absolute Paths: None detected. The scripts use relative path resolution (e.g., Path(__file__).resolve().parents[1]) to ensure portability.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 在 ComputerSystems 类别下引入了一个名为 EdgeServiceReplicaPlacement 的新基准测试任务。这是一个基于模拟的工程问题，侧重于在各种压力条件（工作负载突发、节点故障和链路退化）下动态边缘服务副本的放置和请求路由。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: 更新了任务注册表，在 ComputerSystems 部分增加了 EdgeServiceReplicaPlacement。
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/README.md & README_zh-CN.md: 提供了高层级的任务描述、验证命令和环境变通方法。
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/Task.md & Task_zh-CN.md: 定义了技术接口（decide 函数）、观测/动作模式、硬性无效条件以及多目标评分函数。
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/baseline/result_log.txt: 记录了校准过程、基准性能和敏感性分析结果。
  • benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/calibration/analyze_scoring.py: 一个用于重现评分流水线、测试极端策略（如零副本、仅本地路由）以及执行权重敏感性检查的综合脚本。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 代码和文档表现出极高的领域特定细微差别（例如 P99 SLO 归一化、基于 RTT 的路由以及“单周期冷启动”等特定边缘计算约束）。analyze_scoring.py 脚本包含针对不同策略模式（sla_first、cost_first 等）的复杂启发式逻辑，反映了刻意的工程设计，而非通用的 AI 模板。AI 可能辅助生成了双语文档模板和标准的 Python 样板代码（如 argparse 设置和 OrderedDict 使用），但核心逻辑专业性极强。

3. 工程与经济评估

• 工程现实检验: 这是一个生产级的模拟任务。与“玩具”示例不同，它考虑了现实的分布式系统挑战：
  • 状态性: 新副本的冷启动。
  • 网络现实: 跨区域 RTT 和带宽限制。
  • 故障模式: 同时发生的节点故障和工作负载突发。
  • 隔离性: 使用进程边界执行 Agent，以防止状态泄漏。
• 经济价值: 高。该任务直接建模了基础设施成本（计算/带宽）与用户体验（延迟/可用性）之间的权衡。优化这些指标的解决方案在降低云流量成本和提高全球边缘网络 SLA 合规性方面具有直接应用价值。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: ComputerSystems/EdgeServiceReplicaPlacement
  • 运行与依赖: README.md 清晰地记录了直接验证（python verification/evaluator.py）和通过 frontier_eval 进行统一评估的执行命令。它还指出了特定环境的要求，如 Windows 下的 PYTHONUTF8=1。
• 文档质量: 优秀。文档非常全面，既提供了高层摘要，又提供了深度技术规范。包含评分校准报告和 tiny_oracle 分析证明了其严谨性。未检测到明显的语法错误或格式不一致。
• 组织结构: 逻辑清晰且模块化。将 baseline、calibration 和 verification 逻辑分离，符合基准测试开发的标准软件工程最佳实践。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未发现 .env、API 密钥或 IDE 特定配置。
• 绝对路径: 未检测到。脚本使用相对路径解析（如 Path(__file__).resolve().parents[1]）以确保可移植性。

[Copilot]

🟡 Changes recommended

The tiny oracle hard-codes a scoring reference that the production scorer derives from config, making the “independent transcription” brittle and prone to drift as configuration evolves.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new unified ComputerSystems/EdgeServiceReplicaPlacement benchmark that evaluates dynamic edge-service replica placement and traffic routing policies via a deterministic simulator, isolated candidate runtime, and comprehensive verification suite. This extends the repository’s benchmark taxonomy and provides an end-to-end, CPU-only evaluation loop suitable for unified runs.

Changes:

• Register the new EdgeServiceReplicaPlacement benchmark in the repository task listings and ComputerSystems README docs (EN + zh-CN).
• Introduce the benchmark package: task interface docs, deterministic simulator + evaluator, candidate process runtime, calibration policies, and Frontier Eval metadata.
• Add a verification suite (unit + adversarial tests) plus a multi-timestep tiny oracle and supporting documentation.

File summaries

File	Description
TASK_DETAILS.md	Adds the new benchmark to the top-level task taxonomy table.
TASK_DETAILS_zh-CN.md	Adds the new benchmark to the Chinese task taxonomy table.
benchmarks/ComputerSystems/README.md	Documents the benchmark under the ComputerSystems domain list.
benchmarks/ComputerSystems/README_zh-CN.md	Documents the benchmark under the ComputerSystems domain list (zh-CN).
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/README.md	Benchmark overview, how to validate, and editable boundary description.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/README_zh-CN.md	Benchmark overview (zh-CN).
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/Task.md	Defines candidate observation/action schema and validity rules (EN).
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/Task_zh-CN.md	Defines candidate observation/action schema and validity rules (zh-CN).
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/scripts/init.py	Provides the “reasonable” baseline policy inside the EVOLVE-BLOCK.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/references/config.json	Declares the task’s fixed topology, service params, and score budgets.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/references/design_notes.md	Records model scope and reduced-order assumptions.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/simulator.py	Implements the deterministic state machine, action validation, and metrics.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/evaluator.py	Runs isolated candidates across scenarios and computes the combined score + artifacts.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/policy_runtime.py	Parent-side process runtime (timeouts, bounded I/O, process-tree cleanup).
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/policy_worker.py	Worker-side JSON-lines RPC wrapper around candidate code.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/tiny_oracle.py	Adds a brute-force multi-timestep oracle for a tiny enumerable control case.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/test_simulator.py	Tests determinism, action validation, and a dimensional link/capacity check.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/test_policy_runtime.py	Tests runtime isolation, timeout handling, protocol robustness, stderr draining.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/test_evaluator.py	Tests scoring properties and baseline validity/determinism.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/test_tiny_oracle.py	Ensures tiny oracle matches production simulator/scoring exactly.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/verification/requirements.txt	Declares stdlib-only evaluator dependency intent.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/docs/parameter_assumptions.md	Documents parameter provenance, units, and calibration/maintainer decisions.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/docs/scoring-calibration-report.md	Documents scoring normalization/cap fixes and sensitivity analysis.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/docs/tiny_oracle.md	Explains the oracle case and the exact-match verification approach.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/docs/evaluator-threat-model.md	Documents reliability/isolation threats and defenses (non-OS-sandbox statement).
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/calibration/weak.py	Provides a weak static calibration policy for scoring comparisons.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/calibration/strong.py	Provides a stronger calibration policy for score spread calibration.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/calibration/analyze_scoring.py	Reproduces score pipeline, extreme policies, and weight sensitivity checks.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/run_eval.sh	Frontier Eval wrapper hook for running the evaluator.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/eval_command.txt	Declares the unified evaluation command template.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/initial_program.txt	Declares the initial editable program path.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/constraints.txt	Declares agent-edit constraints and runtime limits for unified runs.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/copy_files.txt	Declares which files are copied into the unified runtime.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/readonly_files.txt	Declares which files must be mounted readonly in unified runs.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/agent_files.txt	Declares which files are exposed to agents in unified runs.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/artifact_files.txt	Declares which artifacts are emitted by the benchmark.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/frontier_eval/candidate_destination.txt	Declares where edited candidates are written for evaluation.
benchmarks/ComputerSystems/EdgeServiceReplicaPlacement/baseline/result_log.txt	Captures baseline calibration/validation results and reproducibility notes.

Review details

• Files reviewed: 38/38 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.