diff --git a/TASK_DETAILS.md b/TASK_DETAILS.md index b331f7d8..5ecf73d8 100644 --- a/TASK_DETAILS.md +++ b/TASK_DETAILS.md @@ -82,7 +82,11 @@ We welcome new engineering problem ideas — even without complete verification C++ SHA3-256 throughput (OpenSSL verified) - CommunicationEngineering + CommunicationEngineering + AdaptiveLinkScheduling + Downlink resource-block scheduling with MCS and power control under queue, latency, fairness, and power-budget constraints + + LDPCErrorFloor LDPC code error floor estimation via importance sampling on trapping sets @@ -298,7 +302,11 @@ We welcome new engineering problem ideas — even without complete verification pyMOTO-based 2D beam topology optimization (SIMP + OC/MMA) under a volume-fraction constraint - Robotics + Robotics + AGVWarehouseRouting + Warehouse AGV pick-sequence optimization with aisle obstacles, congestion, and turn costs + + DynamicObstacleAvoidanceNavigation Navigate a differential-drive robot from start to goal in a dynamic environment @@ -337,10 +345,14 @@ We welcome new engineering problem ideas — even without complete verification Importance-sampling BER estimator for Hamming(127,120) - PowerSystems + PowerSystems EV2GymSmartCharging Upstream-aligned EV smart charging scheduling + + MicrogridBatteryDispatch + Commercial microgrid battery dispatch under solar forecasts, tariffs, demand charges, and degradation cost + AdditiveManufacturing DiffSimThermalControl diff --git a/TASK_DETAILS_zh-CN.md b/TASK_DETAILS_zh-CN.md index 44d1b1fa..f6fdbd99 100644 --- a/TASK_DETAILS_zh-CN.md +++ b/TASK_DETAILS_zh-CN.md @@ -82,7 +82,11 @@ Frontier-Eng 目前已覆盖以下领域的任务。每个任务均配有可运 C++ SHA3-256 吞吐量（OpenSSL 验证） - CommunicationEngineering + CommunicationEngineering + AdaptiveLinkScheduling + 在队列、时延、公平性与功率预算约束下进行下行资源块调度、MCS 与功率控制 + + LDPCErrorFloor 使用 importance sampling 针对 trapping sets 估计 LDPC 码 error floor @@ -298,7 +302,11 @@ Frontier-Eng 目前已覆盖以下领域的任务。每个任务均配有可运 基于 pyMOTO 的 2D 梁拓扑优化（SIMP + OC/MMA），体积分数约束 - Robotics + Robotics + AGVWarehouseRouting + 考虑巷道障碍、拥堵与转向成本的仓储 AGV 拣货顺序优化 + + DynamicObstacleAvoidanceNavigation 在动态环境中控制差分轮机器人从起点到终点 @@ -337,10 +345,14 @@ Frontier-Eng 目前已覆盖以下领域的任务。每个任务均配有可运 使用 importance sampling 估计 Hamming(127,120) 的 BER - PowerSystems + PowerSystems EV2GymSmartCharging 上游对齐的电动车智能充电调度 + + MicrogridBatteryDispatch + 在太阳能预测、分时电价、需量电费与退化成本下优化商业微电网电池调度 + AdditiveManufacturing DiffSimThermalControl diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/README.md b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/README.md new file mode 100644 index 00000000..8f4cd933 --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/README.md @@ -0,0 +1,47 @@ +# AdaptiveLinkScheduling + +Optimize per-resource-block user scheduling, modulation/coding selection, and +transmit power for a small wireless downlink under queue, channel, latency, and +power-budget constraints. + +## Files + +- `Task.md`: task contract and scoring details. +- `scripts/init.py`: editable seed scheduler. +- `verification/evaluator.py`: deterministic link simulator and scorer. +- `verification/requirements.txt`: evaluator dependencies. +- `frontier_eval/`: unified-task metadata. + +## Candidate Interface + +Edit `scripts/init.py` only. The evaluator imports: + +```python +schedule_frame(frame: dict) -> list[dict] +``` + +Return one decision per resource block. Each decision should contain: + +```python +{"user": , "mcs": , "power_dbm": } +``` + +## Quick Run + +From this directory: + +```bash +python verification/evaluator.py scripts/init.py +``` + +From the repository root: + +```bash +python -m frontier_eval \ + task=unified \ + task.benchmark=CommunicationEngineering/AdaptiveLinkScheduling \ + algorithm=openevolve \ + algorithm.iterations=0 +``` + +No GPU, Docker, API key, or external assets are required. diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/Task.md b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/Task.md new file mode 100644 index 00000000..40af9b5f --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/Task.md @@ -0,0 +1,57 @@ +# Adaptive Link Scheduling + +## Background + +Wireless base stations continually decide which users receive resource blocks, +what modulation and coding scheme (MCS) to use, and how much power to transmit. +Good schedulers exploit favorable channel states while protecting users with +urgent queues and respecting power budgets. These decisions directly affect +throughput, latency, energy use, and fairness. + +## Objective + +For each fixed frame snapshot, produce one scheduling decision per resource +block. The verifier simulates packet delivery using deterministic SNR thresholds +and scores weighted throughput minus outage, power, budget, and fairness costs. + +## Candidate API + +The evaluator imports `schedule_frame(frame)` from `scripts/init.py`. + +`frame` contains: + +- `frame_id` +- `num_resource_blocks` +- `mcs_table`: list of `{mcs, snr_threshold_db, bits_per_rb}` +- `power_min_dbm`, `power_max_dbm`, `power_budget_mw` +- `users`: each user has `id`, `queue_bits`, `latency_weight`, + `min_service_bits`, and `snr_estimate_db` per resource block + +Return a list with `num_resource_blocks` entries. Each entry should be a mapping +with `user`, `mcs`, and `power_dbm`. + +## Constraints + +- Do not import external packages. +- Do not read or write files. +- Keep the public `schedule_frame(frame)` interface. +- Keep all editable logic inside the EVOLVE block. +- The scheduler must be deterministic for the same input frame. + +## Scoring + +For each resource block, the evaluator computes: + +```text +effective_snr = snr_estimate_db + (power_dbm - 20) +``` + +The selected MCS succeeds if `effective_snr` reaches the MCS threshold plus a +small implementation margin. Successful transmissions deliver `bits_per_rb` +subject to the selected user's remaining queue. Failed transmissions consume +power and incur outage penalty. + +The frame utility combines latency-weighted delivered bits, minimum-service +satisfaction, Jain fairness, power cost, and budget violations. + +`combined_score = mean(frame_utility across frames)`, so higher is better. diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/agent_files.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/agent_files.txt new file mode 100644 index 00000000..e60e972a --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/agent_files.txt @@ -0,0 +1,5 @@ +README.md +Task.md +scripts/init.py +verification/evaluator.py +frontier_eval/constraints.txt diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/artifact_files.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..76dc893a --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/artifact_files.txt @@ -0,0 +1,2 @@ +metrics.json +artifacts.json diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/candidate_destination.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/constraints.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/constraints.txt new file mode 100644 index 00000000..4c698f62 --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/constraints.txt @@ -0,0 +1,6 @@ +AdaptiveLinkScheduling constraints: +1) Modify only scripts/init.py. +2) Keep schedule_frame(frame) as the public entrypoint. +3) Return one decision per resource block with user, mcs, and power_dbm fields. +4) Do not use external packages, file I/O, network access, or randomness. +5) Keep EVOLVE-BLOCK markers intact and place editable scheduler logic inside them. diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/copy_files.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/copy_files.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/copy_files.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/eval_command.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/eval_command.txt new file mode 100644 index 00000000..527d23b5 --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +{python} verification/evaluator.py {candidate} --metrics-out metrics.json --artifacts-out artifacts.json diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/eval_cwd.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/eval_cwd.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/eval_cwd.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/initial_program.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/initial_program.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/readonly_files.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..c3f5eb23 --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/frontier_eval/readonly_files.txt @@ -0,0 +1,5 @@ +README.md +Task.md +verification/evaluator.py +verification/requirements.txt +frontier_eval/ diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/scripts/init.py b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/scripts/init.py new file mode 100644 index 00000000..63952e46 --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/scripts/init.py @@ -0,0 +1,79 @@ +from __future__ import annotations + +from typing import Any + + +def _choose_mcs(snr_db: float, mcs_table: list[dict[str, Any]], margin_db: float = 2.0) -> int: + selected = int(mcs_table[0]["mcs"]) + for item in mcs_table: + if snr_db >= float(item["snr_threshold_db"]) + margin_db: + selected = int(item["mcs"]) + return selected + + +def schedule_frame(frame: dict[str, Any]) -> list[dict[str, float | int]]: + """Return one scheduling decision per resource block.""" + + # EVOLVE-BLOCK-START + users = frame["users"] + mcs_table = frame["mcs_table"] + num_rbs = int(frame["num_resource_blocks"]) + power_min = float(frame["power_min_dbm"]) + power_max = float(frame["power_max_dbm"]) + nominal_power = min(power_max, max(power_min, 20.0)) + + remaining = {int(user["id"]): float(user["queue_bits"]) for user in users} + served = {int(user["id"]): 0.0 for user in users} + decisions: list[dict[str, float | int]] = [] + + for rb in range(num_rbs): + best_user = None + best_score = None + for user in users: + user_id = int(user["id"]) + if remaining[user_id] <= 0: + continue + snr = float(user["snr_estimate_db"][rb]) + latency = float(user["latency_weight"]) + deficit = max(0.0, float(user["min_service_bits"]) - served[user_id]) + score = snr + 3.0 * latency + 0.002 * deficit + 0.0005 * remaining[user_id] + if best_score is None or score > best_score: + best_score = score + best_user = user + + if best_user is None: + decisions.append({"user": int(users[0]["id"]), "mcs": 0, "power_dbm": power_min}) + continue + + user_id = int(best_user["id"]) + snr_at_power = float(best_user["snr_estimate_db"][rb]) + (nominal_power - 20.0) + mcs = _choose_mcs(snr_at_power, mcs_table, margin_db=2.0) + bits = 0.0 + for item in mcs_table: + if int(item["mcs"]) == int(mcs): + bits = float(item["bits_per_rb"]) + break + delivered_estimate = min(bits, remaining[user_id]) + remaining[user_id] -= delivered_estimate + served[user_id] += delivered_estimate + decisions.append({"user": user_id, "mcs": int(mcs), "power_dbm": nominal_power}) + + return decisions + # EVOLVE-BLOCK-END + + +if __name__ == "__main__": + demo = { + "num_resource_blocks": 2, + "power_min_dbm": 5.0, + "power_max_dbm": 24.0, + "mcs_table": [ + {"mcs": 0, "snr_threshold_db": -3.0, "bits_per_rb": 180}, + {"mcs": 1, "snr_threshold_db": 1.0, "bits_per_rb": 300}, + ], + "users": [ + {"id": 0, "queue_bits": 1000, "latency_weight": 1.0, "min_service_bits": 200, "snr_estimate_db": [2.0, 1.0]}, + {"id": 1, "queue_bits": 800, "latency_weight": 1.4, "min_service_bits": 200, "snr_estimate_db": [0.0, 4.0]}, + ], + } + print(schedule_frame(demo)) diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/verification/evaluator.py b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/verification/evaluator.py new file mode 100644 index 00000000..2a03d8cf --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/verification/evaluator.py @@ -0,0 +1,268 @@ +from __future__ import annotations + +import argparse +import importlib.util +import json +import math +import statistics +import sys +import time +import traceback +from pathlib import Path +from types import ModuleType +from typing import Any + + +INVALID_COMBINED_SCORE = -1e18 +MCS_TABLE = [ + {"mcs": 0, "snr_threshold_db": -3.0, "bits_per_rb": 180.0}, + {"mcs": 1, "snr_threshold_db": 1.0, "bits_per_rb": 300.0}, + {"mcs": 2, "snr_threshold_db": 4.0, "bits_per_rb": 480.0}, + {"mcs": 3, "snr_threshold_db": 8.0, "bits_per_rb": 720.0}, + {"mcs": 4, "snr_threshold_db": 12.0, "bits_per_rb": 960.0}, + {"mcs": 5, "snr_threshold_db": 16.0, "bits_per_rb": 1200.0}, +] + + +def _write_json(path: str | None, payload: dict[str, Any]) -> None: + if not path: + return + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") + + +def _load_candidate(candidate_path: Path) -> ModuleType: + spec = importlib.util.spec_from_file_location("link_scheduler_candidate", candidate_path) + if spec is None or spec.loader is None: + raise ImportError(f"failed to load candidate module from {candidate_path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _frames() -> list[dict[str, Any]]: + frames: list[dict[str, Any]] = [] + for frame_id in range(18): + num_rbs = 12 + users: list[dict[str, Any]] = [] + for user_id in range(5): + queue = 1100.0 + 260.0 * ((frame_id + 2 * user_id) % 5) + 180.0 * math.sin(0.6 * frame_id + user_id) + latency = 0.8 + 0.25 * ((2 * user_id + frame_id) % 4) + min_service = 360.0 + 120.0 * ((user_id + frame_id) % 3) + snrs: list[float] = [] + for rb in range(num_rbs): + slow = 7.5 + 5.0 * math.sin(0.37 * frame_id + 0.73 * user_id) + selective = 4.3 * math.cos(0.61 * rb + 0.41 * user_id + 0.19 * frame_id) + edge_loss = -2.0 if user_id == 4 and frame_id % 3 == 1 else 0.0 + snrs.append(slow + selective + edge_loss) + users.append( + { + "id": user_id, + "queue_bits": max(250.0, queue), + "latency_weight": latency, + "min_service_bits": min_service, + "snr_estimate_db": snrs, + } + ) + frames.append( + { + "frame_id": f"frame_{frame_id:02d}", + "num_resource_blocks": num_rbs, + "mcs_table": [dict(item) for item in MCS_TABLE], + "power_min_dbm": 5.0, + "power_max_dbm": 24.0, + "power_budget_mw": 1450.0, + "implementation_margin_db": 1.25, + "users": users, + } + ) + return frames + + +def _mcs_by_id() -> dict[int, dict[str, float]]: + return {int(item["mcs"]): dict(item) for item in MCS_TABLE} + + +def _dbm_to_mw(power_dbm: float) -> float: + return 10.0 ** (power_dbm / 10.0) + + +def _coerce_decision(raw: Any) -> tuple[int, int, float]: + if not isinstance(raw, dict): + raise ValueError("each schedule entry must be a dict") + user = int(raw["user"]) + mcs = int(raw["mcs"]) + power = float(raw["power_dbm"]) + if not math.isfinite(power): + raise ValueError("power_dbm must be finite") + return user, mcs, power + + +def _public_frame(frame: dict[str, Any]) -> dict[str, Any]: + return { + "frame_id": frame["frame_id"], + "num_resource_blocks": frame["num_resource_blocks"], + "mcs_table": [dict(item) for item in frame["mcs_table"]], + "power_min_dbm": frame["power_min_dbm"], + "power_max_dbm": frame["power_max_dbm"], + "power_budget_mw": frame["power_budget_mw"], + "users": [ + { + "id": user["id"], + "queue_bits": user["queue_bits"], + "latency_weight": user["latency_weight"], + "min_service_bits": user["min_service_bits"], + "snr_estimate_db": list(user["snr_estimate_db"]), + } + for user in frame["users"] + ], + } + + +def _jain(values: list[float]) -> float: + if not values: + return 0.0 + total = sum(values) + sq = sum(v * v for v in values) + if sq <= 0.0: + return 0.0 + return total * total / (len(values) * sq) + + +def _score_frame(schedule_frame: Any, frame: dict[str, Any]) -> dict[str, Any]: + public = _public_frame(frame) + raw_schedule = schedule_frame(public) + if not isinstance(raw_schedule, (list, tuple)): + raise ValueError("schedule_frame must return a list") + if len(raw_schedule) != int(frame["num_resource_blocks"]): + raise ValueError( + f"expected {frame['num_resource_blocks']} decisions, got {len(raw_schedule)}" + ) + + mcs_lookup = _mcs_by_id() + users = {int(user["id"]): dict(user) for user in frame["users"]} + remaining = {user_id: float(user["queue_bits"]) for user_id, user in users.items()} + delivered = {user_id: 0.0 for user_id in users} + weighted_kbits = 0.0 + power_mw_total = 0.0 + outage_count = 0 + invalid_penalty = 0.0 + + for rb, raw in enumerate(raw_schedule): + try: + user_id, mcs_id, power_dbm = _coerce_decision(raw) + except Exception: + invalid_penalty += 20.0 + continue + if user_id not in users or mcs_id not in mcs_lookup: + invalid_penalty += 20.0 + continue + + clipped_power = max(float(frame["power_min_dbm"]), min(float(frame["power_max_dbm"]), power_dbm)) + invalid_penalty += abs(power_dbm - clipped_power) * 0.5 + power_dbm = clipped_power + power_mw = _dbm_to_mw(power_dbm) + power_mw_total += power_mw + + user = users[user_id] + mcs = mcs_lookup[mcs_id] + effective_snr = float(user["snr_estimate_db"][rb]) + (power_dbm - 20.0) + threshold = float(mcs["snr_threshold_db"]) + float(frame["implementation_margin_db"]) + if effective_snr + 1e-9 < threshold: + outage_count += 1 + continue + + bits = min(float(mcs["bits_per_rb"]), remaining[user_id]) + if bits <= 0.0: + continue + remaining[user_id] -= bits + delivered[user_id] += bits + weighted_kbits += float(user["latency_weight"]) * bits / 1000.0 + + service_bonus = 0.0 + service_shortfall = 0.0 + for user_id, user in users.items(): + target = float(user["min_service_bits"]) + got = delivered[user_id] + service_bonus += min(got, target) / 1000.0 + service_shortfall += max(0.0, target - got) / 1000.0 + + fairness = _jain([delivered[user_id] for user_id in sorted(delivered)]) + budget_excess = max(0.0, power_mw_total - float(frame["power_budget_mw"])) + utility = ( + weighted_kbits + + 1.25 * service_bonus + + 3.5 * fairness + - 0.0018 * power_mw_total + - 0.010 * budget_excess + - 0.85 * outage_count + - 1.7 * service_shortfall + - invalid_penalty + ) + return { + "frame_id": frame["frame_id"], + "utility": utility, + "delivered_bits_total": sum(delivered.values()), + "power_mw_total": power_mw_total, + "outage_count": outage_count, + "fairness": fairness, + "service_shortfall_kbits": service_shortfall, + "invalid_penalty": invalid_penalty, + "delivered_by_user": delivered, + } + + +def evaluate(candidate_path: str) -> tuple[dict[str, float], dict[str, Any]]: + module = _load_candidate(Path(candidate_path).resolve()) + schedule_frame = getattr(module, "schedule_frame", None) + if schedule_frame is None: + raise AttributeError("candidate must define schedule_frame(frame)") + frame_results = [_score_frame(schedule_frame, frame) for frame in _frames()] + mean_utility = statistics.fmean(float(item["utility"]) for item in frame_results) + mean_bits = statistics.fmean(float(item["delivered_bits_total"]) for item in frame_results) + mean_outages = statistics.fmean(float(item["outage_count"]) for item in frame_results) + metrics = { + "combined_score": float(mean_utility), + "valid": 1.0, + "mean_frame_utility": float(mean_utility), + "mean_delivered_bits": float(mean_bits), + "mean_outage_count": float(mean_outages), + } + artifacts = { + "frame_results": frame_results, + "score_direction": "higher_is_better", + } + return metrics, artifacts + + +def main(argv: list[str]) -> int: + parser = argparse.ArgumentParser() + parser.add_argument("candidate") + parser.add_argument("--metrics-out", default="metrics.json") + parser.add_argument("--artifacts-out", default="artifacts.json") + args = parser.parse_args(argv) + + start = time.time() + try: + metrics, artifacts = evaluate(args.candidate) + except Exception as exc: + metrics = { + "combined_score": INVALID_COMBINED_SCORE, + "valid": 0.0, + "runtime_s": float(time.time() - start), + } + artifacts = { + "error_message": str(exc), + "traceback": traceback.format_exc(), + } + metrics["runtime_s"] = float(time.time() - start) + artifacts["candidate_path"] = str(Path(args.candidate).resolve()) + _write_json(args.metrics_out, metrics) + _write_json(args.artifacts_out, artifacts) + print(json.dumps(metrics, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/verification/requirements.txt b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/verification/requirements.txt new file mode 100644 index 00000000..821e76fa --- /dev/null +++ b/benchmarks/CommunicationEngineering/AdaptiveLinkScheduling/verification/requirements.txt @@ -0,0 +1 @@ +# Python standard library only. diff --git a/benchmarks/CommunicationEngineering/README.md b/benchmarks/CommunicationEngineering/README.md index 97c8d3ae..160adb45 100644 --- a/benchmarks/CommunicationEngineering/README.md +++ b/benchmarks/CommunicationEngineering/README.md @@ -19,6 +19,10 @@ Communication engineering tasks involve optimizing various aspects of communicat ## Subtask Index +- `AdaptiveLinkScheduling/`: Optimize downlink resource-block scheduling, MCS selection, and transmit power under queue, channel, latency, fairness, and power-budget constraints. + - `frontier_eval` task name: `task=unified task.benchmark=CommunicationEngineering/AdaptiveLinkScheduling` + - quick run: `python -m frontier_eval task=unified task.benchmark=CommunicationEngineering/AdaptiveLinkScheduling algorithm.iterations=0` + - `LDPCErrorFloor/`: Estimate error floor for LDPC codes using importance sampling to handle rare trapping set events. - `frontier_eval` task name: `ldpc_error_floor` - quick run: `python -m frontier_eval task=ldpc_error_floor algorithm.iterations=0` @@ -30,4 +34,3 @@ Communication engineering tasks involve optimizing various aspects of communicat - `PMDSimulation/`: Simulate Polarization Mode Dispersion (PMD) in optical fiber systems using importance sampling for rare outage events. - `frontier_eval` task name: `pmd_simulation` - quick run: `python -m frontier_eval task=pmd_simulation algorithm.iterations=0` - diff --git a/benchmarks/CommunicationEngineering/README_zh-CN.md b/benchmarks/CommunicationEngineering/README_zh-CN.md index 0b825bce..0cc2c7c3 100644 --- a/benchmarks/CommunicationEngineering/README_zh-CN.md +++ b/benchmarks/CommunicationEngineering/README_zh-CN.md @@ -19,6 +19,10 @@ ## 子任务索引 +- `AdaptiveLinkScheduling/`: 在队列、信道、时延、公平性与功率预算约束下优化下行资源块调度、MCS 选择和发射功率。 + - `frontier_eval` 任务: `task=unified task.benchmark=CommunicationEngineering/AdaptiveLinkScheduling` + - 快速运行: `python -m frontier_eval task=unified task.benchmark=CommunicationEngineering/AdaptiveLinkScheduling algorithm.iterations=0` + - `LDPCErrorFloor/`: 使用重要性采样估计LDPC码的错误地板，处理罕见的trapping set事件。 - `frontier_eval` 任务: `task=unified task.benchmark=CommunicationEngineering/LDPCErrorFloor` - 快速运行: `python -m frontier_eval task=unified task.benchmark=CommunicationEngineering/LDPCErrorFloor algorithm.iterations=0` @@ -30,4 +34,3 @@ - `PMDSimulation/`: 使用重要性采样仿真光纤系统中的极化模色散(PMD)，处理罕见的停机事件。 - `frontier_eval` 任务: `task=unified task.benchmark=CommunicationEngineering/PMDSimulation` - 快速运行: `python -m frontier_eval task=unified task.benchmark=CommunicationEngineering/PMDSimulation algorithm.iterations=0` - diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/README.md b/benchmarks/PowerSystems/MicrogridBatteryDispatch/README.md new file mode 100644 index 00000000..be94c8ad --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/README.md @@ -0,0 +1,45 @@ +# MicrogridBatteryDispatch + +Optimize battery dispatch for a grid-connected commercial microgrid with solar PV, +time-varying tariffs, demand charges, round-trip losses, and degradation cost. + +## Files + +- `Task.md`: task contract, model, and scoring details. +- `scripts/init.py`: editable seed policy used by agents. +- `verification/evaluator.py`: deterministic simulator and scorer. +- `verification/requirements.txt`: evaluator dependencies. +- `frontier_eval/`: unified-task metadata. + +## Candidate Interface + +Edit `scripts/init.py` only. The evaluator imports: + +```python +dispatch_action(state: dict) -> float +``` + +The returned value is battery power in kW. Positive values discharge the battery +to serve load, and negative values charge the battery. The evaluator enforces +all physical limits and penalizes requests that exceed feasible charge or +discharge bounds. + +## Quick Run + +From this directory: + +```bash +python verification/evaluator.py scripts/init.py +``` + +From the repository root: + +```bash +python -m frontier_eval \ + task=unified \ + task.benchmark=PowerSystems/MicrogridBatteryDispatch \ + algorithm=openevolve \ + algorithm.iterations=0 +``` + +No GPU, Docker, API key, or external assets are required. diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/Task.md b/benchmarks/PowerSystems/MicrogridBatteryDispatch/Task.md new file mode 100644 index 00000000..6d47f2cf --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/Task.md @@ -0,0 +1,64 @@ +# Microgrid Battery Dispatch + +## Background + +Commercial microgrids with rooftop solar and battery storage can reduce +electricity cost by shifting energy away from expensive hours, absorbing local +solar surplus, and limiting monthly peak demand. A practical controller must +respect battery limits, round-trip losses, and degradation cost while reacting to +load, solar, and tariff forecasts. + +## Objective + +Write a deterministic battery dispatch policy that minimizes operating cost over +several fixed microgrid cases. Each case contains hourly load, PV generation, +buy/sell tariff, demand charge, battery capacity, charge/discharge limits, and +efficiency values. + +## Candidate API + +The evaluator imports `dispatch_action(state)` from `scripts/init.py`. + +`state` contains: + +- `case_id`, `t`, `hour` +- `soc_kwh`, `min_soc_kwh`, `capacity_kwh` +- `max_charge_kw`, `max_discharge_kw` +- `charge_efficiency`, `discharge_efficiency` +- `load_forecast_kw`, `pv_forecast_kw` +- `buy_price_forecast`, `sell_price_forecast` +- `demand_charge_per_kw` +- `degradation_cost_per_kwh` + +The function returns a single finite float: + +- positive: discharge kW +- negative: charge kW +- zero: idle + +The simulation timestep is one hour. + +## Constraints + +- Do not import external packages. +- Do not read or write files. +- Keep the public `dispatch_action(state)` interface. +- Keep all editable logic inside the EVOLVE block. +- The policy must be deterministic for the same input state. + +## Scoring + +For each case the evaluator simulates the dispatch policy and computes: + +```text +total_cost = + grid_import_cost + - export_credit + + demand_charge + + battery_degradation_cost + + infeasible_action_penalty +``` + +`combined_score = -mean(total_cost across cases)`, so higher is better. +Feasible policies receive `valid = 1.0`; non-finite actions or evaluator errors +receive an invalid score. diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/agent_files.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/agent_files.txt new file mode 100644 index 00000000..e60e972a --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/agent_files.txt @@ -0,0 +1,5 @@ +README.md +Task.md +scripts/init.py +verification/evaluator.py +frontier_eval/constraints.txt diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/artifact_files.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..76dc893a --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/artifact_files.txt @@ -0,0 +1,2 @@ +metrics.json +artifacts.json diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/candidate_destination.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/constraints.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/constraints.txt new file mode 100644 index 00000000..873c9d0e --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/constraints.txt @@ -0,0 +1,6 @@ +MicrogridBatteryDispatch constraints: +1) Modify only scripts/init.py. +2) Keep dispatch_action(state) as the public entrypoint. +3) Return one finite float: positive for discharge kW, negative for charge kW. +4) Do not use external packages, file I/O, network access, or randomness. +5) Keep EVOLVE-BLOCK markers intact and place editable policy logic inside them. diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/copy_files.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/copy_files.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/copy_files.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/eval_command.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/eval_command.txt new file mode 100644 index 00000000..527d23b5 --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +{python} verification/evaluator.py {candidate} --metrics-out metrics.json --artifacts-out artifacts.json diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/eval_cwd.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/eval_cwd.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/eval_cwd.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/initial_program.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/initial_program.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/readonly_files.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..c3f5eb23 --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/frontier_eval/readonly_files.txt @@ -0,0 +1,5 @@ +README.md +Task.md +verification/evaluator.py +verification/requirements.txt +frontier_eval/ diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/scripts/init.py b/benchmarks/PowerSystems/MicrogridBatteryDispatch/scripts/init.py new file mode 100644 index 00000000..67b723d6 --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/scripts/init.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +from typing import Any + + +def dispatch_action(state: dict[str, Any]) -> float: + """Return battery power in kW: positive discharges, negative charges.""" + + # EVOLVE-BLOCK-START + soc = float(state["soc_kwh"]) + capacity = float(state["capacity_kwh"]) + min_soc = float(state["min_soc_kwh"]) + max_discharge = float(state["max_discharge_kw"]) + discharge_eff = float(state["discharge_efficiency"]) + + load_forecast = [float(x) for x in state["load_forecast_kw"][:24]] + pv_forecast = [float(x) for x in state["pv_forecast_kw"][:24]] + net_forecast = [load - pv for load, pv in zip(load_forecast, pv_forecast)] + net_load = net_forecast[0] if net_forecast else 0.0 + + discharge_room_kw = max(0.0, (soc - min_soc) * max(discharge_eff, 1e-9)) + reserve_soc = min(capacity, min_soc + 0.05 * (capacity - min_soc)) + available_discharge = min(discharge_room_kw, max(0.0, (soc - reserve_soc) * discharge_eff)) + future_peak = max(net_forecast) if net_forecast else net_load + peak_target = max(90.0, 0.90 * future_peak) + shave_kw = max(0.0, net_load - peak_target) + + if shave_kw > 0.0 and available_discharge > 0.0: + return min(max_discharge, shave_kw, available_discharge) + + return 0.0 + # EVOLVE-BLOCK-END + + +if __name__ == "__main__": + demo_state = { + "soc_kwh": 90.0, + "min_soc_kwh": 20.0, + "capacity_kwh": 160.0, + "max_charge_kw": 45.0, + "max_discharge_kw": 45.0, + "charge_efficiency": 0.94, + "discharge_efficiency": 0.93, + "load_forecast_kw": [120.0] * 24, + "pv_forecast_kw": [80.0] * 24, + "buy_price_forecast": [0.18] * 24, + "sell_price_forecast": [0.05] * 24, + } + print(dispatch_action(demo_state)) diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/verification/evaluator.py b/benchmarks/PowerSystems/MicrogridBatteryDispatch/verification/evaluator.py new file mode 100644 index 00000000..6af5f1b1 --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/verification/evaluator.py @@ -0,0 +1,265 @@ +from __future__ import annotations + +import argparse +import importlib.util +import json +import math +import statistics +import sys +import time +import traceback +from pathlib import Path +from types import ModuleType +from typing import Any + + +INVALID_COMBINED_SCORE = -1e18 + + +def _write_json(path: str | None, payload: dict[str, Any]) -> None: + if not path: + return + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") + + +def _load_candidate(candidate_path: Path) -> ModuleType: + spec = importlib.util.spec_from_file_location("microgrid_candidate", candidate_path) + if spec is None or spec.loader is None: + raise ImportError(f"failed to load candidate module from {candidate_path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _case_profiles(case_id: str, hours: int, scale: float, pv_scale: float, phase: float) -> dict[str, Any]: + load: list[float] = [] + pv: list[float] = [] + buy: list[float] = [] + sell: list[float] = [] + for t in range(hours): + hour = t % 24 + weekday = (t // 24) % 7 + morning = math.exp(-((hour - 8.0) / 3.0) ** 2) + evening = math.exp(-((hour - 18.0) / 4.0) ** 2) + base = scale * (86.0 + 16.0 * morning + 34.0 * evening) + weather = 1.0 + 0.08 * math.sin(0.47 * t + phase) + 0.04 * math.cos(0.19 * t) + load.append(max(25.0, base * weather + (6.0 if weekday >= 5 else 0.0))) + + sun = max(0.0, math.sin(math.pi * (hour - 6.0) / 12.0)) + cloud = 0.72 + 0.20 * math.sin(0.31 * t + phase) + 0.08 * math.cos(0.11 * t + 1.7) + pv.append(max(0.0, pv_scale * 92.0 * sun * max(0.15, cloud))) + + peak = 1.0 if 16 <= hour <= 21 else 0.0 + shoulder = 1.0 if 7 <= hour <= 15 else 0.0 + night = 1.0 if hour <= 5 or hour >= 22 else 0.0 + buy_price = 0.105 + 0.175 * peak + 0.055 * shoulder - 0.018 * night + buy_price += 0.012 * math.sin(0.23 * t + phase) + buy.append(max(0.05, buy_price)) + sell.append(max(0.025, 0.42 * buy[-1] - 0.015)) + + return { + "case_id": case_id, + "load_kw": load, + "pv_kw": pv, + "buy_price": buy, + "sell_price": sell, + } + + +def _build_cases() -> list[dict[str, Any]]: + cases = [ + _case_profiles("office_summer_peak", 96, 1.05, 1.05, 0.2), + _case_profiles("hospital_cloudy_week", 120, 1.28, 0.62, 1.4), + _case_profiles("warehouse_solar_rich", 96, 0.82, 1.38, 2.6), + ] + params = [ + { + "capacity_kwh": 260.0, + "initial_soc_kwh": 132.0, + "min_soc_kwh": 31.0, + "max_charge_kw": 70.0, + "max_discharge_kw": 72.0, + "charge_efficiency": 0.94, + "discharge_efficiency": 0.93, + "demand_charge_per_kw": 15.5, + "degradation_cost_per_kwh": 0.018, + }, + { + "capacity_kwh": 420.0, + "initial_soc_kwh": 250.0, + "min_soc_kwh": 84.0, + "max_charge_kw": 95.0, + "max_discharge_kw": 92.0, + "charge_efficiency": 0.93, + "discharge_efficiency": 0.92, + "demand_charge_per_kw": 18.0, + "degradation_cost_per_kwh": 0.022, + }, + { + "capacity_kwh": 190.0, + "initial_soc_kwh": 78.0, + "min_soc_kwh": 23.0, + "max_charge_kw": 54.0, + "max_discharge_kw": 58.0, + "charge_efficiency": 0.95, + "discharge_efficiency": 0.94, + "demand_charge_per_kw": 12.5, + "degradation_cost_per_kwh": 0.016, + }, + ] + for case, extra in zip(cases, params, strict=True): + case.update(extra) + return cases + + +def _forecast(values: list[float], t: int, horizon: int = 24) -> list[float]: + if not values: + return [] + return [float(values[min(len(values) - 1, t + k)]) for k in range(horizon)] + + +def _simulate_case(dispatch_action: Any, case: dict[str, Any]) -> dict[str, Any]: + soc = float(case["initial_soc_kwh"]) + capacity = float(case["capacity_kwh"]) + min_soc = float(case["min_soc_kwh"]) + charge_eff = float(case["charge_efficiency"]) + discharge_eff = float(case["discharge_efficiency"]) + max_charge = float(case["max_charge_kw"]) + max_discharge = float(case["max_discharge_kw"]) + + import_cost = 0.0 + export_credit = 0.0 + degradation_cost = 0.0 + infeasible_penalty = 0.0 + peak_import_kw = 0.0 + throughput_kwh = 0.0 + actions: list[float] = [] + soc_trace: list[float] = [soc] + + hours = len(case["load_kw"]) + for t in range(hours): + state = { + "case_id": case["case_id"], + "t": t, + "hour": t % 24, + "soc_kwh": soc, + "min_soc_kwh": min_soc, + "capacity_kwh": capacity, + "max_charge_kw": max_charge, + "max_discharge_kw": max_discharge, + "charge_efficiency": charge_eff, + "discharge_efficiency": discharge_eff, + "demand_charge_per_kw": float(case["demand_charge_per_kw"]), + "degradation_cost_per_kwh": float(case["degradation_cost_per_kwh"]), + "load_forecast_kw": _forecast(case["load_kw"], t), + "pv_forecast_kw": _forecast(case["pv_kw"], t), + "buy_price_forecast": _forecast(case["buy_price"], t), + "sell_price_forecast": _forecast(case["sell_price"], t), + } + raw_action = dispatch_action(state) + try: + requested = float(raw_action) + except Exception as exc: + raise ValueError(f"dispatch_action returned non-numeric value at t={t}: {raw_action!r}") from exc + if not math.isfinite(requested): + raise ValueError(f"dispatch_action returned non-finite value at t={t}: {requested!r}") + + charge_limit = min(max_charge, max(0.0, (capacity - soc) / max(charge_eff, 1e-9))) + discharge_limit = min(max_discharge, max(0.0, (soc - min_soc) * max(discharge_eff, 1e-9))) + applied = max(-charge_limit, min(discharge_limit, requested)) + infeasible_penalty += abs(requested - applied) * 1.5 + + if applied >= 0.0: + soc -= applied / max(discharge_eff, 1e-9) + else: + soc += (-applied) * charge_eff + soc = min(capacity, max(min_soc, soc)) + + load = float(case["load_kw"][t]) + pv = float(case["pv_kw"][t]) + grid_kw = load - pv - applied + grid_import = max(0.0, grid_kw) + grid_export = max(0.0, -grid_kw) + import_cost += grid_import * float(case["buy_price"][t]) + export_credit += grid_export * float(case["sell_price"][t]) + throughput_kwh += abs(applied) + degradation_cost += abs(applied) * float(case["degradation_cost_per_kwh"]) + peak_import_kw = max(peak_import_kw, grid_import) + actions.append(applied) + soc_trace.append(soc) + + demand_charge = peak_import_kw * float(case["demand_charge_per_kw"]) + total_cost = import_cost - export_credit + degradation_cost + demand_charge + infeasible_penalty + return { + "case_id": case["case_id"], + "total_cost": total_cost, + "import_cost": import_cost, + "export_credit": export_credit, + "degradation_cost": degradation_cost, + "demand_charge": demand_charge, + "infeasible_penalty": infeasible_penalty, + "peak_import_kw": peak_import_kw, + "throughput_kwh": throughput_kwh, + "final_soc_kwh": soc, + "mean_action_kw": statistics.fmean(actions) if actions else 0.0, + "soc_min_kwh": min(soc_trace), + "soc_max_kwh": max(soc_trace), + } + + +def evaluate(candidate_path: str) -> tuple[dict[str, float], dict[str, Any]]: + module = _load_candidate(Path(candidate_path).resolve()) + dispatch_action = getattr(module, "dispatch_action", None) + if dispatch_action is None: + raise AttributeError("candidate must define dispatch_action(state)") + + case_results = [_simulate_case(dispatch_action, case) for case in _build_cases()] + mean_cost = statistics.fmean(float(item["total_cost"]) for item in case_results) + mean_peak = statistics.fmean(float(item["peak_import_kw"]) for item in case_results) + mean_penalty = statistics.fmean(float(item["infeasible_penalty"]) for item in case_results) + metrics = { + "combined_score": -float(mean_cost), + "valid": 1.0, + "mean_total_cost": float(mean_cost), + "mean_peak_import_kw": float(mean_peak), + "mean_infeasible_penalty": float(mean_penalty), + } + artifacts = { + "case_results": case_results, + "score_direction": "higher_is_better", + } + return metrics, artifacts + + +def main(argv: list[str]) -> int: + parser = argparse.ArgumentParser() + parser.add_argument("candidate") + parser.add_argument("--metrics-out", default="metrics.json") + parser.add_argument("--artifacts-out", default="artifacts.json") + args = parser.parse_args(argv) + + start = time.time() + try: + metrics, artifacts = evaluate(args.candidate) + except Exception as exc: + metrics = { + "combined_score": INVALID_COMBINED_SCORE, + "valid": 0.0, + "runtime_s": float(time.time() - start), + } + artifacts = { + "error_message": str(exc), + "traceback": traceback.format_exc(), + } + metrics["runtime_s"] = float(time.time() - start) + artifacts["candidate_path"] = str(Path(args.candidate).resolve()) + _write_json(args.metrics_out, metrics) + _write_json(args.artifacts_out, artifacts) + print(json.dumps(metrics, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/benchmarks/PowerSystems/MicrogridBatteryDispatch/verification/requirements.txt b/benchmarks/PowerSystems/MicrogridBatteryDispatch/verification/requirements.txt new file mode 100644 index 00000000..821e76fa --- /dev/null +++ b/benchmarks/PowerSystems/MicrogridBatteryDispatch/verification/requirements.txt @@ -0,0 +1 @@ +# Python standard library only. diff --git a/benchmarks/PowerSystems/README.md b/benchmarks/PowerSystems/README.md index 6a3f9d2e..e0e2ca90 100644 --- a/benchmarks/PowerSystems/README.md +++ b/benchmarks/PowerSystems/README.md @@ -9,3 +9,8 @@ Current tasks emphasize realistic operational constraints, economic objectives, - Unified benchmark: `task=unified task.benchmark=PowerSystems/EV2GymSmartCharging` - Quick run: `python -m frontier_eval task=unified task.benchmark=PowerSystems/EV2GymSmartCharging task.runtime.env_name=frontier-eval-driver algorithm.iterations=0` - Description: upstream-aligned EV smart charging with transformer constraints in the real `EV2Gym` simulator + +- `MicrogridBatteryDispatch` + - Unified benchmark: `task=unified task.benchmark=PowerSystems/MicrogridBatteryDispatch` + - Quick run: `python -m frontier_eval task=unified task.benchmark=PowerSystems/MicrogridBatteryDispatch algorithm.iterations=0` + - Description: battery dispatch for a solar commercial microgrid under tariffs, demand charges, degradation, and physical limits diff --git a/benchmarks/PowerSystems/README_zh-CN.md b/benchmarks/PowerSystems/README_zh-CN.md index 30fcfd26..db65486e 100644 --- a/benchmarks/PowerSystems/README_zh-CN.md +++ b/benchmarks/PowerSystems/README_zh-CN.md @@ -9,3 +9,8 @@ - `frontier_eval` 任务：`task=unified task.benchmark=PowerSystems/EV2GymSmartCharging` - 快速运行：`python -m frontier_eval task=unified task.benchmark=PowerSystems/EV2GymSmartCharging task.runtime.env_name=frontier-eval-driver algorithm.iterations=0` - 简介：在真实上游 `EV2Gym` 模拟器中进行、与上游数据对齐的 EV 智能充电与变压器约束优化 + +- `MicrogridBatteryDispatch` + - `frontier_eval` 任务：`task=unified task.benchmark=PowerSystems/MicrogridBatteryDispatch` + - 快速运行：`python -m frontier_eval task=unified task.benchmark=PowerSystems/MicrogridBatteryDispatch algorithm.iterations=0` + - 简介：在分时电价、需量电费、电池退化和物理功率约束下优化商业微电网电池调度 diff --git a/benchmarks/Robotics/AGVWarehouseRouting/README.md b/benchmarks/Robotics/AGVWarehouseRouting/README.md new file mode 100644 index 00000000..303512cf --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/README.md @@ -0,0 +1,44 @@ +# AGVWarehouseRouting + +Optimize the pick sequence for an automated guided vehicle (AGV) in warehouse +aisles with obstacles, congestion zones, turn penalties, and route-length limits. + +## Files + +- `Task.md`: task contract and scoring details. +- `scripts/init.py`: editable seed routing heuristic. +- `verification/evaluator.py`: deterministic route simulator and scorer. +- `verification/requirements.txt`: evaluator dependencies. +- `frontier_eval/`: unified-task metadata. + +## Candidate Interface + +Edit `scripts/init.py` only. The evaluator imports: + +```python +plan_order(instance: dict) -> list[int] +``` + +The returned list must contain every pick id exactly once. The evaluator computes +the least-cost path between consecutive stops on the fixed grid and scores the +resulting total travel cost. + +## Quick Run + +From this directory: + +```bash +python verification/evaluator.py scripts/init.py +``` + +From the repository root: + +```bash +python -m frontier_eval \ + task=unified \ + task.benchmark=Robotics/AGVWarehouseRouting \ + algorithm=openevolve \ + algorithm.iterations=0 +``` + +No GPU, Docker, API key, or external assets are required. diff --git a/benchmarks/Robotics/AGVWarehouseRouting/Task.md b/benchmarks/Robotics/AGVWarehouseRouting/Task.md new file mode 100644 index 00000000..6600f39a --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/Task.md @@ -0,0 +1,49 @@ +# AGV Warehouse Routing + +## Background + +Warehouses often run fleets of automated guided vehicles to pick parts, bins, or +parcels. Even for one vehicle, the best visit order depends on aisle topology, +blocked cells, turn cost, congestion zones, and required return location. Better +ordering reduces cycle time and battery usage without changing warehouse layout. + +## Objective + +Given a fixed warehouse instance, return the order in which the AGV should visit +all required pick locations before ending at the outbound station. The evaluator +uses a deterministic shortest-path model between visits and sums travel, +congestion, and turn costs. + +## Candidate API + +The evaluator imports `plan_order(instance)` from `scripts/init.py`. + +`instance` contains: + +- `instance_id` +- `rows`, `cols` +- `grid`: list of strings, where `#` marks blocked cells +- `start`: `[row, col]` +- `goal`: `[row, col]` +- `picks`: list of `{id, row, col, priority}` +- `traffic`: list of `{row, col, extra_cost}` +- `turn_penalty` + +Return a list of pick ids. Every id must appear exactly once. + +## Constraints + +- Do not import external packages. +- Do not read or write files. +- Keep the public `plan_order(instance)` interface. +- Keep all editable logic inside the EVOLVE block. +- The policy must be deterministic for the same input instance. + +## Scoring + +The evaluator computes the least-cost grid path from start through the returned +pick sequence and then to goal. Cell traversal cost includes base travel cost, +traffic surcharge, and heading-change penalties. + +`combined_score = -mean(route_cost across instances)`, so higher is better. +Invalid permutations or unreachable routes receive an invalid score. diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/agent_files.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/agent_files.txt new file mode 100644 index 00000000..e60e972a --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/agent_files.txt @@ -0,0 +1,5 @@ +README.md +Task.md +scripts/init.py +verification/evaluator.py +frontier_eval/constraints.txt diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/artifact_files.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..76dc893a --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/artifact_files.txt @@ -0,0 +1,2 @@ +metrics.json +artifacts.json diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/candidate_destination.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/constraints.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/constraints.txt new file mode 100644 index 00000000..115db7d1 --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/constraints.txt @@ -0,0 +1,6 @@ +AGVWarehouseRouting constraints: +1) Modify only scripts/init.py. +2) Keep plan_order(instance) as the public entrypoint. +3) Return a permutation containing every pick id exactly once. +4) Do not use external packages, file I/O, network access, or randomness. +5) Keep EVOLVE-BLOCK markers intact and place editable routing logic inside them. diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/copy_files.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/copy_files.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/copy_files.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/eval_command.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/eval_command.txt new file mode 100644 index 00000000..527d23b5 --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +{python} verification/evaluator.py {candidate} --metrics-out metrics.json --artifacts-out artifacts.json diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/eval_cwd.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/eval_cwd.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/eval_cwd.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/initial_program.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/initial_program.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/readonly_files.txt b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..c3f5eb23 --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/frontier_eval/readonly_files.txt @@ -0,0 +1,5 @@ +README.md +Task.md +verification/evaluator.py +verification/requirements.txt +frontier_eval/ diff --git a/benchmarks/Robotics/AGVWarehouseRouting/scripts/init.py b/benchmarks/Robotics/AGVWarehouseRouting/scripts/init.py new file mode 100644 index 00000000..48489bb6 --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/scripts/init.py @@ -0,0 +1,51 @@ +from __future__ import annotations + +from typing import Any + + +def _manhattan(a: tuple[int, int], b: tuple[int, int]) -> int: + return abs(a[0] - b[0]) + abs(a[1] - b[1]) + + +def plan_order(instance: dict[str, Any]) -> list[int]: + """Return a permutation of pick ids for the AGV to visit.""" + + # EVOLVE-BLOCK-START + current = tuple(int(x) for x in instance["start"]) + remaining = [ + { + "id": int(item["id"]), + "pos": (int(item["row"]), int(item["col"])), + "priority": float(item.get("priority", 1.0)), + } + for item in instance["picks"] + ] + goal = tuple(int(x) for x in instance["goal"]) + order: list[int] = [] + + while remaining: + def score(item: dict[str, Any]) -> float: + dist = _manhattan(current, item["pos"]) + finish_bias = 0.20 * _manhattan(item["pos"], goal) + priority_bonus = 2.0 * item["priority"] + return dist + finish_bias - priority_bonus + + best = min(remaining, key=score) + order.append(int(best["id"])) + current = best["pos"] + remaining = [item for item in remaining if int(item["id"]) != int(best["id"])] + + return order + # EVOLVE-BLOCK-END + + +if __name__ == "__main__": + demo = { + "start": [0, 0], + "goal": [0, 4], + "picks": [ + {"id": 0, "row": 2, "col": 1, "priority": 1.0}, + {"id": 1, "row": 3, "col": 3, "priority": 1.0}, + ], + } + print(plan_order(demo)) diff --git a/benchmarks/Robotics/AGVWarehouseRouting/verification/evaluator.py b/benchmarks/Robotics/AGVWarehouseRouting/verification/evaluator.py new file mode 100644 index 00000000..4e1df617 --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/verification/evaluator.py @@ -0,0 +1,296 @@ +from __future__ import annotations + +import argparse +import heapq +import importlib.util +import json +import math +import statistics +import sys +import time +import traceback +from pathlib import Path +from types import ModuleType +from typing import Any + + +INVALID_COMBINED_SCORE = -1e18 +DIRS = [(-1, 0), (0, 1), (1, 0), (0, -1)] + + +def _write_json(path: str | None, payload: dict[str, Any]) -> None: + if not path: + return + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") + + +def _load_candidate(candidate_path: Path) -> ModuleType: + spec = importlib.util.spec_from_file_location("agv_candidate", candidate_path) + if spec is None or spec.loader is None: + raise ImportError(f"failed to load candidate module from {candidate_path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _traffic_map(entries: list[dict[str, Any]]) -> dict[tuple[int, int], float]: + return { + (int(item["row"]), int(item["col"])): float(item["extra_cost"]) + for item in entries + } + + +def _instances() -> list[dict[str, Any]]: + return [ + { + "instance_id": "narrow_aisles_morning", + "grid": [ + "..............", + ".####..####...", + "..............", + "...####..####.", + "..............", + ".####..####...", + "..............", + ], + "start": [0, 0], + "goal": [6, 13], + "picks": [ + {"id": 0, "row": 2, "col": 2, "priority": 1.0}, + {"id": 1, "row": 0, "col": 9, "priority": 1.3}, + {"id": 2, "row": 4, "col": 5, "priority": 0.8}, + {"id": 3, "row": 6, "col": 10, "priority": 1.1}, + {"id": 4, "row": 2, "col": 12, "priority": 0.9}, + ], + "traffic": [ + {"row": 2, "col": 6, "extra_cost": 2.2}, + {"row": 4, "col": 7, "extra_cost": 1.8}, + {"row": 6, "col": 8, "extra_cost": 1.4}, + ], + "turn_penalty": 0.35, + }, + { + "instance_id": "crossdock_afternoon", + "grid": [ + "................", + "..####....####..", + "................", + ".##..######..##.", + "................", + "..####....####..", + "................", + "....###..###....", + "................", + ], + "start": [8, 0], + "goal": [0, 15], + "picks": [ + {"id": 0, "row": 6, "col": 3, "priority": 1.2}, + {"id": 1, "row": 2, "col": 1, "priority": 1.0}, + {"id": 2, "row": 0, "col": 6, "priority": 1.4}, + {"id": 3, "row": 4, "col": 14, "priority": 0.7}, + {"id": 4, "row": 8, "col": 11, "priority": 1.1}, + {"id": 5, "row": 2, "col": 13, "priority": 0.9}, + ], + "traffic": [ + {"row": 4, "col": 7, "extra_cost": 2.8}, + {"row": 4, "col": 8, "extra_cost": 2.8}, + {"row": 2, "col": 8, "extra_cost": 1.5}, + {"row": 6, "col": 8, "extra_cost": 1.5}, + ], + "turn_penalty": 0.42, + }, + { + "instance_id": "returns_lane_congestion", + "grid": [ + ".............", + ".###.###.###.", + ".............", + ".###.....###.", + ".............", + ".###.....###.", + ".............", + ".###.###.###.", + ".............", + ], + "start": [4, 0], + "goal": [4, 12], + "picks": [ + {"id": 0, "row": 0, "col": 3, "priority": 0.9}, + {"id": 1, "row": 2, "col": 8, "priority": 1.2}, + {"id": 2, "row": 6, "col": 2, "priority": 0.8}, + {"id": 3, "row": 8, "col": 9, "priority": 1.1}, + {"id": 4, "row": 3, "col": 6, "priority": 1.5}, + {"id": 5, "row": 5, "col": 8, "priority": 1.0}, + ], + "traffic": [ + {"row": 4, "col": 5, "extra_cost": 3.0}, + {"row": 4, "col": 6, "extra_cost": 3.0}, + {"row": 4, "col": 7, "extra_cost": 3.0}, + {"row": 2, "col": 6, "extra_cost": 1.2}, + {"row": 6, "col": 6, "extra_cost": 1.2}, + ], + "turn_penalty": 0.38, + }, + ] + + +def _public_instance(instance: dict[str, Any]) -> dict[str, Any]: + grid = list(instance["grid"]) + return { + "instance_id": instance["instance_id"], + "rows": len(grid), + "cols": len(grid[0]), + "grid": grid, + "start": list(instance["start"]), + "goal": list(instance["goal"]), + "picks": [dict(item) for item in instance["picks"]], + "traffic": [dict(item) for item in instance["traffic"]], + "turn_penalty": float(instance["turn_penalty"]), + } + + +def _shortest_cost( + grid: list[str], + traffic: dict[tuple[int, int], float], + start: tuple[int, int], + goal: tuple[int, int], + turn_penalty: float, +) -> float: + rows = len(grid) + cols = len(grid[0]) + pq: list[tuple[float, int, int, int]] = [] + best: dict[tuple[int, int, int], float] = {} + for heading in range(4): + state = (start[0], start[1], heading) + best[state] = 0.0 + heapq.heappush(pq, (0.0, start[0], start[1], heading)) + + while pq: + cost, row, col, heading = heapq.heappop(pq) + if cost > best.get((row, col, heading), math.inf) + 1e-12: + continue + if (row, col) == goal: + return cost + for next_heading, (dr, dc) in enumerate(DIRS): + nr, nc = row + dr, col + dc + if nr < 0 or nr >= rows or nc < 0 or nc >= cols: + continue + if grid[nr][nc] == "#": + continue + turn_cost = 0.0 if next_heading == heading else turn_penalty + step_cost = 1.0 + traffic.get((nr, nc), 0.0) + turn_cost + new_cost = cost + step_cost + key = (nr, nc, next_heading) + if new_cost + 1e-12 < best.get(key, math.inf): + best[key] = new_cost + heapq.heappush(pq, (new_cost, nr, nc, next_heading)) + return math.inf + + +def _coerce_order(raw: Any) -> list[int]: + if isinstance(raw, dict): + raw = raw.get("order") + if not isinstance(raw, (list, tuple)): + raise ValueError("plan_order must return a list of pick ids") + order: list[int] = [] + for item in raw: + if isinstance(item, bool): + raise ValueError("pick ids must be integers, not booleans") + order.append(int(item)) + return order + + +def _score_instance(plan_order: Any, instance: dict[str, Any]) -> dict[str, Any]: + public = _public_instance(instance) + order = _coerce_order(plan_order(public)) + expected = {int(item["id"]) for item in instance["picks"]} + if set(order) != expected or len(order) != len(expected): + raise ValueError( + f"invalid pick permutation for {instance['instance_id']}: expected {sorted(expected)}, got {order}" + ) + + pick_by_id = {int(item["id"]): (int(item["row"]), int(item["col"])) for item in instance["picks"]} + stops = [tuple(instance["start"])] + [pick_by_id[item] for item in order] + [tuple(instance["goal"])] + traffic = _traffic_map(instance["traffic"]) + total_cost = 0.0 + leg_costs: list[float] = [] + for src, dst in zip(stops, stops[1:]): + leg = _shortest_cost( + list(instance["grid"]), + traffic, + (int(src[0]), int(src[1])), + (int(dst[0]), int(dst[1])), + float(instance["turn_penalty"]), + ) + if not math.isfinite(leg): + raise ValueError(f"unreachable leg {src}->{dst} in {instance['instance_id']}") + leg_costs.append(leg) + total_cost += leg + + priority_weighted_lateness = 0.0 + for index, pick_id in enumerate(order): + priority = next(float(item.get("priority", 1.0)) for item in instance["picks"] if int(item["id"]) == pick_id) + priority_weighted_lateness += priority * index + total_cost += 0.15 * priority_weighted_lateness + return { + "instance_id": instance["instance_id"], + "order": order, + "route_cost": total_cost, + "leg_costs": leg_costs, + "priority_weighted_lateness": priority_weighted_lateness, + } + + +def evaluate(candidate_path: str) -> tuple[dict[str, float], dict[str, Any]]: + module = _load_candidate(Path(candidate_path).resolve()) + plan_order = getattr(module, "plan_order", None) + if plan_order is None: + raise AttributeError("candidate must define plan_order(instance)") + case_results = [_score_instance(plan_order, instance) for instance in _instances()] + mean_cost = statistics.fmean(float(item["route_cost"]) for item in case_results) + metrics = { + "combined_score": -float(mean_cost), + "valid": 1.0, + "mean_route_cost": float(mean_cost), + "max_route_cost": max(float(item["route_cost"]) for item in case_results), + } + artifacts = { + "case_results": case_results, + "score_direction": "higher_is_better", + } + return metrics, artifacts + + +def main(argv: list[str]) -> int: + parser = argparse.ArgumentParser() + parser.add_argument("candidate") + parser.add_argument("--metrics-out", default="metrics.json") + parser.add_argument("--artifacts-out", default="artifacts.json") + args = parser.parse_args(argv) + + start = time.time() + try: + metrics, artifacts = evaluate(args.candidate) + except Exception as exc: + metrics = { + "combined_score": INVALID_COMBINED_SCORE, + "valid": 0.0, + "runtime_s": float(time.time() - start), + } + artifacts = { + "error_message": str(exc), + "traceback": traceback.format_exc(), + } + metrics["runtime_s"] = float(time.time() - start) + artifacts["candidate_path"] = str(Path(args.candidate).resolve()) + _write_json(args.metrics_out, metrics) + _write_json(args.artifacts_out, artifacts) + print(json.dumps(metrics, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/benchmarks/Robotics/AGVWarehouseRouting/verification/requirements.txt b/benchmarks/Robotics/AGVWarehouseRouting/verification/requirements.txt new file mode 100644 index 00000000..821e76fa --- /dev/null +++ b/benchmarks/Robotics/AGVWarehouseRouting/verification/requirements.txt @@ -0,0 +1 @@ +# Python standard library only. diff --git a/benchmarks/Robotics/README.md b/benchmarks/Robotics/README.md index b3e834a5..28e1eab1 100644 --- a/benchmarks/Robotics/README.md +++ b/benchmarks/Robotics/README.md @@ -7,6 +7,7 @@ This domain contains robotics control and planning tasks for unified evaluation. - `CoFlyersVasarhelyiTuning` - Unified benchmark: `task=coflyers_vasarhelyi_tuning` - Quick run: `python -m frontier_eval task=coflyers_vasarhelyi_tuning algorithm.iterations=0` +- `AGVWarehouseRouting` - `DynamicObstacleAvoidanceNavigation` - `PIDTuning` - `QuadrupedGaitOptimization` @@ -15,6 +16,7 @@ This domain contains robotics control and planning tasks for unified evaluation. ### Unified quick runs +- `AGVWarehouseRouting`: `python -m frontier_eval task=unified task.benchmark=Robotics/AGVWarehouseRouting algorithm.iterations=0` - `DynamicObstacleAvoidanceNavigation`: `python -m frontier_eval task=unified task.benchmark=Robotics/DynamicObstacleAvoidanceNavigation algorithm.iterations=0` - `PIDTuning`: `python -m frontier_eval task=unified task.benchmark=Robotics/PIDTuning algorithm.iterations=0` - `QuadrupedGaitOptimization`: `.venvs/frontier-eval-driver/bin/python -m frontier_eval task=unified task.benchmark=Robotics/QuadrupedGaitOptimization task.runtime.env_name=frontier-v1-main algorithm.iterations=0`