Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,46 @@ jobs:
python -m build
twine check dist/*
python scripts/verify_publishable_dist.py dist

gepa-release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: pnpm/action-setup@v4

- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm

- uses: actions/setup-python@v5
with:
python-version: '3.12'

- uses: astral-sh/setup-uv@v6
with:
enable-cache: true
cache-dependency-glob: clients/python/uv.lock

- run: pnpm install --frozen-lockfile

- name: Install published GEPA
working-directory: clients/python
run: uv sync --frozen --extra dev --group gepa-release

- name: Test published GEPA compatibility
working-directory: clients/python
env:
AGENT_EVAL_EXPECT_GEPA_RELEASE: '1'
run: >-
uv run --frozen --extra dev --group gepa-release
pytest -v
tests/test_gepa_release_compatibility.py
tests/test_gepa_bridge.py
tests/test_official_optimizer_compatibility.py::test_agent_eval_gepa_bridge_resumes_state_from_a_real_prior_run

- name: Test TypeScript bridge with published GEPA
env:
AGENT_EVAL_TEST_PYTHON: ${{ github.workspace }}/clients/python/.venv/bin/python
run: pnpm exec vitest run tests/campaign/gepa-official-integration.test.ts
20 changes: 20 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,26 @@ jobs:
tests/campaign/gepa-official-integration.test.ts
tests/campaign/skillopt-official-integration.test.ts

- name: Install published GEPA
working-directory: clients/python
run: uv sync --frozen --extra dev --group gepa-release

- name: Test published GEPA compatibility
working-directory: clients/python
env:
AGENT_EVAL_EXPECT_GEPA_RELEASE: '1'
run: >-
uv run --frozen --extra dev --group gepa-release
pytest -v
tests/test_gepa_release_compatibility.py
tests/test_gepa_bridge.py
tests/test_official_optimizer_compatibility.py::test_agent_eval_gepa_bridge_resumes_state_from_a_real_prior_run

- name: Test TypeScript bridge with published GEPA
env:
AGENT_EVAL_TEST_PYTHON: ${{ github.workspace }}/clients/python/.venv/bin/python
run: pnpm exec vitest run tests/campaign/gepa-official-integration.test.ts

- name: Install DSPy environment
working-directory: clients/python
run: uv sync --frozen --extra dev --extra dspy
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-

---

## [0.126.5] - 2026-07-24 - published GEPA compatibility

### Fixed

- The standard GEPA engine now runs against the published `gepa[full]==0.1.4` package instead of requiring an unreleased source API.
- GEPA source-only engines and composition functions still fail explicitly unless the documented official source revision is installed.

## [0.126.4] - 2026-07-24 - train-only GEPA optimization

### Fixed
Expand Down
18 changes: 14 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,21 @@ python -m pip install agent-eval-rpc
python -m pip install \
"skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90"

# GEPA Optimize Anything at the source revision tested by this release
# Standard GEPA engine from the published package
python -m pip install agent-eval-rpc
python -m pip install "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f"
python -m pip install "gepa[full]==0.1.4"

# GEPA Omni and source-only engines
python -m pip install \
"gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f"

# DSPy 3.2.1 with Agent Eval metrics
python -m pip install "agent-eval-rpc[dspy]"
```

The published `gepa==0.1.4` wheel does not contain the Optimize Anything API used here.
The Git revision is intentional and should move only after compatibility tests pass.
The published GEPA package supports the standard `gepa` engine.
Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N currently require the tested official source revision.
Move that revision only after both release and source compatibility tests pass.
The published `skillopt==0.2.0` wheel omits the prompt files required by `ReflACTTrainer`, so the tested SkillOpt source revision is also intentional.
DSPy 3.2.1 requires GEPA 0.0.27, while the general bridge requires GEPA 0.1.4.
Install the DSPy adapter and the general GEPA bridge in separate Python environments.
Expand Down Expand Up @@ -373,6 +378,11 @@ Python compatibility tests use the locked dependencies:

```sh
cd clients/python
uv sync --frozen --extra dev --group gepa-release
AGENT_EVAL_EXPECT_GEPA_RELEASE=1 \
uv run --frozen --extra dev --group gepa-release \
pytest tests/test_gepa_release_compatibility.py tests/test_gepa_bridge.py

uv sync --frozen --extra dev --group skillopt-source --group gepa-source
uv run --frozen pytest

Expand Down
20 changes: 15 additions & 5 deletions clients/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,20 +146,25 @@ The Python client does not reimplement either algorithm.

### GEPA

Install the client and the exact GEPA source revision tested by Agent Eval:
Install the client and published GEPA package for the standard engine:

```sh
python -m pip install agent-eval-rpc
python -m pip install "gepa[full]==0.1.4"
```

The published package runs direct recipes with the standard `gepa` engine.
Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N currently require this tested official source revision:

```sh
python -m pip install \
"gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f"
```

The published `gepa==0.1.4` wheel does not contain the Optimize Anything API used by `gepaOptimizationMethod()`.
PyPI package metadata cannot depend directly on a Git URL, so GEPA is a separate install.

From an Agent Eval source checkout, install the same revision with:
From an Agent Eval source checkout:

```sh
uv sync --frozen --group gepa-release
uv sync --frozen --group gepa-source
```

Expand Down Expand Up @@ -245,6 +250,11 @@ DSPy 3.2.1 pins GEPA 0.0.27.
The general Optimize Anything bridge uses GEPA 0.1.4, so repository checks install them in separate environments:

```sh
uv sync --frozen --extra dev --group gepa-release
AGENT_EVAL_EXPECT_GEPA_RELEASE=1 \
uv run --frozen --extra dev --group gepa-release \
pytest tests/test_gepa_release_compatibility.py tests/test_gepa_bridge.py

uv sync --frozen --extra dev --group skillopt-source --group gepa-source
uv run --frozen pytest

Expand Down
13 changes: 12 additions & 1 deletion clients/python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "agent-eval-rpc"
version = "0.126.4"
version = "0.126.5"
description = "Python RPC client, official optimizer bridge, and DSPy metric adapter for @tangle-network/agent-eval."
readme = "README.md"
requires-python = ">=3.10"
Expand Down Expand Up @@ -45,6 +45,9 @@ skillopt-source = [
gepa-source = [
"gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f",
]
gepa-release = [
"gepa[full]==0.1.4",
]

[project.urls]
Homepage = "https://github.com/tangle-network/agent-eval"
Expand All @@ -63,6 +66,14 @@ conflicts = [
{ extra = "dspy" },
{ group = "gepa-source" },
],
[
{ extra = "dspy" },
{ group = "gepa-release" },
],
[
{ group = "gepa-source" },
{ group = "gepa-release" },
],
]

[tool.ruff]
Expand Down
2 changes: 1 addition & 1 deletion clients/python/src/agent_eval_rpc/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
try:
__version__ = version("agent-eval-rpc")
except PackageNotFoundError:
__version__ = "0.126.4"
__version__ = "0.126.5"

__all__ = [
"Client",
Expand Down
56 changes: 56 additions & 0 deletions clients/python/src/agent_eval_rpc/gepa_api.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""Resolve the installed GEPA Optimize Anything API without replacing its search."""

from __future__ import annotations

import importlib
from dataclasses import dataclass
from types import ModuleType
from typing import Any, Literal


@dataclass(frozen=True)
class GepaApi:
module: ModuleType
config_class: Any
config_shape: Literal["engine", "launcher"]

@property
def optimize_anything(self) -> Any:
return self.module.optimize_anything

def composition(self, name: str) -> Any | None:
return getattr(self.module, name, None)


def load_gepa_api() -> GepaApi:
try:
module = importlib.import_module("gepa.optimize_anything")
except ImportError as error:
raise RuntimeError(
"GEPA bridge requires the official gepa package. "
'Install "gepa[full]==0.1.4" or the documented source revision.'
) from error

optimize_anything = getattr(module, "optimize_anything", None)
if not callable(optimize_anything):
raise RuntimeError("Installed GEPA package does not expose optimize_anything()")

engine_config = getattr(module, "OptimizeAnythingConfig", None)
if engine_config is not None:
return GepaApi(
module=module,
config_class=engine_config,
config_shape="engine",
)

launcher_config = getattr(module, "GEPAConfig", None)
if launcher_config is not None:
return GepaApi(
module=module,
config_class=launcher_config,
config_shape="launcher",
)

raise RuntimeError(
"Installed GEPA package exposes neither OptimizeAnythingConfig nor GEPAConfig"
)
Loading
Loading