Skip to content

Add reflector instrumentation, pluggable payload mapper, v2 templates - #2

Draft
Teq2412 wants to merge 1 commit into
mainfrom
reflector-instrumentation-and-payload-mapper
Draft

Teq2412 wants to merge 1 commit into
mainfrom
reflector-instrumentation-and-payload-mapper

Conversation

@Teq2412

@Teq2412 Teq2412 commented May 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Three small, independent additions to the agentic optimization pipeline:

  • BaseAgenticOptimizer._invoke_agent — wraps the inner Strands agent(...) call to record wall-clock time and accumulated metrics (tokens, cycles, tool calls, latency) onto self.last_wall_clock_s / self.last_metrics. Lets callers read step cost/latency without subclassing. ContrastiveReflectionOptimizer._run_reflection routes through it.

  • AgentCoreRolloutEngine.payload_mapper — the engine still builds the canonical {"data_sample": ..., "params": ...} payload. New payload_mapper: Callable[[dict], dict] init kwarg is a transform applied to that canonical dict before the HTTP call, so deployments expecting a different shape (flat fields, renamed keys, nested envelopes) can remap without subclassing the engine. Default is None (identity), fully backward-compatible.

  • New built-in templates at harness_optimizer/templates/contrastive_reflection_v2/. Differences from v1:

    • Forbids the ## Learned Behaviors appendix; requires holistic integration of insights into the original prompt's existing structure.
    • Adds a non-negotiable safety invariant (require user confirmation before consequential actions; remove guidance that tells the agent to act autonomously without confirmation).
    • Adds a length guardrail (<= max(1.1 × original, original + 500 chars)).
    • Submission flow still uses the existing submit_optimized_params tool contract.

Test plan

  • pytest tests/rollout_engines/ tests/optimizers/ passes locally (excluding tests/optimizers/test_multi_agent.py, which is broken on main independently of this PR — it imports context_alchemy.datamodels).
  • Runtime smoke: load_builtin_template("contrastive_reflection_v2/...") loads + renders both new templates.
  • Runtime smoke: AgentCoreRolloutEngine(..., payload_mapper=fn) stores the mapper; default constructor leaves _payload_mapper = None.
  • Runtime smoke: ContrastiveReflectionOptimizer exposes last_metrics, last_wall_clock_s, _invoke_agent.
  • End-to-end reflector run on a real benchmark — verified separately on AppWorld (Sonnet 4 reflector produced a valid optimized prompt; metrics captured wall=224.7s, in=758644 / out=15092 tokens).

🤖 Generated with Claude Code

- BaseAgenticOptimizer: add `_invoke_agent` helper that times the agent
  invocation and captures Strands metrics (tokens, cycles, tool calls,
  latency) onto `last_wall_clock_s` / `last_metrics`. ContrastiveReflectionOptimizer
  routes through it so callers can read step cost without subclassing.

- AgentCoreRolloutEngine: the engine still builds the canonical
  `{"data_sample": ..., "params": ...}` payload. New `payload_mapper:
  Callable[[dict], dict]` init kwarg is a transform applied to that
  canonical payload before the HTTP call, so deployments expecting a
  different shape (flat fields, renamed keys, nested envelopes) can remap
  without subclassing the engine. Default is None (no transform), fully
  backward-compatible.

- New built-in templates `contrastive_reflection_v2/{system_prompt,task_message_system_prompt}.jinja`:
  forbids the "Learned Behaviors" appendix, requires holistic integration
  of insights, adds a safety invariant (require user confirmation before
  consequential actions), and a length guardrail (<= max(1.1x original,
  original + 500 chars)). Submission flow still uses the existing
  `submit_optimized_params` tool contract.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@Teq2412
Teq2412 requested a review from jnzs1836 May 29, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants