Conversation
…rics, and persistence
- MetricResult contract codified: raw score in metric-native units, normalized_score always [0,1] higher-is-better, errors carry no meaningful score, version must match definition - MetricDefinition gains ScoreDirection + default_threshold - MetricResult.passed_against() threshold semantics (inclusive) - All embedding-metric failure paths now set explicit error instead of silent zero scores marked as success - token_usage/cost/latency: missing measurement metadata is an explicit error, no more fabricated perfect scores - tool_call_correctness: absent tool_calls data is an explicit error instead of a vacuous 1.0 pass - embedding metrics record embedding_model in execution metadata - new test_score_contract.py enforces the contract across all 22 registered metrics with deterministic fakes
…rovider - new OpenAIEmbeddingAdapter mapping SDK embedding responses through the existing map_embedding_response mapper - OpenAIProvider now implements the EmbeddingProvider contract and declares EMBEDDING / EMBEDDING_DIMENSIONS capabilities - adapter honors EmbeddingOptions (dimensions, encoding_format) - tests cover SDK response mapping, options passthrough, error propagation, capability declaration, and provider delegation
- ExecutionMetadata gains embedding_provider/embedding_model config - MetricDispatchStage resolves the embedding provider through the ProviderRegistry (explicit name, or fallback to an embedding- capable chat provider); unregistered names degrade to per-metric errors instead of crashing dispatch - Temporal activities inject the embedding-capable provider for embedding-backed metrics - run metadata serialization now round-trips judge and embedding configuration (judge fields were previously dropped on reload) - wiring tests prove registry resolution, fallback, degradation, and real metric execution over the injected boundary
- JudgeResponse gains explicit error field; provider failures and malformed judge output now surface as errors instead of silent zero scores with fabricated confidence - strict structured-output validation: JSON object with numeric score/confidence required; regex score-extraction fallbacks removed entirely (they invented scores from prose) - out-of-range values clamped and documented, raw output preserved for debugging on parse failures - judge token usage cost-accounted through the default cost calculator (unknown pricing yields 0.0 without failing the call) - llm_judge_base maps judge errors to explicit metric error results while keeping raw output, tokens, and rubric/prompt versions - new judge hardening test suite; existing metric tests updated to return structured verdicts
- shared deterministic fakes: scripted embedding vectors and judge verdicts at the provider boundary - embedding metrics proven against exact trigonometry (identical -> 1.0, orthogonal -> 0.0, 45 degrees -> 1/sqrt(2)); previous tests were vacuous (both texts embedded to the same mock vector) - all 9 judge metrics parametrized: verdict propagation with confidence/model/tokens metadata, malformed verdict -> explicit error, provider failure -> explicit error - deterministic metrics pinned to exact formula outputs (token ratios, latency-at-threshold = 0.0, cost-at-cap = 0.0, length triangle profile) - JudgeEngine now records the model actually used by the provider when no explicit judge model is configured
- judge results record executing provider name (configured name or the resolved provider's identity) - embedding results record embedding_provider alongside embedding_model - score-contract tests now enforce provenance metadata for all judge and embedding metrics
correct, incorrect, irrelevant, hallucinated, valid-structured, invalid-structured, context-grounded, ungrounded — all synthetic, version-controlled inputs with invariant tests
- run all 22 registered metrics over the eight canonical fixtures via ProviderInvocation -> MetricDispatch -> Aggregation -> Persistence, fakes only at the chat/embedding provider boundary - fix measurement metrics rejecting string metadata produced by the real invocation stage (tokens/cost/latency now coerce numeric strings, still error on garbage) - fix judge model provenance collapsing to 'None' when judge config carries no explicit model - schema_validation accepts JSON-string schemas from pipeline metadata - MetricEngine stamps run_id/item_id onto results for attribution
'relevance' computed the identical prompt-response cosine similarity as 'answer_relevance' under a second public name, doubling reporting surface with no semantic difference. Removed honestly rather than renamed: use 'answer_relevance'.
Measured with a counting provider: evaluating the four embedding metrics on one item previously issued 8 embed round-trips for 4 unique texts. A per-MetricInput cache in EmbeddingMetric cuts this to 4 (50% fewer provider calls) and ~2x wall-clock improvement at realistic latency, with scores bit-identical to the uncached path.
…er (B.4) - Deterministic evaluation fingerprinting (SHA-256) - Environment capture for reproducibility (git hash, requirements) - Provider call accounting (target/judge/embedding breakdown) - Cost/latency/token tracking with type-safe accounting - Model comparison with metric-level deltas - Golden evaluation suite with deterministic expectations - Metric invariant tests (NaN, infinity, range, directionality) - Benchmark runner with regression detection - Replay trace serialization consistency tests - Embedding call reuse regression protection - 238 new tests, all passing
- Add domain contracts: AdaptiveCampaign aggregate, CampaignRound, TargetExecution, AttackEffectiveness, CampaignBudget, AttackLineage, CampaignResult, CampaignState enum, MutationPhase enum - Add TargetExecutor: routes attack prompts to providers via ProviderRegistry → ChatProvider.chat() - Add AttackEvaluator: combines MetricEngine + safety scoring for effectiveness evaluation - Add MutationStrategySelector: adaptive strategy selection with exploration/exploitation/adaptive phases - Add AdaptiveCampaignEngine: orchestrates the full campaign loop (Attack → Target Execution → Response → Effectiveness Evaluation → Mutation/Strategy Selection → Next Attack → Campaign Result) - Add 48 deterministic integration tests covering all components - All tests pass (1766 passed, 558 skipped), ruff clean, mypy clean
… metrics, evaluator, integration tests B.6: Agent / Trajectory Evaluation — first working implementation. New files: - app/agents/domain/trajectory.py: AgentTrajectory, TrajectoryStep, ToolCallRecord, LLMCallRecord, TrajectoryMetrics, compute_trajectory_metrics - app/agents/domain/tool_execution.py: ToolDefinition, ToolRegistry, SafeToolExecutor - app/agents/runtime/trajectory_recorder.py: TrajectoryRecorder for building trajectories during execution - app/agents/runtime/agent_loop.py: AgentLoop — LLM ↔ tool interaction loop with provider integration - app/evaluation/metrics/trajectories/: TrajectoryCompletenessMetric, TrajectoryEfficiencyMetric, TrajectoryErrorRecoveryMetric, TrajectoryToolSelectionMetric - app/evaluation/metrics/trajectories/evaluator.py: TrajectoryEvaluator orchestrating trajectory evaluation via MetricEngine - tests/agents/trajectory/test_trajectory_evaluation.py: 38 deterministic integration tests proving the full path Proven path: AGENT TASK → AGENT EXECUTION → MODEL ACTION → TOOL CALL → TOOL RESULT → NEXT ACTION → FINAL RESPONSE → COMPLETE TRAJECTORY → TRAJECTORY METRICS → PERSISTED EVALUATION RESULT Test results: 1804 passed, 558 skipped (async), 0 failures. Ruff clean, mypy clean.
- Replaced stubbed _invoke_step with execute_run_sync using AgentLoop - Added execute_sync wrapper to AgentLoop for synchronous callers - Updated AgentRuntimeCoordinator to delegate to AgentLoop in one call - Updated tests to match new AgentExecutor signature B.7 Audit Fix: Path C now has real LLM execution instead of hardcoded stubs
- Added execute_agent_loop_activity that runs the full LLM ↔ tool loop - Added configure_agent_provider_registry for worker startup - Replaced step-by-step lifecycle loop with single activity call - Added ExecuteAgentLoopInput/Result dataclasses B.7 Audit Fix: Path C Temporal workflow now executes real AI, not just lifecycle updates
…zation - conversation_history is tuple[dict], not str — serialize to role:content format - Fixed default value from [-1] to empty tuple - MetricInput.response now receives proper string representation B.7 Audit Fix: trajectory evaluator no longer passes raw list as response string
- Removed app/providers/contracts/audio.py (no implementations) - Removed app/providers/contracts/vision.py (no implementations) - Updated contracts __init__.py to remove stale exports B.7 Audit: dead code cleanup, no production references
- Resource was created after first provider was already set - Now create provider with resource from the start - Single trace.set_tracer_provider call instead of two B.7 Audit Fix: OpenTelemetry no longer silently overwrites its own provider
- Added PrometheusMiddleware that records REQUEST_COUNT and REQUEST_LATENCY - Metrics were registered but never incremented — now they produce data - /metrics endpoint continues to work as before B.7 Audit Fix: Prometheus metrics are now reachable and operational
- agent_trajectory.py: removed index=True from run_id (duplicate of explicit Index) - attack_run.py: removed index=True from status (duplicate of explicit Index)
Deleted: - tests/providers/runtime/test_cache.py - tests/providers/runtime/test_context.py - tests/providers/runtime/test_health.py - tests/providers/selection/ (entire directory) Test regression: 1772 passed, 524 skipped, 0 failures (135 tests removed - all tested dead code)
- Add GET /replay/regression/{baseline_run_id}/{current_run_id} endpoint
- Add MetricRegressionResponse and RegressionResultResponse schemas
- Add replay/comparison/regression API methods to frontend client
- Add TypeScript types for replay, comparison, and regression data
- Add verdict column to run list with color-coded badges - Show verdict prominently in run details header - Connect cancel button with real API call - Connect retry button with real API call - Add compare button for completed runs
- Create ReplayViewer component showing item execution traces - Display prompt, provider response, metric scores per item - Show timing, cost, and error information - Integrate into run details page for completed runs
- Create /runs/compare page for side-by-side run comparison - Display provider, model, winner, and confidence - Show per-metric deltas with baseline vs comparison scores - Support URL params for pre-filling run IDs
- Create RegressionResultView component for regression analysis - Show verdict, regression count, error count, fingerprint compatibility - Display per-metric analysis with status, scores, and reasoning - Integrate into comparison page
- Remove hardcoded mock safety scores and attack scenarios - Wire cancel button to real api.cancelAttackRun() with invalidation - Add clickable links to attack definition IDs - Add evaluation run link when present - Add back-to-runs navigation - Add status color for failed state
- Remove unused useCallback and Play imports from run details - Remove unused Button import from comparison page - Type compareRuns query with TraceComparison return type
- Fix compute_fingerprint call in api/replay.py to use keyword args - Fix analyze_regression call to pass .fingerprint strings not objects - Fix redis_repository.py type stubs for sadd/srem/smembers - Fix attack_evaluator.py dict type parameter - Fix agents/router.py total_steps -> max_steps - Fix cli/__init__.py: rewrite _run_compare to use async session factory, fix metrics dict type annotation, compute fingerprint from config
…ory files - Fix import sorting in test_campaign_engine.py - Remove unused imports in test_kernel.py, test_campaign_engine.py - Fix loop variable and .items() usage in test_campaign_engine.py - Reformat trajectory_tool_selection.py, test_trajectory_evaluation.py
- Format dashboard page, redteam run details, compare page - Format replay viewer, regression result components - Format API TypeScript types
- Format agent workflow, trajectory recorder, trajectory domain - Format trajectory evaluator and metric implementations - Format migration and results files
Register finalize_run_integrity_activity and execute_agent_loop_activity in the ActivityRegistry. These activities were defined and called by their respective workflows but never registered, causing runtime failures when the worker tried to execute them.
…or Temporal activities The agent Temporal activities (execute_agent_loop_activity and lifecycle activities) require a configured session factory and provider registry. These were never wired during bootstrap, causing RuntimeError at runtime when the worker tried to execute agent workflows.
…ties Add bounded retry policies (max 3 attempts) to all Temporal activities with exponential backoff. Activities are categorized by failure mode: - Lifecycle activities (queue/start/complete/fail/cancel): idempotent state transitions, retry for transient DB errors. - Item execution: retry for transient provider failures (rate limits, network), activity has internal idempotency checks. - Agent loop: bounded to 2 attempts (long-running), retry only for transient infrastructure errors. - Persistence/integrity: retry for transient DB and I/O errors. Non-retryable error types (ValueError, KeyError) prevent infinite retries on invalid input.
…liability 16 tests covering: - P0-1: finalize_run_integrity_activity registration in ActivityRegistry - P0-2: execute_agent_loop_activity registration in ActivityRegistry - P1-1: Agent session factory and provider registry configuration - P1-2: Explicit retry policies in evaluation and agent workflows - Container registration verification (all 18 activities registered)
P0-3: The shared API client request() function never read the access
token from localStorage. All authenticated API calls were sent without
Authorization headers, causing 401 responses from the backend.
- Read token from localStorage key 'redops-access-token'
- Add 'Authorization: Bearer <token>' header when token exists
- Omit header when token is absent (preserves public endpoint behavior)
P0-4: SSE stream URLs stripped /api/v1 from BASE_URL via .replace(),
producing URLs like /runs/{id}/events/stream instead of
/api/v1/runs/{id}/events/stream. The backend mounts all routes under
/api/v1, so SSE connections hit 404.
- Use BASE_URL directly for SSE URLs (includes /api/v1)
- Replace native EventSource with fetch-based AuthenticatedEventSource
to support Authorization headers (native EventSource cannot set
custom headers)
- AuthenticatedEventSource implements EventSource interface (onmessage,
onerror, close) for drop-in compatibility with existing consumers
11 tests covering: - Authorization Bearer header sent when token exists in localStorage - Authorization header omitted when token is absent - No fabricated Authorization header for missing token - SSE streamEvents URL contains /api/v1 prefix - SSE streamProgress URL contains /api/v1 prefix - SSE connection includes Authorization header via fetch-based reader - SSE connection omits Authorization when token missing
Resolve merge conflicts from origin/main revert (cee7dfd) while preserving all B.1-B.12.2 develop work. Conflicts resolved by keeping develop versions: - backend/app/agents/api/router.py (config-based task_queue) - backend/app/api/evaluation_run.py (config-based task_queue) - backend/app/infrastructure/composition/container.py (full activity registry) - backend/app/infrastructure/observability/prometheus.py (middleware imports) - backend/tests/integration/test_temporal_queue_alignment.py (existing tests) Restored from develop (deleted by main revert): - opencode.md - backend/tests/integration/test_provider_registration.py - backend/tests/integration/test_provider_resolution.py Reverted harmful auto-merged changes: - backend/app/core/config.py (keep provider credential fields) - backend/app/infrastructure/composition/bootstrap.py (keep DatabaseEngine init) - backend/app/main.py (keep app = create_app() instance) - docker/docker-compose.yml (keep newer temporal-admin-tools tag) Accepted beneficial changes from main: - README.md and Dockerfile: uvicorn --factory pattern - Alembic migrations: added status index, normalized revision IDs - frontend/.gitignore: removed duplicate entries
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Remove 4 unused type: ignore suppressions: - redis_repository.py: sadd, srem, smembers (3x [misc]) - api/replay.py: aioredis.from_url (1x [no-untyped-call]) CI reports these as [unused-ignore] because current redis-py stubs properly type these async methods.
…mpare Extract search-params-consuming logic into CompareContent component and wrap it with React.Suspense in ComparePage. This satisfies Next.js App Router requirement that useSearchParams() must be inside a Suspense boundary. Preserves existing behavior: URL query parameters ?baseline=<id>&comparison=<id> continue to work.
…now() in evaluation workflow Temporal workflow sandbox rejects datetime.now() because workflow execution must be deterministic. Replace both started_at and completed_at timestamps with workflow.now() which provides the deterministic workflow time API. Also register finalize_run_integrity_activity in the integration test worker so the full workflow chain executes correctly.
CI has no .env file so APP_SECRET_KEY defaults to empty string, causing PyJWT to reject HMAC signing in test_create_access_token. Add a session-scoped autouse fixture that sets a test-only key and clears the cached AppConfig before any test runs.
|
🎉 This PR is included in version 0.2.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.