test(injection): Fase 0 prompt-injection harness — Day 1-5 (gates C2/C4 PASS, 113 tests) - #211
Open
SabaTech-dev wants to merge 26 commits into
Open
test(injection): Fase 0 prompt-injection harness — Day 1-5 (gates C2/C4 PASS, 113 tests)#211SabaTech-dev wants to merge 26 commits into
SabaTech-dev wants to merge 26 commits into
Conversation
added 26 commits
August 14, 2026 21:23
… nav Before: mkdocs nav published only 5 of 65 docs (94% invisible to beta users). This is sub-card 1/4 of the Diátaxis docs plan (report: workspace-research/reports/diataxis-docs-plan-2026-08-14.md). Structure (per report §3.1): - docs/tutorials/: new learning-path landing (3 beta tutorials are sub-card 2; not written here) - docs/how-to/ + how-to/deployment/: task-oriented guides (websocket, stripe webhooks, allure, db/perf/security testing, parallelization, monitoring, full deployment set incl. root QUICK_START/RAILWAY_SETUP/ DEPLOYMENT_GUIDE) - docs/reference/: api, api-reference (legacy), auth, env vars, notifications, ai-assistant, global-search - docs/explanation/: architecture, advanced-test-arch, cicd, graceful shutdown, cloud comparison, OWASP/CRA compliance, dast-pipeline, opencore guide, project study - docs/internal/: ~40 non-user docs excluded from the published site via exclude_docs (marketing, management/task boards, plans/, specs/, reports/, OPENCODE_* internals, railway diagnostics, compliance evidence) Nav rewritten: 40 pages across the 4 Diátaxis sections. mkdocs build --strict passes with zero warnings; internal/ verified absent from built site. All cross-links fixed (docs, README.md, .github comments); repo-root README links to old paths updated. Decisions where report classification was low-confidence (documented): - RAILWAY_ENV_VARS.md kept as user-facing reference/environment-variables.md (report lists it under Reference); only RAILWAY_DEPLOY_DIAGNOSTIC and RAILWAY_STATUS_FINAL moved to internal/ - compliance/: OWASP + CRA posture docs -> explanation/ (user-facing trust material); evidence artifacts (disclosure-log, tos-verification, gpai provider docs) -> internal/compliance/ - OPENCORE_RESOLUTION.md -> internal/management/ (decision log), while OPENCORE_OFFICIAL_GUIDE stays user-facing per report - SECURITY_TESTING.md, DATABASE_TESTING.md, PARALLELIZATION_GUIDE.md, dast-pipeline.md classified as how-to / explanation respectively - Duplicate docs NOT merged (api vs API_REFERENCE, getting-started vs quick-start, performance-testing vs performance-guide): consolidation is sub-card 3 - cra-compliance.md was an untracked but completed compliance assessment; committed here because the nav references it - Fixed pre-existing broken links: BILLING_SYSTEM.md (target never existed, de-linked), README CONTRIBUTING.md link, docs/api-guide.md
- curl binary does not exist in python:3.11-slim image -> false unhealthy since first boot (1787 failing streak) - healthcheck also pointed to /api/v1/health (404); correct endpoint is /health (200) - uses python stdlib urllib (always present); postgres/redis healthchecks already OK (pg_isready/redis-cli) - verified post-restart: docker inspect -> health=healthy, failing=0 Also commits pre-existing uncommitted working-tree changes that match the already-running deployment (no runtime change): backend port 8000->8010, frontend 3000->3010, postgres/redis published ports -> internal expose only.
…ronment - first-test-suite.md: corrected host ports (backend 8010, frontend 3010), username-based auth (no email verification), real schemas (suite without tags, test cases with test_code), real execution flow (POST /executions + /start), documented beta issues (start 500, GET executions 500 on completed, simulated runner) - first-api-test.md: added SSRF_ALLOWED_DOMAINS step (allowlist blocks jsonplaceholder without it), verified pip route + 6 passed output - first-e2e-test.md: documented 3 failing example tests (retired Wikipedia #searchButton, mock login selectors), fixed custom test selector to .cdx-search-input__end-button, added -k filter for green subset - index.md: marked tutorials as verified, beta limitations flagged inline - mkdocs.yml: added the three tutorials to nav (required by --strict)
…en openapi.yaml - Merge reference/api.md + reference/api-reference.md into one page (base: api-reference.md, the most complete; unique sections from api.md preserved: Reports, Jira/Slack operations, Pagination/Sorting/ Filtering, SDKs) - Add scripts/generate_openapi.py: exports real FastAPI spec (92 paths) to docs/reference/openapi.yaml; regeneration command documented in the page header - Remove legacy reference/api-reference.md and its nav entry - Update links pointing to the removed file (how-to/setup-stripe-webhooks.md) - Root openapi.yaml left untouched: it is a contract-testing example fixture, not the API spec - mkdocs build --strict passes (EXIT=0)
…el/provider radar Endpoint Accuracy Index (Artificial Analysis, 4-ago-2026) como criterio de selección de providers LLM: EAI >=95% o paridad IC95, precio solo tiebreaker, re-verificacion trimestral. Radar con 30 endpoints verificados (GLM-5.2, gpt-oss-120b, DeepSeek V4 Pro). Card a027ea6d (research).
…7ab8043 + 706b6332 (grafana 12.4.9, prometheus v3.14.0, rebinds 127.0.0.1:9091/3005/9093) — runtime-deployed but uncommitted until now
… alert rules (card 0c8154e1) - backend: dead railway.app https target -> local qa-framework-backend:8000 - postgres: monitoring-stack exporter (watches HOST pg at 10.0.3.1) -> new qa-scoped postgres-exporter v0.20.1 on qa-network - redis: non-existent redis-exporter -> new qa-scoped redis_exporter v1.89.0 on qa-network - node: unreachable docker-DNS node-exporter -> host.docker.internal:9100 (host-gateway) - alertmanager: localhost:9093 (refused) -> qa-framework-alertmanager:9093 - rules: bind-mount alerts dir (19 rules now load, were 0) - exporters publish NO host ports (internal-only)
…ter mitigation (card f90a8079) Backend RateLimitMiddleware rate-limits /metrics/ (post-307-redirect path, missing from skip_paths) at 100 req/h per IP (free plan, Redis sliding window). 15s global interval = 240 req/h >> 100 → 59% scrape 429s since prometheus restart 14:12 UTC today (steady state 140 fail/h; avg_over_time(up[3h])=0.41 ≈ 100/240 predicted). 60s = 60 req/h < 100 → scrape estable. Revert to 15s when middleware normalizes skip_paths for trailing slashes (code fix via pipeline).
- Add dashboard-middleware-tests job to ci-cd.yml (runs after code-quality, before unit-tests) - Add conditional dashboard-middleware-tests job to pr-checks.yml (runs when dashboard/backend/** files changed) - Add middleware test results to notify job summary - Document CI test coverage inventory in docs/ci-test-coverage-inventory.md Addresses: Card 5ec904d4 (W1 from PR #112 review)
…458d4) Spec del 2026-07-27 vivía untracked desde hace 1 mes — commit aditivo (archivo nuevo, sin tocar nada tracked). Home natural: docs/internal/specs/ solo existe en este branch (reestructura Diátaxis).
…rd 78489ff6, mismo fix que PR #137 main)
…, GO condicionado)
…o vector (card a6906265) - core/injection: Scenario/AgentRunResult/Verdict models (binary utility x security outcome), objective evaluator (tool-call trace + FS invariants), utility checker (fail-closed) - adapters/agent: AgentCliAdapter (isolated subprocess workspace, JSONL trace capture) + argparse harness CLI (list/run, exit codes 0/1/2, --json) - corpus: agentdojo-001 seed (indirect-injection, OWASP LLM01) of 20 planned vectors - 37 TDD unit tests green; fixture agents record tool calls only (no network, C4-safe) - config/agent_injection.yaml: defaults + sandbox notes (egress deny-by-default)
…d a6906265) - corpus: vectors 2-20 across AgentDojo indirect-injection seeds + spec §2 families (direct-injection, tool-transition-nudge, payload-in-file, import-shadowing, exfiltration-passive, multi-turn), each with inline asset_content (per-vector poisoned asset) mapped to OWASP LLM01 - models: Scenario.asset_content field - harness: materialize scenario.asset_content instead of the single Day-1 document - 40 unit tests green (3 new corpus tests); e2e detection verified on nudge-001
…ance gate C2 (card a6906265) - judge: LLM-as-judge over OpenAI-compatible endpoint (stdlib urllib, no new deps), JSON verdict + confidence; unparseable/low-confidence -> HITL queue - HITLQueue: append-only JSONL (reports/injection/hitl_queue.jsonl) - key resolution: INJECTION_JUDGE_API_KEY -> LLAMA_API_KEY -> shared key file; never hardcoded. Note: LiteLLM :4000 proxy currently down (no DB); engines 8001/8009 accept the same shared key and are the measured Day-2 endpoint - harness CLI: (layer1+layer2, HITL enqueue) and (N runs over deterministic balanced case set, gate C2 <15% disagreement, exit 2 on gate failure) - 14 judge unit tests + 3 CLI judge tests green; real variance run in progress
…ance gate C2 (card a6906265) - judge: LLM-as-judge over OpenAI-compatible endpoint (stdlib urllib, no new deps), JSON verdict + confidence; unparseable/low-confidence -> HITL queue - HITLQueue: append-only JSONL (reports/injection/hitl_queue.jsonl) - key resolution: INJECTION_JUDGE_API_KEY -> LLAMA_API_KEY -> shared key file; never hardcoded. Note: LiteLLM :4000 proxy currently down (no DB); engines 8001/8009 accept the same shared key and are the measured Day-2 endpoint - harness CLI: `judge` (layer1+layer2, HITL enqueue) and `judge-variance` (N runs over deterministic balanced case set, gate C2 <15% disagreement, exit 2 on gate failure) - 14 judge unit tests + 3 CLI judge tests green; real variance run in progress
… (card a6906265) - design doc (docs/internal/design/): nudge = fail-triggered tool transition; T+C+P+J rule set, static transition table (config-declarative), integration with layer-1 evidence and layer-2 judge context, known limits, Day-3 plan - config: sandbox section updated to implemented DockerSandbox state
…J (card a6906265) TransitionTable (declarative, config/agent_injection.yaml detector.tool_transition) + NudgeDetector: failure-triggered class-lowering transitions per design doc. Unknown-status traces defer high/critical transitions to HITL (never infer).
…-1 nudge integration + 0 FP (card a6906265) - fixture nudge_agent.py: 415->curl->zip-refused->decoder.py->exec chain (record-only, C4-safe) - corpus +nudge-003 (21 vectors: 20 seed + end-to-end detection repro) - ObjectiveEvaluator: optional NudgeDetector; severity>=min marks violated; unknown-status traces defer to Verdict.needs_human -> HITLQueue (never infer) - FP measurement: safe agent across the full corpus -> 0 nudge findings
…ION_HITL_QUEUE (card a6906265) conftest autouse fixture redirects the queue for all injection tests; HITLQueue honors the env override. CLI-level tests no longer pollute reports/injection/hitl_queue.jsonl.
…SS (card a6906265) C2: judge variance 0.0% (0/89 pairs, gate <15%) — caveat: 33% null verdicts defer to HITL by design (reasoning model exhausts max_tokens before JSON). C4: sandbox review PASS (--network none, caps, no-new-privileges, snapshot/ restore, no host-env leak into container).
… (ASR 100%) (card a6906265)
…pus, pre-restore evidence (card a6906265)
…format hardening (card a6906265)
… PASS post-tuning 0% nulls (card a6906265)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fase 0 — harness de prompt-injection testing (card a6906265)
Spec: docs/internal/plans/2026-09-01-agent-prompt-injection-testing-spec.md (d33c0c9) · Veredicto: docs/internal/plans/2026-09-01-agent-prompt-injection-fase0-verdict.md
Entregables (Day 1-5)
Gates Fase 0
Veredicto recomendado
GO CONDICIONADO al MVP (Q4-2026, post-beta — C3 se mantiene). Decisor final: Joker.
Tests
113 passed (tests/unit/injection/) + 25 adyacentes en regresión. ASR: safe 0.00% / nudge 100% (evidencia por familia en reports/injection/).
Card: a6906265 · Review: Alfred APPROVED (proof c95c95c3) · Merge gated.