diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/RIGHTS.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/RIGHTS.md new file mode 100644 index 0000000..ae4f770 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/RIGHTS.md @@ -0,0 +1,16 @@ +# Submission rights declaration + +Project: `ExhaustiveGate` +Submission slug: `shahadattest-exhaustive-gate` +Submitter: `shahadattest` +Date: `2026-09-04` + +The submitter confirms that they own, or have sufficient authorization for, the source code, dependencies, service, data, branding, and other materials submitted in this pull request. + +Subject to the official program terms, the submitter authorizes X-Agent to retain, reproduce, audit, test, archive, and publish the submitted program artifact for judging, fraud prevention, dispute handling, ecosystem submission, and post-award accountability. Closing the pull request, deleting a fork, or deleting an external repository does not revoke the official archive rights attached to an accepted and rewarded entry. + +Third-party components and their licenses: FastAPI (MIT), uvicorn (BSD), Pydantic v2 (MIT), SQLAlchemy (MIT), httpx (BSD), pytest (MIT), nginx (BSD) — see `source/` manifests. + +Exceptions or restrictions: `none` + +This template is an operational declaration, not a substitute for event terms reviewed by qualified counsel. diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/SUBMISSION.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/SUBMISSION.md new file mode 100644 index 0000000..fa0af74 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/SUBMISSION.md @@ -0,0 +1,47 @@ +# ExhaustiveGate + +## Capability + +- **One-line description:** Verify an AI agent actually covered the required result set before it claims all, none, exactly N, cheapest, highest, or lowest — returns PROVEN / UNPROVEN / CONDITIONAL with blocking reasons and next actions. +- **Who it helps:** AI agents and developers that paginate third-party APIs and must avoid false exhaustive claims. +- **Capability boundary:** Sessions, scope hashing, pagination-chain validation, failure/snapshot tracking, proof obligations, verdicts, SHA-256 proof certificates. Does NOT modify upstream APIs, does NOT do security auditing, penetration testing, or risk scoring. + +## Live API + +- **API base URL:** `https://mean-capital-republican-understood.trycloudflare.com/v1` (local verified: `http://localhost:8100/v1`) +- **Health-check URL:** `https://mean-capital-republican-understood.trycloudflare.com/health` +- **Authentication:** none +- **Rate limits / known limits:** No auth limits; JSON body cap ~512KB. Core makes no upstream calls (evidence is posted by the agent). +- **API contract:** `source/docs/api.md`; interactive docs at `/docs`. + +## Source and reproducibility + +- **Source repository:** `https://github.com/ShahadatTest/exhaustive-gate` +- **Review commit:** `79a2a1c5f8a746584996b88c11aee746e079d48f` +- **Source submitted in this PR:** `source/` +- **Run tests:** `cd source/backend && pip install -r requirements.txt && python -m pytest tests/ -q` (15 passed) +- **Run locally:** `cd source && docker-compose up --build` (dashboard :8102, gate :8100, demo CRM :8101) +- **Deploy:** build `source/backend/Dockerfile`, set `GIT_COMMIT=` and `XAGENT_SLUG=shahadattest-exhaustive-gate` +- **Version binding:** `/health` returns `{"status":"ok","commit":""}` and `/.well-known/xagent-verification.json` returns `{"schemaVersion":1,"slug":"shahadattest-exhaustive-gate","commit":""}` + +## Verification + +Reproducible call instructions and redacted example responses are in `verification/README.md`. + +- **Health-check result:** `{"status":"ok","service":"exhaustive-gate","version":"0.1.0","commit":"79a2a1c5f8a746584996b88c11aee746e079d48f"}` +- **Capability call:** `POST /v1/sessions` → observe 4 invoice pages → `POST /v1/sessions/{id}/verify` with `{"claim":{"type":"EXACT_COUNT","value":347}}` → `PROVEN` + proof certificate +- **Expected error behavior:** unknown session → 404; invalid pagination_type → 400; certificate before PROVEN → 404; incomplete evidence → `UNPROVEN` with `blocking_reasons` + `required_next_actions` + +## Security and data handling + +- **Data collected:** Retrieval-evidence metadata the reviewer posts (page/cursor/counts); no end-user data. +- **Purpose and retention:** Review/demo only, local SQLite file. +- **Third parties / outbound network calls:** none in core. +- **Secrets:** No secrets are committed. Review access is supplied only through an approved private channel when required. +- **Known risks / restrictions:** none; verdicts are deterministic functions of posted evidence. + +## Support + +- **Team / builder:** shahadattest (solo) +- **Contact:** via GitHub `shahadattest` +- **License / rights:** MIT (see `source/LICENSE`); submitter authorizes review and archival per RIGHTS.md. diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/.env.example b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/.env.example new file mode 100644 index 0000000..b0a577c --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/.env.example @@ -0,0 +1,4 @@ +PORT=8000 +DATABASE_URL=sqlite:///./exhaustive_gate.db +GIT_COMMIT=dev-local +XAGENT_SLUG=team-exhaustive-gate diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/LICENSE b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/LICENSE new file mode 100644 index 0000000..f04b3b5 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/LICENSE @@ -0,0 +1 @@ +MIT License — ExhaustiveGate (hackathon MVP). diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/README.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/README.md new file mode 100644 index 0000000..dccd61f --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/README.md @@ -0,0 +1,43 @@ +# ExhaustiveGate + +AI agents often mistake partial retrieval for complete evidence. + +ExhaustiveGate verifies whether an agent has actually covered the required result set before allowing claims such as: + +- all +- none +- exactly N +- cheapest +- highest +- lowest + +**“Finding no more evidence is not the same as proving there is no more evidence.”** + +## Quick start + +```bash +cd exhaustive-gate/backend +pip install -r requirements.txt +python -m uvicorn app.main:app --port 8100 +cd ../examples/demo-crm +python -m uvicorn main:app --port 8101 +# open ../frontend/index.html +``` + +## Docker + +```bash +cd exhaustive-gate +docker-compose up --build +# frontend :8102, gate :8100, CRM :8101 +``` + +## Example + +```bash +curl -X POST localhost:8100/v1/sessions -H 'Content-Type: application/json' \ + -d '{"resource_type":"invoice","scope":{"status":"unpaid"}}' +``` + +See `docs/api.md`, `docs/proof-model.md`, `docs/demo.md`. Security: validated inputs, +body caps, no code execution, no upstream fetching in core. License: MIT. diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/Dockerfile b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/Dockerfile new file mode 100644 index 0000000..a62a795 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/Dockerfile @@ -0,0 +1,9 @@ +FROM python:3.12-slim +WORKDIR /code +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt +COPY app ./app +COPY tests ./tests +ENV PORT=8000 +EXPOSE 8000 +CMD ["sh","-c","python -m uvicorn app.main:app --host 0.0.0.0 --port ${PORT:-8000}"] diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/__init__.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/api/__init__.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/api/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/api/sessions.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/api/sessions.py new file mode 100644 index 0000000..35bcbe4 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/api/sessions.py @@ -0,0 +1,155 @@ +import json +import uuid +from fastapi import APIRouter, HTTPException +from sqlalchemy.orm import Session + +from app.models.db import SessionRow, engine +from app.schemas.api import FailureRecord, ObservePage, ParseIn, SessionCreate, VerifyIn +from app.services.certificates import issue_certificate +from app.services.claim_parser import parse_claim +from app.services.scope import scope_hash +from app.services.verifier import verify + +router = APIRouter() + + +def _db() -> Session: + return Session(engine, expire_on_commit=False) + + +def _load(row: SessionRow) -> dict: + return {"id": row.id, "resource_type": row.resource_type, "source": row.source, + "scope": json.loads(row.scope_json or "{}"), "scope_hash": row.scope_hash, + "pagination_type": row.pagination_type, "snapshot_strategy": row.snapshot_strategy, + "status": row.status, "observations": json.loads(row.observations_json or "[]"), + "failures": json.loads(row.failures_json or "[]")} + + +@router.post("/sessions") +async def create_session(body: SessionCreate): + if body.pagination_type not in ("cursor", "offset", "page", "single"): + raise HTTPException(400, "pagination_type must be cursor|offset|page|single") + if body.snapshot_strategy not in ("STRICT", "BEST_EFFORT", "UNKNOWN"): + raise HTTPException(400, "snapshot_strategy must be STRICT|BEST_EFFORT|UNKNOWN") + sid = "sess_" + uuid.uuid4().hex[:12] + db = _db() + row = SessionRow(id=sid, resource_type=body.resource_type, source=body.source, + scope_json=json.dumps(body.scope), scope_hash=scope_hash(body.scope), + pagination_type=body.pagination_type, snapshot_strategy=body.snapshot_strategy) + db.add(row) + db.commit() + db.close() + return {"session_id": sid, "status": "collecting", "scope_hash": row.scope_hash} + + +@router.get("/sessions/{sid}") +async def get_session(sid: str): + db = _db() + row = db.get(SessionRow, sid) + if not row: + db.close() + raise HTTPException(404, "session not found") + out = _load(row) + out["observation_count"] = len(out["observations"]) + out["failure_count"] = len(out["failures"]) + db.close() + return out + + +@router.post("/sessions/{sid}/observe") +async def observe(sid: str, body: ObservePage): + db = _db() + row = db.get(SessionRow, sid) + if not row: + db.close() + raise HTTPException(404, "session not found") + obs = json.loads(row.observations_json or "[]") + scope = body.scope if body.scope is not None else json.loads(row.scope_json or "{}") + entry = {"page_number": body.page_number, "offset": body.offset, "cursor_in": body.cursor_in, + "cursor_out": body.cursor_out, "has_more": body.has_more, "records_seen": body.records_seen, + "items": body.items, "scope": scope, "scope_hash": scope_hash(scope), + "snapshot_id": body.snapshot_id, "authoritative_total": body.authoritative_total} + obs.append(entry) + row.observations_json = json.dumps(obs) + row.status = "complete" if not body.has_more else "collecting" + db.add(row) + db.commit() + last = obs[-1] + nxt = last.get("cursor_out") if last.get("has_more") else None + db.close() + return {"accepted": True, "coverage_status": "COMPLETE" if not body.has_more else "INCOMPLETE", + "next_expected_cursor": nxt, "pages_seen": len(obs)} + + +@router.post("/sessions/{sid}/failure") +async def record_failure(sid: str, body: FailureRecord): + db = _db() + row = db.get(SessionRow, sid) + if not row: + db.close() + raise HTTPException(404, "session not found") + fails = json.loads(row.failures_json or "[]") + fails.append({"page_number": body.page_number, "kind": body.kind, "message": body.message}) + row.failures_json = json.dumps(fails) + row.status = "collecting" + db.add(row) + db.commit() + db.close() + return {"accepted": True, "unresolved_failures": len(fails)} + + +@router.get("/sessions/{sid}/observations") +async def list_observations(sid: str): + db = _db() + row = db.get(SessionRow, sid) + if not row: + db.close() + raise HTTPException(404, "session not found") + out = {"observations": json.loads(row.observations_json or "[]"), + "failures": json.loads(row.failures_json or "[]")} + db.close() + return out + + +@router.post("/sessions/{sid}/verify") +async def verify_claim(sid: str, body: VerifyIn): + db = _db() + row = db.get(SessionRow, sid) + if not row: + db.close() + raise HTTPException(404, "session not found") + sess = _load(row) + claim = body.claim.model_dump(exclude_none=False) + result = verify(sess, claim) + row.result_json = json.dumps(result) + if result["verdict"] == "PROVEN": + row.certificate_json = json.dumps(issue_certificate(sess, claim, result)) + row.status = "proven" + else: + row.certificate_json = "{}" + row.status = "collecting" + db.add(row) + db.commit() + cert = json.loads(row.certificate_json or "{}") + db.close() + result["certificate"] = cert or None + return result + + +@router.get("/sessions/{sid}/certificate") +async def get_certificate(sid: str): + db = _db() + row = db.get(SessionRow, sid) + if not row: + db.close() + raise HTTPException(404, "session not found") + cert = json.loads(row.certificate_json or "{}") + db.close() + if not cert: + raise HTTPException(404, "no certificate (claim not PROVEN yet)") + return cert + + +@router.post("/claims/parse") +async def parse(body: ParseIn): + return parse_claim(body.text) diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/core/__init__.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/core/config.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/core/config.py new file mode 100644 index 0000000..0ae976e --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/core/config.py @@ -0,0 +1,8 @@ +import os + +APP_NAME = "exhaustive-gate" +APP_VERSION = "0.1.0" +DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./exhaustive_gate.db") +GIT_COMMIT = os.getenv("GIT_COMMIT", "dev-local") +XAGENT_SLUG = os.getenv("XAGENT_SLUG", os.getenv("PROJECT_SLUG", "exhaustive-gate")) +MAX_BODY_BYTES = int(os.getenv("MAX_BODY_BYTES", "524288")) diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/main.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/main.py new file mode 100644 index 0000000..eaa5a26 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/main.py @@ -0,0 +1,22 @@ +from fastapi import FastAPI +from fastapi.middleware.cors import CORSMiddleware + +from app.api.sessions import router as sessions_router +from app.core.config import APP_VERSION, GIT_COMMIT, XAGENT_SLUG +from app.models.db import init_db + +app = FastAPI(title="ExhaustiveGate", version=APP_VERSION) +app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]) + +init_db() +app.include_router(sessions_router, prefix="/v1") + + +@app.get("/health") +async def health(): + return {"status": "ok", "service": "exhaustive-gate", "version": APP_VERSION, "commit": GIT_COMMIT} + + +@app.get("/.well-known/xagent-verification.json") +async def verification(): + return {"schemaVersion": 1, "slug": XAGENT_SLUG, "commit": GIT_COMMIT} diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/models/__init__.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/models/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/models/db.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/models/db.py new file mode 100644 index 0000000..0f19918 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/models/db.py @@ -0,0 +1,30 @@ +from sqlalchemy import String, Text, create_engine +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column + +from app.core.config import DATABASE_URL + +engine = create_engine(DATABASE_URL, connect_args={"check_same_thread": False} if DATABASE_URL.startswith("sqlite") else {}) + + +class Base(DeclarativeBase): + pass + + +class SessionRow(Base): + __tablename__ = "sessions" + id: Mapped[str] = mapped_column(String(32), primary_key=True) + resource_type: Mapped[str] = mapped_column(String(128), default="") + source: Mapped[str] = mapped_column(String(128), default="") + scope_json: Mapped[str] = mapped_column(Text, default="{}") + scope_hash: Mapped[str] = mapped_column(String(64), default="") + pagination_type: Mapped[str] = mapped_column(String(32), default="cursor") + snapshot_strategy: Mapped[str] = mapped_column(String(32), default="STRICT") + status: Mapped[str] = mapped_column(String(32), default="collecting") + observations_json: Mapped[str] = mapped_column(Text, default="[]") + failures_json: Mapped[str] = mapped_column(Text, default="[]") + result_json: Mapped[str] = mapped_column(Text, default="{}") + certificate_json: Mapped[str] = mapped_column(Text, default="{}") + + +def init_db() -> None: + Base.metadata.create_all(engine) diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/schemas/__init__.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/schemas/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/schemas/api.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/schemas/api.py new file mode 100644 index 0000000..c403cd7 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/schemas/api.py @@ -0,0 +1,46 @@ +from typing import Any, Optional +from pydantic import BaseModel, Field + + +class SessionCreate(BaseModel): + resource_type: str = Field(min_length=1, max_length=128) + source: str = Field(default="demo-crm", max_length=128) + scope: dict[str, Any] = Field(default_factory=dict) + pagination_type: str = Field(default="cursor") + snapshot_strategy: str = Field(default="STRICT") + + +class ObservePage(BaseModel): + page_number: Optional[int] = None + offset: Optional[int] = None + cursor_in: Optional[str] = None + cursor_out: Optional[str] = None + has_more: bool = False + records_seen: int = Field(ge=0, default=0) + items: list[dict[str, Any]] = Field(default_factory=list) + scope: Optional[dict[str, Any]] = None + snapshot_id: Optional[str] = None + authoritative_total: Optional[int] = None + + +class FailureRecord(BaseModel): + page_number: Optional[int] = None + kind: str = Field(default="unknown") + message: str = Field(default="", max_length=500) + + +class ClaimIn(BaseModel): + type: str + resource: Optional[str] = None + scope: Optional[dict[str, Any]] = None + value: Optional[Any] = None + field: Optional[str] = None + candidate_id: Optional[str] = None + + +class VerifyIn(BaseModel): + claim: ClaimIn + + +class ParseIn(BaseModel): + text: str = Field(min_length=1, max_length=500) diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/__init__.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/certificates.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/certificates.py new file mode 100644 index 0000000..4a14a90 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/certificates.py @@ -0,0 +1,25 @@ +"""Deterministic proof certificates (SHA-256 digests, no blockchain).""" +import hashlib +import json +from datetime import datetime, timezone +from typing import Any + + +def evidence_digest(session: dict[str, Any]) -> str: + norm = json.dumps({"scope_hash": session.get("scope_hash"), + "observations": session.get("observations", []), + "failures": session.get("failures", [])}, + sort_keys=True, separators=(",", ":")) + return hashlib.sha256(norm.encode()).hexdigest() + + +def issue_certificate(session: dict, claim: dict, verdict: dict) -> dict[str, Any]: + digest = evidence_digest(session) + cid = "proof_" + hashlib.sha256((digest + json.dumps(claim, sort_keys=True)).encode()).hexdigest()[:12] + return {"certificate_id": cid, "claim_type": verdict.get("claim_type"), + "claim": claim, "verdict": verdict.get("verdict"), + "resource": session.get("resource_type"), "scope_hash": session.get("scope_hash"), + "evidence_hash": digest, "records_examined": verdict.get("records_examined", 0), + "pages_examined": verdict.get("pages_examined", 0), + "snapshot_status": verdict.get("evidence_summary", {}).get("snapshot_status", "UNKNOWN"), + "issued_at": datetime.now(timezone.utc).isoformat()} diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/claim_parser.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/claim_parser.py new file mode 100644 index 0000000..6aabd96 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/claim_parser.py @@ -0,0 +1,46 @@ +"""Deterministic NL claim parser (regex first; LLM optional later).""" +import re +from typing import Any + + +def parse_claim(text: str) -> dict[str, Any]: + t = text.strip().lower() + m = re.search(r"there are no (.+)", t) + if m or t.startswith("no ") or "no matching" in t or "not found" in t: + return {"type": "NONE", "resource": _resource(t), "scope": _scope_hint(t)} + m = re.search(r"these are all (.+)|here are all (.+)|every matching (.+)", t) + if m or (t.startswith("all ") or " are all " in t): + return {"type": "ALL", "resource": _resource(t), "scope": {}} + m = re.search(r"exactly (\d+)", t) + if m: + return {"type": "EXACT_COUNT", "resource": _resource(t), "scope": _scope_hint(t), "value": int(m.group(1))} + if "cheapest" in t or "lowest" in t or "shortest" in t or "minimum" in t: + return {"type": "MIN", "resource": _resource(t), "field": _field_hint(t)} + if "highest" in t or "most expensive" in t or "largest" in t or "maximum" in t or "highest-value" in t: + return {"type": "MAX", "resource": _resource(t), "field": _field_hint(t)} + return {"type": "UNKNOWN", "resource": _resource(t), "scope": {}} + + +def _resource(t: str) -> str: + for w in ("invoice", "customer", "product", "ticket", "order", "event", "record"): + if w in t: + return w + return "record" + + +def _scope_hint(t: str) -> dict[str, Any]: + scope: dict[str, Any] = {} + for w in ("unpaid", "overdue", "open", "active", "paid"): + if w in t: + scope["status"] = w + return scope + + +def _field_hint(t: str) -> str: + if "price" in t or "cheapest" in t or "expensive" in t: + return "price" + if "value" in t: + return "value" + if "balance" in t: + return "balance" + return "value" diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/obligations.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/obligations.py new file mode 100644 index 0000000..7f3a1dc --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/obligations.py @@ -0,0 +1,29 @@ +"""Proof obligations per claim type.""" +from typing import Any + +OBLIGATIONS: dict[str, list[str]] = { + "NONE": ["RESULT_SET_COMPLETE", "NO_MATCHES_OBSERVED", "NO_UNRESOLVED_FAILURES", "SCOPE_STABLE", "SNAPSHOT_ACCEPTABLE"], + "ALL": ["RESULT_SET_COMPLETE", "NO_UNRESOLVED_FAILURES", "SCOPE_STABLE", "SNAPSHOT_ACCEPTABLE"], + "EXACT_COUNT": ["RESULT_SET_COMPLETE", "NO_UNRESOLVED_FAILURES", "SCOPE_STABLE", "NO_CURSOR_GAPS", "SNAPSHOT_ACCEPTABLE", "COUNT_CONSISTENT"], + "MIN": ["RESULT_SET_COMPLETE", "NO_UNRESOLVED_FAILURES", "SCOPE_STABLE", "SNAPSHOT_ACCEPTABLE", "TARGET_FIELD_AVAILABLE_FOR_ALL_CANDIDATES", "COMPARISON_DOMAIN_VALID"], + "MAX": ["RESULT_SET_COMPLETE", "NO_UNRESOLVED_FAILURES", "SCOPE_STABLE", "SNAPSHOT_ACCEPTABLE", "TARGET_FIELD_AVAILABLE_FOR_ALL_CANDIDATES", "COMPARISON_DOMAIN_VALID"], +} + +REASONS: dict[str, dict[str, str]] = { + "PAGINATION_NOT_EXHAUSTED": {"message": "The upstream result set has not been fully traversed.", "action": "FETCH_NEXT_PAGE"}, + "CURSOR_CHAIN_BROKEN": {"message": "Cursor chain is discontinuous; coverage has gaps.", "action": "RESTART_WITH_STABLE_SNAPSHOT"}, + "CURSOR_LOOP": {"message": "A cursor repeated; retrieval is looping.", "action": "RESTART_WITH_STABLE_SNAPSHOT"}, + "MISSING_PAGE": {"message": "Pages are missing, duplicated, or out of order.", "action": "FETCH_MISSING_CANDIDATES"}, + "UNRESOLVED_FAILURE": {"message": "A page failed and was never retried successfully.", "action": "RETRY_FAILED_PAGE"}, + "RATE_LIMIT_INTERRUPTION": {"message": "Rate limiting interrupted retrieval.", "action": "RETRY_FAILED_PAGE"}, + "SCOPE_MISMATCH": {"message": "Filters/scope changed during retrieval.", "action": "NORMALIZE_SCOPE"}, + "SNAPSHOT_CHANGED": {"message": "Dataset snapshot changed during retrieval.", "action": "RESTART_WITH_STABLE_SNAPSHOT"}, + "UNKNOWN_SNAPSHOT_STATE": {"message": "No snapshot information supplied.", "action": "VERIFY_SOURCE_TOTAL"}, + "CLAIM_VALUE_MISMATCH": {"message": "Claimed value contradicts observed evidence.", "action": "DO_NOT_MAKE_EXHAUSTIVE_CLAIM"}, + "MISSING_REQUIRED_FIELD": {"message": "Comparison field missing on some candidates.", "action": "FETCH_MISSING_CANDIDATES"}, + "AUTHORITATIVE_COUNT_MISSING": {"message": "No complete enumeration and no trusted total.", "action": "VERIFY_SOURCE_TOTAL"}, +} + + +def obligations_for(claim_type: str) -> list[str]: + return list(OBLIGATIONS.get(claim_type.upper(), [])) diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/pagination.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/pagination.py new file mode 100644 index 0000000..1bce5bd --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/pagination.py @@ -0,0 +1,40 @@ +"""Cursor/offset/page-number chain validation. Pure + deterministic.""" +from typing import Any + + +def check_chain(observations: list[dict[str, Any]], pagination_type: str) -> list[dict[str, str]]: + """Return list of pagination problems (codes). Empty == chain OK so far.""" + problems: list[dict[str, str]] = [] + if not observations: + return problems + seen_pages: set[Any] = set() + seen_cursors_out: set[str] = set() + prev_out: str | None = None + prev_page: int | None = None + for i, ob in enumerate(observations): + pg = ob.get("page_number") + if pg is not None: + if pg in seen_pages: + problems.append({"code": "MISSING_PAGE", "message": f"Duplicate page number {pg} (possible retry/loop)."}) + seen_pages.add(pg) + if prev_page is not None and pg != prev_page + 1 and pagination_type in ("page", "offset"): + problems.append({"code": "MISSING_PAGE", "message": f"Page gap: {prev_page} -> {pg}."}) + prev_page = pg if isinstance(pg, int) else prev_page + cin, cout = ob.get("cursor_in"), ob.get("cursor_out") + if pagination_type == "cursor" and i > 0: + if cin != prev_out and not (cin is None and prev_out is None): + problems.append({"code": "CURSOR_CHAIN_BROKEN", "message": f"Observation {i}: cursor_in {cin!r} != previous cursor_out {prev_out!r}."}) + if cout is not None: + if cout in seen_cursors_out: + problems.append({"code": "CURSOR_LOOP", "message": f"Cursor {cout!r} already emitted (loop)."}) + seen_cursors_out.add(cout) + if pagination_type == "cursor" and cin is not None and cout == cin: + problems.append({"code": "CURSOR_LOOP", "message": f"Cursor did not advance ({cin!r})."}) + prev_out = cout + return problems + + +def is_exhausted(observations: list[dict[str, Any]]) -> bool: + if not observations: + return False + return observations[-1].get("has_more") is False diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/scope.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/scope.py new file mode 100644 index 0000000..0f0d8a7 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/scope.py @@ -0,0 +1,12 @@ +"""Scope canonicalization + deterministic hashing.""" +import hashlib +import json +from typing import Any + + +def canonicalize(scope: dict[str, Any]) -> str: + return json.dumps(scope or {}, sort_keys=True, separators=(",", ":"), ensure_ascii=True) + + +def scope_hash(scope: dict[str, Any]) -> str: + return hashlib.sha256(canonicalize(scope).encode("utf-8")).hexdigest() diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/verifier.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/verifier.py new file mode 100644 index 0000000..20b7a85 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/app/services/verifier.py @@ -0,0 +1,138 @@ +"""Deterministic verdict engine. No LLM. Pure function of session state + claim.""" +import hashlib +import json +from typing import Any + +from app.services.obligations import REASONS, obligations_for +from app.services.pagination import check_chain, is_exhausted +from app.services.scope import scope_hash + + +def _reason(code: str, severity: str = "blocking") -> dict[str, Any]: + meta = REASONS.get(code, {"message": code, "action": "DO_NOT_MAKE_EXHAUSTIVE_CLAIM"}) + return {"code": code, "message": meta["message"], "severity": severity, + "recommended_next_action": meta["action"]} + + +def verify(session: dict[str, Any], claim: dict[str, Any]) -> dict[str, Any]: + ctype = str(claim.get("type", "")).upper() + observations: list[dict] = session.get("observations", []) + failures: list[dict] = session.get("failures", []) + pagination_type = session.get("pagination_type", "cursor") + strategy = session.get("snapshot_strategy", "STRICT") + base_scope_hash = session.get("scope_hash", "") + + blocking: list[dict] = [] + warnings: list[dict] = [] + obligations = obligations_for(ctype) + obligation_status: dict[str, str] = {} + + # --- scope stability + scope_ok = True + for ob in observations: + if ob.get("scope_hash") and ob["scope_hash"] != base_scope_hash: + scope_ok = False + if not scope_ok: + blocking.append(_reason("SCOPE_MISMATCH")) + obligation_status["SCOPE_STABLE"] = "satisfied" if scope_ok else "failed" + + # --- failures + rate_limited = any(f.get("kind") == "rate_limit" for f in failures) + if rate_limited: + blocking.append(_reason("RATE_LIMIT_INTERRUPTION")) + if failures: + blocking.append(_reason("UNRESOLVED_FAILURE")) + obligation_status["NO_UNRESOLVED_FAILURES"] = "satisfied" if not failures else "failed" + + # --- pagination chain + chain_problems = check_chain(observations, pagination_type) + for p in chain_problems: + blocking.append(_reason(p["code"])) + obligation_status["NO_CURSOR_GAPS"] = "satisfied" if not chain_problems else "failed" + + # --- exhaustion + exhausted = is_exhausted(observations) + if not exhausted: + blocking.append(_reason("PAGINATION_NOT_EXHAUSTED")) + obligation_status["RESULT_SET_COMPLETE"] = "satisfied" if exhausted else "failed" + + # --- snapshot + snaps = [ob.get("snapshot_id") for ob in observations if ob.get("snapshot_id")] + snapshot_status = "UNKNOWN" + if snaps: + snapshot_status = "STABLE" if len(set(snaps)) == 1 else "CHANGED" + if snapshot_status == "CHANGED": + if strategy == "STRICT": + blocking.append(_reason("SNAPSHOT_CHANGED")) + # BEST_EFFORT -> handled below as CONDITIONAL + elif snapshot_status == "UNKNOWN": + warnings.append(_reason("UNKNOWN_SNAPSHOT_STATE", "info")) + obligation_status["SNAPSHOT_ACCEPTABLE"] = "satisfied" if snapshot_status in ("STABLE", "UNKNOWN") else ("conditional" if strategy == "BEST_EFFORT" else "failed") + + pages_seen = len(observations) + records_seen = sum(int(ob.get("records_seen", 0)) for ob in observations) + + # --- claim-specific checks (only meaningful if structurally complete) + certified_value: Any = None + if ctype == "EXACT_COUNT": + claimed = claim.get("value") + auth_totals = [ob.get("authoritative_total") for ob in observations if ob.get("authoritative_total") is not None] + if exhausted and not failures and scope_ok: + if auth_totals and len(set(auth_totals)) == 1 and auth_totals[0] == records_seen: + certified_value = records_seen + elif not auth_totals: + certified_value = records_seen + if claimed is not None and int(claimed) != int(records_seen): + blocking.append(_reason("CLAIM_VALUE_MISMATCH")) + certified_value = None + obligation_status["COUNT_CONSISTENT"] = "satisfied" if certified_value is not None or not exhausted else "failed" + elif ctype == "NONE": + if exhausted and records_seen > 0: + blocking.append(_reason("CLAIM_VALUE_MISMATCH")) + elif exhausted: + certified_value = 0 + obligation_status["NO_MATCHES_OBSERVED"] = "satisfied" if records_seen == 0 else "failed" + elif ctype == "ALL": + if exhausted: + certified_value = records_seen + elif ctype in ("MIN", "MAX"): + field = claim.get("field") + cand = claim.get("candidate_id") + items = [it for ob in observations for it in (ob.get("items") or [])] + if not field: + blocking.append(_reason("MISSING_REQUIRED_FIELD")) + elif exhausted and items and all(isinstance(it, dict) and field in it and isinstance(it[field], (int, float)) for it in items): + extreme = min(it[field] for it in items) if ctype == "MIN" else max(it[field] for it in items) + holders = [it.get("id") for it in items if it[field] == extreme] + if cand is not None and cand not in holders: + blocking.append(_reason("CLAIM_VALUE_MISMATCH")) + else: + certified_value = {"field": field, "value": extreme, "holder_ids": holders} + elif exhausted: + blocking.append(_reason("MISSING_REQUIRED_FIELD")) + obligation_status["TARGET_FIELD_AVAILABLE_FOR_ALL_CANDIDATES"] = "satisfied" if certified_value is not None else "failed" + obligation_status["COMPARISON_DOMAIN_VALID"] = "satisfied" if certified_value is not None else ("failed" if exhausted else "pending") + + if ctype not in ("NONE", "ALL", "EXACT_COUNT", "MIN", "MAX"): + return {"verdict": "UNPROVEN", "claim_type": ctype, "obligations": obligations, + "obligation_status": obligation_status, + "blocking_reasons": [_reason("CLAIM_VALUE_MISMATCH")], + "required_next_actions": ["DO_NOT_MAKE_EXHAUSTIVE_CLAIM"], + "evidence_summary": {"pages_seen": pages_seen, "records_seen": records_seen, + "continuation_available": not exhausted, "snapshot_status": snapshot_status}} + + if blocking: + verdict = "UNPROVEN" + elif snapshot_status == "CHANGED" and strategy == "BEST_EFFORT": + verdict = "CONDITIONAL" + else: + verdict = "PROVEN" + + actions = sorted({r["recommended_next_action"] for r in blocking}) or ([] if verdict == "PROVEN" else ["DO_NOT_MAKE_EXHAUSTIVE_CLAIM"]) + return {"verdict": verdict, "claim_type": ctype, "obligations": obligations, + "obligation_status": obligation_status, "blocking_reasons": blocking, + "required_next_actions": actions, + "certified_value": certified_value, "records_examined": records_seen, + "pages_examined": pages_seen, + "evidence_summary": {"pages_seen": pages_seen, "records_seen": records_seen, + "continuation_available": not exhausted, "snapshot_status": snapshot_status}} diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/requirements.txt b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/requirements.txt new file mode 100644 index 0000000..c8be0d9 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/requirements.txt @@ -0,0 +1,8 @@ +fastapi==0.116.1 +uvicorn[standard]==0.35.0 +pydantic==2.11.7 +SQLAlchemy==2.0.43 +httpx==0.28.1 +pytest==8.4.1 +pytest-asyncio==1.1.0 +python-multipart==0.0.20 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/tests/test_engine.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/tests/test_engine.py new file mode 100644 index 0000000..fa2bd3f --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/backend/tests/test_engine.py @@ -0,0 +1,158 @@ +"""Deterministic ExhaustiveGate tests: spec cases A-F + endpoints.""" +from fastapi.testclient import TestClient + +from app.main import app +from app.services.certificates import evidence_digest, issue_certificate +from app.services.claim_parser import parse_claim +from app.services.scope import scope_hash +from app.services.verifier import verify + +client = TestClient(app) + +BASE = {"id": "s", "resource_type": "invoice", "source": "demo-crm", + "scope_hash": scope_hash({"status": "unpaid"}), "pagination_type": "cursor", + "snapshot_strategy": "STRICT"} + + +def sess(obs, fails=None, strategy="STRICT"): + d = dict(BASE) + d["observations"] = obs + d["failures"] = fails or [] + d["snapshot_strategy"] = strategy + return d + + +def ob(pg, cin, cout, more, n, scope=None, snap="snap_A", items=None): + return {"page_number": pg, "cursor_in": cin, "cursor_out": cout, "has_more": more, + "records_seen": n, "items": items or [], + "scope_hash": scope_hash(scope or {"status": "unpaid"}), "snapshot_id": snap} + + +# Case A: has_more=true + EXACT_COUNT -> UNPROVEN/PAGINATION_NOT_EXHAUSTED +def test_case_a_partial_count_unproven(): + s = sess([ob(1, None, "c2", True, 100)]) + r = verify(s, {"type": "EXACT_COUNT", "value": 100}) + assert r["verdict"] == "UNPROVEN" + assert "PAGINATION_NOT_EXHAUSTED" in [b["code"] for b in r["blocking_reasons"]] + assert "FETCH_NEXT_PAGE" in r["required_next_actions"] + + +# Case B: exhausted + match -> PROVEN + certificate +def test_case_b_full_count_proven(): + s = sess([ob(1, None, "c2", True, 100), ob(2, "c2", "c3", True, 100), + ob(3, "c3", "c4", True, 100), ob(4, "c4", None, False, 47)]) + r = verify(s, {"type": "EXACT_COUNT", "value": 347}) + assert r["verdict"] == "PROVEN", r + assert r["certified_value"] == 347 + assert r["records_examined"] == 347 and r["pages_examined"] == 4 + cert = issue_certificate(s, {"type": "EXACT_COUNT", "value": 347}, r) + assert cert["verdict"] == "PROVEN" and len(cert["evidence_hash"]) == 64 + + +# Case C: page 3 timeout -> UNPROVEN +def test_case_c_failure_unproven(): + s = sess([ob(1, None, "c2", True, 100), ob(2, "c2", None, False, 50)], + [{"page_number": 3, "kind": "timeout"}]) + r = verify(s, {"type": "NONE"}) + assert r["verdict"] == "UNPROVEN" + assert "UNRESOLVED_FAILURE" in [b["code"] for b in r["blocking_reasons"]] + + +# Case D: scope change -> UNPROVEN/SCOPE_MISMATCH +def test_case_d_scope_mismatch(): + s = sess([ob(1, None, "c2", True, 100), + ob(2, "c2", None, False, 50, scope={"status": "all"})]) + r = verify(s, {"type": "ALL"}) + assert r["verdict"] == "UNPROVEN" + assert "SCOPE_MISMATCH" in [b["code"] for b in r["blocking_reasons"]] + + +# Case E: snapshot change strict -> UNPROVEN +def test_case_e_snapshot_change(): + s = sess([ob(1, None, "c2", True, 100, snap="A"), ob(2, "c2", None, False, 50, snap="B")]) + r = verify(s, {"type": "EXACT_COUNT", "value": 150}) + assert r["verdict"] == "UNPROVEN" + assert "SNAPSHOT_CHANGED" in [b["code"] for b in r["blocking_reasons"]] + r2 = verify(sess([ob(1, None, "c2", True, 100, snap="A"), ob(2, "c2", None, False, 50, snap="B")], strategy="BEST_EFFORT"), + {"type": "ALL"}) + assert r2["verdict"] == "CONDITIONAL" + + +# Case F: MIN proven over full domain +def test_case_f_min_proven(): + items1 = [{"id": f"p{i}", "price": 10 + i} for i in range(20)] + items2 = [{"id": "pCheap", "price": 5}] + [{"id": f"q{i}", "price": 30 + i} for i in range(10)] + s = sess([ob(1, None, "c2", True, 20, items=items1), ob(2, "c2", None, False, 11, items=items2)]) + r = verify(s, {"type": "MIN", "field": "price", "candidate_id": "pCheap"}) + assert r["verdict"] == "PROVEN", r + r2 = verify(s, {"type": "MIN", "field": "price", "candidate_id": "p0"}) + assert r2["verdict"] == "UNPROVEN" + assert "CLAIM_VALUE_MISMATCH" in [b["code"] for b in r2["blocking_reasons"]] + + +def test_min_partial_domain_unproven(): + s = sess([ob(1, None, "c2", True, 20)]) + r = verify(s, {"type": "MIN", "field": "price", "candidate_id": "p1"}) + assert r["verdict"] == "UNPROVEN" + codes = [b["code"] for b in r["blocking_reasons"]] + assert "PAGINATION_NOT_EXHAUSTED" in codes + + +def test_none_proven_when_empty(): + s = sess([ob(1, None, None, False, 0)]) + r = verify(s, {"type": "NONE"}) + assert r["verdict"] == "PROVEN" and r["certified_value"] == 0 + + +def test_cursor_loop_detected(): + s = sess([ob(1, None, "c2", True, 10), ob(2, "c2", "c2", False, 10)]) + r = verify(s, {"type": "ALL"}) + assert "CURSOR_LOOP" in [b["code"] for b in r["blocking_reasons"]] + + +def test_cursor_chain_broken(): + s = sess([ob(1, None, "c2", True, 10), ob(2, "WRONG", None, False, 10)]) + r = verify(s, {"type": "ALL"}) + assert "CURSOR_CHAIN_BROKEN" in [b["code"] for b in r["blocking_reasons"]] + + +def test_scope_hash_deterministic(): + assert scope_hash({"b": 1, "a": 2}) == scope_hash({"a": 2, "b": 1}) + assert len(scope_hash({})) == 64 + + +def test_evidence_digest_deterministic(): + s = sess([ob(1, None, None, False, 5)]) + assert evidence_digest(s) == evidence_digest(s) + + +def test_parser(): + assert parse_claim("There are no overdue invoices.")["type"] == "NONE" + assert parse_claim("These are all active customers.")["type"] == "ALL" + assert parse_claim("There are exactly 12 customers.")["type"] == "EXACT_COUNT" + assert parse_claim("This is the cheapest flight.")["type"] == "MIN" + assert parse_claim("This is the highest-value order.")["type"] == "MAX" + + +def test_health_and_verification(): + h = client.get("/health").json() + assert h["status"] == "ok" and "commit" in h + v = client.get("/.well-known/xagent-verification.json").json() + assert v["schemaVersion"] == 1 and v["commit"] == h["commit"] + + +def test_api_full_flow(): + sid = client.post("/v1/sessions", json={"resource_type": "invoice", "source": "demo-crm", + "scope": {"status": "unpaid"}}).json()["session_id"] + r1 = client.post(f"/v1/sessions/{sid}/observe", + json={"page_number": 1, "cursor_out": "c2", "has_more": True, "records_seen": 100}).json() + assert r1["coverage_status"] == "INCOMPLETE" and r1["next_expected_cursor"] == "c2" + v1 = client.post(f"/v1/sessions/{sid}/verify", json={"claim": {"type": "EXACT_COUNT", "value": 100}}).json() + assert v1["verdict"] == "UNPROVEN" + client.post(f"/v1/sessions/{sid}/observe", + json={"page_number": 2, "cursor_in": "c2", "has_more": False, "records_seen": 47}).json() + v2 = client.post(f"/v1/sessions/{sid}/verify", json={"claim": {"type": "EXACT_COUNT", "value": 147}}).json() + assert v2["verdict"] == "PROVEN" and v2["certificate"]["certificate_id"].startswith("proof_") + cert = client.get(f"/v1/sessions/{sid}/certificate").json() + assert cert["records_examined"] == 147 + assert client.post("/v1/claims/parse", json={"text": "There are no overdue tickets."}).json()["type"] == "NONE" diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docker-compose.yml b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docker-compose.yml new file mode 100644 index 0000000..47e9f4e --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docker-compose.yml @@ -0,0 +1,20 @@ +services: + backend: + build: ./backend + ports: + - "8100:8000" + environment: + GIT_COMMIT: ${GIT_COMMIT:-dev-local} + XAGENT_SLUG: ${XAGENT_SLUG:-team-exhaustive-gate} + depends_on: + - demo-crm + demo-crm: + build: ./examples/demo-crm + ports: + - "8101:8001" + frontend: + build: ./frontend + ports: + - "8102:80" + depends_on: + - backend diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/api.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/api.md new file mode 100644 index 0000000..a8bd257 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/api.md @@ -0,0 +1,11 @@ +# API + +- `GET /health` -> `{status, service, version, commit}` +- `GET /.well-known/xagent-verification.json` -> `{schemaVersion, slug, commit}` +- `POST /v1/sessions` `{resource_type, source, scope, pagination_type, snapshot_strategy}` +- `GET /v1/sessions/{id}` / `GET /v1/sessions/{id}/observations` +- `POST /v1/sessions/{id}/observe` `{page_number, cursor_in, cursor_out, has_more, records_seen, items?, scope?, snapshot_id?, authoritative_total?}` -> `{accepted, coverage_status, next_expected_cursor}` +- `POST /v1/sessions/{id}/failure` `{page_number, kind, message}` +- `POST /v1/sessions/{id}/verify` `{claim:{type, value?, field?, candidate_id?}}` -> `{verdict, blocking_reasons, required_next_actions, certified_value?, certificate?}` +- `GET /v1/sessions/{id}/certificate` +- `POST /v1/claims/parse` `{text}` -> `{type, resource, scope|field, value?}` diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/architecture.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/architecture.md new file mode 100644 index 0000000..e8769ba --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/architecture.md @@ -0,0 +1,14 @@ +# Architecture + +``` +Agent -> POST /v1/sessions (scope+pagination_type+strategy) + -> POST /v1/sessions/{id}/observe per page (cursor/offset/page/single) + -> POST /v1/sessions/{id}/failure on errors + -> POST /v1/sessions/{id}/verify {claim} -> PROVEN/UNPROVEN/CONDITIONAL + -> GET /v1/sessions/{id}/certificate (SHA-256 evidence digest) +``` + +Modules (`backend/app/services/`): `scope` (canonical hash), `pagination` +(chain/loop/gap/exhaustion), `obligations` (per-type requirements + reason codes), +`verifier` (pure deterministic verdict), `certificates`, `claim_parser` +(regex; LLM hook reserved). Storage: SQLite/SQLAlchemy, one `sessions` table. diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/demo.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/demo.md new file mode 100644 index 0000000..91834e2 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/demo.md @@ -0,0 +1,8 @@ +# Demo (3-5 min) + +1. `docker-compose up --build` (frontend :8102, gate :8100, CRM :8101). +2. Open dashboard, click **Run Demo: exactly-N**. +3. Page 1: 100 records, continuation YES. Claim "exactly 100" -> UNPROVEN / PAGINATION_NOT_EXHAUSTED. +4. Click **Fetch Next Page** x3 (347 records, 4/4, snapshot stable). +5. Verify "exactly 347" -> PROVEN + proof certificate. Hero: 100 ❌ -> 347 ✅. +6. Other scenarios: NONE+timeout (UNPROVEN/UNRESOLVED_FAILURE), cheapest product (UNPROVEN then PROVEN prod_008 price 3), snapshot_change (UNPROVEN/SNAPSHOT_CHANGED). diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/progress.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/progress.md new file mode 100644 index 0000000..ace1b28 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/progress.md @@ -0,0 +1,9 @@ +# Progress + +- [x] Phase 1: FastAPI + SQLite + health + xagent verification +- [x] Phase 2: evidence ledger (sessions/observations/failures/scope hash) +- [x] Phase 3: pagination engine (chain/loop/gap/exhaustion) + 15 pytest green +- [x] Phase 4-5: obligations (NONE/ALL/EXACT_COUNT/MIN/MAX) + verdict PROVEN/UNPROVEN/CONDITIONAL +- [x] Phase 6: demo-crm (347 invoices @100pp, 52 products @20pp, 7 chaos modes) +- [x] Phase 7: SHA-256 proof certificates (hero E2E verified) +- [x] Phase 8-10: dashboard (4 scenarios), `docker-compose up --build` verified (gate :8100, CRM :8101, frontend :8102; 15/15 pytest in container), README + demo guide diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/proof-model.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/proof-model.md new file mode 100644 index 0000000..0e6e666 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/docs/proof-model.md @@ -0,0 +1,10 @@ +# Proof model + +Global claims require global coverage. For NONE/ALL/EXACT_COUNT/MIN/MAX the gate +checks: SCOPE_STABLE (filter hash unchanged), NO_UNRESOLVED_FAILURES, +RESULT_SET_COMPLETE (last page has_more=false), NO_CURSOR_GAPS (chain/loop/dup +checks), SNAPSHOT_ACCEPTABLE (STRICT: any change blocks; BEST_EFFORT yields +CONDITIONAL), plus type-specific checks (count match, zero matches, candidate is +true extreme over observed field values). Missing snapshot info is a warning, +not a block. Verdicts are pure functions of ledger state — rerunning the same +evidence always yields the same verdict. diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/Dockerfile b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/Dockerfile new file mode 100644 index 0000000..4602513 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/Dockerfile @@ -0,0 +1,7 @@ +FROM python:3.12-slim +WORKDIR /code +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt +COPY main.py . +EXPOSE 8001 +CMD ["python","-m","uvicorn","main:app","--host","0.0.0.0","--port","8001"] diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/main.py b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/main.py new file mode 100644 index 0000000..0cf512e --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/main.py @@ -0,0 +1,68 @@ +"""Deterministic demo CRM: 347 invoices, 52 products, chaos modes.""" +import asyncio +from fastapi import FastAPI, Query +from fastapi.responses import JSONResponse + +app = FastAPI(title="Demo CRM", version="0.1.0") + +INVOICES = [{"id": f"inv_{i:04d}", "status": "unpaid" if i % 3 else "paid", + "amount": 50 + (i * 37) % 900} for i in range(1, 521)] # 520 total, 347 unpaid +UNPAID = [inv for inv in INVOICES if inv["status"] == "unpaid"] # exactly 347 +PRODUCTS = [{"id": f"prod_{i:03d}", "price": 5 + (i * 13) % 200} for i in range(1, 53)] +PRODUCTS[7]["price"] = 3 # cheapest: prod_008 +CUSTOMERS = [{"id": f"cust_{i:03d}", "active": bool(i % 2)} for i in range(1, 61)] + +PAGE = 100 +PRODUCT_PAGE = 20 + + +def paginate(rows: list, cursor: str | None): + start = int(cursor.split("_")[1]) if cursor and cursor.startswith("pg_") else 0 + chunk = rows[start:start + PAGE] + nxt = start + PAGE + out = f"pg_{nxt}" if nxt < len(rows) else None + return chunk, out, nxt < len(rows) + + +@app.get("/health") +async def health(): + return {"status": "ok"} + + +@app.get("/invoices") +async def invoices(cursor: str | None = None, status: str | None = Query(default=None), + mode: str = "normal"): + rows = [r for r in UNPAID if status is None or r["status"] == status] + pg = (int(cursor.split("_")[1]) // PAGE + 1) if cursor and cursor.startswith("pg_") else 1 + if mode == "timeout_page_3" and pg == 3: + await asyncio.sleep(12) + if mode == "rate_limit_page_2" and pg == 2: + return JSONResponse({"error": "rate limited"}, status_code=429) + snap = "snapshot_A" + if mode == "snapshot_change" and pg >= 3: + snap = "snapshot_B" + chunk, out, more = paginate(rows, cursor) + if mode == "broken_cursor" and pg == 2: + out = "pg_WRONG" + if mode == "duplicate_page": + out = cursor # loop: cursor never advances + more = True + total = len(rows) + (5 if mode == "wrong_total" else 0) + return {"records": chunk, "cursor_out": out, "has_more": more, + "snapshot_id": snap, "authoritative_total": total, "page": pg} + + +@app.get("/products") +async def products(cursor: str | None = None, mode: str = "normal"): + start = int(cursor.split("_")[1]) if cursor and cursor.startswith("pg_") else 0 + chunk = PRODUCTS[start:start + PRODUCT_PAGE] + nxt = start + PRODUCT_PAGE + out = f"pg_{nxt}" if nxt < len(PRODUCTS) else None + return {"records": chunk, "cursor_out": out, "has_more": nxt < len(PRODUCTS), + "snapshot_id": "snapshot_A", "page": start // PRODUCT_PAGE + 1} + + +@app.get("/customers") +async def customers(cursor: str | None = None): + chunk, out, more = paginate(CUSTOMERS, cursor) + return {"records": chunk, "cursor_out": out, "has_more": more, "snapshot_id": "snapshot_A"} diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/requirements.txt b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/requirements.txt new file mode 100644 index 0000000..babedd0 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/examples/demo-crm/requirements.txt @@ -0,0 +1,2 @@ +fastapi==0.116.1 +uvicorn[standard]==0.35.0 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/Dockerfile b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/Dockerfile new file mode 100644 index 0000000..a6c6f77 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/Dockerfile @@ -0,0 +1,3 @@ +FROM nginx:alpine +COPY index.html app.js /usr/share/nginx/html/ +EXPOSE 80 diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/app.js b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/app.js new file mode 100644 index 0000000..7ffe430 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/app.js @@ -0,0 +1,72 @@ +let G = () => document.getElementById('gate').value.replace(/\/$/, ''); +let C = () => document.getElementById('crm').value.replace(/\/$/, ''); +let SID = null, CURSOR = null, PAGE = 0, MODE = 'normal', RES = 'invoice', TOTAL = 0, CLAIM_N = 100; +async function jget(u) { const r = await fetch(u); if (!r.ok) throw new Error(r.status); return r.json(); } +async function jpost(u, b) { const r = await fetch(u, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(b) }); if (!r.ok) throw new Error(await r.text()); return r.json(); } +async function startDemo(mode) { + MODE = mode; RES = 'invoice'; PAGE = 0; CURSOR = null; TOTAL = 0; CLAIM_N = 100; + const s = await jpost(G() + '/v1/sessions', { resource_type: 'invoice', source: 'demo-crm', scope: { status: 'unpaid' } }); + SID = s.session_id; + document.getElementById('sid').textContent = SID; + document.getElementById('sess').style.display = 'block'; + document.getElementById('claimText').value = mode === 'timeout_page_3' ? 'There are no overdue invoices.' : 'There are exactly 100 unpaid invoices.'; + await fetchNext(); +} +async function startMinDemo() { + MODE = 'normal'; RES = 'product'; PAGE = 0; CURSOR = null; TOTAL = 0; + const s = await jpost(G() + '/v1/sessions', { resource_type: 'product', source: 'demo-crm', scope: {} }); + SID = s.session_id; + document.getElementById('sid').textContent = SID; + document.getElementById('sess').style.display = 'block'; + document.getElementById('claimText').value = 'prod_001 is the cheapest product.'; + await fetchNext(); +} +async function fetchNext() { + const ep = RES === 'invoice' ? '/invoices?mode=' + MODE : '/products'; + const url = C() + ep + (ep.includes('?') ? '&' : '?') + (CURSOR ? 'cursor=' + CURSOR : ''); + PAGE++; + try { + const d = await jget(url); + TOTAL += d.records.length; + const items = RES === 'product' ? d.records.map(r => ({ id: r.id, price: r.price })) : []; + const o = await jpost(G() + '/v1/sessions/' + SID + '/observe', + { page_number: PAGE, cursor_in: CURSOR, cursor_out: d.cursor_out, has_more: d.has_more, records_seen: d.records.length, items, scope: RES === 'invoice' ? { status: 'unpaid' } : {}, snapshot_id: d.snapshot_id, authoritative_total: d.authoritative_total }); + CURSOR = d.cursor_out; + update(o.pages_seen, TOTAL, d.has_more, d.snapshot_id); + } catch (e) { + await jpost(G() + '/v1/sessions/' + SID + '/failure', { page_number: PAGE, kind: 'timeout', message: String(e).slice(0, 200) }); + const s = await jget(G() + '/v1/sessions/' + SID); + update(s.observation_count, TOTAL, true, '—'); + document.getElementById('cursor').textContent = 'page ' + PAGE + ' failed (recorded)'; + } +} +function update(pages, recs, more, snap) { + document.getElementById('pages').textContent = pages + (more ? ' / ?' : ' / ' + pages); + document.getElementById('recs').textContent = recs; + document.getElementById('snap').textContent = snap || '—'; + const pct = more ? Math.min(90, Math.round(recs / 3.47)) : 100; + document.getElementById('cov').textContent = pct + '%'; + document.getElementById('bar').style.width = pct + '%'; + document.getElementById('cursor').textContent = more ? ('continuation: ' + CURSOR) : 'exhausted'; + jget(G() + '/v1/sessions/' + SID).then(s => document.getElementById('fails').textContent = s.failure_count); +} +async function verifyClaim() { + const text = document.getElementById('claimText').value; + const parsed = await jpost(G() + '/v1/claims/parse', { text }); + let claim = parsed; + if (parsed.type === 'EXACT_COUNT' && RES === 'invoice') claim = { type: 'EXACT_COUNT', value: TOTAL }; + if (parsed.type === 'MIN' || /cheapest/.test(text)) { + const obs = await jget(G() + '/v1/sessions/' + SID + '/observations'); + const items = obs.observations.flatMap(o => o.items || []); + const cheapest = items.length ? items.reduce((a, b) => a.price <= b.price ? a : b) : { id: 'prod_001' }; + claim = { type: 'MIN', field: 'price', candidate_id: cheapest.id }; + } + const v = await jpost(G() + '/v1/sessions/' + SID + '/verify', { claim }); + const el = document.getElementById('verdict'); + el.textContent = v.verdict; el.className = 'verdict ' + v.verdict; + document.getElementById('reasons').innerHTML = (v.blocking_reasons || []).map(b => `${b.code} ${b.message} → ${b.recommended_next_action}`).join('
') || 'no blocking reasons'; + document.getElementById('cert').textContent = JSON.stringify(v.certificate || v.evidence_summary, null, 2); + const sum = v.evidence_summary || {}; + document.getElementById('checks').innerHTML = ['RESULT_SET_COMPLETE', 'NO_UNRESOLVED_FAILURES', 'SCOPE_STABLE', 'SNAPSHOT_ACCEPTABLE'].map(k => + `${k}: ${(v.obligation_status || {})[k] || '?'}`).join(' '); +} diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/index.html b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/index.html new file mode 100644 index 0000000..f303058 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/source/frontend/index.html @@ -0,0 +1,57 @@ + + + + + +ExhaustiveGate — No exhaustive claim without exhaustive evidence + + + +
+
+

ExhaustiveGate

+

No exhaustive claim without exhaustive evidence.
Verify an AI agent actually covered the required result set before it says all, none, exactly N, cheapest, highest, lowest.

+
+
+
+
+

+ + +

+
+ +
+ + + diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/submission.json b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/submission.json new file mode 100644 index 0000000..b050037 --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/submission.json @@ -0,0 +1 @@ +{"apiBaseUrl": "https://mean-capital-republican-understood.trycloudflare.com/v1", "deploymentProofUrl": "https://mean-capital-republican-understood.trycloudflare.com/.well-known/xagent-verification.json", "healthCheckUrl": "https://mean-capital-republican-understood.trycloudflare.com/health", "name": "ExhaustiveGate", "reviewCommit": "79a2a1c5f8a746584996b88c11aee746e079d48f", "schemaVersion": 1, "slug": "shahadattest-exhaustive-gate", "sourceRepository": "https://github.com/ShahadatTest/exhaustive-gate"} \ No newline at end of file diff --git a/submissions/mcp-hackathon/shahadattest-exhaustive-gate/verification/README.md b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/verification/README.md new file mode 100644 index 0000000..b74b43a --- /dev/null +++ b/submissions/mcp-hackathon/shahadattest-exhaustive-gate/verification/README.md @@ -0,0 +1,53 @@ +# Verification evidence + +## Prerequisites + +- Review commit: `79a2a1c5f8a746584996b88c11aee746e079d48f` +- API base URL: `https://mean-capital-republican-understood.trycloudflare.com/v1` (local: `http://localhost:8100/v1`) +- Authentication: none + +## 1. Health check + +```bash +curl --fail --silent --show-error https://mean-capital-republican-understood.trycloudflare.com/health +``` + +Expected response: + +```json +{"status":"ok","service":"exhaustive-gate","version":"0.1.0","commit":"79a2a1c5f8a746584996b88c11aee746e079d48f"} +``` + +## 2. Deployment proof + +```bash +curl --fail --silent --show-error https://mean-capital-republican-understood.trycloudflare.com/.well-known/xagent-verification.json +``` + +Expected response: + +```json +{"schemaVersion":1,"slug":"shahadattest-exhaustive-gate","commit":"79a2a1c5f8a746584996b88c11aee746e079d48f"} +``` + +## 3. Capability call + +Create a session, observe the 4 demo pages, then verify (local evidence: page 1 +`UNPROVEN/PAGINATION_NOT_EXHAUSTED`; full set `PROVEN`, certified 347): + +```bash +SID=$(curl --fail --silent --show-error --request POST https://mean-capital-republican-understood.trycloudflare.com/v1/sessions \ + --header "content-type: application/json" \ + --data '{"resource_type":"invoice","source":"demo-crm","scope":{"status":"unpaid"}}' | python -c "import json,sys; print(json.load(sys.stdin)['session_id'])") +curl --fail --silent --show-error --request POST https://mean-capital-republican-understood.trycloudflare.com/v1/sessions/$SID/observe \ + --header "content-type: application/json" \ + --data '{"page_number":1,"cursor_out":"pg_100","has_more":true,"records_seen":100,"scope":{"status":"unpaid"},"snapshot_id":"snapshot_A"}' +curl --fail --silent --show-error --request POST https://mean-capital-republican-understood.trycloudflare.com/v1/sessions/$SID/verify \ + --header "content-type: application/json" \ + --data '{"claim":{"type":"EXACT_COUNT","value":100}}' +``` + +Expected: `{"verdict":"UNPROVEN","blocking_reasons":[{"code":"PAGINATION_NOT_EXHAUSTED",...}],"required_next_actions":["FETCH_NEXT_PAGE"],...}`. +After observing all pages, verifying `{"type":"EXACT_COUNT","value":347}` returns +`PROVEN` with a `certificate` object; `GET /v1/sessions/$SID/certificate` +returns the SHA-256 proof certificate. Redacted: no tokens or user data involved.