Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
133 lines (90 loc) · 6.57 KB

ARCHITECTURE.md

File metadata and controls

133 lines (90 loc) · 6.57 KB
Raw

MidSphere shipping architecture

Closed-loop context circuit-breaker for DataHub. This document describes only behaviors that ship today.

Summary

MidSphere runs a multi-stage closed loop on a root dataset URN: audit findings, selective blast radius, advisory quarantine, dry-run fix artifact, MBOM attestation path.

Primary workforce: Google Antigravity SDK for Python (PyPI: google-antigravity; imports Agent, LocalAgentConfig from google.antigravity). MidSphere registers five custom stage tools and runs on Vertex Gemini. Official DataHub MCP attaches when GMS is reachable. Stage tools call live MCP reads and writes when mutations are enabled.

After the first full name, docs may say Antigravity for the agent runtime. Code identifiers stay ANTIGRAVITY_* / antigravity_closed_loop.

When GMS is down or mutations are disabled, quarantine stays job-local with mutations_applied=false. Consumer Gate is MidSphere policy. It is not platform-wide MCP auto-denial.

Challenge fit

Agents That Do Real Work: closed loop with graph write-back when GMS and mutations are enabled.
Secondary: dry-run fix artifacts under examples/prs/.
Three packs: healthcare, ecommerce (PII), NYC taxi (freshness).

Runtime

POST /v1/jobs
  -> run_closed_loop
       PRIMARY: Google Antigravity SDK Agent (google-antigravity)
                tools = [midsphere_*] + optional MCP (mcp-server-datahub)
                stage tools -> get_lineage / add_tags / save_document when configured
       FALLBACK: deterministic pipeline.py if Agent fails
  -> JobArtifact
Stage Tool Shipping behavior
Audit midsphere_context_sentinel_audit Scenario findings (spine); live MCP get_lineage / list_schema_fields when GMS is up (telemetry + enrichment only)
Cascade midsphere_cascade_analyzer Selective nodes and risk from packs (healthcare 61 / ecommerce 88 / nyctaxi 65); live lineage annotated on root when MCP succeeds
Break midsphere_circuit_breaker_trip Live add_tags when mutations are on; else job-local
Fix midsphere_fix_architect Dry-run path under examples/prs/*
MBOM midsphere_mbom_engine Live save_document when mutations are on; else examples/mboms/*
Lift POST /v1/jobs/{id}/lift → lift_quarantine Live official MCP remove_tags when GMS + mutations; else honest sim remove_tags job-local. Gate always updates.

Shared packs: backend/agents/scenarios.py.

execution_mode values: antigravity_closed_loop | demo_closed_loop | live_mcp.

Selective halt

Scenario Quarantine target Open fork
Healthcare mart_billing mart_demographics
Ecommerce mart_customer_360 mart_product_catalog
NYC taxi mart_monthly_revenue mart_trip_distance_stats

Risk scores and paths are the closed-loop defaults defined in scenarios.py (not live warehouse computation).

DataHub MCP (real names only)

Read: search, get_entities, list_schema_fields, get_lineage, get_lineage_paths_between, get_dataset_queries

Write (TOOLS_IS_MUTATION_ENABLED=true): add_tags, remove_tags, add_structured_properties, save_document

Never invent datahub_* tools or MCP create_incident.
Client: uvx mcp-server-datahub@latest (midsphere_mcp/official.py).

Live GMS (hackathon)

Cloud Run does not use laptop localhost for GMS. Shipping approach:

  1. GCE VM runs official datahub docker quickstart.
  2. Firewall allows GMS port 8080 (and UI 9002 if needed).
  3. midsphere-api env: DATAHUB_GMS_URL=http://EXTERNAL_IP:8080, mutations enabled, ANTIGRAVITY_ATTACH_MCP=true.
  4. Cost control: instance schedule via scripts/gce_datahub_schedule.sh (start-now also re-points Cloud Run).

Heavier alternative: DataHub on GKE.

Consumer Gate

GET /v1/gate/{urn} against the latest job quarantine set.

Lift: POST /v1/jobs/{id}/lift always updates job-local gate (LIFTED when no entities remain). When GMS is reachable and TOOLS_IS_MUTATION_ENABLED=true, it also calls official MCP remove_tags (mirrors trip add_tags) and emits real TOOL/WRITE events; otherwise logs honest sim remove_tags.

GET /v1/gate-default-blocked is a healthcare mart_billing convenience probe only. After an ecommerce latest job it correctly returns OPEN for billing (quarantine target is mart_customer_360). Use GET /v1/gate/{urn} for the actual target.

GET /v1/jobs/latest returns 404 when the process store is empty (does not auto-run a pipeline). Clients fall back to sample fixtures or POST /v1/jobs.

Command Center (frontend)

Minimal monochrome shell (no emoji). Default layout:

Region Role
Top Scenario segments (Hospital, Retail, Taxi), API · Job / API · Sample / Offline + GMS, Run
Left Pipeline Run kind (Sample / Agent / Live MCP), metrics strip, stage stepper; Telemetry collapsed by default
Center Selective lineage graph; optional Timeline (narrative only)
Right Decision Findings, Gate (job view + optional live GET /v1/gate probe), Attestation (plus More); sticky Trip / Lift

Live MCP success is recorded under blast_radius.live_enrichment and telemetry. Selective impact flags remain pack-driven so demos stay reliable when GMS is partial.

VITE_API_BASE_URL is baked at build time to Cloud Run. Empty or incorrect base yields Offline status and sample fixtures.

Scenario roots for live Run match packs and demos: raw_patients, raw_customers, raw_yellow_trips (see docs/MANUAL_TEST.md).

Deploy isolation

Tree Target
frontend/ Vercel midsphere (VITE_API_BASE_URL to Cloud Run)
backend/ Cloud Run midsphere-api

Claims this project does not make

  • Platform-wide MCP auto-denial of quarantined entities
  • Multiple independent AI agents (one Agent, five stage tools)
  • Stock sample URNs always matching without ingest
  • Fix PRs always opened on GitHub (dry-run paths are first-class)
  • Live risk scores computed from warehouse metrics (scenario defaults apply)

Layout

frontend/     Command Center
backend/      FastAPI, Google Antigravity SDK, gate, MCP client
examples/     MBOMs, PR diffs, job fixture, sample telemetry
skills/       Upstream skill mirrors (PR #88)
scripts/      Bootstrap and GCE schedule helpers
ARCHITECTURE.md
README.md
LICENSE

OSS skills

Open PR: datahub-project/datahub-skills#88.