Clarify or expose a stable structured read contract for facts, evidence, proofs, history, and conflicts

[ansonnmm]

Summary

Utopia already stores and exposes a rich epistemic chain internally and through several structured HTTP endpoints:

entity
→ asserted fact
→ evidence / quote
→ source document
→ validity
→ history / correction
→ derived fact
→ proof
→ conflict / review state

However, the documented MCP read surface currently returns much of this information as formatted human-readable text rather than identity-rich structured data.

I would like to clarify what Utopia considers its supported external machine-readable contract for this information.

Tested version

Tested against stock Utopia at:

b03a0f20a3d053323a3ec2669c7c9b4aeb9a8e14

No source, schema, or migration modifications were made.

What I verified

Using the existing HTTP and MCP interfaces, I was able to verify that Utopia has enough underlying information to reconstruct:

• entity identity;
• asserted fact identity and world-time validity;
• evidence quote and chunk identity;
• source document identity and version;
• correction / history events;
• derived fact identity;
• ordered proof premises;
• conflicts / review items.

Most of this information is already available through structured /api/v1/... HTTP routes.

The uncertainty is whether those routes are intended to be stable external integration contracts.

Current MCP limitation

The documented MCP interface is clearly intended for external clients, but tools such as entity_facts primarily return formatted text.

For example, the result can describe validity and identify a derived result with something similar to:

[rule: ...]

but the external caller does not receive a machine-composable structure containing identities such as:

fact_id
evidence_id
document_id
derived_fact_id
proof / premise ids
conflict identity / status

Similarly, change/history output contains useful human-readable information, but does not expose enough stable identities to reliably join it back to the fact/evidence/proof chain.

This makes the MCP surface useful for an interactive agent, but difficult to use as a durable machine-to-machine knowledge interface without parsing formatted prose.

Questions for maintainers

Could you clarify which of the following is the intended direction?

1. Existing structured /api/v1 routes are public external contracts

If so:

• Are integrations expected to depend on the entity, fact, evidence, history, proof, graph, and review routes?
• Is there a compatibility or deprecation policy for those response shapes?
• Are opaque entity/fact/evidence/derived IDs intended to remain usable across versions?

2. MCP is the intended public integration contract

If so, would you consider exposing structured content alongside the human-readable text, including stable opaque identities and lineage relationships?

3. Another external API layer is planned

If neither of the above is intended as the durable contract, is there a planned structured read API for this use case?

Minimal information that seems sufficient

A useful structured external contract would not need to expose database rows or internal implementation details.

Opaque IDs plus typed relationships would be sufficient. For example:

EntityRef {
  id
  name
  type
}

AssertedFact {
  id
  subject
  predicate
  object / value
  valid_from
  valid_to
}

EvidenceRef {
  id
  fact_id
  quote
  chunk_id
  document_id
}

DerivedFact {
  id
  subject
  predicate
  object / value
  rule
}

Proof {
  derived_fact_id
  premises[]
}

HistoryEvent {
  id
  timestamp
  entity_id
  fact_id?
  event_type
}

Conflict {
  id
  fact_ids[]
  kind
  status
}

The exact schema above is only illustrative.

The important properties are:

• stable opaque identity;
• machine-composable lineage;
• asserted and derived facts remain distinguishable;
• evidence and source provenance remain recoverable;
• history/corrections can be joined back to the affected facts;
• conflicts can be addressed as structured objects rather than reconstructed from text.

Why this may be useful beyond one integration

One of Utopia's differentiating properties is that it preserves more than the current graph state.

It also preserves:

• evidence;
• provenance;
• temporal validity;
• changing belief/history;
• derivations and proof chains;
• conflicts;
• auditability.

A structured external read contract would allow downstream agents and applications to preserve those guarantees instead of flattening the result into text and then attempting to reconstruct structure from prose.

It would also provide a clean boundary between Utopia's internal schema—which can continue to evolve—and external integrations that should not depend directly on PostgreSQL tables, migrations, or Rust implementation details.

Related documentation

Current MCP documentation at the tested commit:

utopia/web/src/docs/mcp.md

Lines 20 to 52 in b03a0f2

	## The endpoint
	```
	POST /api/v1/kbs/{kb_id}/mcp
	Authorization: Bearer utp_pat_…
	Content-Type: application/json
	```
	One endpoint per knowledge base, speaking JSON-RPC 2.0 (MCP protocol version `2025-06-18`, stateless HTTP). The `kb_id` is in the base's URL in the browser: `/kb/{kb_id}/…`.
	Three methods are served:
	- `initialize` — capabilities and protocol version
	- `tools/list` — the tools below, with their JSON schemas
	- `tools/call` — run one
	## The tools
	| Tool | What it answers |
	|---|---|
	| `search_chunks` | Full-text + semantic search over the base's documents. Returns the six best-matching passages, each cut at 800 characters and carrying its `document_id`. Pass `as_of` to search the base as it stood at that moment — earlier versions, documents deleted since; full-text recall stays current, so hits are right but may be incomplete |
	| `get_document` | The full text of one document, all sections in order, by `document_id`. Use it when a search hit is the right document but the excerpt does not carry the answer. Capped at 24,000 characters, and says so when it cuts |
	| `find_entities` | Entities by (partial) name: id, type, and a disambiguator when several share a name |
	| `entity_facts` | One entity's facts with validity ranges. Pass `at` (a date) to see the world as of that day; this is the tool for "who was X in 2024". Pass `as_of` (a date or an RFC3339 moment) to see the facts **as the base held them then**, before later corrections, retractions and merges — "what did we have on record before the memo arrived". The two combine: `at` for the date asked about, `as_of` for when |
	| `changes` | What the graph learned or revised in a window of **record** time: asserted, corrected, rejected, merged. Needs no entity; use it when the question names a period, not a subject |
	| `search_docs` | Utopia's own manual, for questions about how the platform works. Never the user's documents |
	| `remember` | Record one sentence into the base's memory. **Needs a `write` token held by an editor**; a token without it does not see this tool in `tools/list`, and calling it anyway says why |
	The two time axes matter here. `at` reads **world time** (when something was true); `as_of` reads **record time** (what Utopia held at that moment, before it revised it), and `changes` lists what moved on that axis in a window. They are separate parameters on purpose: folded into one they would answer "what happened in March" with "what we learned in March", and both look plausible.
	## What an agent records waits for a nod
	`remember` stores the sentence immediately — searchable at once, attributed to the token's owner. The facts extracted from it do **not** enter the graph. They queue in Review as proposals, each shown beneath the sentence it came from, and a person confirms or rejects them one at a time.

Request

The immediate request is primarily clarification:

What external structured read surface, if any, should integrations treat as a supported compatibility boundary?

If MCP is intended to be that boundary, structured identity-rich responses for the epistemic chain above would close the current gap.

If the existing structured HTTP routes are already intended to be that boundary, documenting their compatibility expectations would also resolve the uncertainty.

[WaylandYang]

Thanks for laying this out so carefully. The short answer to your question: none of the three surfaces is documented as a compatibility boundary today, and that is the gap. Here is what each one is meant to be, and what changes.

The RDF export is the supported machine-readable read contract for the chain you describe. GET /api/v1/kbs/{id}/export?format=turtle|jsonld streams the whole base (decision record 0020). Every entity and fact has an addressable IRI, minted under urn:utopia:kb:{kb_id}:... from the row's uuid, so two exports a year apart line up; ?base= overrides the namespace for a deployment with a public address. A fact is an rdf:Statement carrying world time (schema:validFrom / validThrough with precision), record time (prov:generatedAtTime / prov:invalidatedAtTime), confidence, supersedes for a correction, the quoted sentence, and prov:wasDerivedFrom the documents it came from. A derived fact is a statement flagged derived, prov:wasGeneratedBy its rule, with prov:used on each premise, asserted or derived. Classes and relations keep the IRI they were imported with. That file is the boundary between our schema and an integration: the tables and the Rust types can move, the export cannot without a record saying why. It is written for a triple store, and it deliberately uses vocabularies a reader already has (PROV-O, schema.org, RDF reification) instead of a private JSON. We will say this in the docs, next to the MCP page you linked, so the question does not have to be asked again.

Three things on your list are missing from the export, and each becomes its own issue: conflicts and review state, the chunk identity behind a quote (today the quote and the document are there, the chunk is not), and fetching one entity's chain on demand (the export is whole-base; a SPARQL endpoint is the planned answer and is not built).

MCP stays the agent surface, and structured content beside the text is the right addition. The prose is deliberate: it is what a model reads well, and entity_facts is tuned for that. But the protocol version we speak (2025-06-18) defines structuredContent, and the read tools could return it alongside content: entity ids from find_entities; fact, derived and document ids with both clocks from entity_facts; document and chunk ids from search_chunks; fact ids from changes. Same opaque uuids as the export, so a client can join the two. It is a bounded change and we would take a PR for it: keep the text as it is, add the structure next to it, and cover each tool with a test that reads the ids back.

/api/v1 is the UI's API and is not a compatibility promise. Its shapes change with the interface, there is no OpenAPI description and no deprecation policy, and we would rather not pretend otherwise. The ids inside those responses are the same uuids the export and MCP use, so an id you keep stays usable; the response shapes around it do not.

On the derived-facts point you raised in #549: fixed in #557, and the record now says what a rewound proof can and cannot show.

[ansonnmm]

Thanks — this clarifies the integration boundary cleanly.

I’ll treat:

• RDF export as the supported machine-readable read contract;
• MCP as the agent-facing surface;
• /api/v1 as UI-internal rather than a compatibility boundary.

I’ve opened #564 for the conflict/review-state gap you identified. I’ll leave chunk identity, MCP structuredContent, and per-entity/SPARQL access separate for now rather than expanding this thread.

Thanks again for the detailed clarification.