Follow-up: large-scale research graph session design

[huangy22]

Follow-up: large-scale research graph session design

Context

This issue captures the design direction for the future large-scale graph-session mode. It is intentionally separate from the current Goal A work:

• Build a standalone gaia-research repo.
• Keep Gaia core and gaia-research connected through public APIs and plugin entry points.
• Preserve parity for the existing review-run workflow.
• Keep .gaia/research/** namespace ownership clear.

Goal A should not be considered blocked on implementing large-scale graph sessions. The split should only avoid architectural choices that would make graph sessions hard or impossible later.

Product Shape

Graph-session mode is for long-running agent-framework research, not short final-report generation. It should support:

• thousands to tens of thousands of durable nodes;
• node, edge, focus, obligation, and evidence-reference records;
• pause, inspect, resume, and continue;
• optional final report generation;
• continuation cost proportional to newly explored frontier/input, rather than repeatedly scanning the whole graph.

Core Design Direction

Use a single Gaia-native research workflow with a session execution layer:

research session kernel
  -> search accepted source/paper/chain -> gaia add / materialize
  -> obligation/focus/gap -> gaia inquiry
  -> assess relation hypothesis -> candidate_relation
  -> stable knowledge -> explicit promote + gaia build check

The session layer is not a second canonical knowledge store. Gaia package and inquiry state remain the live workspace. The session records execution state, cursors, indexes, provenance, repair context, and projection refs needed for continuation.

State Classes

Different research objects should enter different stability layers:

Object	Live projection	Session-private state	Promotion boundary
Accepted search source/paper/chain	gaia add / current materialization surface	search query, raw hit, dedup key, package ref	Already a package dependency or source package, not necessarily a stable claim
Obligation/focus/gap	gaia inquiry	why it was created, source refs, cursor, linked bucket/focus	Later may become stable question/claim only through explicit action
Assessment relation hypothesis	candidate_relation	grounding refs, validation status, repair history	Candidate relation is provisional
Stable claim/relation	Explicit promote plus gaia build check	source candidate id, decision, check result	Stable Gaia source only after promotion

Raw search hits, frontier queues, invalid candidates, provider drafts, repair contexts, and cursor/index data stay under .gaia/research/sessions/**.

Linear Growth Principle

Normal continuation should be incremental:

delta search -> delta landscape -> delta bucket/focus update -> bounded assess -> candidate relation

It must not run this on every step:

read all landscapes -> rebuild global field map -> recompute all focuses -> compare all relations -> rewrite full report

Full rebuild, global field-map rewrite, global contradiction scan, report synthesis, mass promotion, and full gaia build check are explicit checkpoint or maintenance operations. They may be non-linear, but they must be named and not hidden inside ordinary continuation.

Landscape

Landscape should be linear by processing each new search result once:

normalize hit
-> dedup by source/paper/package_ref key
-> create/update session node
-> optional Gaia projection
-> assign to field-map buckets
-> append event

Required indexes:

• seen_source_index
• paper_index
• package_ref_index
• node_index
• bucket_membership_index

Continuation should read the new frontier/input batch and these compact indexes, not all historical landscape artifacts.

Field Map

Current review-run field maps are one-shot synthesis artifacts. Graph-session field maps should be running state:

field bucket = incrementally maintained state

Each bucket should track:

• bucket id;
• label;
• criteria or semantic signature;
• counts;
• top-k exemplars;
• coverage status;
• open gaps;
• last updated cursor.

New nodes are assigned to existing buckets, new-bucket candidates, or an ambiguous assignment queue. Updating counters, exemplars, and coverage status is part of normal continuation. Renaming, merging, reclustering, and rewriting a field-wide narrative are checkpoint operations.

Focus

Focus should be updated from changed buckets and obligations, not recomputed globally every turn.

Trigger examples:

• bucket coverage changed;
• new controversy signal appeared;
• unresolved obligation stayed open;
• high-quality evidence touched an existing focus.

Useful indexes:

• focus_id -> bucket_ids
• bucket_id -> focus_ids
• obligation_id -> focus_id
• bounded priority queue for focus candidates

Normal continuation updates only affected focuses. Global reprioritization is a checkpoint operation.

Assessment Relations

Assessment must avoid all-pairs relation mining. It should use bounded evidence packets:

for each changed focus:
    select top K new/changed evidence
    produce candidate relations grounded in K
    validate refs
    write candidate_relation
    append assessment event

If K is fixed and each new node touches a bounded number of focuses, assessment cost stays proportional to the changed area.

Required Session Files

First implementation should define a minimal, versioned contract under:

.gaia/research/sessions/<session-id>/
  state.json
  events.ndjson
  frontier.jsonl
  nodes.jsonl
  edges.jsonl
  focuses.jsonl
  obligations.jsonl
  field_map.json
  checkpoints/

state.json is a compact summary and cursor/index pointer. Append-only records are the audit source. Derived indexes must be rebuildable by an explicit rebuild command.

Verifiers

Future graph-session implementation should include tests like:

• test_open_session_writes_state_and_schema_version
• test_submit_frontier_batch_appends_records_and_cursor
• test_resume_continues_from_saved_cursor
• test_search_projection_records_gaia_add_refs
• test_obligation_projection_updates_inquiry_state
• test_assess_projection_writes_candidate_relations
• test_stable_promotion_requires_explicit_gate
• test_field_map_updates_from_delta_records
• test_invalid_candidate_produces_repair_context
• test_incremental_continuation_does_not_scan_historical_records

The complexity verifier should seed a large historical log, submit a small new batch, and instrument storage reads. Normal continuation should read only state/cursor/index files and the new input batch. It should fail if it opens every historical node or edge record.

Explicit Non-goals For Goal A

Do not require the repo split to implement:

• large-scale graph-session execution;
• O(N)-verified field-map delta updates;
• global graph clustering;
• mass relation promotion;
• final report generation from a graph session.

Goal A should provide the extraction boundary and public connection points so this issue can be implemented in gaia-research without changing Gaia core internals again.

[huangy22]

Reopening because #768 only narrowed Goal A and moved graph-session design out of the split implementation. This issue remains the follow-up tracking home for the large-scale graph-session implementation idea.