Sqlite rebuild tier does not cross-check molt_count, so a stale sidecar silently seeds wrong session token totals

[huangzesen]

Summary

Tier 1 of the agent-session rebuild (_rebuild_via_sqlite in src/lingtai_kernel/agent_session.py) selects the latest psyche_molt boundary by ORDER BY ts DESC LIMIT 1 and never compares the boundary row's json_extract(fields_json,'$.molt_count') against the molt_count the caller passed in. If logs/log.sqlite lags logs/events.jsonl (importer behind after a crash, or a copied/restored agent_dir), the aggregate for an older molt generation — or even the lifetime total, when the sidecar has no boundary row at all — is returned labeled with the current molt_count, and _start (src/lingtai_kernel/base_agent/lifecycle.py) seeds restore_token_state from those wrong totals with no diagnostic. Filtering/cross-checking the boundary by molt_count and falling through to the tier-2 reverse scan on mismatch would make tier 1 self-validating, matching the module's own "events.jsonl is the source of truth" contract.

Narrative

The kernel distinguishes two "session" notions (src/lingtai_kernel/agent_session.py:1-27): the runtime session (a fresh in-memory object per process start/refresh) and the agent session, the mind segment bounded by molt_count. The agent session survives refresh/restart: on _start, the current generation's since-molt token aggregate is rebuilt from the durable trajectory, where "logs/events.jsonl is the source of truth for that rebuild" (module docstring, lines 16-18). Because a large events.jsonl must not be full-scanned in the normal case, the rebuild uses a three-tier strategy: an indexed sqlite sidecar first, then a bounded reverse JSONL scan, then a full scan as an explicit last resort.

Tier 1 is where the trust boundary is drawn wrong. The sqlite sidecar logs/log.sqlite is a derived index of events.jsonl — an importer copies events into an events table (src/lingtai_kernel/services/logging.py:266-274, columns include type, ts, fields_json, source_offset). Derived indexes can lag: the importer can be behind after a crash mid-run, or an agent_dir can be copied/backed-up/restored with a sidecar snapshot older than its events.jsonl. Yet _rebuild_via_sqlite treats whatever the sidecar contains as current:

# src/lingtai_kernel/agent_session.py:234-238
boundary_rows = query(
    "SELECT ts, source_offset, fields_json FROM events "
    f"WHERE type = '{MOLT_BOUNDARY_EVENT}' "
    "ORDER BY ts DESC LIMIT 1"
)

The function receives molt_count as a parameter (_rebuild_via_sqlite(query, molt_count)) and stamps it onto the returned AgentSession (molt_count=molt_count, line 267), but never checks that the boundary row it found actually belongs to that generation — even though both molt paths persist molt_count in the event fields, so it is sitting right there in fields_json:

# src/lingtai_kernel/intrinsics/psyche/_molt.py:434-441 (agent path; system path ~700-706 is identical)
agent._log(
    "psyche_molt",
    before_tokens=before_tokens,
    after_tokens=after_tokens,
    molt_count=agent._molt_count,
    ...
)

Concrete failure: an agent is at molt generation 5 (per its manifest, so BaseAgent.rebuild_agent_session() at src/lingtai_kernel/base_agent/__init__.py:1961-1965 passes molt_count=5), but the sidecar importer died after indexing generation 4's boundary. Tier 1 finds the generation-4 boundary, sums every llm_response row since that timestamp — which spans all of generation 4 plus whatever of generation 5 got indexed — and returns it as generation 5's aggregate with rebuild_tier="sqlite". There is a worse degenerate case already latent in the code: if the sidecar predates the first molt (no psyche_molt row at all) while the caller's molt_count is positive, boundary_ts stays 0.0 and the aggregate query (... AND ts >= 0.0, lines 251-261) sums every token event ever recorded — the lifetime total, attributed to the current molt. That is precisely the #679-class defect this module exists to fix, resurfacing through its own fast path (see the comment at src/lingtai_kernel/base_agent/lifecycle.py:170-181).

The consumer makes the damage durable and invisible. _start seeds the live counters from the rebuilt aggregate whenever the tier is not "none":

# src/lingtai_kernel/base_agent/lifecycle.py:183-194
agent_session = agent.rebuild_agent_session()
seeded = False
if agent_session is not None and agent_session.rebuild_tier != "none":
    agent.restore_token_state({
        "input_tokens": agent_session.input_tokens,
        ...
    })

From there the inflated numbers flow into the injected _meta.tool_meta.token_usage.session half that the agent itself reads to reason about its context economy, and into the agent_session_rebuilt log line — which happily records the wrong totals as fact. Nothing anywhere compares the sidecar's view of "current generation" against the caller's; there is no agent_session_rebuild_sqlite_stale diagnostic, no fall-through.

The design got here honestly: tier 1 was written for speed ("two indexed lookups, no scan") and the tier ladder was framed around availability — fall to tier 2 when the sidecar is absent or the query errors — not around consistency. But the module's contract already names events.jsonl as the source of truth, and tier 2 reads events.jsonl directly, so the fix is cheap and self-contained: a stale sidecar is detectable in the very row tier 1 already fetches (its fields_json carries molt_count), and detection should demote the rebuild to tier 2 exactly as if the sidecar were absent. Better looks like: tier 1 answers only when it can prove it is looking at the caller's generation; otherwise it logs why and steps aside.

Current behavior

• rebuild_agent_session_from_events (src/lingtai_kernel/agent_session.py:143-199) resolves a read-only sqlite query fn and, if available, returns whatever _rebuild_via_sqlite produces (lines 180-183). Fall-through to tier 2 happens only when the query fn is unresolvable or a query raises.
• _rebuild_via_sqlite (src/lingtai_kernel/agent_session.py:229-278) fetches the newest psyche_molt row without any molt_count predicate, reads ts/source_offset/initiator from it, and never inspects $.molt_count in fields_json (it already parses fields_json for initiator via _boundary_source_from_fields, lines 433-442). When no boundary row exists it proceeds with boundary_ts = 0.0 and boundary_source = "boot", so the token aggregate covers the whole table.
• The returned AgentSession is stamped with the caller's molt_count (line 267) regardless of which generation the data came from.
• _rebuild_via_sqlite receives no logger_fn, so it cannot emit a staleness diagnostic even if it detected one; logger_fn is threaded only to the tier-3 fall-through log (agent_session_rebuild_fullscan, lines 189-196).
• BaseAgent.rebuild_agent_session (src/lingtai_kernel/base_agent/__init__.py:1945-1967) installs the result on SessionManager, and _start (src/lingtai_kernel/base_agent/lifecycle.py:182-210) seeds restore_token_state from it and logs agent_session_rebuilt with the (possibly wrong) totals.
• Tiers 2 and 3 are not affected: they read events.jsonl directly, so their "latest boundary" is by construction the current generation.

Detailed implementation plan

1. Thread the logger into tier 1. In rebuild_agent_session_from_events (src/lingtai_kernel/agent_session.py:180-183), pass logger_fn through:

   session = _rebuild_via_sqlite(resolved_query, molt_count, logger_fn=logger_fn)

   Change _rebuild_via_sqlite's signature to (query, molt_count, *, logger_fn=None).

2. Add a fields-parsing helper next to _boundary_source_from_fields (src/lingtai_kernel/agent_session.py:433), e.g. _boundary_molt_count_from_fields(fields_json) -> int | None, returning the integer molt_count from the boundary row's fields_json or None when the field is absent/unparseable (legacy events). Consider refactoring so fields_json is parsed once and both initiator and molt_count are read from the same dict.

3. Cross-check in _rebuild_via_sqlite. After fetching boundary_rows (line 245), validate before running the aggregate query:

   if boundary_rows:
       row = boundary_rows[0]
       found = _boundary_molt_count_from_fields(row.get("fields_json"))
       if found is not None and found != molt_count:
           if logger_fn is not None:
               logger_fn(
                   "agent_session_rebuild_sqlite_stale",
                   expected_molt_count=molt_count,
                   sidecar_molt_count=found,
                   reason="boundary_molt_count_mismatch",
               )
           return None  # fall through to reverse scan (tier 2)
   elif molt_count > 0:
       # Caller says we have molted, sidecar has never seen a boundary:
       # the sidecar is stale (or from another agent_dir). Do NOT aggregate
       # ts >= 0.0 — that would attribute LIFETIME totals to this molt.
       if logger_fn is not None:
           logger_fn(
               "agent_session_rebuild_sqlite_stale",
               expected_molt_count=molt_count,
               sidecar_molt_count=None,
               reason="boundary_missing_for_positive_molt_count",
           )
       return None

   Note the query itself is deliberately left unfiltered rather than adding AND json_extract(fields_json,'$.molt_count') = {molt_count} to the WHERE clause: filtering would silently select an older matching row on weird trajectories (e.g. a reset manifest replaying molt counts) and would skip validation for legacy rows; fetching the true latest boundary and comparing keeps the check honest and keeps back-compat explicit.

4. Back-compat for legacy boundary rows. When found is None (a psyche_molt row whose fields_json predates the molt_count field, or unparseable JSON), keep today's behavior — trust the row and proceed — so old trajectories do not regress to slower tiers. Both current emit sites (src/lingtai_kernel/intrinsics/psyche/_molt.py:434-441 and ~700-706) already persist molt_count, so new trajectories always validate.

5. No schema or migration changes. The events table already stores fields_json (src/lingtai_kernel/services/logging.py:266-274) and the boundary event already carries molt_count; this is purely a read-side check. _rebuild_via_sqlite returning None already means "tier unavailable" to the caller (lines 180-183), so fall-through needs no new plumbing.

6. Docs. Update the tier-1 description in the rebuild_agent_session_from_events docstring (src/lingtai_kernel/agent_session.py:155-160), the module entries in src/lingtai_kernel/ANATOMY.md and src/lingtai_kernel/base_agent/ANATOMY.md, and docs/references/runtime-vs-agent-session-objects.md (§4) to state that tier 1 self-validates against molt_count and demotes to tier 2 on mismatch, logging agent_session_rebuild_sqlite_stale.

Risks and alternatives

• Performance on stale sidecars: a mismatch demotes the rebuild to the bounded reverse scan (tier 2). That is exactly the tier used today when the sidecar file is absent, is bounded to DEFAULT_MAX_SCAN_BYTES (8 MiB) of tail, and only triggers when the sidecar is genuinely wrong — correct-but-slower strictly beats fast-but-wrong for state that seeds live counters.
• Legacy rows without molt_count: treating found is None as "trust" preserves current behavior for old trajectories but leaves them unvalidated. The alternative (fall through whenever the field is absent) is safer but would permanently demote every legacy trajectory off tier 1; trust-with-log is the better default, and the diagnostic event makes any residual bad case findable.
• Residual staleness within the correct generation: if the sidecar has the current boundary but the importer is behind on token rows, tier 1 still undercounts slightly. This is a much milder failure (undercount within the right generation, self-healing as the importer catches up) than cross-generation attribution; a fuller fix would compare MAX(source_offset) in the sidecar against the events.jsonl size, which is worth a follow-up but out of scope here.
• Alternative: validate in the caller (lifecycle._start compares agent_session.molt_count against agent._molt_count) — does not work, because _rebuild_via_sqlite stamps the caller's molt_count onto the result, so the mismatch is unobservable downstream. The check must happen where the boundary row is in hand.
• Alternative: filter the SQL by molt_count (WHERE ... AND json_extract(...) = N) — simpler, but conflates "stale sidecar" with "legacy rows" (both yield zero rows), silently degrades legacy trajectories, and can match a stale duplicate boundary on replayed counters. Fetch-then-compare keeps the semantics explicit.
• Alternative: always run tier 2 as a verification of tier 1 — defeats the purpose of the indexed tier; the point of the cheap cross-check is to keep tier 1 O(1) while making it self-validating.

Testing plan

Add to tests/test_agent_session.py (fixtures for building a sidecar already exist alongside the tier-agreement tests):

• test_sqlite_tier_falls_through_on_stale_boundary_molt_count — sidecar contains a psyche_molt boundary with fields_json molt_count=1 plus generation-1 token rows; events.jsonl contains the generation-2 boundary and its token events; call rebuild_agent_session_from_events(..., molt_count=2); assert rebuild_tier == "reverse_scan" and the totals equal only the generation-2 events from the JSONL (not the sidecar's generation-1 sums).
• test_sqlite_tier_falls_through_when_boundary_missing_but_molt_count_positive — sidecar has token rows but no psyche_molt row; caller passes molt_count=1; assert tier 1 is skipped (result tier is reverse_scan/full_scan) and the result does not equal the lifetime sum of all sidecar token rows.
• test_sqlite_tier_used_when_boundary_molt_count_matches — sidecar boundary molt_count equals the caller's; assert rebuild_tier == "sqlite" and totals unchanged from today (no regression of the fast path).
• test_sqlite_tier_trusts_legacy_boundary_without_molt_count_field — boundary row whose fields_json has no molt_count key; assert rebuild_tier == "sqlite" (back-compat trust path).
• test_sqlite_stale_fallthrough_logs_diagnostic — inject a recording logger_fn; on mismatch assert exactly one agent_session_rebuild_sqlite_stale call with expected_molt_count, sidecar_molt_count, and reason fields.
• test_molt_count_zero_with_empty_sidecar_stays_boot — molt_count=0 and a sidecar with token rows but no boundary; assert current pre-first-molt behavior is preserved (tier 1 still answers, boundary_source == "boot").
• Wiring-level test in tests/test_agent_session_wiring.py: with a deliberately stale sidecar and a correct events.jsonl, assert the seeded token state after the lifecycle restore path matches the JSONL-derived since-molt totals.

Generated by a Claude Fable 5 deep review of the lingtai-kernel codebase.