Skip to content

feat: add DSH source for searching ~/.dsh session history - #112

Closed
catoncat wants to merge 5 commits into
mainfrom
feat/dsh-source
Closed

feat: add DSH source for searching ~/.dsh session history#112
catoncat wants to merge 5 commits into
mainfrom
feat/dsh-source

Conversation

@catoncat

@catoncat catoncat commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a new public source dsh to Sherlog so shlog sync --source dsh can index zstd-compressed DSH session transcripts under ~/.dsh/sessions, then search them with find, read-page, read-range, list, and stats.

What changed

  • SourceId::Dsh with --source dsh and dsh:<native-id> session refs
  • Default root: ~/.dsh/sessions
  • zstd JSONL metadata/projection helpers
  • New rust/src/sources/dsh.rs adapter:
    • Accepts user/message only when data.source.kind == "user"
    • Accepts assistant/message text blocks only
    • Rejects injected plugin/skill-catalog/agent-instructions context, reasoning, tool calls/results
  • Full-only projection (DeltaUnsupported) like the Pi adapter
  • Latest-wins session title/model (first title is a raw paste; model can switch mid-session)
  • Shared zstd decode loop tolerates a torn final frame (in-flight writes); corrupt data stays a hard error
  • TS differential oracle stub so cross-source find remains consistent
  • Docs, skill reference, and tests updated

Tests

  • cargo fmt --all -- --check
  • cargo test --workspace --all-targets --all-features --locked
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo build --release --locked --bin shlog
  • npm run check
  • Release binary smoke with a zstd DSH fixture: sync/find/read-page

Support --source dsh for shlog by indexing zstd-compressed DSH session
JSONL under ~/.dsh/sessions. The adapter accepts only real user turns
(data.source.kind == "user") and assistant text blocks, rejecting
injected runtime/plugin context, reasoning, and tool payloads. Includes
zstd JSONL helpers, full-only projection, catalog wiring, config/identity
updates, TS oracle stub, docs, and Rust integration tests.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@catoncat

Copy link
Copy Markdown
Owner Author

Review follow-up: all findings from two review rounds are fixed in the working tree (commits incoming).

Round 1 — initial review (8 findings)

  • catalog.rs no longer accepts plain .jsonl for dsh (format drift now skips instead of failing strict sync — verified end-to-end: drift file planted in root → sync exit 0, coverage written).
  • dsh.rs acceptance tightened to match docs/SOURCE_CONTRACTS.md: session records require createdAt, user/assistant records require time; incomplete records rejected. Real-data proof: 111 sessions / 5,600+ messages, 0 rejected.
  • eval/acceptance-gate.ts: replaced the silent roots[sourceId as AcceptanceSourceId] cast with a checked lookup that throws a clear error; full dsh acceptance-gate coverage stays out because the TS oracle dsh adapter is a documented stub (can't read zstd).
  • Reverted the Chinese trigger list in skill-packages/sherlog/SKILL.md (d996932): it violates the no-trigger-list rule without invocation-eval evidence; net skill diff returns to zero.
  • Added tests: agent-instructions rejection, incomplete-record rejection, list/stats e2e assertions.
  • Dedup refactors: shared for_each_zstd_record decode loop + shared write_zstd_lines test helper.

Round 2 — deep integration review (5 findings)

  • Torn tail (the load-bearing one): DSH writes sessions incrementally and repairs torn tails itself; a truncated final zstd frame previously hard-failed the whole strict sync (incomplete frame). The shared decode loop now tolerates exactly UnexpectedEof + incomplete frame (projects the complete prefix; next sync replays in full after repair). Corrupt headers/blocks remain hard errors. Verified on a real 1.1 MB session: torn copy syncs clean, coverage written.
  • Latest-wins title/model: first session/title is a truncated paste of the first user message, later refined by the LLM (14/40 sampled sessions); model can switch mid-session (16/111). Both now take the latest record, matching the pi adapter convention. Spot-check: Investigate pi-rlm GitHub issue / deepseek-v4-pro.
  • Fallback identity: without a session record, the native id now falls back to the session directory name (the real dsh id) instead of the constant file stem session.jsonl.
  • Scan-failure messages no longer duplicate the operation prefix.

Verification

cargo fmt --check · cargo test --workspace --all-targets (189+12) · clippy -D warnings · cargo build --release · npm run check — all green. Release-binary smoke on the real ~/.dsh/sessions library: 105 files, 0 errors; find (CJK), read-page, read-range, anchor_not_found typed error, and evidenceRead argv replay all verified.

Deliberately not changed

  • Plain .jsonl in a dsh root stays fail-closed if injected via failure hooks (conservative over silent loss; revisit with real-world evidence).
  • The dsh inventory_metadata/project cascade mirrors the pi.rs two-read convention; the TS oracle dsh stub remains a documented won't-fix.

Merge-readiness: review-side this is sufficient. Procedurally the fixes need to land as commits + push, then CI green on the new head.

catoncat and others added 2 commits August 17, 2026 15:20
- catalog: reject plain .jsonl for dsh so format drift skips instead of failing strict sync
- dsh: require createdAt/time per SOURCE_CONTRACTS; latest-wins title/model; parent-dir fallback native id
- jsonl: tolerate torn final zstd frame (UnexpectedEof + incomplete frame); share one decode loop
- sync: stop duplicating the operation prefix in scan failure messages
- eval: checked acceptance fixture root lookup instead of a silent cast
- skill: revert Chinese trigger list (no invocation-eval evidence)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The legacy TS oracle wraps non-fresh coverage into a check_coverage_then_retry
find nextAction; native v8 emits find nextAction only for zero-result diagnosis
and surfaces the same state via coverageBySource. The dsh oracle stub can never
be synced in the fixture, so the oracle always emitted the advice for the
all-source and unscoped find cases, failing the gate since e40df2f.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@catoncat

Copy link
Copy Markdown
Owner Author

#112 cannot merge onto current `main`. After #114 the TypeScript CLI is gone and `rust/src` is now `src/`.

A rebased landing is in a follow-up PR that keeps the reviewed native DSH work and drops the oracle-only TS files. Please merge that PR and close this one as superseded.

@catoncat

Copy link
Copy Markdown
Owner Author

Superseded by #115 (rebased onto flattened crate after #114). Closing without merge.

@catoncat catoncat closed this Aug 17, 2026
@catoncat
catoncat deleted the feat/dsh-source branch August 17, 2026 08:17
pull Bot pushed a commit to CrazyForks/sherlog that referenced this pull request Aug 17, 2026
Replay reviewed PR catoncat#112 onto current main: rust/src → src, drop the
deleted TypeScript oracle edits, add dsh to eval/types, keep zstd
decode and native tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant