Skip to content

feat(extractor): filter redundant ghost user_ids during cc_session sync - #1

Open
hrdAI3 wants to merge 1 commit into
mainfrom
fix/extractor-ghost-filter
Open

feat(extractor): filter redundant ghost user_ids during cc_session sync#1
hrdAI3 wants to merge 1 commit into
mainfrom
fix/extractor-ghost-filter

Conversation

@hrdAI3

@hrdAI3 hrdAI3 commented May 19, 2026

Copy link
Copy Markdown
Owner

Summary

Skip "redundant ghost" cc-session uploads from the Matrix-Riven collector so the workboard doesn't double-count the same physical CC session under two user_ids.

Background

Matrix-Riven's realtime path falls back to \${unix_user}@\${hostname} when git config user.email is unset (and pre-PR #3 over there, also when the digital-twin config had a real identity — that PR fixes new sessions but historical data + clients that haven't reinstalled yet still produce ghosts). On the collector this surfaces as same-machine pairs:

  • Stop hook → transcript → tagged hrdai@qq.com (real)
  • Realtime hooks → cc-status → tagged 19723@hut (ghost)

After cleanup the live /api/users list still has 4 sole-identity ghosts whose data only exists under the hostname form:

lv@lvjiawendeMacBook-Air.local              → 吕嘉文 (12MB+ transcripts here)
blink@BlinkdeMacBook-Air.local              → 孙润峰
zhangziyi@zhangziyideMacBook-Air-2.local    → 张紫翊 (16 transcripts here)
alexpeng@pengchengdeMacBook-Air.local       → 彭丞

These four MUST NOT be filtered — they ARE those people's only upload identity until they set git config user.email and reinstall the client.

What this does

Two pure helpers in src/lib/ghost_user.ts:

  • isGhostUserId(userId) — shape predicate. Real-email TLD allowlist + .local hostname suffix + bare-hostname (no dot) classification. Conservative on unknown TLDs (leans real).
  • canonicalSessionId(rawId) — strips the trailing .cc-status the collector appends to snapshot listing entries.

syncCcSessions gets a pre-pass:

  1. List sessions for every non-ghost user inside the lookback window.
  2. Collect transcript session_ids (the entries WITHOUT .cc-status suffix) into a global Set.

Main loop then for each ghost user's session: if canonicalSessionId(sess.id) is in that Set → skip ingestion (don't fetch raw, don't emit events, don't write to events.jsonl), bump a new summary.redundantGhostsSkipped counter, advance lastMtime so we don't re-evaluate the same entry next sync.

Sole-identity ghosts (their session_ids have no non-ghost peer) pass through untouched and emit cc.* events under their hostname-form user_id, same as before.

Tests

8 cases in tests/lib/ghost_user.test.ts — real email TLDs (qq.com / outlook.com / 163.com / nb-ai.com / gmail.com / GitHub users.noreply.github.com), bare hostname (19723@hut), .local mDNS (blink@..., lv@..., zhangziyi@..., alexpeng@...), edge cases (empty, no-at, trailing-at), and the unknown-TLD conservative-lean (person@company.example → not flagged).

All 8 pass.

bun run test shows 5 pre-existing failures (bootstrap/extract.parseOrgChart, evolution/diff.path-allowlist, lib/agents.writeProfile+updateState) — confirmed unrelated by stash-and-rerun on clean main.

Not done in this PR

  • Backfilling the existing redundant cc-status events already in events.jsonl from earlier syncs. Pre-existing data is left alone; the filter only affects future syncs. If the events.jsonl is large enough that the historical ghost noise matters, that's a separate one-shot script.
  • Mapping ghost user_id → real Chinese name. Out of scope here; lives in the team's identity registry if/when added.

Test plan

  • bun run typecheck — no new errors in changed files (pre-existing errors in vector_match + workboard are unrelated)
  • bun run test tests/lib/ghost_user.test.ts — 8/8 pass
  • Stash-and-rerun confirms unrelated test failures are pre-existing
  • Manual: next live sync run should report redundantGhostsSkipped > 0 if any ghost-paired sessions land in window

🤖 Generated with Claude Code

Matrix-Riven's getUserId() in the realtime path falls back to
`${unix_user}@${hostname}` when `git config user.email` is unset on the
client machine. Pre-PR #3 (libz-renlab-ai/Matrix-Riven), that fallback
ALSO fired even when `~/.riven/digital-twin.json` carried a real
identity.user_id — producing same-machine pairs where the Stop hook
(transcript upload) tagged data with the real email and the realtime
hooks (cc-status snapshots) tagged data with the hostname form.

On the collector this surfaces as "ghost user_id" directories sitting
beside real-email directories. Two flavors:

  1. Redundant ghost — same session_ids as a real-email peer on the
     same machine; ghost only holds cc-status fragments, real peer
     holds the full transcript. Ingesting the ghost double-counts.

  2. Sole-identity ghost — that hostname form IS the person's only
     upload identity (their git config user.email genuinely unset,
     they never ran `riven digital-twin login`). Their transcripts
     ONLY exist under the ghost id. Must be preserved.

Real-world snapshot taken from prod (192.168.22.88:8933):

  19723@hut                                    → redundant (paired with
                                                  horton2048@users.noreply.github.com)
  lv@lvjiawendeMacBook-Air.local               → sole identity (12MB+ transcripts)
  blink@BlinkdeMacBook-Air.local               → sole identity
  zhangziyi@zhangziyideMacBook-Air-2.local     → sole identity (16 transcripts)
  alexpeng@pengchengdeMacBook-Air.local        → sole identity

Implementation:

  - `src/lib/ghost_user.ts` — pure shape predicate `isGhostUserId()`
    (allowlist of email TLDs + `.local` hostname suffix) and a
    `canonicalSessionId()` that strips the `.cc-status` listing
    suffix the collector uses to distinguish snapshot entries from
    transcripts.
  - `syncCcSessions` adds a pre-pass scanning all non-ghost users'
    transcript session_ids (NOT cc-status entries) into a global
    Set. Main loop then skips any ghost session whose canonical id
    is in that Set — emitting nothing to events.jsonl for it. The
    summary surfaces a new `redundantGhostsSkipped` counter.
  - Sole-identity ghosts pass through untouched.

Tests: 8 cases on `isGhostUserId` + `canonicalSessionId` covering
real emails (qq/outlook/163/nb-ai/gmail/github noreply), bare-hostname
fallback (`19723@hut`), `.local` mDNS, edge cases, and the unknown-TLD
conservative-lean.

All 8 new tests pass; pre-existing test failures on main
(bootstrap/extract, evolution/diff, lib/agents) are unrelated to this
change — confirmed by stash-and-rerun comparison.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant