Skip to content

Feat/eng 2971#721

Closed
cuongdo-byterover wants to merge 5 commits into
proj/analytics-system-tool-modefrom
feat/ENG-2971
Closed

Feat/eng 2971#721
cuongdo-byterover wants to merge 5 commits into
proj/analytics-system-tool-modefrom
feat/ENG-2971

Conversation

@cuongdo-byterover
Copy link
Copy Markdown
Collaborator

Summary

  • Problem:
  • Why it matters:
  • What changed:
  • What did NOT change (scope boundary):

Type of change

  • Bug fix
  • New feature
  • Refactor (no behavior change)
  • Documentation
  • Test
  • Chore (build, dependencies, CI)

Scope (select all touched areas)

  • TUI / REPL
  • Agent / Tools
  • LLM Providers
  • Server / Daemon
  • Shared (constants, types, transport events)
  • CLI Commands (oclif)
  • Hub / Connectors
  • Cloud Sync
  • CI/CD / Infra

Linked issues

  • Closes #
  • Related #

Root cause (bug fixes only, otherwise write N/A)

  • Root cause:
  • Why this was not caught earlier:

Test plan

  • Coverage added:
    • Unit test
    • Integration test
    • Manual verification only
  • Test file(s):
  • Key scenario(s) covered:

User-visible changes

List user-visible changes (including defaults, config, or CLI output).
If none, write None.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording

Checklist

  • Tests added or updated and passing (npm test)
  • Lint passes (npm run lint)
  • Type check passes (npm run typecheck)
  • Build succeeds (npm run build)
  • Commits follow Conventional Commits format
  • Documentation updated (if applicable)
  • No breaking changes (or clearly documented above)
  • Branch is up to date with main

Risks and mitigations

List real risks for this PR. If none, write None.

  • Risk:
    • Mitigation:

…sk types

Adds curate-tool-mode, query-tool-mode, dream-scan, dream-finalize to
TaskTypes + TASK_TYPE_VALUES. After ENG-2925's rename the daemon
dispatches these four types but the analytics enum still predates v4.0
— task_created / task_completed / task_failed silently rejected every
tool-mode emit at the Zod boundary.

- TaskTypes keeps the legacy 'curate' / 'query' / 'dream' values for
  back-compat with any constructor still building those payloads
- per-event schemas (task_created / task_completed / task_failed) pick
  up the new types automatically via z.enum(TASK_TYPE_VALUES)
- M12 per-flavor schemas (curate_run_completed / query_completed) still
  hardcode their own literals and continue to reject tool-mode types
  here — M14.2 migrates them to the canonical enum as a follow-up

TDD:
- new task-types tests assert TaskCreated/Completed/Failed accept all
  four new types
- regression tests pin the M12 schemas' continued rejection so M14.2
  has a clear flip point
…sk_type to TASK_TYPE_VALUES

Migrates the two M12 per-flavor schemas from hardcoded literal task_type
values to the canonical TASK_TYPE_VALUES enum so v4.0 tool-mode tasks
round-trip the wire boundary instead of being silently Zod-rejected
inside AnalyticsHook.

- curate_run_completed: z.enum(['curate', 'curate-folder']) →
  z.enum(TASK_TYPE_VALUES). curate-tool-mode payloads now validate.
- query_completed: z.literal('query') → z.enum(TASK_TYPE_VALUES).
  query-tool-mode payloads now validate.

The schemas no longer structurally constrain task_type to the curate
or query family; the hook is trusted to only emit each event for the
right flavor. Docblocks call out the widening for the next maintainer.

TDD:
- curate-run-completed.test asserts curate-tool-mode + legacy values
  both succeed; an unknown task_type still rejects
- query-completed.test mirrors the same coverage for query-tool-mode
- task-types.test M14.1 regression assertions flipped from rejection
  to acceptance for the M12 schemas; M14.1 docblock comments updated

Without this, M14.3's hook code would land but the M12 emits that fire
alongside the new generic task_* emits would silently disappear on
every tool-mode task — that's the bug operators noticed in Mixpanel.
…ed in AnalyticsHook

Adds the three generic funnel-event emits described in M14's customer
ask. Every daemon task (curate / curate-folder / curate-tool-mode
(aliased from curate-html-direct) / query / query-tool-mode /
dream-scan / dream-finalize / search) now produces, in order:

  task_created      onTaskCreate (funnel entry — unconditional)
  ...optional per-op + M12 per-flavor terminal emit
  task_completed    onTaskCompleted (terminal-event-last on success)
  task_failed       onTaskError / onTaskCancelled (terminal-event-last)

Three coupled changes shipped together so the wire stays consistent:

1. analytics-hook.ts grows three emits and a `toAnalyticsTaskType`
   alias-translator. The daemon still dispatches the pre-ENG-2925 name
   `curate-html-direct`; analytics canonicalises it to the post-rename
   `curate-tool-mode` so the wire enum matches TASK_TYPE_VALUES. Once
   ENG-2925 lands, the alias becomes a no-op identity.
2. CURATE_TASK_TYPES + QUERY_TASK_TYPES gain the tool-mode names so M12
   per-flavor state init kicks in for tool-mode curates / queries. M12
   counters stay all-zero on tool-mode today (no LLM tool calls fire) —
   that's a separate follow-up (FU-1 in plans/analytics-m14/follow-ups).
3. Both M12 payload builders route `task_type` through the same alias
   so curate-tool-mode tasks emit `task_type='curate-tool-mode'` on the
   curate_run_completed / query_completed events too.

TDD:
- new analytics-hook-m14.test.ts: 15 simulation tests covering every
  task type, the curate-html-direct → curate-tool-mode alias, has_files
  / has_folder semantics, both terminal paths, and the ordering invariant
- existing analytics-hook.test.ts updated to filter out the new generic
  emits via `filterM12()` so M12-focused assertions keep their intent
- integration stress tests updated for the new 13-event sequence
  (task_created → ops → curate_run_completed → task_completed)
- repo full suite: 9121 passing, 0 failing
…ords/tags, structured related_paths

Tightens the curate / query M12 payloads per code review:

curate_operation_applied:
- rename absolute_path → relative_path (project-relative via path.relative
  against the task's projectPath; falls back to identity when projectPath
  is unset so search tasks at the daemon root still emit a usable string)
- keywords / tags promoted from optional to required arrays (default [])
  so the wire shape stays uniform regardless of frontmatter read success

query_completed.read_paths_with_metadata:
- same absolute_path → relative_path rename
- same keywords / tags promotion (required arrays, default [])
- flat related: string[] → structured related_paths: [{relative_path,
  keywords, tags}] so each linked topic carries its own metadata slot.
  Keywords/tags default to [] until a future FU cascade-reads each
  linked file's frontmatter — the wire shape doesn't need to change
  when that lands.

Hook implementation:
- new `toRelativePath(filePath, projectPath)` helper using node:path.relative
- CurateTaskAnalyticsState stores projectPath captured at onTaskCreate so
  per-op emits can relativize without threading task through processToolResult
- all four payload sites (curate-op, curate-run-completed, query-completed
  read_paths_with_metadata) route through the helper

Inspection test added at test/unit/.../analytics-hook-toolmode-inspection
that pretty-prints every event + payload for curate-tool-mode / query
/ query-tool-mode flows — gives PMs a single place to verify the wire
shape end-to-end.

Privacy win: relative paths drop the /Users/{name} prefix from every
file-touched event, keeping host-identifiable PII off the analytics
wire while still letting PMs reason about which file inside a project
an operation touched.

Tests: full repo 9132 passing.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 27, 2026

Claude finished @cuongdo-byterover's task in 6m 12s —— View job


Review summary

Overall score: 3.5 / 5 — Solid M14 follow-up work with good test coverage for the new task_* funnel emits and per-task FIFO async correctness. Held back from a higher score by one wire-format-breaking change that needs explicit backend coordination, an unsafe as TaskType cast that papers over a real failure mode, and a counter/emit-stream inconsistency on filePath-less ops.

What's strong

  • AnalyticsHook async lifecycle is tightly reasoned. The per-task FIFO chain (pendingByTask) with a never-rejecting tail + a returned next that preserves rejection is exactly the right shape for ordered emits without poisoning the chain. The integration stress test (analytics-hook-async-stress.test.ts) drives 50 ops + multi-task interleaving through the real TaskRouter and verifies arrival-order + terminal-last sequencing. Good defense-in-depth.
  • M14.3 funnel emits placement. TASK_CREATED fires before M12 state init so it lands even if downstream throws; TASK_COMPLETED / TASK_FAILED fire after per-flavor emits so terminal-last ordering is preserved. The cancellation→task_failed mapping matches the schema docblock.
  • Test coverage on the M14 hot paths is thorough (analytics-hook-m14.test.ts covers all four task-lifecycle combinations × curate/query/dream-scan/search).
  • TASK_TYPE_VALUES consolidation removes the duplicated literal arrays across curate-run-completed / query-completed / task-* schemas. One-place-to-update for new daemon task types is the right invariant.

Issues raised inline

# Severity File What
1 concern analytics-hook.ts:31-34 daemonType as TaskType cast bypasses validation; a new daemon TaskTypeSchema value not mirrored in TASK_TYPE_VALUES would silently emit an unvalidatable payload. Violates CLAUDE.md ("Avoid as Type assertions").
2 concern analytics-hook.ts:43-50 toRelativePath doesn't fully neutralise host-path PII — paths outside the project return ../../Users/.... Schema header sells this change as "privacy + portability"; reality is only partial.
3 blocking curate-operation-applied.ts:19-32 Wire-format breaking change: absolute_pathrelative_path, keywords/tags optional → required, schema is .strict(). Needs explicit backend coordination — PR body should confirm telemetry backend deploy ordering + rollback plan. Same applies to query-completed.read_paths_with_metadata.
4 concern analytics-hook.ts:510-516 When op.filePath is missing, the type counter bumps but no per-op emit fires → curate_run_completed.operations_added (etc.) will diverge from COUNT(*) curate_operation_applied. Move the early continue before the switch, or emit with a sentinel.
5 nitpick analytics-hook-toolmode-inspection.test.ts:64-74 dumpEvents writes ~8 console.log lines per test × 6 tests = ~200 lines of noise in every npm test run. CLAUDE.md: "Suppress console logging in tests." Either gate on env var, mark suite as .skip by default, or delete if it was PR-review scaffolding.

Things I didn't flag inline but worth mentioning

  • The class docblock at analytics-hook.ts:115-130 only mentions M12.2 / M12.3 — could be updated to call out the M14.3 funnel emits.
  • Asymmetric projectPath capture: curate state snapshots it at onTaskCreate; query state pulls from task.projectPath at terminal time. Harmless today (task.projectPath isn't mutated mid-task) but the asymmetry is surprising.
  • dream is in TASK_TYPE_VALUES (task-types.ts:18) but not in the daemon's TaskTypeSchema (schemas.ts:427-436). Dead value in the analytics enum — fine, but the task-types.ts docstring at line 12 says "Mirrors the daemon's TaskInfo.type union" which is no longer literally true.
  • Curate-log-handler now creates log rows for curate-html-direct tasks (curate-log-handler.ts:58), but tool-mode curate doesn't fire onToolResult yet — these entries persist with all-zero summaries. The toolmode-inspection test calls this out as a known FU-1 deferred fix, which is reasonable.

Checklist

  • Read changed files
  • Analyze analytics-hook.ts changes (M14.3 funnel emits + relative-path conversion + per-task FIFO)
  • Analyze event schema changes (wire-format break)
  • Analyze test coverage
  • Post inline comments for specific issues (5 inline comments)
  • Post summary with score
    · Branch: feat/ENG-2971

Comment on lines +31 to +34
function toAnalyticsTaskType(daemonType: string): TaskType {
if (daemonType === 'curate-html-direct') return TaskTypes.CURATE_TOOL_MODE
return daemonType as TaskType
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (concern, non-blocking): the daemonType as TaskType cast bypasses validation. If a new daemon TaskTypeSchema value is added (e.g. a future vc-commit) but not mirrored in TASK_TYPE_VALUES, this silently emits a payload that will be rejected by the per-event schemas downstream (since they use z.enum(TASK_TYPE_VALUES)). Per CLAUDE.md: "Avoid as Type assertions — use type guards or proper typing instead."

A safer pattern:

const TASK_TYPE_SET: ReadonlySet<string> = new Set(TASK_TYPE_VALUES)
function toAnalyticsTaskType(daemonType: string): TaskType | undefined {
  if (daemonType === 'curate-html-direct') return TaskTypes.CURATE_TOOL_MODE
  return TASK_TYPE_SET.has(daemonType) ? (daemonType as TaskType) : undefined
}

…and have emit skip the call when task_type is undefined (with a processLog for visibility). Today the daemon's TaskTypeSchema only diverges by curate-html-direct, so the practical fallout is bounded — but the comment promises this becomes a "no-op identity" once rename lands, while in practice it would still need the safety net.

Comment on lines +43 to +50
function toRelativePath(filePath: string, projectPath?: string): string {
if (!projectPath) return filePath
const rel = relativePath(projectPath, filePath)
// `path.relative` returns '' when paths are identical — defensively
// surface a leaf token rather than emit a zero-length wire string that
// would fail `z.string().min(1)`.
return rel === '' ? '.' : rel
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (concern): The PII-removal claim is only partly true. path.relative('/project', '/Users/dev/foo.md') returns '../../Users/dev/foo.md' — the full host path is still on the wire, just rebased. The tests at analytics-hook.test.ts:149 (relative_path === '../a.md') and analytics-hook.test.ts:336 (['../a.md', '../b.md', '../c.md']) confirm this — these payloads still leak the parent dir info.

In real curate runs the affected files almost always live inside projectPath, so this only bites pathological inputs. But the comment header on the schema (curate-operation-applied.ts:13) sells this change as "privacy + portability across hosts" — the privacy claim doesn't hold when a path can escape the project root.

Two options worth considering:

  1. Clamp escapes: if rel.startsWith('..'), emit a marker like <outside-project> (or null) instead of the host-leaking string.
  2. Document the residual leak explicitly in the schema header.

Also: the '.' fallback emits a directory marker for a file operation — surprising shape for downstream consumers, but currently unreachable (filePath always points to a file).

Comment on lines 19 to 32
export const CurateOperationAppliedSchema = z
.object({
absolute_path: z.string().min(1),
confidence: z.enum(['high', 'low']).optional(),
impact: z.enum(['high', 'low']).optional(),
keywords: z.array(z.string().max(256)).max(50).optional(),
keywords: z.array(z.string().max(256)).max(50),
knowledge_path: z.string().min(1),
needs_review: z.boolean(),
operation_type: z.enum(['ADD', 'UPDATE', 'DELETE', 'MERGE', 'UPSERT']),
related: z.array(z.string().max(256)).max(50).optional(),
tags: z.array(z.string().max(256)).max(50).optional(),
relative_path: z.string().min(1),
tags: z.array(z.string().max(256)).max(50),
task_id: z.string().min(1),
})
.strict()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking) — wire-format breaking change:

This rev does three things on the same emit:

  • renames absolute_pathrelative_path
  • promotes keywords / tags from optional → required
  • keeps .strict() so unknown extras (incl. the old absolute_path) are rejected

If the analytics backend (telemetry-dev / prod) was previously consuming absolute_path or treating keywords / tags as nullable, this PR will start dropping rows post-deploy until the backend is updated in lockstep. The PR description doesn't mention coordination with the telemetry team.

Please confirm in the PR body:

  1. Backend has been (or will be) updated to read relative_path and require keywords/tags.
  2. Whether any backfill is needed for existing dashboards.
  3. Roll-forward / roll-back plan if the deploy ordering slips.

The same review tightening applies to query-completed.ts (read_paths_with_metadata[*] entries).

Comment on lines +64 to +74
const dumpEvents = (label: string, trackStub: sinon.SinonStub): void => {
console.log(`\n┌─ ${label} ${'─'.repeat(Math.max(0, 70 - label.length))}`)
for (const [i, call] of trackStub.getCalls().entries()) {
const eventName = call.args[0] as string
const props = call.args[1] as Record<string, unknown>
console.log(`│ [${i}] ${eventName}`)
console.log(`│ ${JSON.stringify(props, null, 2).replaceAll('\n', '\n│ ')}`)
}

console.log(`└${'─'.repeat(72)}\n`)
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (nitpick): dumpEvents writes 8+ console.log lines per test, and the file calls it from 6 of the 7 test cases. Per CLAUDE.md: "Suppress console logging in tests to keep output clean."

If the dumps are scaffolding for manual review against a fixed event shape that you'll keep eyeballing, that's fine — but please mark the suite as inspection-only (e.g. .skip by default + a script in package.json to run it), or guard dumpEvents on process.env.BRV_DUMP_ANALYTICS_EVENTS. As-is this adds ~200 lines of noise to every npm test run.

If the dumps were only to validate the M14 wire shape during PR review and aren't intended long-term, consider deleting dumpEvents and keeping only the expect(...) assertions that already cover the same ground in analytics-hook-m14.test.ts.

Comment on lines +510 to +516
keywords: frontmatter.keywords ?? [],
knowledge_path: op.path,
needs_review: op.needsReview ?? false,
operation_type: op.type,
...(frontmatter.related ? {related: frontmatter.related} : {}),
...(frontmatter.tags ? {tags: frontmatter.tags} : {}),
relative_path: toRelativePath(op.filePath, state.projectPath),
tags: frontmatter.tags ?? [],
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (concern): When op.filePath is missing, processToolResult continues out at line 498 — but only AFTER the switch at lines 464–490 has already bumped the type counter. The aggregate curate_run_completed.operations_added (etc.) will count an op for which no curate_operation_applied event was ever emitted.

A downstream dashboard that reconstructs counts from curate_operation_applied (e.g. COUNT(*) GROUP BY operation_type per task_id) will diverge from the aggregate counter on curate_run_completed.

Two reasonable fixes:

  1. Skip the counter bump too: move the if (!op.filePath) continue before the switch (~line 463).
  2. Emit with a sentinel like relative_path: '<unknown>' so the per-op stream stays complete.

The comment at line 497 explicitly says these ops are rare, so (1) is the simplest reconciliation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant