Skip to content

Memory quality: generalizability-scoped surfacing and auto-consolidation #373

@toejough

Description

@toejough

Problem

Narrow memories clutter surfacing across projects, and clusters of similar narrow memories dilute each other's signal.

Sub-features

1. Cross-project generalizability penalty

When surfacing memories, penalize relevance score for memories from a different project based on generalizability:

  • generalizability 5 (universal): no penalty
  • generalizability 4 (similar projects): small penalty
  • generalizability 3 (moderate): moderate penalty
  • generalizability 2 (narrow): large penalty
  • generalizability 1 (this project only): near-zero surfacing cross-project

Current state: project_slug and generalizability fields are written to TOML at extraction time, but NOT read back into the Stored struct at surfacing time. No penalty logic exists in the scoring pipeline. surface.Options lacks a CurrentProjectSlug field.

Work needed:

  1. Add Generalizability and ProjectSlug to memory.Stored struct
  2. Update retrieve.go to extract these fields from TOML
  3. Add CurrentProjectSlug to surface.Options
  4. Wire current project slug through CLI → surface pipeline
  5. Implement penalty in frecency/scoring layer

2. Consolidation signal in maintenance

During maintain triage, detect clusters of 3+ semantically similar memories and emit a consolidation signal to the proposal queue. This follows the existing signal pattern (UC-28): detect at Stop/maintain → queue → surface at next session → model interviews user → user confirms/rejects.

Signal flow:

  • maintain runs semantic clustering (TF-IDF, already in codebase)
  • Identifies clusters where 3+ memories share a theme
  • Emits signal: {type: "consolidation_candidate", members: [...], theme: "..."}
  • Next session: model presents "I notice 3 memories about X — want me to consolidate them into one?"
  • On approval: create generalized memory with higher generalizability, archive originals
  • On rejection: suppress signal for those members (don't re-propose)

Current state: Clustering infrastructure exists in internal/signal/consolidate.go (BeforeStore, OnIrrelevant, BeforeRemove, findCluster). Needs integration as a maintenance signal rather than an active command.

Work needed:

  1. Add consolidation detection to maintain triage pass
  2. Emit consolidation signal to proposal queue
  3. Handle approval: create generalized memory, archive originals, inherit feedback history
  4. Handle rejection: suppress signal for those members

Why these pair

Consolidating narrow memories into a generalized principle automatically raises generalizability, which solves the cross-project surfacing problem. Low-generalizability memories from the same project are natural consolidation candidates.

Prerequisites

  • Generalizability scoring populated on memories (done — extraction-time)
  • Current project slug available at surfacing time (not yet wired)

Subsumes

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions