feat: concept-level confidence scoring — extend article-level confidence to per-claim with concept identity

[manavgup]

Problem

#422 added article-level numeric confidence with decay. That's the right foundation, but it scores at too coarse a granularity: an article that mixes one well-sourced claim with one weakly-sourced claim gets a single blended number, and there's no way for the Q&A agent to weight individual claims when synthesizing answers.

The original #422 issue called for per-claim scoring; we deliberately deferred it because the harder problem isn't the math — it's concept identity. Without a way to recognize that "Redis is single-threaded" in article A is the same claim as "Redis runs on a single thread" in article B, reinforcement and contradiction can't fire at the claim level.

Prerequisites already in place (from #422)

• Article.confidence_score and Article.last_reinforced_at columns
• Pure confidence.py module with compute_confidence and apply_decay
• _refresh_confidence_score() runs in Compiler.save_article()
• Decay surfaced on ArticleResponse, GraphNode, CitationResponse
• CompiledClaim Pydantic model with claim, categorical confidence, optional quote

What's missing

Three additions, in order:

1. Per-claim source attribution

CompiledClaim currently has no source_ids field — attribution is implicit via the article's ArticleSource join. Add:

class CompiledClaim(BaseModel):
    claim: str
    confidence: ConfidenceLevel  # categorical, unchanged
    quote: str | None
    # New:
    source_ids: list[str]
    confidence_score: float  # numeric, 0.0–1.0
    last_reinforced_at: datetime

Compiler change: when extracting claims, ask the LLM to attach the specific source IDs each claim came from (subset of the article's sources). Persist claims to a new compiled_claim SQLModel table (currently key_claims is stored as JSON on the article).

2. Concept identity layer

Two viable approaches; recommend starting with the cheap one:

(a) Embedding-cluster IDs (recommended MVP)

• On compile, embed each claim text using the existing embedding pipeline (or add one if absent).
• Maintain a concept_cluster table: each row is a representative embedding + canonical text + member claim IDs.
• New claims with cosine similarity > threshold (e.g. 0.85) join an existing cluster; otherwise spawn a new one.
• concept_id becomes a foreign key on CompiledClaim.

(b) Canonical Concept entity (full version, defer)

• A Concept SQLModel with id, canonical_name, aliases, etc.
• Entity linking step during compilation (LLM-based NER + dedup).
• Many-to-many between Concepts and Articles.
• This overlaps heavily with feat: self-model layer — Person, Project, Goal, Thread, Commitment as first-class entities #444 (self-model layer) and is closer to a quarter of work — track separately.

3. Reinforcement on ingest

When a new source is compiled and produces a claim that joins an existing concept cluster:

1. Append the new source ID to every CompiledClaim in that cluster
2. Update last_reinforced_at = now() on those claims
3. Recompute confidence_score per claim using compute_confidence (already exists in confidence.py — works at any granularity)
4. Article-level confidence_score becomes a weighted aggregate over its claims (mean weighted by claim length or fixed equal weight).

Contradictions: when the LLM compiler emits a CONTRADICTS backlink, also tag which concept cluster the contradiction is about, so we can penalize confidence on the specific contradicted claims rather than the whole article.

Q&A integration

When qa_agent synthesizes an answer:

• Retrieve claims (not articles) by relevance
• Weight high-confidence claims more heavily in the prompt
• Surface per-claim confidence in CitationResponse: {claim: "...", confidence: 0.82, sources: [...]}
• Optionally: explicit confidence language in answers ("supported by 4 sources" vs "based on a single 2022 article")

Out of scope (separate issues)

• Frontend per-claim confidence badges
• Manual user override / "I know this is true" boost
• Configurable per-source-type quality weights (currently a 0.7 stub in compute_confidence)
• Linter dashboard for stale claims
• Full Concept entity / entity linking (overlaps with feat: self-model layer — Person, Project, Goal, Thread, Commitment as first-class entities #444)

Testing approach

• Unit tests for embedding-cluster join logic at known thresholds
• Integration: ingest two paraphrased confirmations of the same fact, verify both attach to one cluster and confidence increases
• Integration: ingest a contradicting source, verify only the contradicted claim's confidence drops, not the whole article
• Snapshot test for Q&A CitationResponse shape with per-claim fields

References

• feat: confidence scoring for wiki claims based on source authority and recency #422 (article-level scoring — foundation)
• feat: self-model layer — Person, Project, Goal, Thread, Commitment as first-class entities #444 (self-model layer — Concept entity belongs there if we go full)
• @rohitg00 LLM Wiki v2: per-claim scoring + retention curves
• Karpathy gist comment by @a-a-k: "lossy compression — rewriting raw documents can drop caveats, dates, edge cases"

Complexity

Medium-high. Embedding pipeline + clustering + per-claim persistence is the bulk of the work; the math reuses #422's confidence.py verbatim. Estimate: 1–2 weeks for the MVP (approach 2a), separately from the full Concept entity (approach 2b, defer).