Skip to content

Add provenance filtering to RecallRequest / pgkg_retrieve() #24

Description

@Mearman

RecallRequest (pgkg/api.py) has no field for filtering by provenance. Its current fields (besides the ScopedRequest base) are query, k, session_id, with_rerank, with_mmr, expand_graph, valid_at, sources, k_rerank, corpus_fraction, memory_floor — nothing that reaches the provenance table.

provenance (added in 021_provenance_bitemporal.sql) already carries publisher, published_at, licence, source_url, source_authority, retrieved_at. Both propositions and chunks carry a provenance_id FK into it. But pgkg_retrieve() (031_corpus_retrieval.sql) never joins provenance anywhere in its candidate-gathering CTEs (pgkg_bm25_candidates, pgkg_vector_candidates, pgkg_graph_candidates) or its scoring/quota pipeline — the only scope-shaped filters it takes are p_org_ids, p_collection_ids, p_user_id, p_acl_groups, p_valid_at. A caller who wants "only BSI-licensed content" or "nothing published before 2020" has no way to ask for that at retrieval time; they'd have to filter client-side after the fact, discarding rows the quota/rerank budget already spent on.

Proposal: add RecallRequest.filter: PredicateNode (name open to bikeshedding), compiled via a trilean-sql package against the relevant provenance columns (publisher, published_at, licence, source_authority, etc.), and appended to pgkg_retrieve()'s WHERE clause(s) — most naturally by joining provenance into the BM25/vector candidate CTEs (or filtering the fused candidate set against it before quota application) so a provenance filter narrows the candidate pool rather than post-filtering a result page.

trilean-sql doesn't exist yet: ExaDev/trilean is currently a single package exporting PredicateNode/PredicateNodeSchema (src/index.ts), with no SQL compiler and no monorepo split as of this issue. This is a proposal to pick up once a trilean-sql package (or equivalent predicate-to-SQL compiler) is published, not something to build against today.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions