Projects & Cross-Paper Project Chat

[kokonut121]

Projects & Cross-Paper Project Chat

Versioning note: This PRD is delivered in two versions. v1 (projects + RAG-powered cross-paper chat) is the scope below. v2 (knowledge/semantic graph + user-facing graph visualization) is described in Out of Scope and is intentionally deferred. When a version is implemented, update this issue — check off the relevant version, link the PRs, and record any decisions that changed during the build.

Problem Statement

Annotagent today treats every paper in isolation. A user's library is a flat list, and the chat panel is scoped to a single paper at a time (it stuffs that one paper's full text into the system prompt). Researchers don't read one paper in a vacuum — they read clusters of related papers and need to connect ideas, methods, and findings across those papers. There is currently no way to:

• group related papers together for a given line of research, and
• ask questions that synthesize across multiple papers ("how do these three approaches to X differ?", "which of these papers supports claim Y?").

The user is left to hold the cross-paper picture in their own head and to copy-paste between single-paper chats.

Solution

Introduce Projects: private, user-owned collections that group papers from the user's library, plus a per-project chat that can answer questions using context drawn from all papers in the project.

From the user's perspective:

• They create a project, give it a name (and optional description), and add papers to it from their library. Adding a paper that isn't in their library yet automatically adds it to the library first.
• Each project has its own persistent chat. They ask cross-paper questions and get answers that cite which paper and which page each claim came from. Citations render inline as [1], [2], hovering a citation shows the paper title + page, and clicking it opens that paper at that page.
• The chat is accurate and fast regardless of how many papers are in the project, because answers are grounded in retrieved passages, not the entire corpus.

Under the hood, v1 is a retrieval-augmented generation (RAG) system over a persistent per-paper chunk index:

• At ingest time, every paper is chunked, each chunk gets a document-level contextual blurb, and each chunk is embedded. These chunk records (text + blurb + embedding + page) are persisted per paper.
• At chat time, the user's question is embedded, a vector search runs across only the chunks belonging to the project's papers, a cross-paper diversity cap ensures breadth, and the top passages are assembled into the prompt with numbered references. The model answers using those passages and cites them.

This is the "custom context layer" — for v1 it is RAG with a cross-paper-aware retrieval step, not a knowledge graph. The knowledge graph (and its visualization) is v2.

User Stories

Project management

1. As a researcher, I want to create a named project, so that I can group papers around a line of research.
2. As a researcher, I want to add an optional description to a project, so that I can remember its purpose later.
3. As a researcher, I want to see a list of all my projects, so that I can navigate between different research threads.
4. As a researcher, I want to open a project and see all the papers in it, so that I can review what I've collected.
5. As a researcher, I want to rename a project, so that I can keep its title accurate as my research evolves.
6. As a researcher, I want to delete a project, so that I can clean up threads I no longer need — without deleting the underlying papers.
7. As a researcher, I want my projects to be private to my account, so that my research organization stays confidential.

Adding/removing papers

8. As a researcher, I want to add a paper from my library to a project, so that it becomes part of that project's context.
9. As a researcher, I want to add a paper to a project even if it's not yet in my library, and have it auto-added to my library, so that I don't have to do two steps.
10. As a researcher, I want to add the same paper to more than one project, so that a paper relevant to multiple threads isn't duplicated or locked to one project.
11. As a researcher, I want to remove a paper from a project without removing it from my library or other projects, so that membership edits are non-destructive.
12. As a researcher, I want a paper that I remove from my library to also disappear from all my projects, so that projects never point at papers I no longer have.
13. As a researcher, I want to be told when a project is at its paper limit, so that I understand why I can't add more.
14. As a researcher, I want to open any paper in a project in the normal reader workspace, so that I can read and annotate it as usual.

Project chat (cross-paper RAG)

15. As a researcher, I want each project to have its own chat, so that my cross-paper questions live with the relevant project.
16. As a researcher, I want to ask a question and get an answer synthesized from multiple papers in the project, so that I can connect ideas across sources.
17. As a researcher, I want every claim in the answer to cite the specific paper and page it came from, so that I can trust and verify it.
18. As a researcher, I want citations rendered inline as [1], [2], so that I can see which part of the answer is supported by which source.
19. As a researcher, I want to hover a citation and see the paper title and page in a small popup, so that I can identify the source without leaving the chat.
20. As a researcher, I want to click a citation and be taken to that paper opened at the cited page, so that I can read the source passage in context.
21. As a researcher, I want the chat to tell me when the project's papers don't support a claim, so that I'm not misled by confident but unsupported answers.
22. As a researcher, I want the project chat to persist across sessions, so that I can return to my accumulated reasoning later.
23. As a researcher, I want to clear a project chat, so that I can start a fresh line of questioning.
24. As a researcher, I want chat answers to feel responsive (streamed), so that I'm not staring at a blank screen.
25. As a researcher, I want answers to draw from across the project's papers rather than collapsing onto a single dominant paper, so that the synthesis is genuinely cross-paper.
26. As a researcher with a large project, I want the chat to stay fast and affordable, so that project size doesn't degrade the experience.
27. As a researcher, I want the per-paper chat to keep working as before, so that I still have a focused single-paper deep-dive option.
28. As a researcher opening an empty project chat, I want a few suggested prompts, so that I know how to start.

Indexing / freshness

29. As a researcher, I want a paper I just added to become usable in chat once it's indexed, so that the chat reflects my current project.
30. As a researcher, I want a clear "indexing…" indicator on a paper that isn't yet ready for chat, so that I understand why it isn't contributing answers yet.
31. As a researcher who reprocesses a paper's annotations, I want its chat index refreshed too, so that the chat stays consistent with the paper.
32. As a researcher with papers added before this feature existed, I want them to get indexed when I add them to a project, so that older papers also work in project chat.

Implementation Decisions

Scope & versioning

• v1 = projects + RAG project chat. No knowledge graph in v1.
• v2 = knowledge/semantic graph + user-facing graph visualization (deferred — see Out of Scope). The graph is intended to (a) power richer retrieval and (b) be a browsable, clickable artifact for the user, not just a backend trick.
• Update this issue as each version lands.

Ownership & privacy

• Projects are private to a single user. No sharing, collaborators, or roles in v1. Authorization follows the existing per-user pattern (every project read/write checks ownership against the session user), mirroring how userpapers gates paper access today.

Data model (new collections)

• projects — one document per project: { _id, userId, name, description?, createdAt, updatedAt }.
• projectpapers — join table between projects and papers: { _id, projectId, paperId, userId, createdAt }, unique on (projectId, paperId). References only — it points at the shared, globally-deduplicated papers documents. No copies of paper content.
• projectchats — one rolling persistent thread per project: { _id, projectId, userId, messages[], createdAt, updatedAt }. No TTL (unlike the per-paper chats collection, which keeps its 24h TTL). A message-count/token trim policy bounds growth instead of time-based deletion.
• paperchunks — per-paper retrieval index: { _id, paperId, chunkIndex, pageNumber, text, contextualBlurb, embedding }. Keyed by paperId, shared across all users/projects that reference the paper. This is the only place embeddings are persisted.

Membership semantics

• A paper must be in the user's library to be in a project; adding a paper to a project auto-creates the userpapers link if missing.
• A paper may belong to multiple projects and always remains in the flat library.
• Removing a paper from a project deletes only the projectpapers row. Library membership, other projects, and paperchunks are untouched.
• Removing a paper from the library cascades: delete all of that user's projectpapers rows for that paper, so projects never dangle.
• Deleting a project deletes its projectpapers rows and its projectchats doc. It does not delete papers or paperchunks.
• paperchunks lifecycle is tied to the paper: written on ingest and rebuilt on reprocess; deleted only when the paper document itself is deleted (which today only happens for user-uploads when the last library link is removed).
• Paper limit per project: soft cap, default 10, configurable via env var.

Chunk indexing pass (Python ingestion)

• A chunk+embed pass runs unconditionally at ingest and reprocess, decoupled from annotationMode. (Today only dense/Contextual Retrieval produces embeddings, and they're discarded; fast/LongRAG produces none. Project chat needs chunks for every paper regardless of the mode the user ingested under.)
• The pass chunks fullText, builds a document-level contextual blurb per chunk (reusing the existing contextual-blurb approach so cross-paper retrieval gets document-grounded chunks), and embeds each chunk (reusing the shared embedding utility).
• Where chunks are written: the chunk records (text + blurb + embedding + page + index) are returned in the ingest response payload, carried by the browser to the apply route, and written to paperchunks by the server orchestration layer — keeping all Mongo writes in the server layer, consistent with current architecture. (Tradeoff accepted: a heavier apply payload carrying the float vectors.)
• Reprocess regenerates the paper's paperchunks so the index never goes stale relative to the paper.
• Backfill of pre-existing papers: lazy. When a paper without paperchunks is added to a project, trigger indexing and show an "indexing…" state on that paper; exclude it from retrieval until ready. An opportunistic batch backfill script may also be run.

Retrieval / the context layer (server layer)

• At chat time: embed the user's query → Atlas Vector Search over paperchunks filtered to the project's paperId set → apply a cross-paper diversity cap → assemble the top passages into the prompt with numbered references.
• Fixed retrieval budget regardless of project size: top-K ≈ 15 chunks total across all papers (not per-paper), so cost per turn is flat whether the project has 2 or 10 papers. Project size affects only index size, which the vector index handles.
• Cross-paper diversity: cap chunks-per-paper (≈4–5) and/or guarantee at least one hit from each paper that clears a relevance threshold, so retrieval doesn't collapse onto one dominant paper. This is the accuracy-critical behavior that makes the chat genuinely cross-paper.
• K, per-paper cap, and the relevance threshold are tunable via env vars.

Citations contract

• Retrieval produces a numbered reference list ([1] → {paperId, title, page}, …) injected into the system prompt.
• The model is instructed to cite using those exact [N] markers, and to refuse or explicitly qualify any claim the project's papers don't support (same posture as the current per-paper chat).
• The response streams; [N] markers are parsed out of the stream and rendered as interactive citations (inline marker → hover popup with title+page → click opens /paper/[paperId] at the cited page).
• Granularity is page-level for v1 (matches the data already on chunks/annotations and the level humans verify at). Exact in-page passage highlighting is out of scope.
• Duplicate citations are de-duplicated; a soft guideline limits citations-per-sentence to avoid clutter.

Project chat surface

• Coexists with the per-paper chat; does not replace it. Per-paper chat keeps full-paper-text context and its 24h TTL.
• One rolling persistent thread per project in v1 (no multi-thread/sessions). User can clear it.
• Model defaults to the same OPENAI_CHAT_MODEL as per-paper chat, but is independently overridable via a new env var so the project chat can be upgraded to a stronger model without affecting per-paper chat.
• Empty-state shows static suggested prompts (no AI-generated starter questions in v1).

UI / navigation

• New routes: /projects (grid of project cards, styled like the existing Library grid) and /projects/[projectId] (project detail).
• Project detail layout: left column = papers in the project (cards that open the reader workspace; show an "indexing…" chip when not yet retrievable); right column = the project chat panel with citations.
• An "Add papers" control opens a library picker (multi-select, respects the paper cap).
• Entry points: a Projects link in the header nav, and an "Add to project" action on Library paper cards (choose existing project or create new).

Cross-boundary contract sync

• Because this adds fields that originate in the Python service and flow to Mongo, the ingest payload contract is extended end-to-end: Python output model → ingest/apply route validation → server-data write → new paperchunks model. These move together, per the repo's cross-boundary rule.

Testing Decisions

What makes a good test here: tests assert external behavior through a module's public interface, not internal implementation details. Favor pure-input/output tests over tests that reach into private state or mock everything.

In scope for v1 tests — the project retrieval / context layer:

• This is the accuracy-critical core and the most valuable thing to test. Given a fixed set of candidate chunks (with paper ids, pages, and scores) and a query, the retrieval module should return a ranked, citation-ready result that:
  • respects the fixed top-K budget,
  • enforces the cross-paper diversity cap (never returns more than the per-paper cap from a single paper when other qualifying papers exist),
  • produces a stable, correct mapping from [N] marker → {paperId, page, title}.
• These tests exercise pure logic and should not require a live database or model — feed in candidate chunks, assert the selection and the citation mapping.
• Infrastructure note: the main tree has no JavaScript test runner today. Standing up a test runner (e.g. vitest) is part of this work, since the retrieval/context layer is TypeScript in the server layer.

Prior art: the Python service's python_service/tests/test_annotation_prompts.py (pytest) is the closest existing example of isolated, behavior-focused unit testing in this repo — follow its spirit for the TS tests.

Explicitly not specified for automated tests in v1 (per decision): the chunk+embed Python pass, citation rendering UI, project CRUD/membership wiring, and persistence. These are covered by lint/typecheck and manual verification per current repo norm; they can gain tests later.

Infrastructure & Configuration Changes (Non-Code) — DO THIS FIRST

Everything in this section is something you (the developer/operator) must do outside the application code, mostly in MongoDB Atlas and your environment config. Collected here so nothing is missed.

MongoDB Atlas — new collections

These are created implicitly on first write by Mongoose, but their indexes must be set up (some via code/Mongoose, some — the vector index — only in Atlas):

1. projects — no special index strictly required for v1 beyond the default _id. Recommended: index on userId for listing a user's projects.
2. projectpapers — unique compound index on (projectId, paperId); secondary index on projectId (list a project's papers) and on (userId, paperId) (for the library-removal cascade).
3. projectchats — index on projectId (one rolling thread lookup). No TTL index (intentionally persistent — do not copy the TTL from the chats collection).
4. paperchunks — index on paperId (fetch/delete a paper's chunks) plus the Atlas Vector Search index below.

MongoDB Atlas — Vector Search index (manual, Atlas UI/API)

Atlas Vector Search indexes are not created by Mongoose — you must create one in the Atlas console (or via the Atlas Admin API) on the paperchunks collection:

• Index type: Vector Search.
• Field: embedding (the chunk embedding array).
• Dimensions: must match the embedding model — text-embedding-3-small = 1536 dimensions. (If OPENAI_EMBED_MODEL is changed, this must change to match, and existing chunks must be re-embedded.)
• Similarity: cosine.
• Filter field: include paperId as a filterable field so retrieval can restrict the search to a project's papers.
• Confirm your Atlas tier supports Vector Search (Atlas Search/Vector Search availability depends on cluster tier).

MongoDB Atlas — capacity

• paperchunks will be the largest collection (≈ dozens–hundreds of 1536-float vectors per paper). Check storage headroom on your cluster before backfilling existing papers.

Environment variables (new)

Add these to your env config (and document them in the deployment env):

Variable	Purpose	Suggested default
PROJECT_MAX_PAPERS	Soft cap on papers per project	10
OPENAI_PROJECT_CHAT_MODEL	Model for project chat (falls back to OPENAI_CHAT_MODEL if unset)	unset → uses OPENAI_CHAT_MODEL
PROJECT_CHAT_TOP_K	Total chunks retrieved per chat turn	15
PROJECT_CHAT_MAX_CHUNKS_PER_PAPER	Cross-paper diversity cap	5
PROJECT_CHAT_MIN_RELEVANCE	Minimum similarity score for a chunk to be eligible	tune during build

Existing relevant vars to be aware of (no change required unless you tune them): OPENAI_EMBED_MODEL (must stay consistent with the vector index dimensions), OPENAI_CHAT_MODEL.

One-time data backfill (optional but recommended)

• Papers ingested before this feature have no paperchunks. Lazy backfill happens on first add-to-project, but you may also run an opportunistic batch backfill script to index existing papers ahead of time. Mind the embedding API cost and the Atlas storage impact above.

Out of Scope

• v2 — Knowledge/semantic graph + visualization. Entity/relation extraction across papers, cross-paper entity resolution (e.g. recognizing the same method/dataset/concept across papers), graph storage, graph-aware retrieval fused with vector search, and a user-facing, browsable/clickable graph visualization. This is a deliberate v2 because (a) good RAG already delivers the core "connect ideas across papers" value at the 2–10 paper scale, and (b) the graph is a much larger build whose payoff grows with corpus size and with the visualization being a first-class user artifact.
• Sharing / collaboration. Projects are single-user in v1. No invites, roles, shared chats, or team workspaces.
• Multi-thread / chat sessions per project. One rolling thread per project in v1.
• Exact in-page passage highlighting from citations. Citations resolve to page level; deep-linking to the exact highlighted passage is a separate future enhancement.
• AI-generated project starter questions / project-level summaries. Static suggested prompts only in v1.
• Replacing the per-paper chat. Per-paper chat is retained unchanged.
• A separate vector database (Pinecone/Qdrant/etc.). v1 uses Atlas Vector Search; revisit only if Atlas is outgrown.
• Project-scoped (re)indexing. Indexing is paper-scoped and computed once per paper; no per-project re-index step in v1.

Further Notes

• Why paper-scoped (not project-scoped) chunks: a paper can live in many projects and many users' libraries. Indexing per paper means embedding each paper once and fanning out at query time, avoiding duplication and repeated embedding cost.
• Why flat cost per turn: retrieval always pulls a fixed top-K, so a chat turn costs roughly the same whether the project has 2 papers or the 10-paper max. Project size grows the index, not the per-turn context.
• Trust is the feature. Cross-paper chat without reliable citations is worse than the single-paper chat it competes with. The citation contract and the "refuse/qualify when unsupported" posture are not polish — they're load-bearing.
• Migration safety: all changes are additive (new collections, new optional payload fields). Existing papers, libraries, annotations, and the per-paper chat are unaffected until a paper is added to a project (which triggers lazy indexing).
• Remember to update this issue when v1 ships (link PRs, note any changed decisions) and again when v2 (the graph) is scoped/built.