From f9c67b23004e11e31a10b190e3a0978d7e514fea Mon Sep 17 00:00:00 2001 From: castor-agent Date: Tue, 19 May 2026 18:36:31 +0200 Subject: [PATCH 1/6] docs(mcp): add guidance for GitHub entity extraction from email records Adds a [GITHUB ENTITY EXTRACTION] section to MCP instructions and a new docs/subsystems/github_entities.md documenting how to extract and store GitHub issue, PR, org, and project entities when encountered in email records. Also adds a pull_request entity type schema to ENTITY_SCHEMAS. Co-Authored-By: Claude Sonnet 4.6 --- docs/developer/mcp/instructions.md | 12 +- docs/subsystems/github_entities.md | 216 +++++++++++++++++++++++++++++ src/services/schema_definitions.ts | 53 +++++++ 3 files changed, 280 insertions(+), 1 deletion(-) create mode 100644 docs/subsystems/github_entities.md diff --git a/docs/developer/mcp/instructions.md b/docs/developer/mcp/instructions.md index 1a19b046f..81c0ce683 100644 --- a/docs/developer/mcp/instructions.md +++ b/docs/developer/mcp/instructions.md @@ -159,6 +159,15 @@ Display rule — empty state: before rendering the empty-state message or any `S Display rule — override scope: the display rule overrides the silent-storage default and the no-emoji communication style for this disclosure only; do not narrate internal store sequencing. Weekly value surfacing: when the conversation is the first of the day or the user has not interacted for several days, proactively run a bounded retrieval (list_recent_changes, retrieve_entities with a recent time window, or list_timeline_events for the past 7 days) and surface a brief summary: "You have N entities in Neotoma. Here is what changed this week: [2 new contacts, 3 tasks completed, 5 observations added]." Keep it to 1–2 sentences. Do not surface this more than once per day. +[GITHUB ENTITY EXTRACTION] +Apply this section during the per-record scan (step 5 of the per-record extraction checklist in [COMMUNICATION & DISPLAY]) whenever an email, calendar invite, chat message, or web page body contains a GitHub issue URL, PR URL, org name, or project link. Full field reference and extraction examples: docs/subsystems/github_entities.md. +GitHub issue: entity_type "issue", fields github_number (number), repo ("owner/name"), github_url (full URL), title (if parseable), data_source ("email message_id= "), source_quote (verbatim supporting snippet). Identity rule: composite [github_number, repo] — always use these canonical fields so the reducer merges updates to the same row. FORBIDDEN: using ad hoc fields (github_issue_number, repository, url) instead of github_number + repo when the canonical fields are recoverable. If canonical fields cannot be populated, store as a "note" or "technical_research" entity until they are known. See also GitHub issue URL extraction rule in [ISSUE REPORTING]. +GitHub PR: entity_type "pull_request" (aliases: pr, github_pr, merge_request), fields number (number, required), repo ("owner/name", required), url (full URL), title, status ("open" | "merged" | "closed"), author (GitHub login), base_branch, head_branch, created_at, merged_at, closed_at, data_source, source_quote. Identity rule: composite [number, repo]. URL pattern: github.com///pull/. +GitHub organization: use entity_type "organization" (reuse "company" per entity-type reuse check — do NOT create a new "github_org" type). Fields: name (required), external_id (GitHub login — most stable dedup key), website ("https://github.com/"), data_source. Identity rule: [external_id, website, email, legal_name, name] in priority order. +GitHub project: use entity_type "project" (do NOT create a new "github_project" type). Fields: name (required), status ("active" required by schema), notes (GitHub Projects URL), data_source ("GitHub Projects email reference "). Identity rule: [name]. +Linking: include REFERS_TO from the email entity (source) to each extracted GitHub entity (target) in the same store call via the relationships array. Use index-based references when batching in one store call. FORBIDDEN: storing a GitHub entity extracted from email without a REFERS_TO edge back to the originating email record. +data_source per entity: every GitHub entity stored from email MUST carry a per-entity data_source embedding the originating email's message_id (e.g. "email message_id= ") or sender+date when message_id is unavailable. FORBIDDEN: reusing the same data_source string on multiple distinct GitHub entities in one batch — this triggers heuristic identity collapse. + [ATTRIBUTION & AGENT IDENTITY] Identify yourself: every write to Neotoma (observations, relationships, timeline events, sources, interpretations) is attributed per row. Attribution shows up in `/stats`, entity and relationship views, and audit trails. Anonymous writes are accepted but flagged as `anonymous` tier. Preferred — AAuth: sign requests with AAuth (RFC 9421 HTTP Message Signatures plus an `aa-agent+jwt` agent token). Use `@aauth/local-keys` or equivalent. Successful verification records the public-key thumbprint, algorithm, and JWT subject/issuer, and renders the agent with a `hardware` or `software` trust badge (ES256/EdDSA → `hardware`; other algorithms → `software`). AAuth is honoured on every HTTP surface — `/mcp`, direct write routes (`/store`, `/observations/create`, `/create_relationship`, `/correct`, …), and `/session` — and the same identity is threaded into the write-path services regardless of transport (HTTP `/mcp`, MCP stdio, CLI-over-MCP, CLI-over-HTTP). @@ -233,7 +242,7 @@ Optional `relationships` on **`store`** is an array of relationship entries. Use The instruction block is tuned so agents can complete a turn (retrieval → user-phase store → attachment EMBEDS → assistant reply → closing store) without opening tool schemas or exploring the MCP tool set: -1. **Labelled sections.** Bracket-prefixed labels ([TURN LIFECYCLE], [DATA MODEL], [GUEST ENTITY SUBMISSION] (includes PII stripping checklist before issue filing), [CROSS-INSTANCE SYNC — PEERS], [SUBSTRATE SUBSCRIPTIONS], [STORE RECIPES], [RETRIEVAL], [PROVENANCE], [TASKS & COMMITMENTS], [ENTITY TYPES & SCHEMA], [ENTITY & RELATIONSHIP LIFECYCLE], [COMMUNICATION & DISPLAY], [ATTRIBUTION & AGENT IDENTITY], [CONVENTIONS], [ISSUE REPORTING], [QA REFLECTION], [ERRORS & RECOVERY], [ONBOARDING]) let agents locate rules by topic and cross-reference from one section to another without restating them. +1. **Labelled sections.** Bracket-prefixed labels ([TURN LIFECYCLE], [DATA MODEL], [GUEST ENTITY SUBMISSION] (includes PII stripping checklist before issue filing), [CROSS-INSTANCE SYNC — PEERS], [SUBSTRATE SUBSCRIPTIONS], [STORE RECIPES], [RETRIEVAL], [PROVENANCE], [TASKS & COMMITMENTS], [ENTITY TYPES & SCHEMA], [ENTITY & RELATIONSHIP LIFECYCLE], [COMMUNICATION & DISPLAY], [GITHUB ENTITY EXTRACTION], [ATTRIBUTION & AGENT IDENTITY], [CONVENTIONS], [ISSUE REPORTING], [QA REFLECTION], [ERRORS & RECOVERY], [ONBOARDING]) let agents locate rules by topic and cross-reference from one section to another without restating them. 2. **No exploration.** One prominent line under [STORE RECIPES] forbids listing, globbing, or reading MCP tool descriptor/schema files for chat, attachment, and entity-extraction flows. All parameter names and response paths used by the recipes (`structured.entities[].entity_id`, `unstructured.asset_entity_id`) are inline, so agents never need to open schemas. 3. **Unified store shape.** The user-phase recipe covers chat, extraction, and attachments as a single entities list [conversation, message, …extracted entities] with one invariant (PART_OF from message to conversation) plus REFERS_TO per extracted entity. Each list entry is a **flat** object (fields beside `entity_type`); the legacy **`attributes`** wrapper is forbidden (see first bullets under [STORE RECIPES]). Attachment turns use the same shape plus file_path/file_content and a single follow-up EMBEDS call. 4. **Turn-ordered rules.** [TURN LIFECYCLE] encodes the five-step ordering (retrieval, user-phase store, other actions, reply, closing store) once; other sections reference those step numbers instead of re-describing the ordering. @@ -246,6 +255,7 @@ Keeping the recipes in sync with server response shapes (`structured.entities[]. ## Related documents - `docs/specs/MCP_SPEC.md` — Action catalog and entity type rules +- `docs/subsystems/github_entities.md` — Canonical field reference for GitHub entity types extracted from external records - `docs/developer/mcp_overview.md` — Overview and setup - `docs/developer/mcp/unauthenticated.md` — Unauthenticated instructions - `docs/developer/mcp/tool_descriptions.yaml` — Per-tool descriptions diff --git a/docs/subsystems/github_entities.md b/docs/subsystems/github_entities.md new file mode 100644 index 000000000..938b8297a --- /dev/null +++ b/docs/subsystems/github_entities.md @@ -0,0 +1,216 @@ +# GitHub Entities + +When email records, calendar invites, or other external sources reference GitHub resources (issues, pull requests, organizations, or projects), agents extract and store those resources as first-class Neotoma entities and link them to the originating record via REFERS_TO. + +## Scope + +This document covers: + +- Canonical entity types and field names for GitHub resources extracted from external records. +- Extraction rules for each resource class. +- Linking conventions (email entity as source, REFERS_TO edges, observation `data_source`). + +It does NOT cover: + +- The `issue` subsystem's Neotoma-native issue tracking and GitHub mirror pipeline. See [`issues.md`](issues.md). +- Generic external-entity submission. See [`entity_submission.md`](entity_submission.md). + +## Entity Types + +### GitHub Issue (`issue`) + +Use `entity_type: "issue"` for GitHub issues referenced in email or other external records. This is the same type used by the Neotoma issue subsystem; identity is `(github_number, repo)`. + +**Required fields:** + +| Field | Type | Description | +|-------|------|-------------| +| `github_number` | number | Issue number (e.g. `42`) | +| `repo` | string | `owner/name` (e.g. `markmhendrickson/neotoma`) | + +**Optional fields:** + +| Field | Type | Description | +|-------|------|-------------| +| `github_url` | string | Full issue URL | +| `title` | string | Issue title when parseable | +| `status` | string | `open` or `closed` when known | +| `data_source` | string | Provenance string (tool + id + date) | +| `source_quote` | string | Verbatim snippet from the email body supporting extraction | + +**Identity rule:** `[{ composite: ["github_number", "repo"] }]` — re-stores update the existing row rather than creating a duplicate. + +**Do NOT use** a generic `note` or invent ad hoc fields (`github_issue_number`, `repository`, `url`) when the canonical fields are recoverable. If only partial context is available and `github_number` + `repo` cannot be populated, store a `note` or `technical_research` entity instead until canonical fields are known. See `[ISSUE REPORTING]` GitHub issue URL extraction rule in MCP instructions. + +### Pull Request (`pull_request`) + +Use `entity_type: "pull_request"` for GitHub pull requests referenced in email or other external records. + +**Required fields:** + +| Field | Type | Description | +|-------|------|-------------| +| `number` | number | PR number (e.g. `57`) | +| `repo` | string | `owner/name` (e.g. `markmhendrickson/neotoma`) | + +**Optional fields:** + +| Field | Type | Description | +|-------|------|-------------| +| `url` | string | Full PR URL | +| `title` | string | PR title when parseable | +| `status` | string | `open`, `merged`, or `closed` when known | +| `author` | string | GitHub login of PR author | +| `base_branch` | string | Target branch | +| `head_branch` | string | Source branch | +| `created_at` | date | PR creation timestamp | +| `merged_at` | date | Merge timestamp | +| `closed_at` | date | Close timestamp | +| `data_source` | string | Provenance string | +| `source_quote` | string | Verbatim snippet from the email body | + +**Identity rule:** `[{ composite: ["number", "repo"] }]` with `url` as fallback. + +**URL pattern for recognition:** `github.com///pull/`. + +**Aliases accepted by resolver:** `pr`, `github_pr`, `merge_request`. + +### GitHub Organization (`organization` / `company`) + +Use `entity_type: "organization"` (or reuse `company` per entity-type reuse check) for GitHub organizations mentioned as email senders, vendors, sponsors, or named collaborators. + +**Fields (from `company` schema):** + +| Field | Type | Description | +|-------|------|-------------| +| `name` | string | Organization display name (required) | +| `website` | string | `https://github.com/` | +| `external_id` | string | GitHub login (e.g. `octocat`) — use as the stable identifier | +| `description` | string | Organization description when available | +| `data_source` | string | Provenance string | + +**Identity rule:** `["external_id", "website", "email", "legal_name", "name"]` in priority order. Use `external_id` = GitHub login for the most stable deduplication key. + +**Do NOT** create a new `github_org` type. Use the established `organization` / `company` type with `external_id` set to the GitHub login and `website` set to the GitHub URL. + +### GitHub Project (`project`) + +Use `entity_type: "project"` for GitHub Projects referenced in email (e.g. project board links or project-context subject lines). This is a general-purpose project type shared with non-GitHub projects; disambiguate using `data_source`. + +**Fields (from `project` schema):** + +| Field | Type | Description | +|-------|------|-------------| +| `name` | string | Project name (required) | +| `status` | string | `active`, `closed`, etc. (required by schema) | +| `description` | string | Project description | +| `notes` | string | GitHub project URL or other notes | +| `data_source` | string | Provenance string (e.g. `GitHub Projects email reference 2026-05-19`) | + +**Identity rule:** `["name"]`. + +**Do NOT** create a new `github_project` type. Use `project` with `data_source` to distinguish GitHub Projects from other project types. + +## Extraction Rules + +### When to Extract + +Run the GitHub entity extraction pass as part of the per-record scan (see `[COMMUNICATION & DISPLAY]` per-record extraction checklist) whenever an email, calendar invite, chat message, or web page body contains: + +- A GitHub issue URL: `github.com///issues/` +- A GitHub PR URL: `github.com///pull/` +- An issue or PR reference: `#` (when repo context is available from subject or sender) +- A GitHub organization name or `github.com/` URL +- A GitHub Projects board link: `github.com/orgs//projects/` or `github.com/users//projects/` + +### Extraction per Entity Class + +**GitHub issue from URL:** + +``` +entity_type: "issue" +github_number: +repo: "/" +github_url: "" +title: "" +data_source: "email message_id=<id> <ISO-date>" +source_quote: "<verbatim URL or surrounding sentence>" +``` + +**GitHub PR from URL:** + +``` +entity_type: "pull_request" +number: <number from URL> +repo: "<owner>/<name>" +url: "<full URL>" +title: "<title if parseable>" +data_source: "email message_id=<id> <ISO-date>" +source_quote: "<verbatim URL or surrounding sentence>" +``` + +**GitHub organization from sender or body:** + +``` +entity_type: "organization" +name: "<org display name>" +external_id: "<github login>" +website: "https://github.com/<login>" +data_source: "email message_id=<id> <ISO-date>" +``` + +**GitHub project from URL or subject:** + +``` +entity_type: "project" +name: "<project name>" +status: "active" +notes: "<full GitHub Projects URL>" +data_source: "GitHub Projects email reference <ISO-date>" +``` + +### Linking + +After storing the extracted entity, link it to the originating email record in the **same `store` call** using the `relationships` array: + +``` +{ relationship_type: "REFERS_TO", source_entity_id: "<email_entity_id>", target_entity_id: "<github_entity_id>" } +``` + +Or, when batching in one store call, use index-based references: + +``` +{ relationship_type: "REFERS_TO", source_index: <email_index>, target_index: <github_entity_index> } +``` + +Use the email entity (e.g. `email_message`) as the **source** on the REFERS_TO edge and the GitHub entity as the target. This matches the `[STORE RECIPES]` user-phase relationship convention (message → extracted entity). + +### Observation `data_source` + +Every GitHub entity stored from email MUST carry a per-entity `data_source` field identifying the originating email: + +``` +"email message_id=<gmail_message_id> <ISO-date>" +``` + +When the `message_id` is unavailable, use the sender address and date: + +``` +"email from=<sender> <ISO-date>" +``` + +This satisfies the multi-row `data_source` identity requirement in `[PROVENANCE]` and prevents distinct email records from collapsing into the same GitHub entity row when the same issue is mentioned in multiple emails. + +## Schema Registration + +- `issue` — defined in `src/services/issues/seed_schema.ts` (global, seeded at startup). +- `pull_request` — defined in `src/services/schema_definitions.ts` (static bootstrap, `ENTITY_SCHEMAS`). +- `organization` / `company` — defined in `src/services/schema_definitions.ts`. +- `project` — defined in `src/services/schema_definitions.ts`. + +## Related Documents + +- [`issues.md`](issues.md) — Neotoma-native issue tracking and GitHub mirror pipeline +- [`docs/developer/mcp/instructions.md`](../developer/mcp/instructions.md) — `[GITHUB ENTITY EXTRACTION]` section with inline extraction rules for agents +- [`record_types.md`](record_types.md) — Full catalog of application-level entity types +- [`relationships.md`](relationships.md) — Relationship types (REFERS_TO, EMBEDS, PART_OF) diff --git a/src/services/schema_definitions.ts b/src/services/schema_definitions.ts index 2c437fdcb..c57ee3455 100644 --- a/src/services/schema_definitions.ts +++ b/src/services/schema_definitions.ts @@ -3105,6 +3105,59 @@ export const ENTITY_SCHEMAS: Record<string, EntitySchema> = { }, }, }, + + pull_request: { + entity_type: "pull_request", + schema_version: "1.0", + metadata: { + label: "Pull Request", + description: + "A GitHub pull request referenced in email, chat, or other external records. " + + "Use for PRs extracted from email notifications, code-review emails, or other " + + "external sources. Identity is repo + number (e.g. 'markmhendrickson/neotoma#42').", + category: "productivity", + aliases: ["pr", "github_pr", "merge_request"], + }, + schema_definition: { + fields: { + schema_version: { type: "string", required: false }, + number: { type: "number", required: true }, + repo: { type: "string", required: true }, + url: { type: "string", required: false }, + title: { type: "string", required: false, preserveCase: true }, + body: { type: "string", required: false, preserveCase: true }, + status: { type: "string", required: false }, + author: { type: "string", required: false }, + base_branch: { type: "string", required: false }, + head_branch: { type: "string", required: false }, + created_at: { type: "date", required: false }, + merged_at: { type: "date", required: false }, + closed_at: { type: "date", required: false }, + data_source: { type: "string", required: false }, + source_quote: { type: "string", required: false, preserveCase: true }, + }, + // R2: PRs are uniquely identified by repo + number (same pattern as issues). + canonical_name_fields: [{ composite: ["number", "repo"] }, "url"], + temporal_fields: [ + { field: "created_at", event_type: "pull_request_created" }, + { field: "merged_at", event_type: "pull_request_merged" }, + { field: "closed_at", event_type: "pull_request_closed" }, + ], + }, + reducer_config: { + merge_policies: { + title: { strategy: "last_write" }, + body: { strategy: "last_write" }, + status: { strategy: "last_write" }, + url: { strategy: "last_write" }, + author: { strategy: "last_write" }, + base_branch: { strategy: "last_write" }, + head_branch: { strategy: "last_write" }, + merged_at: { strategy: "last_write" }, + closed_at: { strategy: "last_write" }, + }, + }, + }, }; /** From c472771301ae2e521ae0b89ccdaf5648b5e2c280 Mon Sep 17 00:00:00 2001 From: castor-agent <markmhendrickson+castor-agent@gmail.com> Date: Tue, 19 May 2026 18:40:26 +0200 Subject: [PATCH 2/6] docs(mcp): tighten store-first protocol for external tool actions Adds an explicit [STORE-FIRST PROTOCOL] section to the MCP interaction instructions fenced block in docs/developer/mcp/instructions.md. The new section makes the external-write store-first rule binding and unambiguous: agents MUST store intent as a Neotoma entity before executing any external write action (email send, GitHub issue creation, calendar event booking, Slack post, webhook trigger, etc.), update the entity status after the action completes or fails, and never skip the store step regardless of how trivial the action appears. The section includes a three-step checklist, entity-type mapping for common external actions, on-failure behavior, and explicit FORBIDDEN rules. The existing "External tool store-first" line in [COMMUNICATION & DISPLAY] is updated to cross-reference the new section. The design-rationale section label list is updated to include the new section. Also adds a Store-first protocol section to docs/foundation/what_to_store.md explaining the three-step sequence for users and operators. Fixes #174 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- docs/developer/mcp/instructions.md | 22 ++++++++++++++++++++-- docs/foundation/what_to_store.md | 16 ++++++++++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/docs/developer/mcp/instructions.md b/docs/developer/mcp/instructions.md index 81c0ce683..2bd00c78a 100644 --- a/docs/developer/mcp/instructions.md +++ b/docs/developer/mcp/instructions.md @@ -114,6 +114,24 @@ Awaiting-reply task rule (outbound email/message without inbox response): during Scheduling cues in correspondence: when email, chat, screenshot, or pasted message text implies arranging a future meeting or call with a named person (e.g. pencil in, another for [month], book next, sync again, catch up later), create a task in the same extraction/store turn to follow up and schedule it, set due_date when a month or date is inferable (otherwise capture the timeframe in notes), and link the task to the relevant contact or person (REFERS_TO from the user agent_message to the task when batching in one store, plus task→contact if the recipe supports it, or create_relationship after store). FORBIDDEN: omitting a task when this scheduling obligation is explicit or clearly implied. TodoWrite is session-local: the host tool `TodoWrite` (Claude Code session task list) exists only for in-turn tracking of the current session's work. It does NOT satisfy the Neotoma store protocol. When a turn produces follow-up tasks — actions to take in a future session, commitments to carry forward, or work items the user should be able to query later — MUST also store each task via `store` with `entity_type: "task"` in Neotoma in the same turn. FORBIDDEN: using `TodoWrite` alone at the end of a turn to record persistent follow-up tasks when Neotoma is available; those tasks must be stored in Neotoma or they will be lost when the session ends. +[STORE-FIRST PROTOCOL] +Binding rule: before executing ANY external-tool action that mutates state in the outside world — sending email, posting a GitHub issue or PR comment, creating a calendar event, posting to Slack or Discord, triggering a webhook, submitting a form, placing an order, or any equivalent write — the agent MUST first store the intent as a Neotoma entity or observation in the same turn. The store step is non-negotiable and cannot be skipped because the action seems trivial, low-risk, or idempotent. +Rationale: if the external action fails, the stored entity remains as durable evidence of the intent. If the action succeeds, the stored entity is updated to reflect completion. Without the prior store, a failed or interrupted action leaves no audit trail and cannot be recovered or retried deterministically. +Checklist (execute in order for every external-tool write action): + 1. Store intent: call **`store`** with an entity that captures the intent, target, content, and relevant metadata BEFORE executing the external action. Use the most specific applicable entity_type (see mapping below). Include enough detail to reconstruct or retry the action from the stored entity alone. + 2. Execute external action: call the external MCP tool or API (send email, create issue, post message, book event, etc.) only AFTER the store call returns successfully. + 3. Update entity status: in the same turn after the external action returns, update the stored entity with the outcome — set `status: "sent"` / `status: "created"` / `status: "booked"` / `status: "posted"` as appropriate, plus `sent_at` / `created_at` / `external_id` when the tool returns them. Use `correct` on the entity_id returned by step 1 to add the outcome observation without duplicating the entity. +On failure: if the external action fails or is interrupted after the store call, leave the stored entity as-is (it serves as evidence of the intent) and set `status: "failed"` with `failure_reason` via `correct`. Do NOT delete the stored entity on failure. FORBIDDEN: retrying a failed external action without first checking whether the stored intent entity already exists (to avoid duplicate sends/posts). +Entity-type mapping for common external actions: + - Sending email → store `entity_type: "email_draft"` or `entity_type: "email_message"` with `subject`, `to`, `body`, `status: "pending"` before send; update to `status: "sent"` after. + - Creating GitHub issue → store `entity_type: "issue"` with `title`, `body`, `repo`, `status: "pending"` before `submit_issue`; update `github_number` and `github_url` from the response. + - Posting GitHub PR comment → store `entity_type: "pr_comment"` or `entity_type: "note"` with `body`, `pr_number`, `repo`, `status: "pending"` before posting; update to `status: "posted"` after. + - Creating calendar event → store `entity_type: "event"` with `title`, `start_time`, `end_time`, `attendees`, `status: "pending"` before creating; update `external_id` and `status: "booked"` after. + - Posting to Slack/Discord → store `entity_type: "message"` or `entity_type: "note"` with `content`, `channel`, `platform`, `status: "pending"` before posting; update to `status: "posted"` after. + - Triggering a webhook or API write → store `entity_type: "api_action"` with `endpoint`, `method`, `payload_summary`, `status: "pending"` before calling; update to `status: "completed"` or `status: "failed"` after. + - Submitting a form or placing an order → store `entity_type: "order"` or `entity_type: "form_submission"` with relevant fields and `status: "pending"` before submitting; update to `status: "submitted"` after. +FORBIDDEN: calling any external-tool write action before the intent entity is stored in Neotoma. FORBIDDEN: skipping step 3 (status update) after the external action completes or fails. FORBIDDEN: rationalizing a store-first skip because the external action is "just a comment", "just a notification", "idempotent", or "low-stakes" — the protocol is unconditional for all external writes. Note: read-only external-tool calls (fetching email, reading a calendar, web search) are governed by the "External tool store-first" rule in [COMMUNICATION & DISPLAY], not this section; that rule requires extracting and storing entities from the fetched data before responding. + [ENTITY TYPES & SCHEMA] Schema-agnostic for chat: for storage from chat, use a descriptive entity_type and whatever properties the message implies; server accepts arbitrary fields and infers schema. For the well-known types listed here, do NOT call list_entity_types before storing — proceed directly. Examples of entity_type (not fixed shapes): transaction, task, event, person, contact, company, receipt, note, location, place, legal_research, competitive_analysis, market_research, technical_research, report. For any other entity_type not in this list and not already in the session's cached type list, call `get_schema_recommendations` (with the candidate entity_type) or `list_entity_types` with a relevant keyword BEFORE the first store call that uses that type — this catches declared schemas and prevents type proliferation. For non-chat flows (imports, workflow automation, structured data extraction), always call `get_schema_recommendations` or `list_entity_types` before the first store for a given entity_type regardless of whether it appears in the common list above. Schema-check before storing known entity types: before storing with an entity_type that has a registered schema (i.e. it appears in `list_entity_types` results, or a prior store for that type returned schema-field metadata, or the type was retrieved from Neotoma this session), check its declared field names first — use `get_schema_recommendations` with the entity_type, or retrieve one existing entity of that type via `retrieve_entity_by_identifier` or `retrieve_entities` to inspect the snapshot's field names. Use declared fields where they fit the data exactly. When the data has no declared home, invent additional snake_case fields for that content — do not omit data because no declared field matches. FORBIDDEN: storing entities of a known registered type using entirely invented field names without first checking what declared fields exist. @@ -137,7 +155,7 @@ Silent storage default: do not mention storage, memory, or linking unless the us Proactive storage: use MCP actions proactively. Store when the user states relevant information; store first, then respond. Do not skip store because the user did not ask to save. Artifact store triggers: when a concrete artifact is approved or finalized in conversation — including but not limited to a plan, schema_design, architectural_decision, decision_record, feature_spec, policy, design_doc, runbook, or migration_guide — store it in the same turn without waiting for an explicit user instruction to save it. Use a descriptive entity_type matching the artifact kind (e.g. `plan`, `schema_design`, `architectural_decision`, `decision_record`, `feature_spec`). Include a `title`, `content` or `summary`, and any relevant metadata fields. Link the stored artifact entity to the active conversation with REFERS_TO from the user agent_message to the artifact entity, consistent with the Session-derived chat artifacts provenance rule (see [STORE RECIPES] `Session-derived chat artifacts`). FORBIDDEN: ending a turn in which an artifact was approved or finalized without storing it when Neotoma is available. Repo canon is additive, not a replacement: when a user asks to capture a durable principle, tenet, standing rule, mission element, or other strategy-layer canon in a repo document, persist the durable fact in Neotoma in the same turn when Neotoma is available, then update the canonical repo document if the repo is also the requested or established source of truth. Do not treat "this belongs in the repo" as a reason to skip Neotoma. -External tool store-first: when you pull data from any external source (email, calendar, search, web fetch, web scrape, API, file read, or any other tool), extract and store people, companies, locations, events, tasks, notifications, device status, and relationships in the same turn and BEFORE responding. Create tasks for action items (see [TASKS & COMMITMENTS]). Link events/tasks to locations and people. Do not respond with external data until storage is complete. +External tool store-first: when you pull data from any external source (email, calendar, search, web fetch, web scrape, API, file read, or any other tool), extract and store people, companies, locations, events, tasks, notifications, device status, and relationships in the same turn and BEFORE responding. Create tasks for action items (see [TASKS & COMMITMENTS]). Link events/tasks to locations and people. Do not respond with external data until storage is complete. For external-tool WRITE actions (sending email, creating issues, posting messages, booking events, triggering webhooks, etc.) — see [STORE-FIRST PROTOCOL] for the binding store-before-execute checklist; those actions require storing intent first, executing second, and updating status third. Per-record extraction checklist (REQUIRED for every external-source record, not just peek/list queries): for each record encountered — every email opened, every calendar event read, every search result hydrated, every transaction listed — run the full extraction pass before moving to the next record. Do not batch "I'll extract later" or skip extraction because the user only asked for a count or summary. The minimum per-record scan covers, in order: (1) **people and organizations** named in the record (sender, recipient, mentioned third parties, employer, vendor) → `contact` / `person` / `organization` / `company` entities; (2) **temporal commitments** (dates, deadlines, scheduled meetings, follow-ups) → `event` and/or `task` entities; (3) **transactional facts** (amounts, currencies, charges, refunds, transfers) → `transaction` entities with normalized amount/currency/date; (4) **locations** (addresses, venues, places mentioned) → `place` / `location` entities; (5) **referenced artifacts** (linked issues, projects, documents, threads) → see "GH issue / org / project extraction" below; (6) **outreach implications** (outbound replies still awaiting response, follow-up commitments) → see "Awaiting-reply task rule" below. Each extracted entity carries its own `data_source` per the rule above and a `source_quote` per "Embedded entity extraction" below when the extraction is inferred from body content. FORBIDDEN: skipping the per-record scan for "I just want to count my emails" or "show me my last N events" — peek-style queries still hydrate detail and trigger the full extraction pass. GH issue / org / project extraction (extends the per-record scan): when an external-source record (email, calendar invite, chat message, document, web page, or other tool response) names or references a GitHub issue, organization, multi-message project thread, or recurring outreach pattern with a counterparty, store the corresponding entity in the same turn: (1) **GitHub issues** mentioned by URL, number (e.g. "#123"), or title — use `entity_type: "issue"` with `github_number`, `github_url`, `repo` per the canonical identity rule in [ISSUE REPORTING] "GitHub issue URL extraction"; do NOT create a generic `note` placeholder when the canonical fields are recoverable. (2) **Organizations** named as senders' employers, vendors, partners, or sponsors — use `entity_type: "organization"` or `entity_type: "company"` (reuse the established type per "Entity-type reuse check"); link via REFERS_TO from the source `email_message` / `event` / `note` to the organization. (3) **Multi-message projects** — when two or more external records share a project context (same project name in subject, same client engagement, same recurring topic), create a `project` entity with `name`, `start_date`, `status`, and link all member records via REFERS_TO. Reuse the project across turns using bounded retrieval before creating a duplicate. (4) **Outreach interactions** — when an inbound or outbound message represents a substantive interaction with a contact beyond bookkeeping (not a one-line confirmation), create an `outreach_interaction` entity capturing direction (`inbound` | `outbound`), interaction kind (email, call, meeting, dm), contact, summary, and link to the source record via REFERS_TO. Used for relationship-cadence analysis and follow-up tracking. Depth of capture: list/summary tool responses (e.g. Gmail search_emails, calendar list_events, CRM list_contacts, search result pages, HTTP index listings) are index rows, not the final payload. Before persisting each item you intend to keep, call the corresponding detail endpoint (e.g. read_email, get_event, get_contact, fetch page) and store the richest stable fields it returns (for email: `body_text` / `body_html`, attachment metadata; for events: full description, attendees; for web pages: parsed content). Preserve both provenance layers by keeping the list-row JSON under `api_response_data.list` and the detail-row JSON under `api_response_data.detail`. @@ -242,7 +260,7 @@ Optional `relationships` on **`store`** is an array of relationship entries. Use The instruction block is tuned so agents can complete a turn (retrieval → user-phase store → attachment EMBEDS → assistant reply → closing store) without opening tool schemas or exploring the MCP tool set: -1. **Labelled sections.** Bracket-prefixed labels ([TURN LIFECYCLE], [DATA MODEL], [GUEST ENTITY SUBMISSION] (includes PII stripping checklist before issue filing), [CROSS-INSTANCE SYNC — PEERS], [SUBSTRATE SUBSCRIPTIONS], [STORE RECIPES], [RETRIEVAL], [PROVENANCE], [TASKS & COMMITMENTS], [ENTITY TYPES & SCHEMA], [ENTITY & RELATIONSHIP LIFECYCLE], [COMMUNICATION & DISPLAY], [GITHUB ENTITY EXTRACTION], [ATTRIBUTION & AGENT IDENTITY], [CONVENTIONS], [ISSUE REPORTING], [QA REFLECTION], [ERRORS & RECOVERY], [ONBOARDING]) let agents locate rules by topic and cross-reference from one section to another without restating them. +1. **Labelled sections.** Bracket-prefixed labels ([TURN LIFECYCLE], [DATA MODEL], [GUEST ENTITY SUBMISSION] (includes PII stripping checklist before issue filing), [CROSS-INSTANCE SYNC — PEERS], [SUBSTRATE SUBSCRIPTIONS], [STORE RECIPES], [RETRIEVAL], [PROVENANCE], [TASKS & COMMITMENTS], [STORE-FIRST PROTOCOL], [ENTITY TYPES & SCHEMA], [ENTITY & RELATIONSHIP LIFECYCLE], [COMMUNICATION & DISPLAY], [GITHUB ENTITY EXTRACTION], [ATTRIBUTION & AGENT IDENTITY], [CONVENTIONS], [ISSUE REPORTING], [QA REFLECTION], [ERRORS & RECOVERY], [ONBOARDING]) let agents locate rules by topic and cross-reference from one section to another without restating them. 2. **No exploration.** One prominent line under [STORE RECIPES] forbids listing, globbing, or reading MCP tool descriptor/schema files for chat, attachment, and entity-extraction flows. All parameter names and response paths used by the recipes (`structured.entities[].entity_id`, `unstructured.asset_entity_id`) are inline, so agents never need to open schemas. 3. **Unified store shape.** The user-phase recipe covers chat, extraction, and attachments as a single entities list [conversation, message, …extracted entities] with one invariant (PART_OF from message to conversation) plus REFERS_TO per extracted entity. Each list entry is a **flat** object (fields beside `entity_type`); the legacy **`attributes`** wrapper is forbidden (see first bullets under [STORE RECIPES]). Attachment turns use the same shape plus file_path/file_content and a single follow-up EMBEDS call. 4. **Turn-ordered rules.** [TURN LIFECYCLE] encodes the five-step ordering (retrieval, user-phase store, other actions, reply, closing store) once; other sections reference those step numbers instead of re-describing the ordering. diff --git a/docs/foundation/what_to_store.md b/docs/foundation/what_to_store.md index b341ec9f7..5376cf822 100644 --- a/docs/foundation/what_to_store.md +++ b/docs/foundation/what_to_store.md @@ -93,6 +93,22 @@ These before/after examples show what storage looks like in practice. "Before" i - Before: You have an ongoing billing dispute with a vendor. Details are scattered across emails, chat messages, and phone call notes. Reconstructing the timeline requires manual archaeology. - After: Agent stores each interaction as an observation on the dispute entity. `{ entity_type: "dispute", vendor: "Acme Billing", status: "open", amount_disputed: 250.00 }` with observations for each touchpoint. The full timeline is queryable: "What did we know about this dispute on March 15?" +## Store-first protocol for external tool actions + +When an agent is about to execute a write action in an external tool — sending an email, creating a GitHub issue, posting a Slack message, booking a calendar event, triggering a webhook, or any equivalent operation that mutates state outside Neotoma — it MUST store the intent as a Neotoma entity BEFORE executing the action. + +This is not optional. The store step is non-negotiable regardless of how simple or low-risk the external action appears. + +The three-step sequence is: + +1. **Store intent.** Store an entity capturing what will be done, to whom, and with what content. Use the most specific entity_type (e.g. `email_draft`, `issue`, `event`, `message`). Set `status: "pending"`. +2. **Execute the external action.** Call the external tool only after the store returns successfully. +3. **Update entity status.** After the external action completes or fails, update the stored entity with the outcome (`status: "sent"`, `status: "created"`, `status: "failed"`, plus any external IDs returned). + +If the external action fails, the stored entity remains as durable evidence of the intent. If the action succeeds, the entity records what was done, when, and to whom — traceable and auditable. + +This protocol extends the general store-first rule (which applies to reading external data) to cover writes. The agent instructions in `docs/developer/mcp/instructions.md` define the full binding rule and entity-type mapping under `[STORE-FIRST PROTOCOL]`. + ## What NOT to store | Condition | Reason | From 53719fdffbd141739bb12795c39d5b3fe567bd87 Mon Sep 17 00:00:00 2001 From: castor-agent <markmhendrickson+castor-agent@gmail.com> Date: Tue, 19 May 2026 18:52:08 +0200 Subject: [PATCH 3/6] feat(cli): add --project-local and --safe flags to neotoma init Implements issue #169. - --project-local: stores init config in .neotoma/config.json in the current directory (project-scoped) instead of ~/.config/neotoma/config.json (user-scoped). Project-local config takes precedence over user-level config when readEffectiveConfig() is used. - --safe: dry-run mode that reports what init would do (create directories, write config, initialize databases) without making any changes. Outputs a human-readable checklist in pretty mode or a JSON object with dry_run=true and planned_actions[] when --json is set. Exit 0 if all planned actions would succeed. Also adds: - writeProjectLocalConfig() and projectLocalConfigPath() helpers in config.ts - readEffectiveConfig() that merges project-local over user-level config - Tests in tests/cli/cli_init_flags.test.ts covering both flags - cli_reference.md documentation with examples and runtime override tables - Updated automated test catalog Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- docs/developer/cli_reference.md | 30 ++- src/cli/config.ts | 49 +++++ src/cli/index.ts | 100 +++++++++- tests/cli/cli_init_flags.test.ts | 326 +++++++++++++++++++++++++++++++ 4 files changed, 503 insertions(+), 2 deletions(-) create mode 100644 tests/cli/cli_init_flags.test.ts diff --git a/docs/developer/cli_reference.md b/docs/developer/cli_reference.md index dc49736fa..9711e4a07 100644 --- a/docs/developer/cli_reference.md +++ b/docs/developer/cli_reference.md @@ -328,8 +328,10 @@ neotoma session --servers - `--force`: Overwrite existing configuration. - `--skip-db`: Skip database initialization. - `--skip-env`: Skip interactive `.env` creation and variable prompts (e.g. for CI or non-interactive use). + - `--project-local`: Store the Neotoma config in `.neotoma/config.json` in the current directory (project-scoped) instead of the user-level `~/.config/neotoma/config.json`. The project-local config takes precedence over the user-level config when `readEffectiveConfig` is used. Use this when you want per-project Neotoma configuration that is independent of the user-level setup. + - `--safe`: Dry-run mode. Reports what `init` would do (create directories, write config, run migrations) without writing any files or making any changes. Output lists each planned action with a check mark. Exit code is 0 if everything would succeed. Combine with `--json` to get machine-readable output. -**Example:** +**Examples:** ```bash # Basic initialization @@ -337,6 +339,18 @@ neotoma init # Initialize with custom data directory neotoma init --data-dir /path/to/data + +# Store config in current project directory instead of user home +neotoma init --project-local + +# Preview what init would do without making any changes +neotoma init --safe + +# Dry-run with machine-readable output +neotoma init --safe --json + +# Combine: dry-run scoped to current project +neotoma init --safe --project-local ``` **What it creates:** @@ -345,6 +359,20 @@ neotoma init --data-dir /path/to/data - SQLite database: `<data-dir>/neotoma.db` (with WAL mode enabled) - Encryption key (if user chooses key-derived auth when prompted): `~/.config/neotoma/keys/neotoma.key` (mode 0600). - Environment file target: project `<checkout>/.env` when checkout is detected, otherwise `~/.config/neotoma/.env` +- Config file: `~/.config/neotoma/config.json` (default) or `.neotoma/config.json` in the current directory when `--project-local` is given. + +**Runtime overrides** for `neotoma init`: + +| Precedence | Source | Description | +|------------|--------|-------------| +| 1 (highest) | `--data-dir` flag | Explicit data directory path | +| 2 | `NEOTOMA_DATA_DIR` env var | Environment variable override | +| 3 (default) | Auto-detected or `~/neotoma/data` | Resolved at startup | + +| Precedence | Source | Description | +|------------|--------|-------------| +| 1 (highest) | `--project-local` flag | Write to `.neotoma/config.json` in cwd | +| 2 (default) | (no flag) | Write to `~/.config/neotoma/config.json` | ### Harness setup diff --git a/src/cli/config.ts b/src/cli/config.ts index 12d01f781..e42f9ee28 100644 --- a/src/cli/config.ts +++ b/src/cli/config.ts @@ -39,6 +39,55 @@ export const CONFIG_DIR = path.join(os.homedir(), ".config", "neotoma"); export const CONFIG_PATH = path.join(CONFIG_DIR, "config.json"); export const USER_ENV_PATH = path.join(CONFIG_DIR, ".env"); +/** Subdirectory name used for project-local Neotoma config when `--project-local` is given to `neotoma init`. */ +export const PROJECT_LOCAL_CONFIG_DIR_NAME = ".neotoma"; +/** Config file name inside a project-local config directory. */ +export const PROJECT_LOCAL_CONFIG_FILE_NAME = "config.json"; + +/** + * Returns the project-local config path for the given directory (default: cwd). + * Project-local config lives at `<dir>/.neotoma/config.json` and takes + * precedence over the user-level `~/.config/neotoma/config.json` when present. + */ +export function projectLocalConfigPath(cwd?: string): string { + return path.join( + cwd ?? process.cwd(), + PROJECT_LOCAL_CONFIG_DIR_NAME, + PROJECT_LOCAL_CONFIG_FILE_NAME + ); +} + +/** + * Read the merged effective config: project-local (if present) takes precedence + * over user-level config. Call this instead of `readConfig()` when callers want + * the full precedence chain. + */ +export async function readEffectiveConfig( + cwd?: string +): Promise<Config & { _source?: "project-local" | "user" }> { + const localPath = projectLocalConfigPath(cwd); + try { + const raw = await fs.readFile(localPath, "utf-8"); + const local = JSON.parse(raw) as Config; + return { ...local, _source: "project-local" }; + } catch { + // No project-local config; fall through to user-level. + } + const userConfig = await readConfig(); + return { ...userConfig, _source: "user" }; +} + +/** + * Write config to the project-local path (`.neotoma/config.json` in `cwd`). + * Creates the `.neotoma/` directory if needed. + */ +export async function writeProjectLocalConfig(next: Config, cwd?: string): Promise<string> { + const localPath = projectLocalConfigPath(cwd); + await fs.mkdir(path.dirname(localPath), { recursive: true }); + await fs.writeFile(localPath, JSON.stringify(next, null, 2)); + return localPath; +} + const cliEnv = process.env.NEOTOMA_ENV || "development"; /** True when NEOTOMA_ENV is "production". Used for API logs dir, PID path, and CLI log default. */ export const isProd = cliEnv === "production"; diff --git a/src/cli/index.ts b/src/cli/index.ts index df2a8b1cc..65a9a9eb9 100644 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -44,6 +44,8 @@ import { waitForApiReady, waitForHealth, writeConfig, + writeProjectLocalConfig, + projectLocalConfigPath, type ApiInstance, type Config, } from "./config.js"; @@ -4630,6 +4632,14 @@ const initCommand = program "Limit transcript import to a specific harness: claude-code, codex, or cursor" ) .option("--transcript-limit <n>", "Maximum number of transcript files to import per harness") + .option( + "--project-local", + "Store init config in .neotoma/config.json in the current directory (project-scoped) instead of ~/.config/neotoma/config.json (user-scoped)" + ) + .option( + "--safe", + "Dry-run mode: report what init would do (create directories, write config, run migrations) without making any changes. Exit 0 if everything would succeed." + ) .action( async (opts: { dataDir?: string; @@ -4654,6 +4664,8 @@ const initCommand = program importTranscripts?: boolean; transcriptHarness?: string; transcriptLimit?: string; + projectLocal?: boolean; + safe?: boolean; }) => { try { const outputMode = resolveOutputMode(); @@ -4677,6 +4689,80 @@ const initCommand = program // Fall through to normal init on any detection error. } } + // --safe: dry-run mode — report planned actions without making any changes. + if (opts.safe) { + const cwd = process.cwd(); + const homeDir = process.env.HOME || process.env.USERPROFILE || "."; + const dataDirDefault = path.join(homeDir, "neotoma", "data"); + const resolvedDataDir = opts.dataDir?.trim() || dataDirDefault; + const configTarget = opts.projectLocal ? projectLocalConfigPath(cwd) : CONFIG_PATH; + const envTarget = path.join(CONFIG_DIR, ".env"); + const dbPaths = [ + path.join(resolvedDataDir, "neotoma.db"), + path.join(resolvedDataDir, "neotoma.prod.db"), + ]; + const plannedActions: Array<{ label: string; path?: string; blockerReason?: string }> = [ + { + label: "Create data directory", + path: resolvedDataDir, + }, + { + label: "Create sources directory", + path: path.join(resolvedDataDir, "sources"), + }, + { + label: "Create logs directory", + path: path.join(resolvedDataDir, "logs"), + }, + ...dbPaths.map((p) => ({ label: `Initialize database`, path: p })), + { + label: `Write config`, + path: configTarget, + }, + ...(!opts.skipEnv ? [{ label: "Write environment file", path: envTarget }] : []), + ]; + const outputMode = resolveOutputMode(); + if (outputMode === "json") { + writeOutput( + { + ok: true, + dry_run: true, + planned_actions: plannedActions.map((a) => ({ + label: a.label, + ...(a.path ? { path: a.path } : {}), + would_succeed: !a.blockerReason, + ...(a.blockerReason ? { blocker: a.blockerReason } : {}), + })), + }, + outputMode + ); + } else { + process.stdout.write( + bold("neotoma init --safe") + " (dry-run — no files will be written)\n\n" + ); + process.stdout.write( + dim("Config scope: ") + + (opts.projectLocal + ? pathStyle(configTarget) + " (project-local)" + : pathStyle(configTarget) + " (user-level)") + + "\n\n" + ); + for (const action of plannedActions) { + const ok = !action.blockerReason; + const icon = ok ? success("✓") : warn("✗"); + const pathPart = action.path ? " " + pathStyle(action.path) : ""; + const blockerPart = action.blockerReason + ? "\n " + dim("blocker: " + action.blockerReason) + : ""; + process.stdout.write(` ${icon} ${action.label}${pathPart}${blockerPart}\n`); + } + process.stdout.write( + "\n" + dim("All checks passed. Run without --safe to apply.") + "\n" + ); + } + return; + } + const interactiveRequested = Boolean(opts.interactive || opts.advanced); let useAdvancedPrompts = interactiveRequested; const applyDefaultsWithoutPrompts = Boolean(opts.yes || !interactiveRequested); @@ -6397,7 +6483,19 @@ NEOTOMA_MCP_TOKEN_ENCRYPTION_KEY=${mcpTokenEncryptionKey} ...(configRepoRoot ? { project_root: configRepoRoot, repo_root: configRepoRoot } : {}), ...(initAuthSummary.mode !== "skip" ? { init_auth_mode: initAuthSummary.mode } : {}), }; - await writeConfig(nextConfig); + if (opts.projectLocal) { + // Write project-local config to .neotoma/config.json in cwd; this takes + // precedence over user-level config when running Neotoma from this directory. + const localConfigPath = await writeProjectLocalConfig(nextConfig, process.cwd()); + if (outputMode === "pretty") { + process.stdout.write( + bullet(success("Project-local config written: ") + pathStyle(localConfigPath)) + + "\n" + ); + } + } else { + await writeConfig(nextConfig); + } } // If repo root was discovered late (after env setup phase), still ensure diff --git a/tests/cli/cli_init_flags.test.ts b/tests/cli/cli_init_flags.test.ts new file mode 100644 index 000000000..64093aea6 --- /dev/null +++ b/tests/cli/cli_init_flags.test.ts @@ -0,0 +1,326 @@ +/** + * Tests for `neotoma init --safe` (dry-run) and `neotoma init --project-local` + * (project-scoped config) flags. + */ + +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; + +import { afterEach, describe, expect, it, vi } from "vitest"; + +type CliModule = { + runCli: (argv: string[]) => Promise<void>; +}; + +async function loadCli(): Promise<CliModule> { + vi.resetModules(); + return (await import("../../src/cli/index.ts")) as CliModule; +} + +async function withTempHome<T>(callback: (homeDir: string) => Promise<T>): Promise<T> { + const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-init-flags-home-")); + const previousHome = process.env.HOME; + const previousUserProfile = process.env.USERPROFILE; + const previousRepoRoot = process.env.NEOTOMA_REPO_ROOT; + const previousDataDir = process.env.NEOTOMA_DATA_DIR; + process.env.HOME = tempDir; + process.env.USERPROFILE = tempDir; + delete process.env.NEOTOMA_REPO_ROOT; + delete process.env.NEOTOMA_DATA_DIR; + try { + return await callback(tempDir); + } finally { + if (previousHome === undefined) delete process.env.HOME; + else process.env.HOME = previousHome; + if (previousUserProfile === undefined) delete process.env.USERPROFILE; + else process.env.USERPROFILE = previousUserProfile; + if (previousRepoRoot === undefined) delete process.env.NEOTOMA_REPO_ROOT; + else process.env.NEOTOMA_REPO_ROOT = previousRepoRoot; + if (previousDataDir === undefined) delete process.env.NEOTOMA_DATA_DIR; + else process.env.NEOTOMA_DATA_DIR = previousDataDir; + } +} + +function captureStdout(): { lines: () => string; restore: () => void } { + const chunks: string[] = []; + const spy = vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + if (typeof chunk === "string") chunks.push(chunk); + return true; + }); + return { + lines: () => chunks.join(""), + restore: () => spy.mockRestore(), + }; +} + +function mockReadlineForInit(): void { + vi.doMock("node:readline", () => ({ + createInterface: () => ({ + on: () => {}, + question: (_q: string, cb: (v: string) => void) => cb(""), + close: () => {}, + }), + })); +} + +describe("neotoma init --safe (dry-run)", () => { + afterEach(() => { + vi.restoreAllMocks(); + vi.resetModules(); + }); + + it("exits without creating any files", async () => { + await withTempHome(async (homeDir) => { + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-init-safe-cwd-")); + const previousCwd = process.cwd(); + process.chdir(cwd); + try { + mockReadlineForInit(); + const { runCli } = await loadCli(); + const out = captureStdout(); + try { + await runCli([ + "node", + "cli", + "init", + "--safe", + "--skip-db", + "--skip-env", + "--auth-mode", + "dev_local", + ]); + } finally { + out.restore(); + } + + // No files should have been created in the home dir config dir + const configPath = path.join(homeDir, ".config", "neotoma", "config.json"); + await expect(fs.access(configPath)).rejects.toThrow(); + + // The dry-run output should mention dry-run / no-files + const text = out.lines(); + expect(text).toContain("dry-run"); + } finally { + process.chdir(previousCwd); + await fs.rm(cwd, { recursive: true, force: true }); + } + }); + }, 15000); + + it("reports planned actions in pretty mode", async () => { + await withTempHome(async (_homeDir) => { + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-init-safe-pretty-")); + const previousCwd = process.cwd(); + process.chdir(cwd); + try { + mockReadlineForInit(); + const { runCli } = await loadCli(); + const out = captureStdout(); + try { + await runCli([ + "node", + "cli", + "init", + "--safe", + "--skip-db", + "--skip-env", + "--auth-mode", + "dev_local", + ]); + } finally { + out.restore(); + } + const text = out.lines(); + // Should include check marks for planned actions + expect(text).toContain("✓"); + // Should list directory creation + expect(text).toMatch(/Create data directory|data directory/i); + } finally { + process.chdir(previousCwd); + await fs.rm(cwd, { recursive: true, force: true }); + } + }); + }, 15000); + + it("reports project-local config path when combined with --project-local", async () => { + await withTempHome(async (_homeDir) => { + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-init-safe-local-")); + const previousCwd = process.cwd(); + process.chdir(cwd); + try { + mockReadlineForInit(); + const { runCli } = await loadCli(); + const out = captureStdout(); + try { + await runCli([ + "node", + "cli", + "init", + "--safe", + "--project-local", + "--skip-db", + "--skip-env", + "--auth-mode", + "dev_local", + ]); + } finally { + out.restore(); + } + const text = out.lines(); + // Should mention the project-local config path + expect(text).toContain(".neotoma"); + expect(text).toContain("project-local"); + } finally { + process.chdir(previousCwd); + await fs.rm(cwd, { recursive: true, force: true }); + } + }); + }, 15000); + + it("outputs JSON with dry_run=true and planned_actions array when --json is set", async () => { + await withTempHome(async (_homeDir) => { + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-init-safe-json-")); + const previousCwd = process.cwd(); + process.chdir(cwd); + try { + mockReadlineForInit(); + const { runCli } = await loadCli(); + const chunks: string[] = []; + const spy = vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + if (typeof chunk === "string") chunks.push(chunk); + return true; + }); + try { + await runCli([ + "node", + "cli", + "--json", + "init", + "--safe", + "--skip-db", + "--skip-env", + "--auth-mode", + "dev_local", + ]); + } finally { + spy.mockRestore(); + } + const text = chunks.join(""); + const parsed = JSON.parse(text) as Record<string, unknown>; + expect(parsed.ok).toBe(true); + expect(parsed.dry_run).toBe(true); + expect(Array.isArray(parsed.planned_actions)).toBe(true); + } finally { + process.chdir(previousCwd); + await fs.rm(cwd, { recursive: true, force: true }); + } + }); + }, 15000); +}); + +describe("neotoma init --project-local: config module", () => { + /** + * These tests exercise the config module functions directly to avoid + * running the full init flow (which requires SQLite and server setup). + */ + afterEach(() => { + vi.restoreAllMocks(); + vi.resetModules(); + }); + + it("projectLocalConfigPath returns .neotoma/config.json relative to cwd", async () => { + const { projectLocalConfigPath } = await import("../../src/cli/config.ts"); + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-plcp-")); + try { + const result = projectLocalConfigPath(cwd); + expect(result).toBe(path.join(cwd, ".neotoma", "config.json")); + } finally { + await fs.rm(cwd, { recursive: true, force: true }); + } + }); + + it("writeProjectLocalConfig creates .neotoma/config.json with the supplied config", async () => { + const { writeProjectLocalConfig } = await import("../../src/cli/config.ts"); + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-wplc-")); + try { + const testConfig = { init_auth_mode: "dev_local" as const, project_root: "/some/path" }; + const writtenPath = await writeProjectLocalConfig(testConfig, cwd); + expect(writtenPath).toBe(path.join(cwd, ".neotoma", "config.json")); + + const raw = await fs.readFile(writtenPath, "utf-8"); + const parsed = JSON.parse(raw) as Record<string, unknown>; + expect(parsed.init_auth_mode).toBe("dev_local"); + expect(parsed.project_root).toBe("/some/path"); + } finally { + await fs.rm(cwd, { recursive: true, force: true }); + } + }); + + it("writeProjectLocalConfig creates the .neotoma directory if it does not exist", async () => { + const { writeProjectLocalConfig } = await import("../../src/cli/config.ts"); + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-wplc-mkdir-")); + try { + const neotomaDir = path.join(cwd, ".neotoma"); + // Directory should not exist yet + await expect(fs.access(neotomaDir)).rejects.toThrow(); + + await writeProjectLocalConfig({ init_auth_mode: "dev_local" as const }, cwd); + + // Directory should now exist + await expect(fs.access(neotomaDir)).resolves.toBeUndefined(); + } finally { + await fs.rm(cwd, { recursive: true, force: true }); + } + }); + + it("readEffectiveConfig prefers project-local config over user-level config", async () => { + const { readEffectiveConfig, writeProjectLocalConfig, writeConfig } = await import( + "../../src/cli/config.ts" + ); + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-rec-")); + const homeDir = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-rec-home-")); + const previousHome = process.env.HOME; + process.env.HOME = homeDir; + try { + // Write user-level config + await writeConfig({ init_auth_mode: "oauth" as const, project_root: "/user/path" }); + // Write project-local config with different values + await writeProjectLocalConfig( + { init_auth_mode: "dev_local" as const, project_root: "/local/path" }, + cwd + ); + + const effective = await readEffectiveConfig(cwd); + expect(effective._source).toBe("project-local"); + expect(effective.init_auth_mode).toBe("dev_local"); + expect(effective.project_root).toBe("/local/path"); + } finally { + if (previousHome === undefined) delete process.env.HOME; + else process.env.HOME = previousHome; + await fs.rm(cwd, { recursive: true, force: true }); + await fs.rm(homeDir, { recursive: true, force: true }); + } + }); + + it("readEffectiveConfig falls back to user-level config when no project-local config exists", async () => { + const { readEffectiveConfig, writeConfig } = await import("../../src/cli/config.ts"); + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-rec-fallback-")); + const homeDir = await fs.mkdtemp(path.join(os.tmpdir(), "neotoma-rec-fallback-home-")); + const previousHome = process.env.HOME; + process.env.HOME = homeDir; + try { + // Write only user-level config; no project-local + await writeConfig({ init_auth_mode: "oauth" as const, project_root: "/user/path" }); + + const effective = await readEffectiveConfig(cwd); + expect(effective._source).toBe("user"); + expect(effective.init_auth_mode).toBe("oauth"); + } finally { + if (previousHome === undefined) delete process.env.HOME; + else process.env.HOME = previousHome; + await fs.rm(cwd, { recursive: true, force: true }); + await fs.rm(homeDir, { recursive: true, force: true }); + } + }); +}); From 512ee02967e618d3192e9a11c40bfeaa0f8210bd Mon Sep 17 00:00:00 2001 From: castor-agent <markmhendrickson+castor-agent@gmail.com> Date: Tue, 19 May 2026 18:56:37 +0200 Subject: [PATCH 4/6] =?UTF-8?q?docs(mcp):=20auto=20issue=20filing=20?= =?UTF-8?q?=E2=80=94=20remove=20consent=20prompts=20from=20issue=20reporti?= =?UTF-8?q?ng=20instructions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes #160 Changes `issues.reporting_mode` default from `consent` to `proactive` so agents file issues immediately when a reportable condition is detected, without asking for permission. Removes the "mode discovery" prompt that asked the user once to configure their preference on first encounter. Key changes in docs/developer/mcp/instructions.md [ISSUE REPORTING] section: - Default mode is now `proactive` (was `consent`) - Removed mode discovery flow (no longer asks "what reporting mode do you prefer?") - Added standing-consent statement: Neotoma installation constitutes consent - Added explicit opt-out path: preference entity with `auto_file_issues: false` - FORBIDDEN clause explicitly names the consent-prompt patterns agents must not use Also updates [QA REFLECTION] QA-driven issue filing and Issue reporting lines to reflect `proactive` as default and reference the `auto_file_issues` preference opt-out. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- docs/developer/mcp/instructions.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/developer/mcp/instructions.md b/docs/developer/mcp/instructions.md index 2bd00c78a..a7ef713cd 100644 --- a/docs/developer/mcp/instructions.md +++ b/docs/developer/mcp/instructions.md @@ -216,8 +216,8 @@ Summarization after MCP actions: follow the [COMMUNICATION & DISPLAY] display ru Update check: at session start, optionally call npm_check_update with packageName "neotoma" and the client version; if updateAvailable, prompt user to upgrade. Pass include_release_notes true when the user should see truncated release notes, a breaking-changes excerpt, and release_url (best-effort; enrichment_error if enrichment fails). [ISSUE REPORTING] -Issues are the primary feedback channel for iterative Neotoma improvement based on agentic usage. Filing happens in two scenarios: (1) **reactive** — when you hit friction using Neotoma (a failing tool call, an opaque error, a missing surface, or a doc gap); (2) **QA-driven** — when the per-turn self-audit (see [QA REFLECTION] "Auto-file threshold" and "QA-driven issue filing") produces file-worthy findings indicating a Neotoma-side bug or an instruction/interface improvement opportunity. Both scenarios are gated by **`issues.reporting_mode`**: **`proactive`** — file immediately without asking (`neotoma issues config --mode proactive` or `NEOTOMA_ISSUES_REPORTING_MODE=proactive`); **`consent`** (default) — confirm with the user before each `submit_issue`; **`off`** — only submit when the user explicitly asks. Mode discovery: if `reporting_mode` has not been configured (no prior `neotoma issues config --mode` and no `NEOTOMA_ISSUES_REPORTING_MODE` env var) and an issue-filing situation arises, ask the user once: "Neotoma can automatically file issues for product improvements discovered during usage. What reporting mode do you prefer? (proactive / consent / off)". Then immediately persist the choice via `neotoma issues config --mode <choice>` so it applies to all future sessions — do not re-ask. -Proactive mode contract (auto-file mandate): when **`issues.reporting_mode`** is **`proactive`** (set explicitly via `neotoma issues config --mode proactive`, the `NEOTOMA_ISSUES_REPORTING_MODE=proactive` env var, or a future `auto_file_issues` standing-rule preference resolving to true), the agent MUST call `submit_issue` for every reactive friction event AND every file-worthy QA finding in the same turn, with no user confirmation prompt and no "want me to file this?" question. FORBIDDEN: pausing to ask "should I file this?" when proactive mode is set; deferring filing to a later turn; bundling multiple findings into a chat summary instead of separate `submit_issue` calls when their root causes differ. REQUIRED: complete the PII stripping checklist above for every auto-filed issue; run the dedup-before-filing search before each call; render every auto-filed issue in the `Issues` group of the turn report with its `entity_id` / `github_number`. Proactive mode does NOT bypass risk-management hold points (auth, schema migrations, foundation docs, destructive data repair) — for those, file the issue describing the problem but do not auto-execute the fix. +Issues are the primary feedback channel for iterative Neotoma improvement based on agentic usage. Filing happens in two scenarios: (1) **reactive** — when you hit friction using Neotoma (a failing tool call, an opaque error, a missing surface, or a doc gap); (2) **QA-driven** — when the per-turn self-audit (see [QA REFLECTION] "Auto-file threshold" and "QA-driven issue filing") produces file-worthy findings indicating a Neotoma-side bug or an instruction/interface improvement opportunity. Both scenarios are controlled by **`issues.reporting_mode`**: **`proactive`** (default) — file immediately without asking; **`consent`** — confirm with the user before each `submit_issue` (`neotoma issues config --mode consent` or `NEOTOMA_ISSUES_REPORTING_MODE=consent`); **`off`** — only submit when the user explicitly asks. Standing consent: the user's installation and configuration of Neotoma constitutes standing consent for automatic issue filing; do NOT ask for permission before filing an issue when `reporting_mode` is `proactive` (the default). Opt-out: before filing, retrieve any `preference` entity with `key: "auto_file_issues"` for the current user; if `value` is `false`, skip `submit_issue` and render the finding in the `Issues` group only (same as `off` mode for that session). FORBIDDEN: asking "would you like me to file this as an issue?", "should I file this?", or any consent prompt when `reporting_mode` is `proactive`. +Proactive mode contract (auto-file mandate): when **`issues.reporting_mode`** is **`proactive`** (the default when not overridden, or set explicitly via `neotoma issues config --mode proactive` / `NEOTOMA_ISSUES_REPORTING_MODE=proactive`), the agent MUST call `submit_issue` for every reactive friction event AND every file-worthy QA finding in the same turn, with no user confirmation prompt and no "want me to file this?" question. FORBIDDEN: pausing to ask "should I file this?" when proactive mode is active; deferring filing to a later turn; bundling multiple findings into a chat summary instead of separate `submit_issue` calls when their root causes differ. REQUIRED: complete the PII stripping checklist above for every auto-filed issue; run the dedup-before-filing search before each call; check for `preference` entity `auto_file_issues: false` opt-out before each call; render every auto-filed issue in the `Issues` group of the turn report with its `entity_id` / `github_number`. Proactive mode does NOT bypass risk-management hold points (auth, schema migrations, foundation docs, destructive data repair) — for those, file the issue describing the problem but do not auto-execute the fix. Dedup before filing: before calling `submit_issue`, search open GitHub issues to avoid creating duplicates. Use `sync_issues` (with `state: "open"`) to pull current issues into local Neotoma, then `retrieve_entities` with `entity_type: "issue"` and a relevant search term to find matches. If a matching or closely related open issue exists: (1) use `add_issue_message` on the existing issue instead of creating a new one — include the new finding, reproduction context, and any additional detail; (2) if the new finding is related but distinct enough to warrant its own issue, proceed with `submit_issue` but reference the existing issue by GitHub number (e.g. "Related: #42") in the body so both are cross-linked. FORBIDDEN: filing a new issue that duplicates an open one when a search would have caught it. When `sync_issues` is unavailable or stale, a `gh issue list --search "..." --state open` shell command is an acceptable fallback for the search step. PII and secrets: for **`visibility: "public"`** (default when mirrored to GitHub), redact emails, phone numbers, API tokens, UUIDs, and home-directory path fragments with `<LABEL:hash>` placeholders before `submit_issue` so public GitHub text is safe. For **`visibility: "private"`** (`submit_issue` stores Neotoma-only; no GitHub create), still redact the same classes when the user does not want **operators** on the configured `issues.target_url` instance to see them in the issue body or thread — private means no GitHub mirror, not unlimited disclosure to maintainers; omit or generalise fields the user marks sensitive unless they explicitly want them in the report. Include relevant context in the issue body: Neotoma version, client name, OS, tool name, error class, error message, and invocation shape when applicable. `reporter_app_version` is auto-populated by the server from the running package version when not supplied — you do not need to call `npm_check_update` first to obtain the version, though you may pass it explicitly if you have a more specific value (git SHA, build tag). @@ -235,11 +235,11 @@ Session-start health check (source checkout only): on the first turn of a sessio Per-turn self-audit: before finalizing the reply, classify the turn against four tiers: Tier 1 interaction efficiency (one user-phase store, 0–2 bounded retrievals, one closing store, 0–1 separate relationship call unless an out-of-store EMBEDS is needed); Tier 2 data quality (entity-type reuse, scoped turn_key, idempotency key, unknown_fields_count [see below], correction protocol, ErrorEnvelope handling, source fields); Tier 3 interpretation fidelity for source material (complete entity extraction, accurate amounts/dates/names/statuses, raw source preservation, schema consistency, dedup awareness); Tier 4 database health from the session-start check. Tier 2 — mandatory unknown_fields repair: if any entity in a store response has `unknown_fields_count > 0`, this is a mandatory inline repair — immediately re-store or `correct` those entities using declared schema field names; do not proceed to the closing assistant store until all entities in the turn report `unknown_fields_count: 0`. Do not treat `unknown_fields_count > 0` as informational; it means data was silently dropped to `raw_fragments` and is not in the entity snapshot. Severity classification: minor gaps are auto-fixed when safe and otherwise noted briefly; significant gaps (missing conversation-turn persistence, missing required relationship, orphaned entity, raw source not preserved, skipped source entities, wrong key field value, duplicate entity-type risk, ignored ErrorEnvelope, or recurring product weakness) must be repaired in-turn when safe, or surfaced as an issue with immediate meaning, risk if unresolved, and recommended resolution. Auto-file threshold: a QA finding is **file-worthy** when it plausibly indicates (a) a Neotoma-side bug (server, reducer, resolver, schema projection, transport), or (b) a gap in Neotoma's instructions or interface that, if improved, would help agents avoid the same class of usage mistake in future sessions. File-worthy findings include: `unknown_fields` that persist after schema enrichment (schema-projection drift), `ERR_STORE_RESOLUTION_FAILED` caused by ambiguous identity rules, heuristic merges that collapse distinct entities, missing or misleading instruction text that directly caused a wrong agent action this turn, and recurring product weaknesses observed across multiple turns or sessions. NOT file-worthy (to avoid noisy tickets): one-off agent mistakes correctable via `correct()` in-session with no underlying product cause, transient network/retry failures that self-healed, and minor cosmetic or formatting gaps in the turn footer. When in doubt, err toward filing — the issue system is the primary feedback channel for iterative Neotoma improvement based on agentic usage. -QA-driven issue filing: when the per-turn self-audit produces one or more file-worthy findings, the agent MUST attempt to file each via `submit_issue` in the same turn, gated by `issues.reporting_mode`: (1) **`proactive`** — file immediately without asking; (2) **`consent`** (default) — present each file-worthy finding to the user with a one-line summary and ask "File this as a Neotoma issue? (yes/no, public/private)"; batch multiple findings into a single prompt when possible; (3) **`off`** — do not file; render the finding in the `Issues` group only. If `reporting_mode` has not been configured and a file-worthy finding is detected, follow the mode discovery flow in [ISSUE REPORTING] — ask the user once and persist via `neotoma issues config --mode <choice>` before proceeding. Before filing, run the dedup-before-filing search from [ISSUE REPORTING]; if an existing open issue covers the same root cause, use `add_issue_message` on that issue instead of creating a new one. Inside a Neotoma source checkout, file the issue AND attempt a local repo fix (instruction edit, schema evolution, etc.) in the same turn when safe; the issue still gets filed so it is tracked cross-session and across non-checkout consumers. Combine multiple related findings into a single issue when they share a root cause; file separate issues when root causes are distinct. +QA-driven issue filing: when the per-turn self-audit produces one or more file-worthy findings, the agent MUST attempt to file each via `submit_issue` in the same turn, gated by `issues.reporting_mode` (see [ISSUE REPORTING]): (1) **`proactive`** (default) — file immediately without asking or prompting the user; (2) **`consent`** — present each file-worthy finding to the user with a one-line summary and ask "File this as a Neotoma issue? (yes/no, public/private)"; batch multiple findings into a single prompt when possible; (3) **`off`** — do not file; render the finding in the `Issues` group only. Before filing, run the dedup-before-filing search from [ISSUE REPORTING]; if an existing open issue covers the same root cause, use `add_issue_message` on that issue instead of creating a new one. Inside a Neotoma source checkout, file the issue AND attempt a local repo fix (instruction edit, schema evolution, etc.) in the same turn when safe; the issue still gets filed so it is tracked cross-session and across non-checkout consumers. Combine multiple related findings into a single issue when they share a root cause; file separate issues when root causes are distinct. Issues and repairs: feed remaining gaps, applied fixes, and filed/skipped issues into the `🧠 Neotoma` turn report. Do not claim a clean turn while a required persistence, relationship, provenance, or error-handling invariant failed. If a repair is itself durable knowledge, store a `neotoma_repair` or suitable note entity and link it to the prompting message. Compliance-pass diagnosis and auto-fix (source checkout only): when a compliance pass is triggered (by the stop-hook follow-up or by manual user request), the agent MUST (1) diagnose WHY the prior turn missed stores — classify as one of: tooling_unavailable_or_failed, instruction_delivery_missing_or_stale, agent_ignored_available_instructions, hook_state_incomplete, or false_positive_or_no_material_content; cite the signals that led to the classification (reminder injected? tool failures? connection failure? model?); (2) after completing the backfill store, attempt to resolve the underlying repo-level root cause in the same turn — e.g. update client instructions (`docs/developer/mcp/instructions.md`, `docs/developer/cli_agent_instructions.md`, `.cursor/rules/neotoma_cli.mdc`), fix hook delivery paths, clarify ambiguous recipes, or add a missing reminder gate — guided by the stop-hook's `recommended_repairs` when present; (3) render the diagnosis and any applied fix in the `🧠 Neotoma` section under an `Issues` or `Repairs` group. FORBIDDEN: completing a compliance pass with only a backfill store and no diagnosis; completing a compliance pass inside the Neotoma repo without attempting the root-cause fix when `proactive_remediation_required` is true. Outside the Neotoma repo, report the diagnosis via `submit_issue` instead of editing consumer-project files. Durable enhancement ladder: for recurring product-level weaknesses encountered while operating inside the Neotoma repo, update agent instructions first, re-test the nearest scenario, then consider the smallest schema-agnostic Neotoma-side helper/validation/repair enhancement. Ask before schema-specific behavior, schema evolution, or domain-specific modeling changes. Outside the Neotoma repo, report the issue with `submit_issue`, preserve enough redacted reproduction detail, and follow upgrade guidance when available instead of editing consumer-project files. -Issue reporting: every file-worthy `Issues` item MUST have a same-turn `submit_issue` call (gated by `reporting_mode` per QA-driven issue filing above). Inside a Neotoma source checkout, a local repo fix is additive — the issue is still filed so it is tracked. If an issue is not filed because the user declined (consent mode), mode is `off`, or the issue already exists, render that decision in the `Issues (N)` group so the problem is not orphaned. Whenever `submit_issue` or `get_issue_status` is called, or a check is deliberately deferred, render one traffic-light bullet per issue: green for resolved/closed, yellow for open/in-progress/deferred/skipped/local-fix, red for blocked/transport error. Show at most a short `entity_id` or `github_number` (when present) and the issue URL. +Issue reporting: every file-worthy `Issues` item MUST have a same-turn `submit_issue` call (gated by `reporting_mode` per QA-driven issue filing above). Inside a Neotoma source checkout, a local repo fix is additive — the issue is still filed so it is tracked. If an issue is not filed because the user declined (consent mode), mode is `off`, the user has an `auto_file_issues: false` preference, or the issue already exists, render that decision in the `Issues (N)` group so the problem is not orphaned. Whenever `submit_issue` or `get_issue_status` is called, or a check is deliberately deferred, render one traffic-light bullet per issue: green for resolved/closed, yellow for open/in-progress/deferred/skipped/local-fix, red for blocked/transport error. Show at most a short `entity_id` or `github_number` (when present) and the issue URL. [ERRORS & RECOVERY] Store retry policy: if **`store`** fails, (1) retry once with the same payload; (2) if it fails again, surface the error to the user ("Storage failed: [error message]") before responding with any retrieved data; (3) do not silently skip storage and respond as if it succeeded. From 4a3c9d22627d174674cfb93a9d1f663a7f431639 Mon Sep 17 00:00:00 2001 From: castor-agent <markmhendrickson+castor-agent@gmail.com> Date: Tue, 19 May 2026 19:06:05 +0200 Subject: [PATCH 5/6] feat(schema): register complete schema for pull_request entity type (#158) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds reference_fields, linked_issues field, and documentation to the pull_request schema bootstrap declaration. Specifically: - src/services/schema_definitions.ts: add linked_issues field (type: string) and reference_fields declaring author→contact, repo→github_repo, and linked_issues→issue with REFERS_TO edges. All three reference fields reference declared schema fields. Bump schema_version to 1.1. - docs/subsystems/record_types.md: add pull_request to the Productivity Schema Family table and add a full Pull Request section under 4.2 Productivity Types documenting required/optional fields, identity rule, aliases, reference fields, and schema registration location. - tests/unit/pull_request_schema.test.ts: 20 unit tests covering ENTITY_SCHEMAS registration, canonical_name_fields (number+repo composite and url fallback), temporal_fields (created_at, merged_at, closed_at with event types), reference_fields entries (author, repo, linked_issues), field declarations, merge policies, and no-dangling- reference-field constraint. - docs/testing/automated_test_catalog.md: regenerated to include the new test file. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- docs/subsystems/record_types.md | 34 ++++ docs/testing/automated_test_catalog.md | 9 +- src/services/schema_definitions.ts | 23 ++- tests/unit/pull_request_schema.test.ts | 207 +++++++++++++++++++++++++ 4 files changed, 268 insertions(+), 5 deletions(-) create mode 100644 tests/unit/pull_request_schema.test.ts diff --git a/docs/subsystems/record_types.md b/docs/subsystems/record_types.md index 40fd6b13c..fcd6fc75a 100644 --- a/docs/subsystems/record_types.md +++ b/docs/subsystems/record_types.md @@ -58,6 +58,7 @@ Application types for notes, documents, messages, tasks, projects, and events. | `task` | Action items with status | Founders & Small Teams | | `project` | Multi-step initiatives | Founders & Small Teams | | `event` | Meetings, appointments, calendar events | All Tier 1 ICPs | +| `pull_request` | GitHub pull requests extracted from email or chat | Founders & Small Teams | **Rationale:** These types support core Tier 1 workflows: - **AI-Native Operators:** Research synthesis (document, note), communication tracking (message) - **Knowledge Workers:** Due diligence (document), legal research (document, note), client work (message) @@ -302,6 +303,39 @@ const EVENT_PATTERNS = { location: /(?:location|where)[\s:]*([A-Za-z0-9\s,.-]+)/i, }; ``` +#### Pull Request +A GitHub pull request extracted from email, chat messages, or other external records. Identity is `repo + number` (e.g. `markmhendrickson/neotoma#42`). See [`github_entities.md`](./github_entities.md) for extraction rules. + +**Required Fields:** +- `number`: number — PR number (e.g. `42`) +- `repo`: string — `owner/name` (e.g. `markmhendrickson/neotoma`) + +**Optional Fields:** +- `url`: string — Full PR URL (e.g. `https://github.com/owner/repo/pull/42`) +- `title`: string — PR title when parseable +- `body`: string — PR description when available +- `status`: string — `open`, `merged`, or `closed` +- `author`: string — GitHub login of PR author (auto-links to `contact` via reference_fields) +- `base_branch`: string — Target branch +- `head_branch`: string — Source branch +- `linked_issues`: string — Issue reference(s) linked to this PR (auto-links to `issue` via reference_fields) +- `created_at`: ISO 8601 date — PR creation timestamp (emits `pull_request_created` event) +- `merged_at`: ISO 8601 date — Merge timestamp (emits `pull_request_merged` event) +- `closed_at`: ISO 8601 date — Close timestamp (emits `pull_request_closed` event) +- `data_source`: string — Provenance string (e.g. `email message_id=<id> <ISO-date>`) +- `source_quote`: string — Verbatim snippet from the originating record + +**Identity rule:** `[{ composite: ["number", "repo"] }]` with `url` as fallback. + +**Aliases accepted by resolver:** `pr`, `github_pr`, `merge_request`. + +**Reference fields (auto-linked at store time):** +- `author` → `contact` (REFERS_TO) +- `repo` → `github_repo` (REFERS_TO) +- `linked_issues` → `issue` (REFERS_TO) + +**Schema registration:** `src/services/schema_definitions.ts` (`ENTITY_SCHEMAS["pull_request"]`). + ### 4.3 Knowledge Types #### Contact **Required Fields:** diff --git a/docs/testing/automated_test_catalog.md b/docs/testing/automated_test_catalog.md index 64c7b386d..5a02d63dc 100644 --- a/docs/testing/automated_test_catalog.md +++ b/docs/testing/automated_test_catalog.md @@ -61,8 +61,8 @@ flowchart TD - Do not hand-edit suite inventory entries in this file. Update the generator or the repository tree, then regenerate. ## Repo-wide summary -- Total automated test files: **391** -- Backend and repo Vitest files: **358** +- Total automated test files: **392** +- Backend and repo Vitest files: **359** - Frontend Vitest files: **9** - Playwright spec files: **24** @@ -73,7 +73,7 @@ flowchart TD | Vitest service tests | 33 | | Source-adjacent tests | 45 | | Vitest integration tests | 106 | -| Vitest CLI tests | 59 | +| Vitest CLI tests | 60 | | Vitest contract tests | 10 | | Vitest security tests | 1 | | Vitest subscription tests | 3 | @@ -415,7 +415,7 @@ flowchart TD **Runner:** `vitest` **Command:** `npm test -- tests/cli` **Requirements:** Basic `.env`; some tests provision temp config homes automatically. -**Files (59):** +**Files (60):** - `tests/cli/api_client_offline_fallback.test.ts` - `tests/cli/backup_verify.test.ts` - `tests/cli/cli_access_commands.test.ts` @@ -435,6 +435,7 @@ flowchart TD - `tests/cli/cli_ingest_remote_upload.test.ts` - `tests/cli/cli_init_commands.test.ts` - `tests/cli/cli_init_env_targeting.test.ts` +- `tests/cli/cli_init_flags.test.ts` - `tests/cli/cli_init_interactive.test.ts` - `tests/cli/cli_issues_commands.test.ts` - `tests/cli/cli_mcp_commands.test.ts` diff --git a/src/services/schema_definitions.ts b/src/services/schema_definitions.ts index c57ee3455..394b781bf 100644 --- a/src/services/schema_definitions.ts +++ b/src/services/schema_definitions.ts @@ -3108,7 +3108,7 @@ export const ENTITY_SCHEMAS: Record<string, EntitySchema> = { pull_request: { entity_type: "pull_request", - schema_version: "1.0", + schema_version: "1.1", metadata: { label: "Pull Request", description: @@ -3130,6 +3130,7 @@ export const ENTITY_SCHEMAS: Record<string, EntitySchema> = { author: { type: "string", required: false }, base_branch: { type: "string", required: false }, head_branch: { type: "string", required: false }, + linked_issues: { type: "string", required: false }, created_at: { type: "date", required: false }, merged_at: { type: "date", required: false }, closed_at: { type: "date", required: false }, @@ -3143,6 +3144,25 @@ export const ENTITY_SCHEMAS: Record<string, EntitySchema> = { { field: "merged_at", event_type: "pull_request_merged" }, { field: "closed_at", event_type: "pull_request_closed" }, ], + // R3: Schema-driven reference linking — auto-create typed edges at store time + // rather than per-type code branches. See docs/foundation/schema_agnostic_design_rules.md. + reference_fields: [ + { + field: "author", + target_entity_type: "contact", + relationship_type: "REFERS_TO", + }, + { + field: "repo", + target_entity_type: "github_repo", + relationship_type: "REFERS_TO", + }, + { + field: "linked_issues", + target_entity_type: "issue", + relationship_type: "REFERS_TO", + }, + ], }, reducer_config: { merge_policies: { @@ -3153,6 +3173,7 @@ export const ENTITY_SCHEMAS: Record<string, EntitySchema> = { author: { strategy: "last_write" }, base_branch: { strategy: "last_write" }, head_branch: { strategy: "last_write" }, + linked_issues: { strategy: "last_write" }, merged_at: { strategy: "last_write" }, closed_at: { strategy: "last_write" }, }, diff --git a/tests/unit/pull_request_schema.test.ts b/tests/unit/pull_request_schema.test.ts new file mode 100644 index 000000000..000e9f136 --- /dev/null +++ b/tests/unit/pull_request_schema.test.ts @@ -0,0 +1,207 @@ +/** + * Unit tests for pull_request schema definition (issue #158). + * + * Verifies that the pull_request entity type is registered in the bootstrap + * schema registry with: + * - canonical_name_fields covering repo + number composite and url fallback + * - temporal_fields for created_at, merged_at, and closed_at + * - reference_fields for author, repo, and linked_issues + * - all expected fields declared in schema_definition.fields + * - merge policies for mutable fields + */ + +import { describe, it, expect } from "vitest"; +import { + ENTITY_SCHEMAS, + getSchemaDefinition, +} from "../../src/services/schema_definitions.js"; + +describe("pull_request schema (#158)", () => { + const schema = ENTITY_SCHEMAS["pull_request"]; + + it("is registered in ENTITY_SCHEMAS", () => { + expect(schema).toBeDefined(); + expect(schema.entity_type).toBe("pull_request"); + }); + + it("is retrievable via getSchemaDefinition", () => { + const result = getSchemaDefinition("pull_request"); + expect(result).not.toBeNull(); + expect(result?.entity_type).toBe("pull_request"); + }); + + it("has productivity category metadata", () => { + expect(schema.metadata?.category).toBe("productivity"); + }); + + it("metadata aliases include pr, github_pr, and merge_request", () => { + const aliases = schema.metadata?.aliases ?? []; + expect(aliases).toContain("pr"); + expect(aliases).toContain("github_pr"); + expect(aliases).toContain("merge_request"); + }); + + // --- canonical_name_fields --- + + it("canonical_name_fields is defined and non-empty", () => { + const { canonical_name_fields } = schema.schema_definition; + expect(canonical_name_fields).toBeDefined(); + expect(Array.isArray(canonical_name_fields)).toBe(true); + expect((canonical_name_fields ?? []).length).toBeGreaterThan(0); + }); + + it("canonical_name_fields includes a composite rule for number + repo", () => { + const rules = schema.schema_definition + .canonical_name_fields as Array<string | { composite: string[] }>; + const compositeRule = rules.find( + (r): r is { composite: string[] } => + typeof r === "object" && "composite" in r, + ); + expect(compositeRule).toBeDefined(); + expect(compositeRule?.composite).toContain("number"); + expect(compositeRule?.composite).toContain("repo"); + }); + + it("canonical_name_fields includes url as a fallback rule", () => { + const rules = schema.schema_definition + .canonical_name_fields as Array<string | { composite: string[] }>; + const hasUrl = rules.some((r) => r === "url"); + expect(hasUrl).toBe(true); + }); + + // --- temporal_fields --- + + it("temporal_fields is defined", () => { + expect(schema.schema_definition.temporal_fields).toBeDefined(); + }); + + it("temporal_fields includes created_at with event_type pull_request_created", () => { + const tf = schema.schema_definition.temporal_fields ?? []; + const entry = tf.find((t) => t.field === "created_at"); + expect(entry).toBeDefined(); + expect(entry?.event_type).toBe("pull_request_created"); + }); + + it("temporal_fields includes merged_at with event_type pull_request_merged", () => { + const tf = schema.schema_definition.temporal_fields ?? []; + const entry = tf.find((t) => t.field === "merged_at"); + expect(entry).toBeDefined(); + expect(entry?.event_type).toBe("pull_request_merged"); + }); + + it("temporal_fields includes closed_at with event_type pull_request_closed", () => { + const tf = schema.schema_definition.temporal_fields ?? []; + const entry = tf.find((t) => t.field === "closed_at"); + expect(entry).toBeDefined(); + expect(entry?.event_type).toBe("pull_request_closed"); + }); + + // --- reference_fields --- + + it("reference_fields is defined and non-empty", () => { + const { reference_fields } = schema.schema_definition; + expect(reference_fields).toBeDefined(); + expect(Array.isArray(reference_fields)).toBe(true); + expect((reference_fields ?? []).length).toBeGreaterThan(0); + }); + + it("reference_fields links author to contact", () => { + const refs = schema.schema_definition.reference_fields ?? []; + const entry = refs.find((r) => r.field === "author"); + expect(entry).toBeDefined(); + expect(entry?.target_entity_type).toBe("contact"); + expect(entry?.relationship_type).toBe("REFERS_TO"); + }); + + it("reference_fields links repo to github_repo", () => { + const refs = schema.schema_definition.reference_fields ?? []; + const entry = refs.find((r) => r.field === "repo"); + expect(entry).toBeDefined(); + expect(entry?.target_entity_type).toBe("github_repo"); + expect(entry?.relationship_type).toBe("REFERS_TO"); + }); + + it("reference_fields links linked_issues to issue", () => { + const refs = schema.schema_definition.reference_fields ?? []; + const entry = refs.find((r) => r.field === "linked_issues"); + expect(entry).toBeDefined(); + expect(entry?.target_entity_type).toBe("issue"); + expect(entry?.relationship_type).toBe("REFERS_TO"); + }); + + // --- field declarations --- + + it("required fields number and repo are declared", () => { + const { fields } = schema.schema_definition; + expect(fields.number).toBeDefined(); + expect(fields.number.type).toBe("number"); + expect(fields.number.required).toBe(true); + expect(fields.repo).toBeDefined(); + expect(fields.repo.type).toBe("string"); + expect(fields.repo.required).toBe(true); + }); + + it("has expected optional fields declared", () => { + const { fields } = schema.schema_definition; + const optionalFields = [ + "url", + "title", + "body", + "status", + "author", + "base_branch", + "head_branch", + "linked_issues", + "created_at", + "merged_at", + "closed_at", + "data_source", + "source_quote", + ]; + for (const f of optionalFields) { + expect(fields, `expected field '${f}' in pull_request schema`).toHaveProperty(f); + } + }); + + // --- merge policies --- + + it("reducer_config has last_write merge policies for mutable fields", () => { + const { merge_policies } = schema.reducer_config; + const mutableFields = [ + "title", + "body", + "status", + "url", + "author", + "base_branch", + "head_branch", + "linked_issues", + "merged_at", + "closed_at", + ] as const; + for (const f of mutableFields) { + expect( + merge_policies[f]?.strategy, + `expected last_write strategy for field '${f}'`, + ).toBe("last_write"); + } + }); + + it("reducer_config merge policies do not reference undeclared fields", () => { + const { fields } = schema.schema_definition; + const { merge_policies } = schema.reducer_config; + for (const key of Object.keys(merge_policies)) { + expect(fields, `merge policy references undeclared field '${key}'`).toHaveProperty(key); + } + }); + + it("reference_fields entries all reference declared fields", () => { + const { fields, reference_fields } = schema.schema_definition; + for (const ref of reference_fields ?? []) { + expect( + fields, + `reference_field '${ref.field}' is not declared in schema fields`, + ).toHaveProperty(ref.field); + } + }); +}); From 37f7c53097b8dae00d2c9f2c4f8cec29b281acf5 Mon Sep 17 00:00:00 2001 From: castor-agent <markmhendrickson+castor-agent@gmail.com> Date: Wed, 20 May 2026 13:16:16 +0200 Subject: [PATCH 6/6] chore: regenerate test catalog Missing commit dcf6235fb (use git ls-files in test catalog generator). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- docs/testing/automated_test_catalog.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/testing/automated_test_catalog.md b/docs/testing/automated_test_catalog.md index 5a02d63dc..c14fa95f5 100644 --- a/docs/testing/automated_test_catalog.md +++ b/docs/testing/automated_test_catalog.md @@ -61,15 +61,15 @@ flowchart TD - Do not hand-edit suite inventory entries in this file. Update the generator or the repository tree, then regenerate. ## Repo-wide summary -- Total automated test files: **392** -- Backend and repo Vitest files: **359** +- Total automated test files: **393** +- Backend and repo Vitest files: **360** - Frontend Vitest files: **9** - Playwright spec files: **24** ### Suite counts | Suite | Files | |---|---:| -| Vitest unit tests | 96 | +| Vitest unit tests | 97 | | Vitest service tests | 33 | | Source-adjacent tests | 45 | | Vitest integration tests | 106 | @@ -107,7 +107,7 @@ flowchart TD **Runner:** `vitest` **Command:** `npm test -- tests/unit` **Requirements:** Basic `.env` if required by the module under test. -**Files (96):** +**Files (97):** - `tests/unit/aauth_admission.test.ts` - `tests/unit/aauth_attestation_apple_se.test.ts` - `tests/unit/aauth_attestation_revocation.test.ts` @@ -181,6 +181,7 @@ flowchart TD - `tests/unit/parquet_reader.test.ts` - `tests/unit/product_feedback_schema.test.ts` - `tests/unit/protected_entity_types.test.ts` +- `tests/unit/pull_request_schema.test.ts` - `tests/unit/relationship_batch_schemas.test.ts` - `tests/unit/relationship_reducer.test.ts` - `tests/unit/request_context.test.ts`