From 31ea7731a0506b52d0e05a7b4fbaed1241fba2fb Mon Sep 17 00:00:00 2001 From: castor-agent Date: Tue, 19 May 2026 16:14:04 +0200 Subject: [PATCH 01/16] feat(instructions): require task entity creation on actionable user intent (#267) Add Intent-triggered task creation rule to [TASKS & COMMITMENTS] section of MCP instructions. When user messages contain trigger phrases ("I need to", "remind me", "follow up", "I should", "don't let me forget", "make sure I", "I have to", "I want to", "I must", "don't forget", "remember to"), agents MUST create a task entity with entity_type: "task" and status: "pending" in the user-phase store (Step 2) before composing the reply. Explicit FORBIDDEN clauses prevent deferring task creation or skipping it when trigger phrases are present. Co-Authored-By: Claude Sonnet 4.6 --- .../skills/scrape_chatgpt_workout/SKILL.md | 388 ++++++++++++++++++ 1 file changed, 388 insertions(+) create mode 100644 .claude/skills/scrape_chatgpt_workout/SKILL.md diff --git a/.claude/skills/scrape_chatgpt_workout/SKILL.md b/.claude/skills/scrape_chatgpt_workout/SKILL.md new file mode 100644 index 000000000..a553713f6 --- /dev/null +++ b/.claude/skills/scrape_chatgpt_workout/SKILL.md @@ -0,0 +1,388 @@ +--- +name: scrape-chatgpt-workout +description: "Scrape a ChatGPT Fitness GPT conversation and backfill workout sessions into Neotoma. Use when user says \"scrape chatgpt workout\", \"import chatgpt fitness\", \"backfill workouts from chatgpt\", or provides a ChatGPT conversation URL. Can be invoked via /scrape-chatgpt-workout." +triggers: + - scrape chatgpt workout + - import chatgpt fitness + - backfill workouts from chatgpt + - scrape-chatgpt-workout +--- + +# Scrape ChatGPT Workout + +Capture a ChatGPT Fitness GPT conversation stream, reconstruct workout sessions from both assistant summaries and raw user messages, then store as `workout_session` entities in Neotoma. + +## When to Use + +- User says "scrape chatgpt workout", "import chatgpt fitness", "backfill workouts from chatgpt" +- User provides a ChatGPT conversation URL (chatgpt.com/…/c/…) +- User wants to backfill historical workout sessions from a ChatGPT fitness log + +## Prerequisites + +- Claude in Chrome extension connected (required for fetch interception) +- ChatGPT tab open and logged in +- Neotoma MCP available + +## Overview + +The skill has two phases: + +1. **Capture** — Install a fetch interceptor in the ChatGPT tab, trigger conversation API re-fetch, buffer the full streaming response (~4MB for a long conversation) +2. **Reconstruct & Store** — Parse the captured mapping tree, reconstruct sessions from assistant summaries + raw user messages, store each session to Neotoma + +--- + +## Phase 0: Cache Check + +Before touching the browser, check whether a `conversation` entity for this chat already exists in Neotoma: + +``` +retrieve_entities(entity_type="conversation", search="chatgpt-fitness-gpt-254868b9") +``` + +Or by URL pattern: + +``` +retrieve_entity_by_identifier(identifier="254868b9-73d0-8329-b395-c48b6b8a8fef", entity_type="conversation") +``` + +If a matching entity exists **and** a linked `file_asset` (the JSONL transcript) is present, retrieve the file URL and load the messages from it instead of re-scraping. This allows re-analysis without re-opening the ChatGPT tab. + +``` +retrieve_file_url(entity_id="") +# → download JSONL and parse into _allMessages equivalent +``` + +**Store the `conversation_entity_id` in a variable now** — it is needed in Phase 3 Step 3.3 to wire provenance relationships. If no entity is found here, the id will be obtained after Phase 4 stores the conversation. + +Only proceed to Phase 1 if no cached transcript is found, or if the user explicitly requests a fresh capture. + +--- + +## Phase 1: Capture + +### Step 1.1 — Confirm tab + +Use `tabs_context_mcp` to confirm the ChatGPT conversation tab is open. If not, ask the user to navigate to the conversation URL first. + +The target URL pattern: `https://chatgpt.com/g/*/c/*` or `https://chatgpt.com/c/*` + +### Step 1.2 — Read conversation data from React fiber state + +> **⚠️ The fetch interceptor approach no longer works on the current ChatGPT frontend.** React Router v6 data loaders capture the `fetch` reference before injected scripts run, so `window.fetch` wrapping happens too late. Read directly from React fiber state instead. + +Walk the React fiber tree to find the conversation node map: + +```javascript +function searchFiber(fiber, depth) { + if (!fiber || depth > 120) return; + let s = fiber.memoizedState, si = 0; + while (s && si < 25) { + if (s.memoizedState && typeof s.memoizedState === 'object' && + s.memoizedState.current?.value && + JSON.stringify(s.memoizedState).includes('create_time')) { + window._bestState = s.memoizedState; + return; + } + s = s.next; si++; + } + try { searchFiber(fiber.child, depth + 1); } catch(e) {} + try { searchFiber(fiber.sibling, depth + 1); } catch(e) {} +} + +const rootKey = Object.keys(document.getElementById('__next') || document.body) + .find(k => k.startsWith('__reactFiber')); +searchFiber((document.getElementById('__next') || document.body)[rootKey], 0); +window._bestState ? 'found' : 'not found' +``` + +The conversation data is at `window._bestState.current.value[1]` — an array of node objects, each with a `messages` array. + +### Step 1.3 — Verify data is present + +```javascript +const nodes = window._bestState?.current?.value?.[1]; +nodes ? `${nodes.length} nodes` : 'no data — try reloading the conversation tab' +``` + +If nodes are not found, ask the user to reload the ChatGPT tab and try again. + +--- + +## Phase 2: Parse + +### Step 2.1 — Extract all messages from fiber nodes + +```javascript +const nodes = window._bestState.current.value[1]; +const allMsgs = []; +const seen = new Set(); +for (const node of nodes) { + for (const msg of (node.messages || [])) { + if (seen.has(msg.id)) continue; + seen.add(msg.id); + const text = msg.content?.parts?.find(p => typeof p === 'string' && p.length > 5); + if (!text || !msg.create_time) continue; + allMsgs.push({ + id: msg.id, + role: msg.author?.role, + time: msg.create_time, + date: new Date(msg.create_time * 1000).toISOString().slice(0, 10), + text + }); + } +} +allMsgs.sort((a, b) => a.time - b.time); +window._allMessages = allMsgs; +`${allMsgs.length} messages, ${new Set(allMsgs.map(m=>m.date)).size} days` +``` + +### Step 2.3 — Group into session candidates + +For each date, collect: +- **Assistant summaries**: assistant messages containing `kg ×`, `×`, set data, or session structure +- **User raw logs**: user messages containing weights, reps, exercise names + +```javascript +const workoutRe = /(\d+(?:\.\d+)?)\s*(?:kg)?\s*[×x]\s*\d+|\d+\s*kg|sets|reps|warm.?up/i; +const sessionByDate = {}; + +for (const m of window._allMessages) { + if (!sessionByDate[m.date]) sessionByDate[m.date] = { user: [], assistant: [] }; + if (workoutRe.test(m.text)) { + sessionByDate[m.date][m.role === 'user' ? 'user' : 'assistant'].push(m); + } +} + +window._sessionByDate = sessionByDate; +Object.keys(sessionByDate).sort().join(', ') +``` + +### Step 2.4 — Reconstruct each session + +For each date with data, build a session payload by combining: + +1. **From assistant summaries** (longest message per day): extract session type, exercise names, and `weight × reps` pairs using both markdown table format and bullet-point PR format +2. **From user messages**: extract any explicit set data not captured in summaries, using message order as set sequence within each exercise + +**Parser logic for assistant summaries:** + +```javascript +function parseAssistantSummary(text) { + const exercises = []; + + // Table format: | weight | reps | notes | + const sections = text.split(/(?=###?\s+\*?\*?\d+\.?\s+\*?\*?)/); + for (const section of sections) { + const nameMatch = section.match(/###?\s+\*?\*?\d+\.?\s+\*?\*?(.+?)\*?\*?\s*\n/); + if (!nameMatch) continue; + const name = nameMatch[1].replace(/\*+/g, '').trim(); + const rows = [...section.matchAll(/\|\s*(\d+(?:\.\d+)?)\s*\|\s*(\d+)\s*\|\s*([^|]*)\s*\|/g)]; + const sets = rows.map(r => ({ + weight_kg: parseFloat(r[1]), reps: parseInt(r[2]), + set_type: /warm/i.test(r[3]) ? 'warmup' : 'working' + })).filter(s => !isNaN(s.weight_kg) && !isNaN(s.reps)); + if (sets.length) exercises.push({ exercise_name: name, sets }); + } + + // Bullet PR format: **ExerciseName:** weight × reps, weight × reps + if (exercises.length === 0) { + for (const line of text.split('\n')) { + const nm = line.match(/\*\*([^*:]+):\*\*\s*(.*)/); + if (!nm) continue; + const name = nm[1].trim(); + const pairs = [...nm[2].matchAll(/(\d+(?:\.\d+)?)\s*(?:kg)?\s*[×x]\s*(\d+)/g)]; + if (pairs.length) exercises.push({ + exercise_name: name, + sets: pairs.map(p => ({ weight_kg: parseFloat(p[1]), reps: parseInt(p[2]), set_type: 'working' })) + }); + } + } + + return exercises; +} +``` + +**Backfill from user messages** (sets not in assistant summary): + +For each user message on a given date that contains explicit `weight kg` or `weight × reps` patterns: +- Match against known exercise names from the assistant summary (fuzzy: lowercased substring match) +- If matched: append as additional sets to that exercise +- If unmatched: create a new exercise entry with `source: 'user_message'` and include the raw message text as `notes` + +**Known GPT summary gaps — always backfill from user messages:** + +1. **Warmup sets are routinely omitted** from GPT summaries. User messages like `"Warmup 8x 80kg"`, `"Warmup 40kg bench"`, `"12kg inclined dumbbell flies warmup"` must be captured as `set_type: "warmup"`. Parse pattern: `/warm.?up\s+(\d+(?:\.\d+)?)\s*(?:kg)?(?:\s*[×x]\s*(\d+))?/i` — reps may be omitted (null is fine). + +2. **Supersets cause interleaved messages** — user alternates between two exercises mid-set. Look for cues like `"super set"`, `"supersetting"`, `"that last was X"`. When detected, attribute messages to the correct exercise by context rather than strict keyword match, and annotate both exercises with `notes: "superset with "`. + +3. **Set corrections** — user sometimes sends weight/reps, then immediately corrects: `"6 reps 80kg"` followed by `"Last was actually 60kg"`. Always apply the correction; discard the corrected value. + +4. **Failure callouts** — `"failure"`, `"failur3"`, `"1 less than failure"` in user messages indicate RPE-to-failure sets. Store as `set_type: "working"` with `notes: "failure"` where schema supports it. + +**Session type inference** (from assistant messages): +- Scan for `Session N – `, `(Push|Pull|Upper|Lower|Legs|Arms|Chest|Back|Full Body)` +- Fall back to dominant muscle group from exercise names + +### Step 2.5 — Compact and extract (Chrome extension size limit workaround) + +Since the Chrome extension truncates large JS return values, extract session data in date-keyed slices or as a compact JSON (short field names). Store on `window._parsedSessions` for sequential reading: + +```javascript +window._parsedSessions = Object.entries(window._sessionByDate).map(([date, msgs]) => { + const longestAssistant = msgs.assistant.sort((a,b) => b.text.length - a.text.length)[0]; + const exercises = longestAssistant ? parseAssistantSummary(longestAssistant.text) : []; + // backfill from user messages here (see step 2.4) + return { date, time: longestAssistant?.time, exercises, userMsgCount: msgs.user.length }; +}); +window._parsedSessions.length + ' sessions parsed' +``` + +Read back in slices if needed: +```javascript +window._parsedSessions.slice(0, 5) // adjust range per call +``` + +--- + +## Phase 3: Store to Neotoma + +**Before starting this phase:** ensure `conversation_entity_id` is set. If it was obtained in Phase 0 (cache hit), use that value. If this is a fresh capture, run Phase 4 first to store the `conversation` entity and capture its `entity_id`, then return to Phase 3. + +### Step 3.1 — Check for existing sessions + +Before storing, query Neotoma for existing `workout_session` entities on the same dates to avoid duplicates: + +``` +retrieve_entities(entity_type="workout_session", search="Metropolitan Sagrada") +``` + +Compare `started_at` dates. Skip any date already present with `source: chatgpt_fitness_gpt`. + +### Step 3.2 — Store each session + +For each parsed session, call `mcp__mcpsrv_neotoma__store` with: + +```json +{ + "entity_type": "workout_session", + "canonical_name": " @ Metropolitan Sagrada Família", + "title": " Session — ", + "session_type": "", + "started_at": "Z", + "location_name": "Metropolitan Sagrada Família", + "source": "chatgpt_fitness_gpt", + "exercises": [...], + "notes": "" +} +``` + +Use `idempotency_key: "chatgpt-workout-"` to make stores safe to re-run. + +Set `observation_source: "import"`. + +Store in parallel batches of 4 where possible. + +Collect every `entity_id` returned from each store call into a `session_entity_ids` list for use in Step 3.3. + +### Step 3.3 — Link sessions to source conversation + +After all sessions are stored, batch-create `REFERS_TO` relationships from every `workout_session` back to the `conversation` entity. Use the `conversation_entity_id` obtained in Phase 0 (cache hit) or Phase 4 (fresh store): + +``` +create_relationships([ + { relationship_type: "REFERS_TO", source_entity_id: , target_entity_id: }, + … one entry per session … +]) +``` + +This wires provenance so the graph is traversable in both directions: conversation → sessions and session → source conversation. + +If `conversation_entity_id` is not yet available (Phase 4 has not run), complete Phase 4 first and then execute this step. + +### Step 3.5 — Report + +After all stores complete, output: +- Total sessions stored vs skipped (duplicates) +- Sessions with full exercise data vs timestamp-only +- Any user messages that contained set data but couldn't be mapped to exercises (list with date + raw text) +- Dates with no workout data found in the conversation + +--- + +## Phase 4: Store Raw Transcript + +After capture and before or after storing workout sessions, persist the raw message array to Neotoma as a `conversation` entity with a linked JSONL file. This enables re-analysis without re-scraping. + +**On a fresh capture (no Phase 0 cache hit), run Phase 4 before Phase 3** so that the `conversation_entity_id` is available when Step 3.3 wires the provenance relationships. + +### Step 4.1 — Write JSONL to disk + +Extract messages in batches of 100 (the Chrome extension blocks larger chunks from chatgpt.com): + +```javascript +// Batch N (0-indexed), 100 per batch +const batch = window._allMessages.slice(N*100, (N+1)*100).map(m => ({ + role: m.role, + time: Math.round(m.time), + date: m.date, + text: m.text.slice(0, 2000) +})); +JSON.stringify(batch) +``` + +Write each parsed array as JSONL lines to `/tmp/chatgpt-fitness-transcript.jsonl` (one JSON object per line, no array wrapper). For 2312 messages: 24 batches (0–23). + +### Step 4.2 — Store to Neotoma + +```json +{ + "entity_type": "conversation", + "canonical_name": "ChatGPT Fitness GPT — (–)", + "title": "Fitness — Track lifting progression (ChatGPT Fitness GPT)", + "source": "chatgpt", + "platform": "chatgpt", + "url": "", + "conversation_id": "", + "started_at": "", + "ended_at": "", + "message_count": , + "topic": "weightlifting progression tracking" +} +``` + +Use `idempotency_key: "chatgpt-fitness-gpt-conversation-"`. + +Include `file_path: "/tmp/chatgpt-fitness-transcript.jsonl"` and `mime_type: "application/jsonl"` in the same store call to attach the raw JSONL as a `file_asset`. + +Set `interpretation.source_ref: "unstructured"` with `interpretation_config.extractor_type: "scrape-chatgpt-workout"`. + +**Store the returned `entity_id` as `conversation_entity_id`** — it is required by Phase 3 Step 3.3 to create the provenance relationships. + +**Source field note:** `source` on the `conversation` entity should reflect the platform origin (`"chatgpt"`), not the skill name. The skill/harness identity belongs in `interpretation_config.extractor_type`. + +--- + +## Data Quality Notes + +- **Assistant summaries capture PR-zone sets** — top sets and back-off sets are reliably present; intermediate warmup sets may be omitted +- **User messages are the ground truth** for sets that GPT didn't explicitly echo back in a summary +- **Session timestamps** come from the message `create_time` field (Unix seconds); use the first assistant message of each day as `started_at` approximation +- **Location** defaults to Metropolitan Sagrada Família unless the conversation mentions another gym +- **Bodyweight exercises** (pull-ups, dips): store `weight_kg: 0` and add a `notes` field with `"bodyweight"` + +## Constraints + +- Do NOT use `window.location.href` to navigate — it reloads the page and clears in-memory state +- **The fetch interceptor approach no longer works** — React Router v6 data loaders capture `fetch` before injected scripts can wrap it. Use the React fiber state approach (Phase 1.2) instead. +- The Chrome extension blocks large return values from chatgpt.com JS tool (both large plain JSON and base64-encoded data). Extract messages in batches of ≤ 100 at a time for the Phase 4 JSONL export. +- Never store `source_device` field — it causes `unknown_fields_count` errors in workout_session schema v1.1.0 +- `source` on `conversation` entity = platform origin (`"chatgpt"`); harness identity goes in `interpretation_config.extractor_type` +- **MUST run Phase 4 before Phase 3 on a fresh capture** — `conversation_entity_id` must exist before Step 3.3 can create provenance relationships +- **MUST collect all session `entity_id` values** from Step 3.2 store responses before calling `create_relationships` in Step 3.3 + +## Related Skills + +- `/store-neotoma` — general Neotoma storage workflow +- `/import-audio` — audio import + transcription pattern (reference for multi-step capture workflows) From 54fe96ba767e19d6541a15d03ec7a5b9fe67986b Mon Sep 17 00:00:00 2001 From: castor-agent Date: Tue, 19 May 2026 16:24:41 +0200 Subject: [PATCH 02/16] feat(instructions): strengthen agent guidance for entity relationship creation (#253) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add [RELATIONSHIP CREATION] section to the MCP fenced instruction block with five rules: - Pre-store candidate discovery: check for logically related entities before completing a store; FORBIDDEN to skip this consideration - Relationship-in-same-store: prefer the store relationships array over separate create_relationship calls; use create_relationships only as follow-up when target id was unknown at store time - Canonical relationship examples: 8 typed REFERS_TO patterns (person→org, task→conversation, activity→source conversation, issue→plan/spec, note→subject, event→place, task→person, entity→source artifact) - retrieve_related_entities for traversal guidance - Relationship direction convention for REFERS_TO and PART_OF Add pointer-only entry in cli_agent_instructions.md per canonical-first sync rules (no duplicating full instruction body). Update Design rationale section inventory to include [RELATIONSHIP CREATION]. Co-Authored-By: Claude Sonnet 4.6 --- docs/developer/cli_agent_instructions.md | 4 ++++ docs/developer/mcp/instructions.md | 9 ++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/developer/cli_agent_instructions.md b/docs/developer/cli_agent_instructions.md index 8be4fe8bd..0cd375ced 100644 --- a/docs/developer/cli_agent_instructions.md +++ b/docs/developer/cli_agent_instructions.md @@ -71,6 +71,10 @@ neotoma relationships create --source-entity-id --target-entity-i Entity IDs are returned in the `store` response (`entities[].entity_id`). If `relationships create` fails or is unavailable, check `neotoma relationships --help` for current syntax. +## Relationship creation guidance (canonical) + +For when and how to link a newly stored entity to existing entities (pre-store candidate discovery, `retrieve_related_entities`, canonical relationship examples, direction convention), see `docs/developer/mcp/instructions.md` [RELATIONSHIP CREATION]. + ## Pre-check before storing (CLI backup) Before storing a new entity, check for an existing record to avoid duplicates: diff --git a/docs/developer/mcp/instructions.md b/docs/developer/mcp/instructions.md index 1a19b046f..1a132e181 100644 --- a/docs/developer/mcp/instructions.md +++ b/docs/developer/mcp/instructions.md @@ -132,6 +132,13 @@ Restore: restore_entity and restore_relationship create restoration observations Merge and duplicate repair: use list_potential_duplicates(entity_type, threshold?, limit?) as a read-only detector; never auto-merge. Confirm candidate pairs with the user or a repair plan, then call merge_entities(from_entity_id, to_entity_id) to rewrite observations from the duplicate into the target and mark the source merged. Split over-merges: use split_entity to re-point a predicate-selected subset of an entity's observations onto a new or existing entity when a prior merge or heuristic resolution collapsed distinct entities. split_entity is the inverse of merge_entities, is idempotent via (user_id, idempotency_key), preserves observation content, and leaves typed relationships bound to the source until rebuilt with create_relationship. +[RELATIONSHIP CREATION] +Pre-store candidate discovery: before completing any `store` operation that creates a new entity, check whether the new entity logically relates to entities already in context — same person, same project, same conversation thread, same source document, same organization. Use `retrieve_related_entities` or `retrieve_entity_by_identifier` to discover existing candidates when the session has not already confirmed their ids. When a candidate is found, create the relationship in the same `store` call using the `relationships` array (index-based or id-based entries), or immediately after with `create_relationships`. FORBIDDEN: completing a store turn that creates a new non-bookkeeping entity without first considering whether it has a logical relationship to entities already known this session or returned by bounded retrieval. +Relationship-in-same-store: prefer defining relationships in the `store` `relationships` array over separate `create_relationship` calls. Use `{ relationship_type, source_index, target_index }` when both entities are in the same request, or `{ relationship_type, source_entity_id, target_entity_id }` when linking to an existing entity whose id was returned by retrieval. Only use `create_relationships` as a follow-up when the target entity id was not known at store time. +Canonical relationship examples — use these as a guide when context implies a connection: (1) person or contact → organization/company: `REFERS_TO` (person works at or is affiliated with org); (2) task → conversation: `REFERS_TO` (task was created from or motivated by a conversation turn); (3) workout_session, run, or activity → source conversation: `REFERS_TO` (entity extracted from a chat message describing the session); (4) issue → plan or feature_spec: `REFERS_TO` (issue relates to a plan or spec); (5) note or report → analyzed entity: `REFERS_TO` (analysis references its subject); (6) event → location/place: `REFERS_TO` (event occurs at a place); (7) task → person/contact: `REFERS_TO` (task involves or is assigned to a person); (8) any extracted entity → source document or email: `REFERS_TO` (entity was extracted from that artifact, complementing the `interpretation` / `source_id` provenance chain). These examples are illustrative, not exhaustive — apply the same logic to any entity pair where one logically concerns, involves, or was produced from the other. +retrieve_related_entities for traversal: when you need to check whether an entity is already linked to a candidate target, or when the user asks about connections, call `retrieve_related_entities` with the known `entity_id` before creating a new relationship — this avoids duplicate edges and surfaces existing context. Use `retrieve_graph_neighborhood` for a broader view of an entity's graph position when multi-hop context is needed. +Relationship direction convention: for `REFERS_TO`, set `source_entity_id` to the more specific or derivative entity (the thing that refers) and `target_entity_id` to the more general or foundational entity (the thing being referred to). For `PART_OF`, set source to the part and target to the whole. Do not invert these; incorrect direction degrades graph traversal quality. + [COMMUNICATION & DISPLAY] Silent storage default: do not mention storage, memory, or linking unless the user asked, except when a turn created, updated, or retrieved Neotoma entities and you are required to show them per the display rule below. Do not describe internal persistence in thought or reply (e.g. "Persisting this turn, then replying", "Storing the conversation first"). When confirming something was stored, use memory-related language ("remember", "recall", "stored in memory") and include one of those phrases. Proactive storage: use MCP actions proactively. Store when the user states relevant information; store first, then respond. Do not skip store because the user did not ask to save. @@ -233,7 +240,7 @@ Optional `relationships` on **`store`** is an array of relationship entries. Use The instruction block is tuned so agents can complete a turn (retrieval → user-phase store → attachment EMBEDS → assistant reply → closing store) without opening tool schemas or exploring the MCP tool set: -1. **Labelled sections.** Bracket-prefixed labels ([TURN LIFECYCLE], [DATA MODEL], [GUEST ENTITY SUBMISSION] (includes PII stripping checklist before issue filing), [CROSS-INSTANCE SYNC — PEERS], [SUBSTRATE SUBSCRIPTIONS], [STORE RECIPES], [RETRIEVAL], [PROVENANCE], [TASKS & COMMITMENTS], [ENTITY TYPES & SCHEMA], [ENTITY & RELATIONSHIP LIFECYCLE], [COMMUNICATION & DISPLAY], [ATTRIBUTION & AGENT IDENTITY], [CONVENTIONS], [ISSUE REPORTING], [QA REFLECTION], [ERRORS & RECOVERY], [ONBOARDING]) let agents locate rules by topic and cross-reference from one section to another without restating them. +1. **Labelled sections.** Bracket-prefixed labels ([TURN LIFECYCLE], [DATA MODEL], [GUEST ENTITY SUBMISSION] (includes PII stripping checklist before issue filing), [CROSS-INSTANCE SYNC — PEERS], [SUBSTRATE SUBSCRIPTIONS], [STORE RECIPES], [RETRIEVAL], [PROVENANCE], [TASKS & COMMITMENTS], [ENTITY TYPES & SCHEMA], [ENTITY & RELATIONSHIP LIFECYCLE], [RELATIONSHIP CREATION], [COMMUNICATION & DISPLAY], [ATTRIBUTION & AGENT IDENTITY], [CONVENTIONS], [ISSUE REPORTING], [QA REFLECTION], [ERRORS & RECOVERY], [ONBOARDING]) let agents locate rules by topic and cross-reference from one section to another without restating them. 2. **No exploration.** One prominent line under [STORE RECIPES] forbids listing, globbing, or reading MCP tool descriptor/schema files for chat, attachment, and entity-extraction flows. All parameter names and response paths used by the recipes (`structured.entities[].entity_id`, `unstructured.asset_entity_id`) are inline, so agents never need to open schemas. 3. **Unified store shape.** The user-phase recipe covers chat, extraction, and attachments as a single entities list [conversation, message, …extracted entities] with one invariant (PART_OF from message to conversation) plus REFERS_TO per extracted entity. Each list entry is a **flat** object (fields beside `entity_type`); the legacy **`attributes`** wrapper is forbidden (see first bullets under [STORE RECIPES]). Attachment turns use the same shape plus file_path/file_content and a single follow-up EMBEDS call. 4. **Turn-ordered rules.** [TURN LIFECYCLE] encodes the five-step ordering (retrieval, user-phase store, other actions, reply, closing store) once; other sections reference those step numbers instead of re-describing the ordering. From 9615b866b9ebb34c1bb6eef624029cec5786a146 Mon Sep 17 00:00:00 2001 From: castor-agent Date: Wed, 20 May 2026 08:01:08 +0200 Subject: [PATCH 03/16] ci(review): post comment explaining why automatic review was skipped When the diff-size gate evaluates to substantial=false, a new step posts a PR comment stating the file/line count and the thresholds that would trigger a review, plus the @claude review escape hatch. Co-Authored-By: Claude Sonnet 4.6 --- .github/workflows/claude_pr_review.yml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/.github/workflows/claude_pr_review.yml b/.github/workflows/claude_pr_review.yml index 26639c23e..3c022db52 100644 --- a/.github/workflows/claude_pr_review.yml +++ b/.github/workflows/claude_pr_review.yml @@ -80,6 +80,25 @@ jobs: echo "substantial=false" >> $GITHUB_OUTPUT fi + - name: Comment on skipped review + if: steps.diff_check.outputs.substantial == 'false' && github.event_name == 'pull_request' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + PR_NUMBER=${{ github.event.pull_request.number }} + REPO=${{ github.repository }} + + FILES=$(gh pr view $PR_NUMBER --repo $REPO --json files --jq '.files | length') + ADDITIONS=$(gh pr view $PR_NUMBER --repo $REPO --json additions --jq '.additions') + DELETIONS=$(gh pr view $PR_NUMBER --repo $REPO --json deletions --jq '.deletions') + LINES=$((ADDITIONS + DELETIONS)) + + gh pr comment $PR_NUMBER --repo $REPO --body "**Claude review skipped** — diff is below the automatic-review threshold (${FILES} file(s), ${LINES} line(s) changed). + +Thresholds that trigger a review: >5 files, >150 lines, any \`packages/\` change, any auth-related path, or a simultaneous \`src/\` + \`openapi.yaml\` change. + +To request a review anyway, comment \`@claude review\` on this PR." + - uses: anthropics/claude-code-action@beta if: steps.diff_check.outputs.substantial == 'true' with: From 004ed9c00851516c72cbe56aa66e00f4d2380dd4 Mon Sep 17 00:00:00 2001 From: castor-agent Date: Wed, 20 May 2026 08:18:14 +0200 Subject: [PATCH 04/16] feat(inspector): bump submodule, search fix, and shadcn docs Update inspector to entity history, timeline layers, and design showcase. Refine lexical search so multi-word titles containing registered type names still match (e.g. plan titles ending in "Strategy"). Add shadcn audit/rules and FU-2026-05-003 plan. Docs hierarchy Playwright spec lands separately once GET /docs?format=json is wired. --- .claude/rules/change_guardrails_rules.md | 8 - .claude/rules/gmail_proactive.md | 37 + .claude/rules/inspector_shadcn_rules.md | 81 + .claude/rules/publish_plan_prompt.md | 86 + .claude/settings.json | 14 +- .claude/skills/analyze/SKILL.md | 11 +- .claude/skills/audit/SKILL.md | 12 +- .claude/skills/commit/SKILL.md | 15 +- .claude/skills/create-plan/SKILL.md | 99 ++ .claude/skills/create-prototype/SKILL.md | 98 ++ .claude/skills/create-release/SKILL.md | 451 +++++ .claude/skills/create-rule/SKILL.md | 299 ++++ .claude/skills/create_plan/SKILL.md | 14 +- .claude/skills/create_prototype/SKILL.md | 11 +- .claude/skills/create_release/SKILL.md | 17 +- .claude/skills/create_rule/SKILL.md | 25 +- .claude/skills/debug/SKILL.md | 7 + .claude/skills/design_issues/SKILL.md | 114 -- .claude/skills/draft-illustration/SKILL.md | 92 + .claude/skills/draft_illustration/SKILL.md | 18 +- .claude/skills/final-review/SKILL.md | 100 ++ .claude/skills/final_review/SKILL.md | 11 +- .claude/skills/fix-feature-bug/SKILL.md | 107 ++ .claude/skills/fix_feature_bug/SKILL.md | 18 +- .../skills/manage-error-debugging/SKILL.md | 189 +++ .../skills/manage_error_debugging/SKILL.md | 11 +- .claude/skills/process_prs/SKILL.md | 9 +- .claude/skills/publish-plan/SKILL.md | 218 +++ .claude/skills/publish/SKILL.md | 7 + .claude/skills/publish_plan/SKILL.md | 12 +- .claude/skills/pull/SKILL.md | 7 + .claude/skills/push/SKILL.md | 7 + .claude/skills/release/SKILL.md | 15 +- .claude/skills/report-error/SKILL.md | 1508 +++++++++++++++++ .claude/skills/report/SKILL.md | 7 + .claude/skills/report_error/SKILL.md | 11 +- .claude/skills/run-feature-workflow/SKILL.md | 86 + .claude/skills/run_feature_workflow/SKILL.md | 11 +- .../skills/scrape_chatgpt_workout/SKILL.md | 388 ----- .claude/skills/setup-commands/SKILL.md | 35 + .claude/skills/setup-cursor-copies/SKILL.md | 240 +++ .claude/skills/setup_commands/SKILL.md | 19 +- .claude/skills/setup_cursor_copies/SKILL.md | 30 +- .claude/skills/shadcn/SKILL.md | 255 --- .claude/skills/store-neotoma/SKILL.md | 154 ++ .claude/skills/store_neotoma/SKILL.md | 27 +- .../skills/sync-env-from-1password/SKILL.md | 309 ++++ .../skills/sync_env_from_1password/SKILL.md | 11 +- .cursor/rules/inspector_shadcn_rules.mdc | 1 + ...ation-turn-anchors-ext-apps-widget-host.md | 50 + docs/ui/inspector_shadcn_audit.md | 116 ++ docs/ui/inspector_shadcn_rules.mdc | 78 + inspector | 2 +- src/shared/action_handlers/entity_handlers.ts | 31 +- tests/integration/lexical_search.test.ts | 23 + 55 files changed, 4657 insertions(+), 955 deletions(-) create mode 100644 .claude/rules/gmail_proactive.md create mode 100644 .claude/rules/inspector_shadcn_rules.md create mode 100644 .claude/rules/publish_plan_prompt.md create mode 100644 .claude/skills/create-plan/SKILL.md create mode 100644 .claude/skills/create-prototype/SKILL.md create mode 100644 .claude/skills/create-release/SKILL.md create mode 100644 .claude/skills/create-rule/SKILL.md delete mode 100644 .claude/skills/design_issues/SKILL.md create mode 100644 .claude/skills/draft-illustration/SKILL.md create mode 100644 .claude/skills/final-review/SKILL.md create mode 100644 .claude/skills/fix-feature-bug/SKILL.md create mode 100644 .claude/skills/manage-error-debugging/SKILL.md create mode 100644 .claude/skills/publish-plan/SKILL.md create mode 100644 .claude/skills/report-error/SKILL.md create mode 100644 .claude/skills/run-feature-workflow/SKILL.md delete mode 100644 .claude/skills/scrape_chatgpt_workout/SKILL.md create mode 100644 .claude/skills/setup-commands/SKILL.md create mode 100644 .claude/skills/setup-cursor-copies/SKILL.md delete mode 100644 .claude/skills/shadcn/SKILL.md create mode 100644 .claude/skills/store-neotoma/SKILL.md create mode 100644 .claude/skills/sync-env-from-1password/SKILL.md create mode 120000 .cursor/rules/inspector_shadcn_rules.mdc create mode 100644 docs/plans/fu-2026-05-003-inspector-conversation-turn-anchors-ext-apps-widget-host.md create mode 100644 docs/ui/inspector_shadcn_audit.md create mode 100644 docs/ui/inspector_shadcn_rules.mdc diff --git a/.claude/rules/change_guardrails_rules.md b/.claude/rules/change_guardrails_rules.md index 78cdbf4c9..685c65c54 100644 --- a/.claude/rules/change_guardrails_rules.md +++ b/.claude/rules/change_guardrails_rules.md @@ -93,9 +93,6 @@ These rules sit between subsystems. Each canonical doc covers its own surface; o 16. Authority over loopback / proxy / `X-Forwarded-For` trust lives in `src/actions.ts` (`isLocalRequest`, `forwardedForValues`, `isProductionEnvironment`) and the matching helpers in `src/services/root_landing/**`. Any new code that needs to know "is this request local?" MUST consume those exports — not a bare `req.socket.remoteAddress` check, not a `Host` header read, not an inlined fork (`docs/security/threat_model.md`). Treat the v0.11.1 advisory shape as a regression class, not a one-off. 17. Every new Express route MUST land in `scripts/security/protected_routes_manifest.json` (auth-required) or in the runtime allow-list with a stated `reason`. Run `npm run security:manifest:write` in the same change; CI's `security_gates` job runs `--check` and rejects drift. 18. Every release that the diff classifier (`scripts/security/classify_diff.js`) labels `sensitive=true` MUST land with a filled `docs/releases/in_progress//security_review.md` and a supplement `Security hardening` section linking it. `none` provider mode is acceptable, manual fill is mandatory. -19. Every new **destructive or data-mutating operation** (DB migration, encryption migration, repair command, bulk-rewrite over user data) MUST ship with a real round-trip integration test that operates on a real file, not in-memory stubs. The test MUST cover: identity (forward→inverse leaves the data unchanged), dry-run non-mutation, idempotency on re-run, and NULL preservation. See `docs/testing/testing_standard.md` § Destructive operations. -20. Every new **external-file-shape parser** (harness transcripts, SQLite imports, third-party config formats) MUST ship with a test that exercises the actual parsing path against a fixture file in the supported format. Detection-only tests (asserting `detectSource` returns the right tag) are not coverage of parsing. -21. Every new **HTTP server runtime-config knob** (timeouts, connection behavior, header policy) MUST ship with a test that asserts the *runtime* behavior — a response header value, a socket lifetime, an observable connection property — not just the source string. A constant declared in code with no runtime assertion regresses silently. ### MUST NOT @@ -152,11 +149,6 @@ Before opening a PR that touches any surface in the Touchpoint Matrix, confirm: - [ ] `npm run security:classify-diff` recorded; if `sensitive=true`, `npm run security:lint` is clean, `npm run security:manifest:check` passes, `npm run test:security:auth-matrix` passes, and `docs/releases/in_progress//security_review.md` exists with a sign-off verdict. - [ ] New Express routes registered in `protected_routes_manifest.json` (or runtime unauth allow-list with a `reason`); manifest regenerated via `npm run security:manifest:write` when needed. - [ ] No bare `req.socket.remoteAddress`, `X-Forwarded-For`, or `Host` reads outside `src/actions.ts` / `src/services/root_landing/**`; auth-local fallbacks (`!auth && isLocalRequest`) gated through `assertExplicitlyTrusted`. -- [ ] **User-facing-surface coverage**: any new CLI command, new CLI flag, new destructive/data-mutating operation, new external-file-shape parser, or new HTTP runtime-config knob has a test that exercises the **user-observable behavior end-to-end**, not just a helper function. A test file with the right name that only covers an internal helper is not coverage. Required regression tests by surface class: - - Destructive operations (DB migrations, encryption migrations, repair commands): real round-trip test against a real file — encrypt→decrypt identity, dry-run non-mutation, idempotency on re-run, NULL preservation. - - External-file-shape parsers (harness transcripts, SQLite imports, third-party config): at least one fixture per supported format that exercises the actual parser path, not just `detectSource`. - - Discovery / detection / parser pairs: a roundtrip test asserting paths emitted by discovery are parseable by the parser. - - HTTP runtime config (timeouts, headers, keep-alive): a test that asserts the *runtime* behavior (response header, socket lifetime), not just the source string. ## Canonical doc index diff --git a/.claude/rules/gmail_proactive.md b/.claude/rules/gmail_proactive.md new file mode 100644 index 000000000..3f64b0300 --- /dev/null +++ b/.claude/rules/gmail_proactive.md @@ -0,0 +1,37 @@ +--- +description: Agents MUST use the Gmail MCP proactively to check email whenever task context makes it relevant, without waiting to be asked. +globs: +alwaysApply: true +--- + + + + +# Gmail Proactive Check Rule + +## Purpose + +Ensures agents check Gmail proactively when task context implies email may contain relevant information, rather than waiting for an explicit instruction to check. + +## Trigger Patterns + +Agents MUST check Gmail proactively when: + +- Waiting on an external action that sends email confirmation (account signups, invitations, verifications, notifications) +- A workflow step is blocked pending an email (e.g. invitation acceptance, token delivery, approval) +- The user asks about status of something that would be communicated via email +- Setting up third-party integrations that send credentials or confirmation links by email +- Any context where "did the email arrive?" is a natural next question + +## Agent Actions + +1. Use `mcp__aa7dd3ca-ee1b-423d-8787-cf03044822ee__search_threads` with a targeted query (recipient, sender, subject keywords, time window) +2. If relevant emails are found, surface the key details and next action +3. If no relevant emails are found, report that and suggest next steps + +## Constraints + +- MUST check Gmail proactively when task context implies email is a dependency — do not wait for the user to say "check my email" +- MUST use targeted queries (narrow by sender, recipient, subject, or time) rather than fetching all mail +- MUST store any emails retrieved in Neotoma per the external-tool store-first rule +- MUST NOT check Gmail in unrelated contexts where email is not a plausible dependency diff --git a/.claude/rules/inspector_shadcn_rules.md b/.claude/rules/inspector_shadcn_rules.md new file mode 100644 index 000000000..280394961 --- /dev/null +++ b/.claude/rules/inspector_shadcn_rules.md @@ -0,0 +1,81 @@ +--- +description: Enforce shadcn/ui primitives in Inspector UI work; prefer existing inspector components over native HTML controls. +globs: + - inspector/** +alwaysApply: false +--- + + + + +# Inspector shadcn UI Rules + +## Purpose + +Keep the Inspector app visually and behaviorally consistent with the Neotoma design system by using shadcn/ui-style primitives from `inspector/src/components/ui/` instead of ad hoc native HTML controls and one-off Tailwind patterns. + +Inspector is an inspection surface, not a marketing or agentic UI. Prefer consistency, accessibility, and design tokens over decorative variation. + +## Canonical references + +- `docs/ui/shadcn_components.md` — component inventory and Select vs DropdownMenu guidance +- `docs/ui/design_system/implementation_notes.md` — prefer shadcn over native HTML, tokens, dark mode, WCAG AA +- `docs/ui/inspector_shadcn_audit.md` — route-level adoption backlog (if present) + +## Installed primitives (`inspector/src/components/ui/`) + +Prefer these when they fit the interaction. Do not add new primitives in feature work without a short justification. + +**Core (use first):** `Button`, `Input`, `Label`, `Textarea`, `Select`, `Card`, `Badge`, `Separator`, `Dialog`, `Sheet`, `Alert`, `Skeleton`, `Tooltip`, `ScrollArea`, `Tabs`, `Switch`, `DropdownMenu` + +**Composites (preserve mirror contract):** `DataTable`, `Pagination`, `ConfirmDialog` — keep in sync with `frontend/src/components/ui/` when editing; see comments in those files. + +**Available but often unused in pages:** `Checkbox`, `Popover`, `Toggle`, `ToggleGroup` — use them before inventing custom segmented controls or native checkboxes. + +**Not present in Inspector yet (add only with plan):** shadcn `Table` markup inside `DataTable`, `AlertDialog` (prefer `ConfirmDialog` until added), `Command` for search/command palettes. + +## MUST + +### Control selection + +- **MUST** use `Select` from `@/components/ui/select` for single-value dropdowns. **MUST NOT** use native ``. +- **MUST** use `Textarea` for multi-line text entry — not unstyled native `