feat: use auto-generated docs in the system prompt (#45)#52
feat: use auto-generated docs in the system prompt (#45)#52SyedHannanMehdi wants to merge 30 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Integrates auto-generated tscircuit component documentation into the system prompt building flow to improve AI benchmark accuracy by providing an up-to-date component/props reference.
Changes:
- Adds a CLI script to fetch and cache auto-generated docs into
assets/tscircuit-docs.md. - Introduces multiple system-prompt builder helpers that append cached (or fetched) docs to a base prompt.
- Adds an eval helper module that re-exports a “prompt with docs” value for eval usage.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/update-docs.ts | Fetches docs from registry/GitHub and writes them into assets/tscircuit-docs.md. |
| lib/system-prompt.ts | Builds a system prompt and (optionally) appends cached docs inside a <tscircuit_docs> block. |
| lib/get-system-prompt.ts | Async system prompt builder that reads cache or fetches docs from registry. |
| lib/get-system-prompt-with-docs.ts | Provides split helpers for base prompt vs base+cached-docs prompt. |
| evals/eval-with-docs.eval.ts | Exposes a systemPrompt value for eval files (currently using the markdown-style docs append). |
| assets/.gitkeep | Keeps assets/ tracked for committing cached docs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| * The main system prompt used in all tscircuit prompt benchmarks. | ||
| * | ||
| * Includes: | ||
| * 1. Core instructions for generating valid tscircuit code | ||
| * 2. Auto-generated component documentation for accurate API references |
There was a problem hiding this comment.
The docstring states this is "the main system prompt used in all tscircuit prompt benchmarks", but the benchmarks currently build the system prompt via createLocalCircuitPrompt() (see benchmarks/benchmark-local-circuit*.eval.ts) and nothing imports lib/system-prompt.ts. Please either wire this prompt builder into the benchmark flow or reword the comment to avoid misleading future readers.
| import * as fs from "fs" | ||
| import * as path from "path" |
There was a problem hiding this comment.
Repository convention is to import Node built-ins via the node: specifier with default imports (e.g. import fs from "node:fs", import path from "node:path" as in scripts/threshold-test.ts:2-3). Please align these imports for consistency.
| import * as fs from "fs" | |
| import * as path from "path" | |
| import fs from "node:fs" | |
| import path from "node:path" |
| import { getSystemPromptWithDocs } from "../lib/get-system-prompt-with-docs" | ||
|
|
There was a problem hiding this comment.
PR description mentions appending cached docs inside a clearly delimited <tscircuit_docs> block, but this eval re-export uses getSystemPromptWithDocs() which appends docs under a markdown heading and does not include the <tscircuit_docs> delimiter. Please align the eval prompt output with the intended wrapper (either change the builder to wrap in <tscircuit_docs> or import the builder that already does so).
| export function getBaseSystemPrompt(): string { | ||
| return `You are an expert at generating tscircuit code. tscircuit is a TypeScript library for designing electronic circuits using a React-like JSX/TSX syntax. | ||
|
|
||
| ## Rules | ||
| - Always import components from "tscircuit" |
There was a problem hiding this comment.
There are now multiple hard-coded base prompt variants across lib/system-prompt.ts, lib/get-system-prompt.ts, and this getBaseSystemPrompt() (different rules/examples and different docs wrapper formats). This duplication is likely to drift over time; consider extracting a single shared base prompt builder and having each variant only customize the docs-loading/wrapping strategy.
| import * as fs from "fs" | ||
| import * as path from "path" |
There was a problem hiding this comment.
Repository convention is to import Node built-ins via the node: specifier with default imports (e.g. import fs from "node:fs", import path from "node:path" as in lib/utils/save-prompt.ts:1-2). Using import * as fs from "fs" / import * as path from "path" is inconsistent and can behave differently depending on TS/Bundler settings; please align these imports with the existing pattern.
| import * as fs from "fs" | |
| import * as path from "path" | |
| import fs from "node:fs" | |
| import path from "node:path" |
| * Builds the system prompt used for AI-based circuit generation benchmarks. | ||
| * Incorporates auto-generated tscircuit component documentation so the model | ||
| * has accurate, up-to-date API references. | ||
| */ | ||
| export async function getSystemPrompt(): Promise<string> { |
There was a problem hiding this comment.
This file exports an async getSystemPrompt(), but lib/system-prompt.ts also exports a sync getSystemPrompt() with different formatting/wrapping. Having two similarly-named public APIs for the same concept is confusing and can lead to importing the wrong one; consider consolidating to one implementation or renaming to reflect behavior (sync vs async / cached vs fetch) and standardizing the docs wrapper format.
| * Builds the system prompt used for AI-based circuit generation benchmarks. | |
| * Incorporates auto-generated tscircuit component documentation so the model | |
| * has accurate, up-to-date API references. | |
| */ | |
| export async function getSystemPrompt(): Promise<string> { | |
| * Builds the system prompt used for AI-based circuit generation benchmarks, | |
| * including auto-generated tscircuit component documentation so the model | |
| * has accurate, up-to-date API references. | |
| * | |
| * This async variant fetches the latest docs (from local cache or remote) | |
| * and should be used when you need a prompt with embedded documentation. | |
| */ | |
| export async function getSystemPromptWithDocs(): Promise<string> { |
| import * as fs from "fs" | ||
| import * as path from "path" |
There was a problem hiding this comment.
Repository convention is to import Node built-ins via the node: specifier with default imports (e.g. import fs from "node:fs", import path from "node:path" as in lib/utils/save-prompt.ts:1-2). Please align these imports for consistency and to avoid ESM interop edge-cases.
| import * as fs from "fs" | |
| import * as path from "path" | |
| import fs from "node:fs" | |
| import path from "node:path" |
| import * as fs from "fs" | ||
| import * as path from "path" |
There was a problem hiding this comment.
Repository convention is to import Node built-ins via the node: specifier with default imports (e.g. import fs from "node:fs", import path from "node:path" as in lib/utils/save-prompt.ts:1-2). Please align these imports for consistency.
| import * as fs from "fs" | |
| import * as path from "path" | |
| import fs from "node:fs" | |
| import path from "node:path" |
… shared BASE_SYSTEM_PROMPT
…rcuit_docs> wrapper
…rcuit_docs> wrapper across all helpers
…with repo convention)
…rcuit_docs> block as intended
…into system prompt
…e prompt, wrap docs in <tscircuit_docs>
Summary
Closes #45
This PR integrates the auto-generated tscircuit component documentation into the system prompt used for AI benchmark evaluations.
What Changed
New files
lib/system-prompt.ts<tscircuit_docs>blocklib/get-system-prompt-with-docs.tsgetSystemPromptWithDocs()andgetBaseSystemPrompt()separatelylib/get-system-prompt.tsscripts/update-docs.tsnpx tsx scripts/update-docs.tsevals/eval-with-docs.eval.tssystemPrompt(with docs) for use in eval filesassets/.gitkeepassets/directory is tracked so the cached docs file can be committedHow it works
assets/tscircuit-docs.mdviascripts/update-docs.ts.lib/system-prompt.ts#getSystemPrompt()reads the cached file at runtime and appends it to the base prompt inside a clearly delimited<tscircuit_docs>XML block.Why
The auto-generated docs give the AI model accurate, up-to-date API references for every tscircuit component and its props, reducing hallucinations and improving benchmark scores without requiring manual doc maintenance.