trajectory-tips

A generic, inference-only memory hook and CLI for agent trajectories. It extracts reusable lessons from past runs, stores them as JSONL, retrieves relevant tips for a new task, and injects them into the next prompt.

This project intentionally stays on Bun plus TypeScript. Hook cold-start matters, the CLI should be easy to inspect, and the deterministic offline path has no required network dependency.

Architecture

flowchart LR
  HookStop[Stop or SessionEnd hook] --> Parser[Transcript parser]
  CLIExtract[CLI extract] --> Parser
  Parser --> Validator[Validation]
  Validator --> Extractor[Tip extractor]
  Extractor --> Store[(Atomic JSONL TipStore)]
  Prompt[UserPromptSubmit or CLI retrieve] --> Config[Config]
  Config --> Retriever[Lexical or optional embedding retriever]
  Store --> Retriever
  Retriever --> Injector[Injection formatter]
  Injector --> Context[additionalContext or CLI output]
  Store --> Bench[Offline benchmark]

Loading

Core modules:

• src/config.ts loads defaults, JSON config, and environment overrides.
• src/errors.ts exposes typed errors with stable codes.
• src/logger.ts emits structured JSON logs.
• src/validation.ts validates trajectories and tips before use.
• src/store.ts provides JSONL persistence with validation, lock-file guarded atomic updates, max-tip pruning, byte limits, and rotation.
• src/retriever.ts keeps deterministic lexical retrieval as the default.
• src/embedding-retriever.ts implements the pluggable async embedding retrieval interface with a local deterministic provider.
• hooks/user-prompt-submit.ts injects retrieved tips.
• hooks/stop.ts extracts tips from complete or partial transcripts.

See ARCHITECTURE.md for design details.

Quickstart

bun install
bun test
bun run typecheck
bun run bench

Try the CLI:

bun run src/cli.ts extract examples/sample-trajectory.json --store .trajectory-tips/example.jsonl
bun run src/cli.ts retrieve "List records after AUTH_REQUIRED" --store .trajectory-tips/example.jsonl
bun run src/cli.ts stats --store .trajectory-tips/example.jsonl

Try the embedding retrieval extension without network access:

bun run src/cli.ts retrieve "List records after AUTH_REQUIRED" --retrieval embedding --store .trajectory-tips/example.jsonl

Configuration

Configuration precedence is defaults, JSON config file, then environment variables.

{
  "storePath": ".trajectory-tips/tips.jsonl",
  "retrieval": {
    "k": 5,
    "minSimilarity": 0.08,
    "mode": "lexical"
  },
  "store": {
    "maxTips": 1000,
    "maxBytes": 1048576,
    "rotateBackups": 3,
    "lockTimeoutMs": 5000
  },
  "logLevel": "silent"
}

Environment overrides:

• TRAJECTORY_TIPS_CONFIG
• TRAJECTORY_TIPS_STORE
• TRAJECTORY_TIPS_K
• TRAJECTORY_TIPS_MIN_SIMILARITY
• TRAJECTORY_TIPS_RETRIEVAL
• TRAJECTORY_TIPS_MAX_TIPS
• TRAJECTORY_TIPS_MAX_BYTES
• TRAJECTORY_TIPS_ROTATE_BACKUPS
• TRAJECTORY_TIPS_LOCK_TIMEOUT_MS
• TRAJECTORY_TIPS_LOG_LEVEL

Hook Installation

Use absolute paths in hook commands:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "bun /absolute/path/to/trajectory-tips/hooks/user-prompt-submit.ts"
          }
        ]
      }
    ],
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "bun /absolute/path/to/trajectory-tips/hooks/stop.ts"
          }
        ]
      }
    ],
    "SessionEnd": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "bun /absolute/path/to/trajectory-tips/hooks/stop.ts"
          }
        ]
      }
    ]
  }
}

Extraction runs on Stop or SessionEnd style events. Injection runs on UserPromptSubmit. The same core is usable as a standalone CLI.

Benchmark Results

Run:

bun run bench

Current deterministic offline result:

Benchmark	Before tips	After tips	Precision@K
Synthetic offline env	0.0%	100.0%	100.0%

The offline benchmark uses a seeded synthetic task family covering authentication, pagination, retry, batching, required filters, and identifier resolution. The simulated agent succeeds only when retrieval surfaces the relevant learned rule.

Optional open-task evaluation is deliberately user-supplied and not run in CI:

bun run src/cli.ts bench --open-eval path/to/tasks.jsonl

Honesty

The default benchmark is synthetic, and the offline agent is simulated. It demonstrates the mechanism: extraction, validation, storage, lexical retrieval, optional embedding retrieval plumbing, injection formatting, and benchmark reporting. It is not evidence of paper-level real-world task performance.

The motivating trajectory-derived tips paper reports AppWorld improvements of +3.6 and +14.3 percentage points. Those numbers are cited as background, not reproduced here.

Lexical retrieval is the production default in v1 because it is deterministic and dependency-free. Embedding retrieval is implemented behind EmbeddingProvider; the included LocalHashEmbeddingProvider is offline and deterministic, while network-backed embedders can be added by consumers without changing the core path.

The optional LLM extraction path lazy-loads @anthropic-ai/sdk and requires a runtime API key. Offline tests and the default benchmark require no API key.

Development

bun install --frozen-lockfile
bun test
bun run typecheck
bun run bench

CI runs the same checks.