Skip to content

feat: add /ai-observe skill — AI observability and monitoring instrumentation#171

Open
HMAKT99 wants to merge 1 commit intogarrytan:mainfrom
HMAKT99:arun/ai-observe-skill
Open

feat: add /ai-observe skill — AI observability and monitoring instrumentation#171
HMAKT99 wants to merge 1 commit intogarrytan:mainfrom
HMAKT99:arun/ai-observe-skill

Conversation

@HMAKT99
Copy link
Contributor

@HMAKT99 HMAKT99 commented Mar 18, 2026

You ship an LLM feature Monday. It degrades by Friday. Nobody notices.

Traditional monitoring checks uptime and errors. AI features degrade differently: output quality drops, costs creep up, latency increases, the model provider updates the model without telling you. By the time a user complains, it's been broken for days.

What /ai-observe does

You:   /ai-observe

Claude: AI OBSERVABILITY INVENTORY
        Integration    Logging    Metrics    Alerts    Drift Detection
        chat.rb        Partial    None ←     None ←    None ←
        classify.py    Full       Basic      None ←    None ←
        
        COVERAGE: 25% — 3 of 4 integrations are flying blind

        RECOMMENDED METRICS:
        llm_request_latency_ms    Histogram   Alert: p99 > 5000ms
        llm_cost_per_request      Gauge       Alert: > $0.10/request
        llm_output_quality        Gauge       Custom scoring via /eval
        
        DASHBOARD DESIGN:
        Row 1: [Request rate] [Error rate] [Latency p50/p99]
        Row 2: [Daily spend] [Cost per request] [Budget remaining]
        Row 3: [Quality score] [Cache hit rate] [Token efficiency]

Where it fits

gstack instruments the development workflow (plan, review, ship, QA). /ai-observe instruments what you shipped — the AI features running in production.

/review      → code quality before merge
/ship        → push to production  
/ai-observe  → is the AI still working in production?    ← NEW

Only .tmpl committed — bun run gen:skill-docs generates the rest.

Test plan

  • .tmpl follows template pipeline — uses {{PREAMBLE}}
  • Registered in gen-skill-docs.ts, skill-check.ts, both test files
  • bun run gen:skill-docs generates valid SKILL.md
  • All existing tests pass with skill added

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant