From 6b2f3519cb155b872a54681bf29c0abda16cdf00 Mon Sep 17 00:00:00 2001 From: Steve Calvert Date: Fri, 13 Feb 2026 18:00:14 -0800 Subject: [PATCH] fix: add vetting phase to skill discovery to filter one-off activities MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The discover command was suggesting skills for one-time events like team mergers and channel migrations. Added a critical vetting phase that evaluates each candidate against: - Recurrence test: Will this happen again? - Frequency threshold: Daily/weekly/monthly prioritization - Cumulative value: frequency × time saved - Disqualifying signals: mergers, migrations, reorgs The agent now shows its vetting work including rejected candidates and reasons. Quality over quantity - a few high-value recommendations beat many weak ones. Co-Authored-By: Claude --- .../agents/work-pattern-analyzer.md | 73 +++++++++++++++---- plugins/glean-skills/commands/discover.md | 71 +++++++++++++----- 2 files changed, 110 insertions(+), 34 deletions(-) diff --git a/plugins/glean-skills/agents/work-pattern-analyzer.md b/plugins/glean-skills/agents/work-pattern-analyzer.md index de753bb..121062a 100644 --- a/plugins/glean-skills/agents/work-pattern-analyzer.md +++ b/plugins/glean-skills/agents/work-pattern-analyzer.md @@ -38,9 +38,37 @@ Look for: - **Workflow sequences**: Steps that often happen together - **Manual processes**: Tasks described in docs that could be automated -### 3. Categorize Opportunities +### 3. Vet Each Candidate (CRITICAL) -Group findings by skill type: +Before recommending any skill, you MUST evaluate each candidate against these criteria: + +**Recurrence Test** - Ask: "Will this person do this again?" +- ✅ Weekly standup prep → YES, happens every week +- ✅ PR review checklist → YES, happens multiple times per week +- ✅ Quarterly planning prep → YES, happens regularly (even if infrequent) +- ❌ Team merger channel migration → NO, one-time organizational event +- ❌ Office move coordination → NO, happens once every few years +- ❌ Annual review writing → MAYBE, but only once per year - low value + +**Frequency Threshold** - Prioritize skills that would be used: +- **High frequency**: Daily or multiple times per week +- **Medium frequency**: Weekly or bi-weekly +- **Low frequency**: Monthly or quarterly (only include if task is very time-consuming) +- **Reject**: One-time events, annual tasks, or organizational changes + +**Cumulative Value Test** - Calculate rough impact: +- Frequency × Time saved per use = Cumulative value +- A 5-minute task done daily (25 min/week) beats a 2-hour task done annually + +**Disqualifying Signals** - REJECT candidates that involve: +- Team mergers, reorgs, or structural changes +- One-time migrations or transitions +- Event-specific planning (unless recurring events) +- Tasks triggered by external one-off circumstances + +### 4. Categorize Remaining Opportunities + +Group vetted findings by skill type: - **Search shortcuts**: Common queries that could be skills - **Preparation workflows**: Meeting prep, code review prep, etc. - **Status generation**: Reports, summaries, digests @@ -61,7 +89,13 @@ Return structured analysis with tables for easy scanning: | **Active Projects** | [from memory] | | **Recent Focus Areas** | [from user_activity] | -### Skill Opportunities Found +### Candidates Considered & Vetted + +| Candidate | Recurrence? | Frequency | Verdict | +|-----------|-------------|-----------|---------| +| [activity] | Yes/No | Daily/Weekly/One-time | ✅ Include / ❌ Reject | + +### Skill Opportunities (Vetted) | # | Pattern | Type | Frequency | Suggested Skill | |---|---------|------|-----------|-----------------| @@ -76,6 +110,8 @@ Return structured analysis with tables for easy scanning: |-----------|---------| | **Type** | Search shortcut / Preparation / Status / Onboarding / Verification | | **Evidence** | [What in the data suggests this] | +| **Recurrence** | [Why this will happen again - be specific] | +| **Frequency** | [How often: daily, weekly, monthly] | | **What it does** | [What the skill would do] | #### 2. [Pattern Name] → `[skill-name]` @@ -84,14 +120,20 @@ Return structured analysis with tables for easy scanning: ### Recommendations #### High Value (frequent + automatable) -| Skill | Why | -|-------|-----| -| `[name]` | [evidence of frequency and time savings] | +| Skill | Frequency | Cumulative Impact | +|-------|-----------|-------------------| +| `[name]` | [daily/weekly] | [time saved × frequency] | + +#### Medium Value (less frequent but significant) +| Skill | Frequency | Cumulative Impact | +|-------|-----------|-------------------| +| `[name]` | [weekly/monthly] | [time saved × frequency] | + +### Rejected Candidates -#### Medium Value (occasional but saves time) -| Skill | Why | -|-------|-----| -| `[name]` | [evidence] | +| Candidate | Reason | +|-----------|--------| +| [one-off task] | One-time event, won't recur | ### Next Steps To create any of these skills, run: @@ -100,8 +142,9 @@ To create any of these skills, run: ## Guidelines -- Focus on patterns, not one-off activities -- Prioritize by frequency × time saved -- Be specific about what the skill would do -- Suggest actionable skill names -- Only report what's found in the data +- **BE SKEPTICAL**: Most activities are NOT good skill candidates +- Reject one-off activities aggressively +- Only recommend skills with clear recurrence evidence +- Prioritize by frequency × time saved (cumulative value) +- A few high-quality recommendations beats many weak ones +- Show your vetting work - explain why candidates were rejected diff --git a/plugins/glean-skills/commands/discover.md b/plugins/glean-skills/commands/discover.md index 50fd30d..11f342b 100644 --- a/plugins/glean-skills/commands/discover.md +++ b/plugins/glean-skills/commands/discover.md @@ -41,7 +41,23 @@ Spawn the `work-pattern-analyzer` agent to analyze the gathered data and identif - Workflow sequences - Manual processes that could be automated -### Phase 3: Present Recommendations +### Phase 3: Vet Recommendations (CRITICAL) + +The agent will critically evaluate each candidate: + +**What Gets Included:** +- Tasks with clear recurrence (daily, weekly, monthly) +- High cumulative value (frequency × time saved) +- Patterns with evidence of repetition in the data + +**What Gets Rejected:** +- One-time events (team mergers, migrations, reorgs) +- Annual or rare tasks with low cumulative value +- Activities triggered by unique circumstances + +The agent shows its vetting work, including rejected candidates and why. + +### Phase 4: Present Recommendations Format findings as actionable recommendations: @@ -53,37 +69,54 @@ Format findings as actionable recommendations: - **Active Projects**: [from memory] - **Recent Focus**: [from activity] +### Vetting Summary +| Candidates Found | Passed Vetting | Rejected | +|------------------|----------------|----------| +| [#] | [#] | [#] | + ### Recommended Skills -#### High Value -| Skill Name | Type | Why | -|------------|------|-----| -| [name] | [type] | [evidence from data] | +#### High Value (Frequent + Automatable) +| Skill Name | Frequency | Cumulative Impact | +|------------|-----------|-------------------| +| [name] | [daily/weekly] | [time saved × frequency] | + +#### Medium Value (Less Frequent but Significant) +| Skill Name | Frequency | Cumulative Impact | +|------------|-----------|-------------------| +| [name] | [monthly] | [time saved × frequency] | -#### Medium Value -| Skill Name | Type | Why | -|------------|------|-----| -| [name] | [type] | [evidence from data] | +### Rejected Candidates +| Candidate | Reason | +|-----------|--------| +| [name] | [why it won't recur] | ### Create a Skill To create any of these skills: -\`\`\` +``` /glean-skills:create -\`\`\` +``` Or describe what you want: -\`\`\` +``` /glean-skills:create "skill that does X when Y happens" -\`\`\` ``` +``` + +## Quality Over Quantity + +A few high-quality recommendations are better than many weak ones. The goal is to find skills that will save significant cumulative time through frequent use. -## Tips +**Good skill candidates:** +- Daily standup prep (5 min × 5 days = 25 min/week saved) +- PR review checklist (10 min × 10 PRs/week = 100 min/week saved) +- Weekly status report (30 min × 1/week = 30 min/week saved) -- Focus on patterns that appear multiple times -- Prioritize by frequency × time saved -- Consider both personal skills and team-sharable skills -- Look for "I always do X before Y" patterns +**Poor skill candidates:** +- Team merger channel migration (2 hours, but happens once) +- Annual performance review (1 hour, but once per year) +- Office relocation checklist (happens once every few years) --- @@ -103,5 +136,5 @@ If user activity data is sparse: ### No Patterns Found If no clear patterns emerge: - This might indicate varied work without repetition +- This is a valid outcome - not everyone has automatable patterns - Ask user about tasks they find tedious or repetitive -- Suggest common starter skills (meeting prep, status updates)