Skip to content

Iteratively refine learned skill instructions from post-run feedback #67

Description

@neooriginal

This issue is AI generated.

Problem

NeoAgent captures skill drafts after 3+ repeated successful runs (server/services/ai/learning.js) but never re-evaluates or improves those instructions over time. Once a draft is written, it is static. Runs that use the skill and succeed or fail do not feed back into the skill text.

Hermes Agent (Nous Research) demonstrates that a closed feedback loop — measuring skill reuse success rate and rewriting instructions from task outcomes — achieves measurably faster task completion over time (reported 40% speedup in their benchmarks). The current approach captures pattern repetition but discards the signal from actual skill invocations.

Required change

  • Track skill usage: when an agent run invokes a skill, record the run's outcome (success/failure/user correction) against that skill.
  • After N uses, trigger a refinement pass: use the model to diff the current instructions against the observed successful and failed invocations and produce improved instructions.
  • Gate refinement behind auto_skill_learning (which should already be fixed per Honor the auto_skill_learning setting before capturing skill drafts #55).
  • Keep a version history of skill instructions so refinements can be rolled back.

Acceptance criteria

  • Skill invocations are tracked with outcome (success / failure / correction).
  • After a configurable threshold of uses, an async refinement pass updates the instructions.
  • The original instructions are preserved and restorable.
  • Users can view the version history and diff for any learned skill.

Metadata

Metadata

Assignees

No one assigned

    Labels

    autonomousGenerated by AIenhancementNew feature or requestpriority: lowNice to have — refactoring, minor UX, or future featuresreliabilityReliability, resilience, or concurrency concern

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions