Skip to content

Add Skill Evaluation & Quality Rubric CI Check #16

@24kchengYe

Description

@24kchengYe

Summary

Add automated quality checks for SKILL.md files to ensure new contributions meet the project's pedagogical and structural standards.

Proposed Checks (CI / GitHub Actions)

1. Structure Validation

  • All 6 required sections present: Description, Triggers, Methodology, Instructions, Examples, References
  • Description is 1-3 paragraphs (not too short, not too long)
  • At least 2 conversation examples
  • At least 3 academic references

2. Pedagogy Lint

  • Contains Socratic questioning patterns (questions before explanations)
  • Contains spaced repetition / review checkpoint language
  • "What NOT to do" section present
  • No lecturing blocks > 500 words without an interactive break

3. Tree Integration

  • Skill slug matches directory name
  • If adding to tree-config.ts, node exists in correct phase
  • Edges reference valid existing skills

4. Quality Metrics (Advisory, non-blocking)

  • Word count report (typical range: 2,000-8,000 words)
  • Readability score (Flesch-Kincaid)
  • Reference quality (DOI links, publication year)

Implementation

  • Add a scripts/validate-skill.ts that runs these checks
  • Integrate into .github/workflows/validate.yml
  • Run on PRs that modify app/content/skills/**
  • Block merge on structural failures, warn on quality metrics

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions