Skip to content

Replace no-op syntax_check eval with real validation, add capability_surface and research_grounding #4

Description

@lukeinglis

Summary

The current eval/score.py has a no-op syntax_check that runs true (always passes) and only two eval dimensions. This inflates the composite score and leaves the factory blind to capability surface and research grounding.

Changes

  • Replace eval_syntax_check: use ast.parse on .py files in scripts/ and bash -n on .sh files
  • Add eval_capability_surface: count slash commands in .claude/commands/, skills with SKILL.md in skills/, and templates in Templates/. Score = count / 50 (capped at 1.0)
  • Add eval_research_grounding: count archive entries in .factory/archive/. Score = entries / 10 (capped at 1.0)
  • Update weights: syntax_check=0.20, observability=0.20, capability_surface=0.30, research_grounding=0.30
  • Preserve existing eval_observability function exactly as-is

From H1 in .factory/strategy/current.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions