Skip to content

feat(verifier): add structured test output parsers for Jest/Vitest and Go test in verifier engine #3

Description

@Kaap10

Problem Statement

Currently, TestOutputParser in src/terminal_agent/verifier/runners.py has dedicated regex matchers for pytest and unittest. When verifying JavaScript/TypeScript projects (npm test, jest, vitest) or Go projects (go test ./...), it falls back to binary exit-code heuristics (1 passed, 0 failed on exit code 0; 0 passed, 1 failed on exit code > 0). This prevents accurate telemetry reporting on how many tests passed vs failed in non-Python projects.

Proposed Solution

Add dedicated parser methods to TestOutputParser:

  1. Jest / Vitest parser:
    • Match: Tests:\s+(\d+)\s+passed,\s+(\d+)\s+failed,\s+(\d+)\s+total
    • Match: Tests:\s+(\d+)\s+passed,\s+(\d+)\s+total
  2. Go test parser:
    • Count lines matching --- PASS: <TestName> and --- FAIL: <TestName>.
    • Match summary lines PASS / FAIL\s+<pkg>\s+[\d\.]+s.

Files to Modify

  • src/terminal_agent/verifier/runners.py
  • tests/unit/test_verifier.py

Acceptance Criteria

  • TestOutputParser.parse(jest_output, exit_code=0) accurately extracts passed, failed, and total test counts.
  • TestOutputParser.parse(vitest_output, exit_code=1) extracts failed test counts and error totals.
  • TestOutputParser.parse(go_test_output, exit_code=0) extracts passing test counts from go test -v.
  • New unit tests in tests/unit/test_verifier.py covering Jest, Vitest, and Go test output samples.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions