analyze-transcript: add test coverage for the analyzer script

[guyoron1]

Upstream: fullsend-ai#5784

Summary

skills/analyze-transcript/analyze-transcript.py has no tests. The script has grown to ~770 lines with several heuristic-based functions (detect_file_type, host/method filtering, error detection, OTLP sniffing) that are easy to regress on.

Test data

• Real transcripts: pull from fullsend-ai/.fullsend GitHub Actions artifacts (gh run download) to get representative agent runs with varied tool usage, error patterns, and network activity.
• Crafted fixtures: build small JSONL files to exercise specific paths -- OTLP telemetry detection, non-dict JSON lines, blank-line-heavy files, stdin input, non-UTF-8 content, edge cases in host matching and method filtering.

Key areas to cover

• detect_file_type: OTLP detection, non-dict JSON, non-JSON lines, unreadable files, blank line budget consumption
• cmd_audit / cmd_summary / cmd_errors: token accounting, error heuristics (tool_result errors, assistant-text mentions, <error> tags), stop reason counting
• cmd_network: --http, --method, --host filter combinations in both text and JSON output modes; DENIED entry handling
• iter_messages / parse_lines: line range filtering, malformed input resilience

Context

PR fullsend-ai#5164 added the audit subcommand, telemetry detection, and HTTP filters. Review rounds surfaced several edge-case bugs that were fixed iteratively. Tests would have caught most of them before review.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 12:56 PM UTC · Completed 1:00 PM UTC
Commit: 9908ddc · View workflow run →

[fullsend-ai-triage]

Triage Summary

Category: Feature (test coverage) · Severity: Medium

Problem

The analyze-transcript.py script (~770 lines) has no tests. Its heuristic-based functions — file type detection, error pattern matching, network filtering, OTLP sniffing — are regression-prone, as demonstrated by edge-case bugs caught during PR review.

Recommended Approach

Create a pytest-based test suite with two categories of test data:

1. Crafted JSONL fixtures for deterministic edge-case coverage:

   • OTLP telemetry detection, non-dict JSON lines, blank-line-heavy files
   • stdin input simulation, non-UTF-8 content
   • Host matching and method filtering edge cases

2. Representative real transcripts (optional, from CI artifacts) for integration-level validation

Key Functions to Cover

Function	Focus areas
detect_file_type	OTLP detection, non-dict JSON, non-JSON lines, unreadable files, blank line budget
cmd_audit / cmd_summary / cmd_errors	Token accounting, error heuristics (tool_result, assistant-text, <error> tags), stop reason counting
cmd_network	--http/--method/--host filter combos (text + JSON modes), DENIED entries
iter_messages / parse_lines	Line range filtering, malformed input resilience

Proposed Test Case

def test_detect_file_type_otlp():
    """JSONL with OTLP span/trace fields detected as OTLP."""

def test_cmd_network_host_and_method_filter():
    """--host + --method filters AND together; DENIED entries handled."""

def test_parse_lines_non_utf8():
    """Non-UTF-8 bytes handled gracefully, no crash."""

def test_cmd_errors_detects_error_tags():
    """<error> tags in assistant messages are counted."""

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[mysticgohan1]

/fs-code

[guyoron1]

/fs-code

[guyoron1]

/fs-code

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 2:40 AM UTC · Completed 2:49 AM UTC
Commit: ba77dbd · View workflow run →

[guyoron1]

Path A complete — closing for Path B run.