Skip to content

TESTING_ROADMAP.md

Latest commit

 

History

History
246 lines (179 loc) · 4.9 KB

TESTING_ROADMAP.md

File metadata and controls

246 lines (179 loc) · 4.9 KB

MONITOR System Testing Roadmap

Generated: 2026-05-23 22:30:22 UTC

Executive Summary

  • Total Use Cases: 10
  • Tested Use Cases: 0
  • Untested Use Cases: 10
  • Current Coverage: 0.0%
  • Target Coverage: 85%
  • Gap: 10 use cases to test

Epic Breakdown

0-data-layer-DL

  • Use Cases: 26
  • Tested: 0
  • Coverage: 0.0%
  • Untested: DL

1-world-M

  • Use Cases: 25
  • Tested: 0
  • Coverage: 0.0%

10-packs-MP

  • Use Cases: 9
  • Tested: 0
  • Coverage: 0.0%
  • Untested: MP

11-system-SYS

  • Use Cases: 12
  • Tested: 0
  • Coverage: 0.0%
  • Untested: SYS

2-ingestion-I

  • Use Cases: 20
  • Tested: 0
  • Coverage: 0.0%
  • Untested: I

3-identity-M

  • Use Cases: 12
  • Tested: 0
  • Coverage: 0.0%

4-autonomous-gm-P

  • Use Cases: 6
  • Tested: 0
  • Coverage: 0.0%

5-rules-RS

  • Use Cases: 9
  • Tested: 0
  • Coverage: 0.0%
  • Untested: P, RS

6-timeline-Q

  • Use Cases: 19
  • Tested: 0
  • Coverage: 0.0%
  • Untested: M, Q

7-copilot-CF

  • Use Cases: 8
  • Tested: 0
  • Coverage: 0.0%
  • Untested: CF

8-planning-ST

  • Use Cases: 7
  • Tested: 0
  • Coverage: 0.0%
  • Untested: ST

Prioritized Testing Plan

Phase 6: Remaining Use Cases

Description: All remaining untested use cases Use Cases (10): DL, M, P, MP, SYS, I, RS, Q, CF, ST Estimated Tests: ~20

Currently Tested Use Cases

Untested Use Cases (by Epic)

0-data-layer-DL

  • DL

10-packs-MP

  • MP

11-system-SYS

  • SYS

2-ingestion-I

  • I

5-rules-RS

  • P
  • RS

6-timeline-Q

  • M
  • Q

7-copilot-CF

  • CF

8-planning-ST

  • ST

Actionable Next Steps

Immediate Actions (This Sprint)

  1. Complete Phase 1 Testing

    • Focus on: None (all complete!)
    • Estimated effort: 0 tests

  2. Review Existing Test Coverage

    • Run: uv run pytest packages/ --cov=packages/ --cov-report=term-missing
    • Identify modules with < 50% coverage
    • Map modules to use cases

  3. Create Test Templates

    • Use existing behavior tests as templates
    • Follow naming convention: test_<USE_CASE>_behavior.py
    • Include unit tests and integration tests

Medium-term Actions (Next 2-3 Sprints)

  1. Complete Phase 2 & 3

    • Phase 2: 0 use cases, ~0 tests
    • Phase 3: 0 use cases, ~0 tests

  2. Improve Coverage in Key Modules

    • Focus on agents package (SceneLoop, ContextAssembly, Narrator)
    • Focus on data-layer MCP tools
    • Aim for 85%+ coverage in core modules

Long-term Actions (Ongoing)

  1. Complete Phases 4-6

    • Remaining: 10 use cases

  2. Maintain Test Coverage

    • Enforce test requirements in PR reviews
    • Use CI gates to reject code without tests
    • Update tests when use cases change

Coverage Analysis

Low-Coverage Modules

To identify low-coverage modules, run:

uv run pytest packages/ --cov=packages/ --cov-report=term-missing --cov-report=html

Then focus on modules with < 50% coverage:

  • packages/agents/: Agent orchestration and LLM calls

    • Test: SceneLoop, ContextAssembly, Narrator agents
    • Use cases: P-1, P-2, P-3, P-4, P-5, P-6

  • packages/data-layer/src/monitor_data/tools/: MCP tools

    • Test: All MCP tool functions
    • Use cases: DL-1 through DL-26

  • packages/cli/: CLI commands

    • Test: Command handlers and user interaction
    • Use cases: SYS-1, SYS-2, SYS-3, M-1, M-2, P-1

Test Function Recommendations

For each untested use case, create the following test functions:

  1. Schema Validation Tests

    • Test that Pydantic schemas validate correctly
    • Test required fields and optional fields
    • Test invalid inputs

  2. Function Existence Tests

    • Verify that MCP tools exist and are callable
    • Verify that agent methods exist

  3. Happy Path Tests

    • Test successful execution with valid inputs
    • Verify expected outputs

  4. Error Path Tests

    • Test error handling for invalid inputs
    • Test error handling for missing dependencies

  5. Integration Tests (where applicable)

    • Test end-to-end workflows
    • Test cross-layer interactions

Example Test Template

# tests/behavior/test_<USE_CASE>_behavior.py
"""Behavior tests for <USE_CASE>: <Use Case Name>."""

from __future__ import annotations

import pytest
from uuid import uuid4
from unittest.mock import patch, MagicMock

# Import relevant schemas and functions


class Test<USE_CASE_ID>HappyPath:
    """Scenario 1: Happy path tests."""

    def test_<function_name>_exists(self):
        """Test that the function exists."""
        assert <function_name> is not None

    def test_<operation>_success(self):
        """Test successful <operation>."""
        # Arrange
        # Act
        # Assert
        pass


class Test<USE_CASE_ID>ErrorPaths:
    """Scenario 2: Error path tests."""

    def test_<operation>_with_invalid_input(self):
        """Test <operation> with invalid input."""
        with pytest.raises(ValueError):
            # Arrange & Act
            pass

This report is auto-generated. Update as use cases and tests evolve.