[MEDIUM] Add monitoring system architecture documentation

[emiperez95]

Documentation Gap

The monitoring system lacks comprehensive documentation, making it difficult for new developers to understand and maintain.

Missing Documentation

1. Architecture Overview

• System components and their relationships
• Data flow diagrams
• Integration points with Claude Code
• Prometheus/Grafana setup

2. Metrics Reference

• Complete list of exposed metrics
• Metric descriptions and purposes
• Label definitions and cardinality
• Query examples for common scenarios

3. Operational Guide

• Installation and setup procedures
• Configuration options
• Troubleshooting common issues
• Performance tuning guidelines

4. Development Guide

• How to add new metrics
• Testing procedures
• Code organization principles
• Contributing guidelines

Proposed Documentation Structure

README Updates

• Quick start guide
• Configuration examples
• Basic troubleshooting

MONITORING.md Enhancements

• Detailed architecture section
• Complete metrics reference
• Advanced configuration

New Documents

• ARCHITECTURE.md - System design
• METRICS_REFERENCE.md - Complete metric docs
• TROUBLESHOOTING.md - Common issues
• DEVELOPMENT.md - Developer guide

Content Examples

Architecture Diagram

Metrics Reference Table

Metric Name	Type	Description	Labels	Example Query
agent_invocation_total	Gauge	Total agent invocations	agent_name, phase, status, model	sum by (agent_name)
session_duration_seconds	Histogram	Session execution time	session_id	histogram_quantile(0.95, rate(...[5m]))

Configuration Examples

Implementation Tasks

1. Update Existing Docs (1 hour)

• Enhance README.md with quick start
• Update MONITORING.md with architecture
• Add configuration examples

2. Create Architecture Guide (2 hours)

• System design documentation
• Component interaction diagrams
• Data flow documentation
• Integration architecture

3. Complete Metrics Reference (1 hour)

• All metrics documented
• Label explanations
• Query examples
• Cardinality guidelines

4. Operational Documentation (1 hour)

• Installation procedures
• Configuration options
• Monitoring and alerting
• Troubleshooting guide

5. Developer Guide (1 hour)

• Code organization
• Adding new metrics
• Testing procedures
• Contribution workflow

Documentation Standards

Format

• Markdown for all documentation
• Mermaid diagrams for architecture
• Code examples with syntax highlighting
• Consistent formatting and structure

Content Guidelines

• Clear, concise explanations
• Working code examples
• Step-by-step procedures
• Screenshots for complex setups

Maintenance

• Update docs with code changes
• Version documentation with releases
• Regular review for accuracy
• Community feedback incorporation

Validation Criteria

• New developer can set up system from docs
• All metrics documented with examples
• Architecture clearly explained
• Troubleshooting guide covers common issues
• Code examples work as written
• Documentation stays current with code

Success Metrics

• Reduced onboarding time for new developers
• Fewer support questions in issues
• Higher community adoption
• Better system understanding

Effort Estimate

6 hours total

• 1 hour: Update existing documentation
• 2 hours: Architecture and design docs
• 1 hour: Complete metrics reference
• 1 hour: Operational procedures
• 1 hour: Developer guidelines

Dependencies

• Should be updated after major refactoring ([HIGH] Refactor complex methods for maintainability #13)
• Benefits from having test coverage ([CRITICAL] Add unit tests for prometheus_exporter.py #11, [CRITICAL] Add integration tests for monitoring system #12)
• Should document security features ([CRITICAL] No authentication on Prometheus metrics endpoint #6-[HIGH] Implement rate limiting for signal handlers #10)

References

• PR feat: Add comprehensive monitoring with Prometheus and Grafana #1: feat: Add comprehensive monitoring with Prometheus and Grafana #1
• Current MONITORING.md
• Prometheus documentation standards
• Grafana dashboard documentation