feat: add React frontend, REST API, CI/CD pipeline, and project documentation

[ChefControl]

Summary

This PR introduces the complete frontend service, restructures the manager into a proper REST API, adds CI/CD automation, and provides comprehensive project documentation.

Frontend (34 new files)

• Full React SPA with TypeScript, Tailwind CSS, and Vite
• Pages: Dashboard, Crawl List, New Crawl, Crawl Detail (with Progress, Graph, and Statistics tabs)
• Force-directed graph visualization (D3) with interactive controls and dynamic legend
• Real-time crawl progress via WebSocket with auto-reconnection
• Independently deployable nginx container with SPA fallback and /api/ reverse proxy
• Custom cobweb favicon replacing the default Vite icon
• Helm subchart with NodePort 30080 for external access

Manager API refactoring (16 files, 908 insertions)

• Restructured into proper modules: routes/, services/, models/, state.rs, config.rs
• REST endpoints: POST/GET/DELETE /api/v1/crawls, /crawls/:id/graph, /crawls/:id/stats, /crawls/:id/ws
• Pagination support for crawl listing (limit, offset query params)
• Liveness (/livez) and readiness (/readyz) probe endpoints (K8s best practices)
• Input validation: crawl depth capped to 1–5
• CANCELLED status tracking across all query endpoints (get_crawl_progress, list_crawls, get_crawl_stats)
• Status derivation: running → completed / cancelled (when all cancelled, none completed)
• Atomic cancel operation (single Cypher query, no race window with feeders)
• WebSocket closes on both completed and cancelled status
• CORS and gzip compression middleware
• Default logging changed from tower_http=debug to tower_http=info
• Removed static file serving (now handled by frontend nginx)

Feeder improvements

• Added HTTP health probe server on port 8081 (/livez) for Kubernetes liveness checks
• URL deduplication scoped by crawl_id (independent crawls no longer interfere)
• Configurable environment variables for poll intervals, stale timeout, max depth
• Graceful shutdown resets claimed jobs to PENDING

Shared library additions

• error.rs: Unified CrawlerError type with HTTP status code mapping
• schema.rs: Idempotent Neo4j index/constraint creation on startup (includes composite index for exact URL lookups)

CI/CD pipeline (new)

• GitHub Actions workflow for master merges
• Independent versioning per service via conventional commits
• Docker image builds pushed to GHCR (ghcr.io/bluedotiya/web-crawler/{service})
• Helm chart packaged and published as OCI artifact
• Frontend CI job: lint, type-check, build

Helm charts

• All liveness probes use /livez, all readiness probes use /readyz
• Frontend readiness probe disabled (no meaningful ready vs alive distinction for static nginx)
• Probe paths consistent across subchart and parent values.yaml

Documentation (11 new files)

• Rewritten README.md with architecture diagram, screenshots, quick start guide
• Corrected security section: removed false robots.txt claim, replaced with honest description
• docs/architecture.md: System overview, data flow, concurrency model, WebSocket protocol
• docs/neo4j-graph-model.md: Node schemas, status lifecycle, indexes aligned with schema.rs
• docs/api-reference.md: All endpoints with request/response schemas, cancelled status documented, depth constraint (1–5)
• docs/deployment.md: Helm install (OCI + local), full config reference, K8s architecture
• docs/development.md: Local dev workflow, actual Helm commands (removed non-existent ./dev.sh reference)
• 6 screenshots captured from the live frontend

Cargo workspace

• Explanatory comments on all reqwest = { workspace = true, default-features = false } lines
• Root Cargo.toml comment explaining default-features = false is ignored at workspace level

Other changes

• .dockerignore to reduce build context
• Updated .pre-commit-config.yaml (removed deleted dev.sh hook, added frontend lint)
• Removed dev.sh integration test script (superseded by CI/CD)
• vite-env.d.ts comment explaining its purpose

Test plan

• cd frontend && npm run dev — Vite proxy works for local dev
• docker build -t frontend-test frontend/ — Frontend image builds
• docker build -t manager-test -f manager/Dockerfile . — Manager image builds
• Deploy to minikube: dashboard loads at NodePort 30080
• Create a crawl via the UI, verify WebSocket progress updates
• Verify graph visualization renders with interactive controls
• Verify crawl list shows progress bars with failed (red) segments
• cargo test --workspace — All 19 Rust tests pass
• cd frontend && npm run lint && npm run type-check — Frontend checks pass
• All Mermaid diagrams in docs render correctly on GitHub
• Manager /livez returns 200 {"status":"ok"}
• Manager /readyz returns 200 {"status":"ready"}
• Feeder /livez on port 8081 returns 200 {"status":"ok"}
• Frontend nginx /livez returns 200 ok
• Old /health and /ready paths return 404
• Atomic cancel: create deep crawl → immediate cancel → status is "cancelled"
• cancelled field present in crawl progress, list, and stats responses
• ?status=cancelled filter works in list crawls endpoint
• Depth validation: depth=0 → 400, depth=6 → 400, depth=1 → 201, depth=5 → 201
• Cross-crawl isolation: two crawls for same URL both complete with independent URL sets
• All pods stable with 0 restarts after deployment
• All 29 PR review comments resolved

🤖 Generated with Claude Code

[ChefControl]

Need to fix comments before merging to master