Skip to content
BranchesTags

Repository files navigation

ContentPulse

High-Throughput Streaming ETL Engine for Web Content Intelligence

An enterprise-grade, real-time web crawling, SEO analysis, and content extraction platform —
built with .NET 9, orchestrated by .NET Aspire, and designed for production reliability.

.NET 9.0 .NET Aspire PostgreSQL Redis Playwright SignalR Docker Bicep

Table of Contents

⚡ Overview

ContentPulse is not a simple web scraper. It is a distributed, streaming ETL system that:

  1. Extracts web content at scale via dual-engine ingestion (static HTTP + headless Playwright)
  2. Transforms raw HTML through a 3-stage in-memory pipeline (Parse → SEO Analyze → Content Extract)
  3. Loads structured results into PostgreSQL with full relational integrity
  4. Broadcasts live progress to a real-time dashboard via SignalR WebSockets

The system is designed around the same principles used in production data platforms: backpressure-aware channels, idempotent deduplication, bounded concurrency, and scoped dependency lifetimes to prevent resource leaks.

🧰 Tech Stack

Layer Technology Purpose
Runtime .NET 9 / C# 13 .NET High-performance, AOT-ready runtime
Orchestration .NET Aspire Aspire Service discovery, health checks, distributed tracing
Persistence PostgreSQL + EF Core PostgreSQL Relational storage with migration support
Caching / Dedup Redis (StackExchange) Redis O(1) URL deduplication with TTL-based expiry
Static Ingestion AngleSharp AngleSharp Fast, in-memory HTML DOM parsing
Dynamic Ingestion Playwright (Chromium) Playwright Headless JS execution for SPA/CSR sites
Streaming System.Threading.Channels Channels Bounded, backpressure-aware in-memory pipelines
Resilience Polly (.NET Resilience) Polly Exponential backoff, circuit breakers, retries
Real-Time SignalR SignalR WebSocket push to connected dashboard clients
Frontend Blazor Server Blazor Interactive SSR dashboard with glassmorphism UI
IaC Azure Bicep Azure Declarative infrastructure provisioning
CI/CD GitHub Actions GitHub Actions Build → Test → Docker → Deploy pipeline
Containers Docker / Docker Compose Docker Reproducible builds and local orchestration

🏗️ System Architecture

graph TB
    subgraph "Ingress"
        User([fa:fa-user User / Browser])
    end

    subgraph "Orchestration Layer (Aspire AppHost)"
        Gateway[fa:fa-network-wired Service Discovery & Health Checks]
    end

    subgraph "Application Layer"
        Dashboard["fa:fa-desktop Dashboard<br/>(Blazor Server)"]
        Api["fa:fa-server API<br/>(REST + SignalR Hub)"]
        Crawler["fa:fa-spider Crawler Worker<br/>(BackgroundService × 5)"]
    end

    subgraph "Data Layer"
        Redis[("fa:fa-bolt Redis<br/>URL Dedup Cache")]
        Postgres[("fa:fa-database PostgreSQL<br/>Jobs · Pages · SEO Reports")]
    end

    Internet((fa:fa-globe Target Websites))

    User -->|HTTPS| Dashboard
    User -->|REST API| Api
    Gateway -.->|Orchestrates & Monitors| Dashboard
    Gateway -.->|Orchestrates & Monitors| Api
    Gateway -.->|Orchestrates & Monitors| Crawler

    Dashboard <-->|"SignalR (WebSocket)"| Api
    Api <-->|EF Core| Postgres

    Crawler -->|"Poll Pending Jobs (5s)"| Postgres
    Crawler -->|"O(1) Dedup Check"| Redis
    Crawler -->|"Mark Visited (TTL 24h)"| Redis
    Crawler <-->|"HTTP GET / Playwright"| Internet
    Crawler -->|"Persist CrawledPage"| Postgres
    Crawler -->|"POST /notify-update"| Api
Loading

🔄 Streaming ETL Pipeline — Data Engineering

The heart of ContentPulse is a 3-stage streaming ETL pipeline built on System.Threading.Channels. Unlike batch processing, data flows continuously through bounded channels with built-in backpressure — if a downstream stage is slow, upstream stages block automatically rather than buffering unbounded data into memory.

graph LR
    subgraph "Extract"
        CW["CrawlerWorker<br/>(5 parallel consumers)"]
    end

    subgraph "Transform"
        S1["Stage 1: HTML Parsing<br/>(AngleSharp DOM)"]
        S2["Stage 2: SEO Analysis<br/>(15-signal scoring)"]
        S3["Stage 3: Content Extraction<br/>(Readability + NLP)"]
    end

    subgraph "Load"
        PG[("PostgreSQL<br/>Structured Results")]
        WS["SignalR<br/>Live Dashboard"]
    end

    CW -->|"Channel&lt;CrawledPage&gt;<br/>capacity: 50"| S1
    S1 -->|"Channel&lt;ParsedDocument&gt;<br/>capacity: 30"| S2
    S2 -->|"Channel&lt;AnalyzedPage&gt;<br/>capacity: 30"| S3
    S3 -->|"Channel&lt;ProcessedContent&gt;<br/>unbounded"| PG
    CW -->|"HTTP POST"| WS
Loading

Key Data Engineering Patterns

Pattern Implementation Why It Matters
Backpressure BoundedChannelOptions with FullMode.Wait Prevents OOM by blocking producers when consumers are slow
Idempotent Dedup Redis SET with SetContainsAsync + 24h TTL Guarantees at-most-once processing per URL per crawl job
Scoped DB Lifetimes IServiceScopeFactory per operation Prevents captive dependency (stale DbContext in Singletons)
Fan-Out Parallelism 5 ProcessQueueAsync workers consuming from single ChannelReader Thread-safe work-stealing pattern with zero lock contention
Graceful Degradation TryComplete() on channel writers in finally blocks Ensures downstream stages terminate cleanly on upstream failure
Atomic Counters Interlocked.Increment for pipeline metrics Lock-free thread-safe counting across parallel workers

SEO Analysis: 15-Signal Scoring Engine

Each crawled page is evaluated against 15 independent SEO signals across 6 categories, producing a score out of 100:

Category Signals Max Points
Meta Title Tag, Meta Description 20
Content H1 Tag, Heading Hierarchy 10
Accessibility Image Alt Text, Language Attr 15
Links Internal Link Count 10
Technical Canonical, Viewport, Favicon, HTTPS 20
Structured Data Schema.org (JSON-LD), Open Graph, Twitter Card 20
Total 95

⚙️ Concurrency & Resource Governance

┌─────────────────────────────────────────────────────────────────┐
│                     CrawlerWorker (BackgroundService)           │
│                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
│  │ Worker 1 │  │ Worker 2 │  │ Worker 3 │  │ Worker 4 │  │ Worker 5 │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘
│       │              │              │              │              │
│       └──────────────┴──────┬───────┴──────────────┴──────────────┘
│                             │
│              ChannelReader<CrawlTask> (shared, thread-safe)
│                             │
│       ┌─────────────────────┴─────────────────────┐
│       │           Mode Router                      │
│       │                                            │
│       │  Static ──► HttpClient (unlimited)         │
│       │  Dynamic ──► Playwright (SemaphoreSlim=3)  │
│       └────────────────────────────────────────────┘
└─────────────────────────────────────────────────────────────────┘
  • 5 parallel workers consume from a single bounded Channel<CrawlTask> (capacity: 1000)
  • Static mode uses pooled HttpClient instances with Polly resilience (retries + circuit breaker)
  • Dynamic mode is gated by SemaphoreSlim(3, 3) — only 3 Chromium contexts can exist simultaneously, preventing the ~250MB-per-instance memory overhead from causing OOM
  • Polling loop checks for new pending jobs every 5 seconds via a separate Task.Run background thread

🚀 DevOps & CI/CD

Infrastructure as Code (Azure Bicep)

All cloud resources are declaratively defined in infra/main.bicep:

infra/
├── main.bicep              # Root orchestration module
├── resources/
│   ├── containerApp.bicep  # Azure Container Apps definitions
│   ├── postgres.bicep      # Azure Database for PostgreSQL Flexible Server
│   └── redis.bicep         # Azure Cache for Redis

CI/CD Pipeline (GitHub Actions)

graph LR
    subgraph "CI (Pull Request)"
        A[Checkout] --> B[Restore]
        B --> C["Build (dotnet build)"]
        C --> D["Test (dotnet test)"]
    end

    subgraph "CD (Main Branch)"
        D --> E["Docker Build<br/>(multi-stage)"]
        E --> F["Push to ACR<br/>(Azure Container Registry)"]
        F --> G["Deploy to ACA<br/>(Azure Container Apps)"]
        G --> H["Health Check<br/>(/health, /alive)"]
    end
Loading

Docker Strategy

Each microservice has its own multi-stage Dockerfile:

# Example: ContentPulse.Crawler
FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build    # Build stage
FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS final  # Runtime stage (~80MB)
Service Port Description
ContentPulse.Api 8080 REST API + SignalR Hub
ContentPulse.Crawler — Headless worker (no HTTP)
ContentPulse.Dashboard 8080 Blazor Server UI

📁 Project Structure

ContentPulse/
├── src/
│   ├── ContentPulse.AppHost/          # .NET Aspire orchestrator
│   │   └── Program.cs                 # Wires Postgres, Redis, API, Crawler, Dashboard
│   │
│   ├── ContentPulse.Core/             # Domain layer (zero dependencies)
│   │   ├── Models/                    # CrawlJob, CrawledPage, SeoReport, ProcessedContent
│   │   ├── Events/                    # PageCrawledEvent, ContentProcessedEvent
│   │   └── Interfaces/               # Repository contracts, IPipelineStage<TIn,TOut>
│   │
│   ├── ContentPulse.Infrastructure/   # Data access layer
│   │   ├── Data/AppDbContext.cs       # EF Core context with full Fluent API config
│   │   ├── Repositories/             # CrawlJob, CrawledPage, SeoReport repositories
│   │   ├── Caching/                   # Redis URL deduplication service
│   │   └── DependencyInjection.cs     # Service registration extension method
│   │
│   ├── ContentPulse.Pipeline/         # Streaming ETL engine
│   │   ├── PipelineOrchestrator.cs    # BackgroundService: 3-stage channel orchestration
│   │   └── Stages/
│   │       ├── HtmlParsingStage.cs    # AngleSharp DOM parsing
│   │       ├── SeoAnalysisStage.cs    # 15-signal SEO scoring engine
│   │       └── ContentExtractionStage.cs  # Readability + keyword extraction
│   │
│   ├── ContentPulse.Crawler/          # Ingestion worker
│   │   └── Services/
│   │       ├── CrawlerWorker.cs       # BackgroundService: 5 parallel consumers + poller
│   │       ├── CrawlScheduler.cs      # Job lifecycle, dedup, enqueue, scoped DB access
│   │       ├── HttpCrawlService.cs    # Static HTTP ingestion
│   │       └── PlaywrightService.cs   # Dynamic JS rendering (SemaphoreSlim-bounded)
│   │
│   ├── ContentPulse.Api/              # REST API + real-time hub
│   │   ├── Controllers/              # CrawlController, AnalyticsController
│   │   └── Hubs/DashboardHub.cs      # SignalR hub for live push
│   │
│   ├── ContentPulse.Dashboard/        # Blazor Server frontend
│   │   ├── Components/Pages/         # Home.razor (real-time UI)
│   │   ├── Services/ApiClient.cs     # Typed HTTP client for API
│   │   └── wwwroot/app.css           # Glassmorphism design system
│   │
│   └── ContentPulse.ServiceDefaults/  # Shared Aspire defaults (health, telemetry)
│
├── infra/                             # Azure Bicep IaC templates
├── .github/workflows/                 # CI/CD pipeline definitions
├── docker-compose.yml                 # Local multi-container dev environment
└── README.md

🛠️ Getting Started

Prerequisites

Tool Version Purpose
.NET 9.0 SDK 9.0+ Build & run
Docker Desktop Latest Aspire provisions Postgres & Redis containers

Quick Start

# 1. Clone the repository
git clone https://github.com/FelixMatrixar/contentpulse.git
cd contentpulse

# 2. Trust the ASP.NET Core dev certificate (first time only)
dotnet dev-certs https --trust

# 3. Run the Aspire AppHost (spins up everything)
dotnet run --project src/ContentPulse.AppHost/ContentPulse.AppHost.csproj

Aspire will automatically:

  • Pull and start PostgreSQL and Redis Docker containers
  • Launch the API, Crawler, and Dashboard services
  • Open the Aspire Dashboard for distributed tracing and log aggregation

📊 Observability & Telemetry

Every layer is instrumented with structured logging at the Trace level in Development:

Component What's Logged
CrawlerWorker Job discovery, task dispatch, mode routing
CrawlScheduler URL enqueue/dedup decisions, scoped DB writes, job lifecycle transitions
PlaywrightService Semaphore acquire/release, render time, page size
HttpCrawlService Request initiation, TTFB, response status/size
PipelineOrchestrator Stage init, stage completion time, fatal errors
HtmlParsingStage Parse duration, document size, skip reasons
SeoAnalysisStage Score breakdown, DB persistence
ContentExtractionStage Word count, readability grade, estimated read time
Repositories Every CREATE, UPDATE, DELETE with entity IDs
UrlDeduplicationService Redis hit/miss per URL, TTL management
API Controllers Request received, response status, SignalR broadcasts
Dashboard ApiClient Outbound HTTP calls, deserialization results

All logs use structured logging with named parameters (e.g., {JobId}, {Url}, {StatusCode}) for seamless integration with log aggregation tools (Seq, Elasticsearch, Azure Monitor).

👥 Contributors

FelixMatrixar

FelixMatrixar — Architecture, Data Engineering, DevOps & Implementation

Contributions, issues, and feature requests are welcome! Feel free to check the issues page.

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

MIT License

Copyright (c) 2026 FelixMatrixar

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

About

.NET Aspire-orchestrated web crawler that extracts content and runs SEO analysis on pages, with a Blazor dashboard for tracking crawl jobs and per-page signal breakdowns.

Topics

dotnet-aspireseo-analysisweb-crawler

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages