Skip to content

feat(content): add ModDB HTML page parser and Playwright headless service - #398

Open
undead2146 wants to merge 2 commits into
community-outpost:developmentfrom
undead2146:feat/moddb-playwright-parser
Open

feat(content): add ModDB HTML page parser and Playwright headless service#398
undead2146 wants to merge 2 commits into
community-outpost:developmentfrom
undead2146:feat/moddb-playwright-parser

Conversation

@undead2146

Copy link
Copy Markdown
Member

Summary

Introduces a managed headless Chromium runtime using Playwright and a robust HTML parser for scraping ModDB mod, addon, and map detail pages.

Note

This is a self-contained feature PR carved out of #265 and must be merged ahead of #265.

Motivation

ModDB pages contain rich metadata, multi-part download mirrors, screenshots, changelogs, and dependency relationships that need structured normalization for client downloads.

Changes

  • Core Contracts & Models:
    • Added IWebPageParser and IPlaywrightService interfaces.
    • Added ModDBFilter, ModDBCategoryMapper, MapDetails, Comment, DownloadableFile, and FileSectionType.
  • Services:
    • Implemented PlaywrightService with managed Chromium browser lifecycle, stealth configuration, and session cookie persistence.
    • Implemented ModDBPageParser for extracting structured game content, direct mirrors, and media galleries.
    • Added HtmlTextHelper for clean string normalization and markup stripping.
  • DI Registration: Registered in ContentPipelineModule.
  • Tests: Added 81 unit tests in ModDBPageParserTests.cs, ManagedChromiumRuntimeTests.cs, ModDBCategoryMapperTests.cs, and HtmlTextHelperTests.cs.

Verification

Comment thread GenHub/GenHub/Features/Content/Services/Parsers/ModDBPageParser.cs Outdated
Comment thread GenHub/GenHub/Features/Content/Services/Parsers/ModDBPageParser.cs
Comment thread GenHub/GenHub/Features/Content/Services/Tools/PlaywrightService.cs
Comment thread GenHub/GenHub/Features/Content/Services/Tools/PlaywrightService.cs Outdated
Comment thread GenHub/GenHub/Features/Content/Services/Tools/PlaywrightService.cs Outdated
@community-outpost community-outpost deleted a comment from coderabbitai Bot Aug 19, 2026
@community-outpost community-outpost deleted a comment from deepsource-io Bot Aug 19, 2026
@community-outpost community-outpost deleted a comment from qodo-code-review Bot Aug 19, 2026
@undead2146
undead2146 force-pushed the feat/moddb-playwright-parser branch from 451d3af to f21f124 Compare August 19, 2026 12:10
@community-outpost community-outpost deleted a comment from qodo-code-review Bot Aug 19, 2026
@community-outpost community-outpost deleted a comment from coderabbitai Bot Aug 19, 2026
@community-outpost community-outpost deleted a comment from deepsource-io Bot Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant