Switch URL discovery to sitemap-driven ingestion#18

Open

Copilot wants to merge 24 commits into

mainfrom

copilot/update-url-extraction-from-sitemap

Commits on Feb 1, 2026

Initial plan
Copilot
committed
Update URL sourcing to sitemap

Copilot
and
Shinoryo
committed
Harden sitemap parsing

Copilot
and
Shinoryo
committed
Improve sitemap fetch robustness

Copilot
and
Shinoryo
committed
Add sitemap request headers

Copilot
and
Shinoryo
committed
Refine sitemap traversal

Copilot
and
Shinoryo
committed
Tighten sitemap XML parsing

Copilot
and
Shinoryo
committed
Refactor sitemap helpers

Copilot
and
Shinoryo
committed
Adjust sitemap parsing loop

Copilot
and
Shinoryo
committed
Handle sitemap decode errors

Copilot
and
Shinoryo
committed
Document sitemap constants

Copilot
and
Shinoryo
committed
Update sitemap user agent

Copilot
and
Shinoryo
committed
Enhance sitemap handling

Copilot
and
Shinoryo
committed
Address sitemap review feedback

Copilot
and
Shinoryo
committed
Clarify sitemap defaults

Copilot
and
Shinoryo
committed
Set default User-Agent

Copilot
and
Shinoryo
committed
Refine sitemap normalization

Copilot
and
Shinoryo
committed
Harden sitemap entry parsing

Copilot
and
Shinoryo
committed
Clarify sitemap docstrings

Copilot
and
Shinoryo
committed
Use module logger

Copilot
and
Shinoryo
committed