Comment only when a link we publish has actually rotted

[mmcky]

You were right that the comment wasn't useful. This makes it useful rather than removing it.

The problem

Every PR to this repo got a comment reading 🚨 1 broken link(s) and 3 redirect(s). The broken one was this-host-cannot-resolve.invalid — a fixture built specifically so it can never resolve. The comment said the same thing on every run, so it carried no information, and a contributor seeing 🚨 would reasonably think something was wrong. Six of them had accumulated across #3, #4 and #7.

That is the signal-to-noise problem #2 was about, pointed at our own pull requests.

The fix

The fixture scan drops to fail-on-broken: 'false', which stops the comment while keeping its real-network smoke coverage.

The comment path moves to a new check-own-docs job that renders this repository's Markdown to HTML and checks the links in it. A finding there is a link we publish that has rotted, and the comment names it.

The key property is what happens when nothing is wrong:

Docs state	broken-links-found	Comment
Clean	false	None posted
A link rotted	true	Posted, naming the link

So the comment appears only when it has something to say.

Rendering does the filtering for free. A grep over the raw Markdown finds 15 URLs, but most are placeholders inside fenced code blocks — github.com/user/repo, YOUR-USERNAME/REPO-NAME, the fred\.stlouisfed\.org/.* regex from a YAML sample. Those never become anchors, so the job checks the 4 genuine links and ignores the rest. No allowlist to maintain.

It found a real bug immediately

The README's Marketplace badge linked to https://github.com/marketplace/actions/ai-link-checker, which returns 404. The action was never published there — the v1.1.0 release notes already recorded that no GitHub Release existed before now, and Marketplace requires one. The badge has been advertising a listing that does not exist.

Badge removed. If you would rather publish to Marketplace instead, restoring one line brings it back — but as things stand the link was false.

Verification

Both directions, run locally against the real renderer and the real checker:

• As committed — 4 links checked, 0 broken, 0 redirects → broken-links-found=false → no comment
• With a rotted link appended to a copy of the README — 1 broken → broken-links-found=true → comment posted, naming this-doc-link-rotted.invalid

I also extracted the render step from the YAML exactly as the runner would and ran it under bash --noprofile --norc -e -o pipefail, confirming all six Markdown files render and the heredoc dedents correctly.

No change to the action itself — action.yml and link_checker.py are untouched.

[Copilot]

Pull request overview

This PR reduces CI noise by ensuring link-check PR comments are only posted when the repository’s own published documentation has broken-link findings, rather than repeatedly reporting known fixture failures. It also removes an invalid README Marketplace badge link discovered by the new docs-check pathway.

Changes:

• Stop fixture-based link checks from posting PR comments by setting fail-on-broken: 'false' for the fixture smoke scan.
• Add a check-own-docs CI job that renders this repo’s Markdown to HTML and scans only rendered anchor links, posting a PR comment only when broken links are found.
• Remove the README GitHub Marketplace badge that pointed to a non-existent listing (404).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
tests/README.md	Documents that PR comments are reserved for the repo’s own docs checks (not fixtures).
README.md	Removes the broken GitHub Marketplace badge link.
CHANGELOG.md	Records the new docs-check job and the change to PR-comment behavior.
.github/workflows/ci.yml	Updates fixture scan to avoid PR comments and introduces the new check-own-docs job.

Suppressed comments (1)

.github/workflows/ci.yml:378

• The new docs link-check job will treat transient transport failures (timeouts/DNS) as broken links (status 0) by default, which can trigger PR comments even when nothing has actually changed in our docs. If the intent is to comment only on actionable doc regressions, silence status 0 for this job while keeping the existing default silent codes (403,503).

      with:
        html-path: 'docs-html'
        fail-on-broken: 'true'
        timeout: 20

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mmcky]

Agreed — fixed in e2ae392. The entry now says the comment is posted "only when a link we publish is reported broken", which is what the checker actually observes.

It also records the related change from your suppressed comment on ci.yml: the job runs with silent-codes: '0,403,503', so a transient timeout or DNS failure stays quiet and the comment is reserved for a real status such as a 404. That part matters for anyone reading the changelog to understand when this job will and will not speak up.