From 9028e40776dc7b00e76b93a52a9fd1b585833a46 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Mon, 10 Aug 2026 13:03:23 +0100 Subject: [PATCH 1/3] DOC-6951 Date each redirect from git history, and say why deletions are not published Item D3, half of it built and half of it deliberately not. The built half is the dates. The map is rendered from .Aliases, which is everything Hugo knows, and Hugo cannot say when a page moved because frontmatter does not record it. git does. build/generate_page_moves.py writes those dates into data/page-moves.json before the build and the template attaches a `moved_on` to each redirect it can date: 570 of 1,061 entries. The rest are vanity or legacy paths that were never a page's location, so there is no date to give -- which the docs now say explicitly, because an absent date could otherwise read as an undated move. Generated rather than committed, gitignored alongside data/examples.json and the other derived files here, so a snapshot of git history cannot go stale. Every lookup tolerates the file being missing, so a bare `hugo` without `make` still builds and simply publishes no dates; verified both ways. The half not built is deleted pages, and the measurement is the reason. git cannot reliably tell a deletion from a move it failed to detect. Of 195 apparent deletions in this history, 83 have a same-named page elsewhere today -- /develop/reference/cluster-spec against operate/oss_and_stack/reference/cluster-spec.md, and so on -- so they almost certainly moved by a delete-plus-add below the rename similarity threshold. The other 112 are no safer, because a page can be renamed and relocated at once and no name-matching heuristic sees that. Publishing those as deleted would tell a consumer to discard a citation that still resolves, which is worse than telling them nothing. So the docs now state that a URL missing from the map is not necessarily gone, and that we deliberately do not claim deletions. That is a better answer than a list that is 40% wrong under the first check anyone would run against it. Verified on a full build with a production baseURL: 1,061 redirects, 27 ambiguous, 10 shadowed, 570 dated, tombstone coverage still 1,061 from 1,061, and the same ten pre-existing REF_NOT_FOUND warnings that main already emits. Learned: the deletions half looked like the easy other half of this item and was the part that could not be done honestly -- 83 of 195 apparent deletions had a live successor, so shipping the list would have instructed consumers to drop working citations Constraint: the map never claims a page was deleted, because git cannot distinguish a deletion from an undetected move, and a wrong deletion is worse for a consumer than no entry at all Constraint: data/page-moves.json is generated before each build and gitignored, so the dates cannot drift from the history they come from Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 1 + Makefile | 11 ++++- build/generate_page_moves.py | 80 +++++++++++++++++++++++++++++++++++ content/ai-agent-resources.md | 12 ++++-- layouts/index.redirects.json | 17 ++++++-- 5 files changed, 113 insertions(+), 8 deletions(-) create mode 100644 build/generate_page_moves.py diff --git a/.gitignore b/.gitignore index 0c7268678d..d67cd86400 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ /resources/ /content/tmp/ /data/examples.json +/data/page-moves.json /data/languages.json /data/repos.json /data/tool_types.json diff --git a/Makefile b/Makefile index 1d0c8f538c..261ee90def 100644 --- a/Makefile +++ b/Makefile @@ -22,7 +22,14 @@ components: components_local: @python3 build/make.py --stack ./data/components_local/index.json -hugo: +# Move dates live only in git history, which Hugo cannot read, so they are written +# into data/ before the build. Generated rather than committed, like the other +# derived files in data/, so a snapshot of history cannot go stale. +page_moves: + @echo "Recording page move dates..." + @python3 build/generate_page_moves.py + +hugo: page_moves @hugo $(HUGO_DEBUG) $(HUGO_BUILD) # json_transform requires hugo to have populated public/ with index.json files @@ -45,7 +52,7 @@ ndjson: redirect_tombstones @echo "Compressing NDJSON feed..." @gzip -kf public/docs.ndjson -serve_hugo: +serve_hugo: page_moves @hugo serve # Passive post-build report of unusually large rendered pages (warn-only). diff --git a/build/generate_page_moves.py b/build/generate_page_moves.py new file mode 100644 index 0000000000..e07002c405 --- /dev/null +++ b/build/generate_page_moves.py @@ -0,0 +1,80 @@ +"""Record when each page move happened, for the published redirect map. + +The map in ``layouts/index.redirects.json`` is rendered from ``.Aliases``, which is +everything Hugo knows. It cannot say *when* a page moved, because frontmatter does +not record that -- only git does. This writes those dates into ``data/page-moves.json`` +before the build, so the template can attach a ``moved_on`` to each redirect it +publishes. + +Why a consumer wants it: a date separates "this redirect is years old, I have surely +seen it" from "this appeared last week, my index is stale". Without one, every entry +in a thousand-line map looks equally new. + +Generated at build time and gitignored, matching ``data/examples.json`` and the other +derived data files in this repo. That keeps it from going stale, which a committed +snapshot of git history would do immediately. + +Deliberately does **not** record deleted pages, though the redirect map would be a +natural home for them. git cannot reliably distinguish a deletion from a move it +failed to detect: of 195 apparent deletions in this repo's history, 83 have a +same-named page somewhere else today, so they almost certainly moved by a +delete-plus-add that fell below git's rename similarity threshold. Publishing those as +deleted would tell a consumer to discard a citation that still resolves, which is +worse than saying nothing at all. The remaining 112 are not safe either, since a page +can be renamed *and* relocated in one go, which no name-matching heuristic can see. +See DOC-6951. +""" + +import json +import logging +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from check_missing_aliases import ( # noqa: E402 + DEFAULT_THRESHOLD, classify, find_moves, git, norm, +) + +logger = logging.getLogger("generate_page_moves") + +OUTPUT = os.path.join("data", "page-moves.json") + + +def main() -> int: + logging.basicConfig(level=logging.INFO, format="%(message)s") + + moves = find_moves(None, DEFAULT_THRESHOLD) + classify(moves) + + # One record per redirect, keyed the way the map keys its entries so the template + # can look a date up directly. Where a page moved more than once the earliest + # date wins, because that is when the old URL stopped resolving -- which is what + # a consumer holding a stale citation actually cares about. + dates: dict[str, str] = {} + for move in sorted(moves, key=lambda m: m.date): + key = "/" + norm(move.old_url) + dates.setdefault(key, move.date) + + try: + head = git("rev-parse", "--short", "HEAD").strip() + except Exception: # noqa: BLE001 - a missing commit must not fail the build + head = "" + + payload = { + "generated_from": head, + "count": len(dates), + "moved_on": dates, + } + + os.makedirs(os.path.dirname(OUTPUT), exist_ok=True) + with open(OUTPUT, "w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=1, sort_keys=True) + handle.write("\n") + + logger.info("generate_page_moves: wrote %d move date(s) to %s.", len(dates), OUTPUT) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/content/ai-agent-resources.md b/content/ai-agent-resources.md index 96541d6558..39579ef7b7 100644 --- a/content/ai-agent-resources.md +++ b/content/ai-agent-resources.md @@ -125,7 +125,7 @@ the page it resolves to: "ambiguous_count": 27, "shadowed_count": 10, "redirects": [ - {"from": "/develop/ai/langcache", "to": "https://redis.io/docs/latest/develop/ai/context-engine/langcache/"} + {"from": "/develop/ai/langcache", "to": "https://redis.io/docs/latest/develop/ai/context-engine/langcache/", "moved_on": "2026-05-11"} ], "ambiguous": [ {"from": "/develop/use/pipelining", "candidates": ["https://redis.io/docs/latest/develop/using-commands/", "https://redis.io/docs/latest/develop/using-commands/pipelining/"]} @@ -143,8 +143,14 @@ Four things worth knowing before you rely on it: record of its own at `/index.json`, so you can resolve one URL without fetching the whole map. - **It is an alias map, not a move log.** Many entries are vanity or legacy paths that - were never a page's location, and there is no date, because the source data does not - record when a page moved. + were never a page's location. Where we can date a move from the repository history, + the entry carries a `moved_on`; entries without one are not undated moves, they are + aliases that were never a page's location in the first place. +- **A URL missing from the map is not necessarily gone.** We publish redirects we can + establish, not a complete account of every URL that ever existed. In particular we + deliberately do not publish a list of deleted pages: the repository history cannot + reliably tell a deletion from a move it failed to detect, and telling you a page was + deleted when it merely moved would be worse than telling you nothing. - `ambiguous` holds the keys that more than one page claims, with every candidate listed. We publish them separately rather than picking one, because the site itself resolves those arbitrarily — so any single answer we gave you would sometimes diff --git a/layouts/index.redirects.json b/layouts/index.redirects.json index c4da604c74..fe488362c9 100644 --- a/layouts/index.redirects.json +++ b/layouts/index.redirects.json @@ -26,8 +26,8 @@ is an even split -- so normalizing here saves every consumer doing it. `to` is absolute, matching the `url` field every per-page record already publishes. - This is an *alias* map, not a move log. Many entries are vanity or legacy paths - that were never a page's location, and it carries no date, because frontmatter - does not record when a page moved. + that were never a page's location, so only the ones git can date carry a + `moved_on`. Only pages Hugo publishes appear, so drafts contribute nothing -- which matches the site, since Hugo emits no stub for a draft's aliases either. @@ -74,6 +74,13 @@ {{- $seen.SetInMap "map" $from ($targets | append $to | uniq) -}} {{- end -}} {{- end -}} +{{- /* Move dates, written into data/ before the build by + build/generate_page_moves.py, because git records when a page moved and + frontmatter does not. Absent when the site is built with bare `hugo` rather + than `make`, so every lookup tolerates a missing file: a date is extra + information, never a precondition. Only moves have one -- a vanity or legacy + alias was never a page's location, so there is no date to give. */ -}} +{{- $movedOn := index (index site.Data "page-moves" | default dict) "moved_on" | default dict -}} {{- $redirects := slice -}} {{- $ambiguous := slice -}} {{- $shadowed := slice -}} @@ -81,7 +88,11 @@ {{- if index ($occupied.Get "url" | default dict) $from -}} {{- $shadowed = $shadowed | append (dict "from" $from "declared" (sort $targets)) -}} {{- else if eq (len $targets) 1 -}} - {{- $redirects = $redirects | append (dict "from" $from "to" (index $targets 0)) -}} + {{- $entry := dict "from" $from "to" (index $targets 0) -}} + {{- with index $movedOn $from -}} + {{- $entry = merge $entry (dict "moved_on" .) -}} + {{- end -}} + {{- $redirects = $redirects | append $entry -}} {{- else -}} {{- $ambiguous = $ambiguous | append (dict "from" $from "candidates" (sort $targets)) -}} {{- end -}} From 52bb287b8b95331a5033aa51b55bb79ce8451fc5 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Mon, 10 Aug 2026 13:52:47 +0100 Subject: [PATCH 2/3] DOC-6951 Fetch full history for the latest build, or the map ships with no dates The dates would have been empty in production. generate_page_moves.py reads git rename records, and the only job that runs `make ci` checks out with actions/checkout's default fetch-depth of 1. A shallow clone has no rename records, so the scan finds nothing, reports zero moves and exits perfectly happily -- the map would have shipped without a single date while every local build showed hundreds. Verified by running the generator inside a --depth 1 clone: 0 dates, no error. Two changes. The latest build now checks out with fetch-depth: 0, and only that job, since the versioned matrix builds run bare `hugo` and never generate the map. And the generator now detects a shallow clone, warns as an Actions annotation, and records `shallow_clone` in the output so nobody has to guess whether a file with no dates means a corpus without moves or a clone that could not see them. This is the third time in this ticket that correct code was defeated by a default in its surroundings: checkout depth in the alias workflow, the baseURL prefix in the map template, and the Actions shell's errexit. All three were invisible locally and none would have failed a test. The pattern is worth more than any of the individual fixes: verify under the conditions CI creates, not the ones a laptop happens to have. Learned: a shallow clone makes any git-history scan return an empty answer rather than an error, so the absence of data has to be distinguishable from an absence of history Constraint: the latest build needs fetch-depth 0 because make ci reads git rename records; the versioned builds do not, and should not pay for it Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/main.yml | 9 +++++++++ build/generate_page_moves.py | 13 +++++++++++++ 2 files changed, 22 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index cdf16725ac..7cbfef381d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -93,6 +93,15 @@ jobs: - name: Check the branch out uses: actions/checkout@v4 + with: + # Full history, because `make ci` runs build/generate_page_moves.py, which + # reads git rename records to date each entry in the published redirect map. + # checkout defaults to fetch-depth: 1, and in a shallow clone that scan finds + # nothing and reports zero moves without failing -- so the map would ship with + # no dates at all while a local full-clone build shows hundreds. Only this job + # needs it; the versioned matrix builds run bare `hugo` and never generate the + # map. See DOC-6951. + fetch-depth: 0 - name: Install dependencies run: make deps diff --git a/build/generate_page_moves.py b/build/generate_page_moves.py index e07002c405..dfefb64dd6 100644 --- a/build/generate_page_moves.py +++ b/build/generate_page_moves.py @@ -44,6 +44,16 @@ def main() -> int: logging.basicConfig(level=logging.INFO, format="%(message)s") + # A shallow clone has no rename records, so the scan below finds nothing and + # reports zero moves perfectly happily -- which is how this shipped with no dates + # at all while local full-clone builds showed hundreds. Say so loudly rather than + # writing an empty file that looks like a corpus with no history. + shallow = git("rev-parse", "--is-shallow-repository").strip() == "true" + if shallow: + logger.warning("::warning::generate_page_moves: this is a shallow clone, so " + "no move dates can be read. The redirect map will publish " + "none. Check out with fetch-depth: 0.") + moves = find_moves(None, DEFAULT_THRESHOLD) classify(moves) @@ -63,6 +73,9 @@ def main() -> int: payload = { "generated_from": head, + # Recorded so a consumer of this file, or anyone reading a build log, can tell + # "this history has no moves" from "this clone could not see the history". + "shallow_clone": shallow, "count": len(dates), "moved_on": dates, } From dd114cef27ea0816bfef6aab9dfd7c8a3d8d8c6b Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Mon, 10 Aug 2026 14:39:07 +0100 Subject: [PATCH 3/3] DOC-6951 Stop classifying moves the date generator never looks at classify() ran on every build and nothing read what it set. It decides whether each move is already aliased, shadowed by a live page, contested or aimed at a draft, and that costs a scan of every published URL plus the frontmatter of every file declaring an alias. Dating needs only old_url and date. Removing the call leaves the output byte-identical -- same 615 dates, same payload -- and takes the generator from 2.78s to 1.04s. Small in absolute terms, but it runs on every build of the site, and work that nothing consumes is worth none of it. It was there because this generator was written by importing the scanner's pipeline wholesale, and find_moves plus classify is how every other caller uses it. Reusing a pipeline is right; reusing the parts of it you do not need is just habit. Learned: importing a pipeline wholesale carried a scan whose results were never read -- reuse the function, not the sequence of calls around it Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) --- build/generate_page_moves.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/build/generate_page_moves.py b/build/generate_page_moves.py index dfefb64dd6..d9694a9c2f 100644 --- a/build/generate_page_moves.py +++ b/build/generate_page_moves.py @@ -33,7 +33,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from check_missing_aliases import ( # noqa: E402 - DEFAULT_THRESHOLD, classify, find_moves, git, norm, + DEFAULT_THRESHOLD, find_moves, git, norm, ) logger = logging.getLogger("generate_page_moves") @@ -54,8 +54,12 @@ def main() -> int: "no move dates can be read. The redirect map will publish " "none. Check out with fetch-depth: 0.") + # Deliberately not classified. classify() decides whether each move is already + # aliased, shadowed, contested or a draft target, which costs a scan of every + # published URL and of the frontmatter of every file declaring an alias -- and + # dating needs only old_url and date. This runs on every build, so the work has to + # be work that is used. moves = find_moves(None, DEFAULT_THRESHOLD) - classify(moves) # One record per redirect, keyed the way the map keys its entries so the template # can look a date up directly. Where a page moved more than once the earliest