From b0306759d05977f505c84847986fa6821f2d6187 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 6 Aug 2026 13:38:00 +0100 Subject: [PATCH] DOC-6939 Document feed coverage and the section role vocabulary Answers the two documentation gaps the applied AI team raised: nothing stated why the feed has fewer URLs than sitemap.xml, and the role values a consumer might filter or rank on were given as five examples followed by "etc." The coverage section explains the exclusion as what it actually is, rather than as a skip list. JSON and Markdown are configured for Hugo's section and page kinds only, so taxonomy listings, term pages and the home page produce no JSON and therefore no record. That is also why the count cannot be reconciled by adding a filter description: there is no filter. It documents the asymmetry in the other direction too, which the assessment spotted but could not explain. A page carrying _build.list: never is dropped from Hugo's page collections, and the sitemap is built from those collections, but the page is still rendered -- so it has JSON and reaches the feed while being absent from the sitemap. integrate/redis-data-integration/reference/data-transformation/cache is the current example. It is genuine reference material reached by direct link, so publishing it is right; only the mismatch needed explaining. The role table is the full fifteen values in the order the patterns are tested, with the trigger words for each, because first match wins and the patterns overlap: a heading of "Version history" is history rather than compatibility purely because history is tested first. Verified against the feed rather than transcribed from the code -- the fifteen documented values and the fifteen the build emits are the same set, with nothing in one and not the other. The caveat about stability is deliberately explicit, and phrased as guidance rather than a promise: treat an unrecognised role as content instead of discarding the section. We cannot offer a contract on these names until there is a schema_version to signal a change, which is still an open item. One detail worth knowing about this page in particular: it is documentation about the feed that also appears in the feed, so a caveat written inside a shortcode has to survive the AI-output path. A paired shortcode does -- the catch-all strip removes the {{< >}} delimiters but not the text between them -- and that was checked rather than assumed, along with the whole section appearing in the JSON. Constraint: the role table on content/ai-agent-resources.md is published as the vocabulary consumers filter on, so any change to ROLE_PATTERNS in build/transform_json_sections.ts has to update it, or we publish a vocabulary the feed no longer uses Ticket: DOC-6939 Co-Authored-By: Claude Opus 5 (1M context) --- content/ai-agent-resources.md | 68 +++++++++++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 2 deletions(-) diff --git a/content/ai-agent-resources.md b/content/ai-agent-resources.md index 74765c041f..36c5973aa6 100644 --- a/content/ai-agent-resources.md +++ b/content/ai-agent-resources.md @@ -37,6 +37,34 @@ Each documentation page has a corresponding JSON file at the same URL with `/ind - Page: `https://redis.io/docs/latest/commands/set/` - JSON: `https://redis.io/docs/latest/commands/set/index.json` +### What the feeds cover + +The feeds contain one record for every documentation page. They deliberately do **not** +contain the taxonomy pages that appear in +[sitemap.xml](https://redis.io/docs/latest/sitemap.xml), so a straight comparison of the two +shows the sitemap with more URLs. + +The excluded pages are: + +- the `categories/` listing and each `categories/` page +- the `tags/` listing and each `tags/` page +- the documentation home page + +These are generated index pages that list other pages. They carry no documentation prose of +their own, so a record for them would add navigation noise without adding content. The +exclusion is a consequence of how the output formats are configured rather than a filter +applied afterwards: JSON and Markdown are produced for Hugo's `section` and `page` kinds +only, and taxonomy, term and home pages are none of those. + +Two consequences worth knowing if you diff the feed against the sitemap: + +- Pages excluded from Hugo's page lists with `_build.list: never` are **absent from the + sitemap but present in the feeds**, because they are still built and rendered. They are + real documentation pages, usually reference material reached by direct link rather than by + browsing. +- Both figures move. The feed is rebuilt at least daily and the corpus grows, so treat any + page count as a snapshot. + ### JSON schema Each document contains: @@ -54,9 +82,11 @@ Each document contains: | `children` | array | Child pages (index pages only) | Each **section** contains: -- `id`: Slugified heading +- `id`: Slugified heading, matching the heading's anchor on the rendered page, so + `#
` links to that section - `title`: Original heading text -- `role`: Semantic role (`overview`, `syntax`, `example`, `parameters`, `returns`, etc.) +- `role`: Semantic role, assigned from the heading text. See + [Section roles](#section-roles) for the current values. - `text`: Section content (code blocks replaced with `[code example]` placeholder) Each **example** contains: @@ -65,6 +95,40 @@ Each **example** contains: - `code`: The code content - `section_id`: Which section this example came from +### Section roles + +Each section carries a `role`, derived from its heading text. These are the values currently +in use: + +| Role | Assigned when the heading begins with | +|------|----------------------------------------| +| `overview` | `overview`, `introduction`, `about`, `description` | +| `syntax` | `syntax`, `usage`, `command`, `signature` | +| `example` | `example`, `demo`, `sample`, `code example` | +| `parameters` | `option`, `parameter`, `argument`, `flag` | +| `returns` | `return`, `response`, `output`, `result` | +| `errors` | `error`, `exception`, `troubleshoot` | +| `performance` | `performance`, `complexity`, `benchmark` | +| `limits` | `limit`, `constraint`, `restriction` | +| `related` | `see also`, `related`, `learn more`, `reference` | +| `setup` | `install`, `setup`, `getting started`, `quickstart` | +| `configuration` | `configur`, `setting` | +| `security` | `security`, `auth`, `permission`, `acl` | +| `history` | `history`, `changelog`, `version history` | +| `compatibility` | `compatib`, `support`, `version` | +| `content` | none of the above | + +The table is in priority order and the first match wins, which matters where the patterns +overlap: a heading of "Version history" is `history` rather than `compatibility`, because +`history` is tested first. + +A page's introductory text, before its first heading, is also given the `overview` role. + +{{< note >}}This vocabulary is descriptive, not a contract. It reflects the values produced +today and may gain entries, or change how a heading maps to a role, without notice. If you +filter or rank on `role`, treat an unrecognized value as `content` rather than discarding the +section, and do not assume a value you rely on will keep its current name.{{< /note >}} + ### Verifying content_hash The `content_hash` can be verified by computing: