docs: document the API reference docs pipeline in DEVELOPER.md

DEVELOPER.md

@@ -111,6 +111,144 @@ This project uses `release-please` for automated releases. The release pipeline
 * [`toolbox-langchain`](https://pypi.org/project/toolbox-langchain/)
 * [`toolbox-llamaindex`](https://pypi.org/project/toolbox-llamaindex/)
 
+## API Reference Documentation
+
+The API reference is published to [py.mcp-toolbox.dev](https://py.mcp-toolbox.dev).
+It is generated with
+[pydoc-markdown](https://niklasrosenstein.github.io/pydoc-markdown/) and rendered
+by [Hugo](https://gohugo.io/) + [Docsy](https://www.docsy.dev/) from the
+`docs-site/` directory. Docs are built **per package, per version** and served at
+`/<package>/<version>/` (e.g. `/core/v1.0.0/`), with a `/<package>/latest/`
+redirect to the newest release. `<package>` is the URL slug: `core`, `adk`,
+`langchain`, or `llamaindex`.
+
+pydoc-markdown parses the SDK source statically — it never imports the packages —
+so no build or install of the SDK is needed before generating docs.
+
+### What gets documented
+
+Each package's public API is a **curated module list** in
+[`scripts/generate-api-docs.sh`](./scripts/generate-api-docs.sh). The `case` block
+there is the single source of truth for both the valid package slugs and the
+modules rendered for each; internal modules (e.g. `utils`, the transport layer)
+are deliberately left out. To document a new module, add it to that package's
+`MODULES`.
+
+#### Adding a new package
+
+The package must already live under `packages/toolbox-<slug>/`. Then:
+
+1. Add a `case` arm (slug → `TITLE` + `MODULES`) in
+   [`scripts/generate-api-docs.sh`](./scripts/generate-api-docs.sh).
+2. In [`.github/workflows/api-docs.yml`](./.github/workflows/api-docs.yml), add a
+   `refs/tags/toolbox-<slug>-v*` arm to the tag router **and** append the slug to
+   the default `packages=` list (the `dev` build of all packages).
+3. Add a `[[params.versions.<slug>]]` block in
+   [`docs-site/hugo.toml`](./docs-site/hugo.toml) so the version picker lists it
+   (see [Adding a version to the picker](#adding-a-version-to-the-picker)).
+
+### Workflows
+
+The `api-docs.yml` workflow deploys to the `gh-pages` branch. It runs only on the
+upstream repository and uses the `api-docs-deploy` concurrency group, so it never
+races another deploy.
+
+The automatic flow is as follows:
+
+* Push to `main` (or manual dispatch) → builds all four packages as `dev`.
+* Push of a per-package tag → builds that one version **and** rebuilds the root
+  landing page. Tags are hyphenated: `toolbox-core-vX.Y.Z` builds `core`,
+  `toolbox-adk-vX.Y.Z` builds `adk`, and likewise for `langchain` and
+  `llamaindex`.
+* Other tags (e.g. `release-please-*`) are skipped.
+
+### Adding a version to the picker
+
+The version dropdown and the `/<package>/latest/` redirect are driven entirely by
+the hand-edited `[params.versions.<pkg>]` list in `docs-site/hugo.toml` — not by
+the build's version. Before each **new release**, add a `[[params.versions.<pkg>]]`
+block for the version (newest first; the first non-`dev` entry becomes `latest`).
+
+Add it in the **same commit you tag from**: the deploy reads `hugo.toml` from the
+tagged ref and regenerates the dropdown/`latest` files in that run. Only list a
+version whose `/<pkg>/<version>/` pages already exist (or will after this run), or
+the dropdown link 404s.
+
+### Backfilling old docs
+
+Use the **`api-docs-backfill.yml`** (API Reference Backfill) workflow to publish
+docs for a version whose pages are missing — typically releases that predate the
+docs tooling, or a deployment that failed. It builds **one historical version per
+run**.
+
+Unlike `api-docs.yml`, this workflow does **not** deploy to production directly.
+Each run opens a **pull request into the `gh-pages` branch**, so the docs are
+reviewed before they go live. The page is published only when you merge that PR.
+
+How a run works:
+
+1.  It checks out `main` for the current docs tooling (layouts, scripts, version
+    picker), then overlays the requested version's package **`src/`** from its
+    release tag (`toolbox-<pkg>-<version>`), so pydoc-markdown documents that
+    version's API. Only `src/` is overlaid, so the package's `pyproject.toml` and
+    the docs tooling stay in sync with `main`.
+2.  It builds `/<package>/<version>/` (plus the package's `releases`/`latest`
+    files), overlays it onto a clone of the live `gh-pages` tree — existing
+    versions, `CNAME`, and `.nojekyll` are preserved — and opens a PR from branch
+    `backfill/<pkg>-<ver>` with `gh-pages` as the base.
+
+Steps to backfill:
+
+1.  Make sure the version is listed in `docs-site/hugo.toml` (see
+    [Adding a version to the picker](#adding-a-version-to-the-picker)), so the
+    dropdown links to it.
+2.  Trigger the workflow from the Actions tab, or with:
+
+    ```bash
+    gh workflow run api-docs-backfill.yml -f package=core -f version=v1.0.0
+    ```
+
+    To catch up several versions, dispatch it once per `package`/`version`. The
+    concurrency group is scoped per version, so the runs are independent and none
+    are cancelled — each opens its own PR.
+3.  Review the resulting `backfill/<pkg>-<ver>` PR (the diff should be just that
+    version's directory) and **merge it into `gh-pages`** to publish. Re-running
+    the workflow for the same version updates the existing PR's branch.
+
+#### Previewing a backfill PR
+
+GitHub won't render the built HTML in the PR diff. Because the PR branch *is* the
+rendered `gh-pages` tree, check it out and serve it statically — exactly what
+Pages will serve after merge:
+
+```bash
+git fetch origin backfill/<pkg>-<ver>
+# Check the branch out somewhere disposable (a detached worktree keeps your
+# current branch untouched).
+git worktree add --detach /tmp/preview-docs origin/backfill/<pkg>-<ver>
+python3 -m http.server 8099 --directory /tmp/preview-docs
+# → http://localhost:8099/<pkg>/<ver>/   e.g. http://localhost:8099/core/v0.6.0/
+```
+
+The version dropdown fetches `/<pkg>/releases.releases` at runtime, so links to
+versions not present in this branch (other backfills) will 404 locally — that's
+expected. When done, clean up:
+
+```bash
+git worktree remove /tmp/preview-docs
+```
+
+### Building locally
+
+```bash
+# Build a single package/version (base URL must end in a slash).
+./scripts/generate-api-docs.sh core dev http://localhost:8080/
+
+# Serve the output.
+(cd docs-site/public && python3 -m http.server 8080)
+# → http://localhost:8080/core/dev/
+```
+
 ## Support
 
 Check the existing [GitHub Issues](https://github.com/googleapis/mcp-toolbox-sdk-python/issues) for any concerns.