Upgrade Registry content, full WranglesPY coverage, and human/agent usability before docs cutover

[ebhills]

Purpose and alignment gate

Upgrade the Registry into the canonical content source for customer documentation and recipe-authoring agents across the supported public WranglesPY capability set. The same reviewed source should generate documentation, discovery/contracts, and recipe-editing schemas.

Assessment and proposed scope only. Implementation is paused until Eric confirms alignment. This issue is the content, coverage, and usability workstream supporting #27: Complete Docs Registry migration and production cutover. Deployment, retirement of the legacy docs, and declaring the Registry production-ready remain separate cutover decisions under #27. WranglesXL functionality is out of scope.

Assessment baseline and limits

Assessed September 7, 2026:

• Docs main: 60016898, Registry 0.2.1 / entry contract 0.2. GitHub comparison confirms the inspected clean local checkout at d2db617 has the same file tree.
• Pinned WranglesPY: 1.20.2 at 7916bf15, matching the clean local implementation inspected. This revision is one manifest-tooling commit ahead of current main 39247724; its changes are manifest generation/schema/tests and workflows, not runtime implementation. Resolve producer provenance on main before a production release.
• Rai: current main 92613c9. The relevant catalog and vendoring files had no local changes; unrelated ongoing work was left untouched.
• Exhaustive static inventory of 98 Registry records and their examples; static inventory of connector operations; focused implementation/docstring/_schema/test-source comparisons for advanced behavior and discovered conflicts.
• Browser review of representative group and individual pages using the existing local build, plus the live conditions and write pages. This was not a fresh site build or a complete responsive/accessibility audit.
• The read-only compiler check passed: 330 generated artifacts are current. No Wrangles recipes, external services, or deployment were executed. Passing compilation is not evidence that examples execute correctly.

Findings

1. Existing coverage is mechanically complete for recipe wrangles, but content is uneven

Measure	Finding
Canonical recipe-wrangle entries	98 / 98 in the pinned runtime manifest
Recipe Writer eligible entries	88; eligibility is distinct from documentation coverage
Descriptions	Mechanical sentence scan: 68 one-sentence, 25 two-sentence, 4 three-sentence, 1 seven-sentence
Entries containing examples	93 / 98; most entries already have examples and sample data
Examples with separate JSON fixtures	3 entries use input/output fixture pairs linked from examples[]; all marked static
Embedded examples and sample data	90 additional entries retain these inside Registry Markdown
No recipe examples	format.price_breaks, standardize.clean, standardize.custom, maths, recipe
Same-example parameter comparison	No scanned recipe example repeats its own wrangle as multiple top-level steps
Missing canonical UUIDs	10; reconcile existing catalog identities, do not invent replacement IDs
Embedded-schema reconciliation	253 differences across 90 entries; these are review items, not 253 proven defects

The work is primarily to improve and consolidate existing examples, not to create examples for 95 empty entries. The separate fixture directory contains three input/output pairs for convert.case, convert.data_type and convert.from_json; generated static copies are the same fixtures, not additional examples.

There is useful industrial content already, but it is inconsistent. For example, convert.case uses product descriptions but demonstrates only upper case; other examples use generic columns, Python lists, colors/shapes, or placeholder functions. The Registry's “verified” status currently establishes mechanical runtime agreement, not complete accepted-value or example correctness. See the reconciliation report.

2. There are correctness gaps to fix alongside the writing

A read-only check parsed the YAML examples and validated their own-wrangle calls against the current generated per-key schemas. 17 of 143 calls failed, across 12 entries: create.jinja, extract.ai, extract.attributes, extract.properties, merge.dictionaries, select.dictionary_element, batch, copy, python, remove_words, rename, and try. Two YAML data literals were excluded. These are example/schema inconsistencies; a failure does not by itself mean WranglesPY rejects the recipe.

Concrete examples:

• extract.ai restricts input to array/null, while its prose/examples and the pinned Python implementation support scalar column names. Its generated schema also loses the embedded output-or-model_id requirement. The first example declares numeric output but illustrates values such as “4.5 inches.”
• All 98 records currently mark ai_powered and requires_external_api_key false, including extract.ai. Access metadata must be reviewed by execution mode and agree with the actual prerequisites.
• The example validator does not perform full recipe-schema validation or execution. Most migrated examples are outside the structured example collection.
• WranglesPY recipe/connector docstrings describe environment-variable precedence differently from implementation and tests. This demonstrates why docstrings and _schema must be double-checked, not copied blindly.

Use the existing authority order: implementation, callable signatures and tests; embedded schema/docstrings; then legacy documentation as migration material. Record upstream disagreements and resolve or explicitly disposition each one.

3. Full WranglesPY coverage needs additional entry types and operation contracts

The current entry schema fixes type to wrangle, and the compiler assumes that every entry maps to a runtime recipe-wrangle key. Changing frontmatter alone cannot add other capability types.

The pinned implementation exposes 38 connector keys with 71 callable read/write/run operations. Including the five runner-owned operations—read.input, read.join, read.union, read.concatenate, write.dataframe—gives 76 operation contracts: 29 read, 26 write, 21 run. These are not represented as operation-specific Registry entries.

The generated recipe schema currently supports only a required top-level wrangles array, with a permissive nested write placeholder. It is not a complete recipe-document schema.

The additional inventory must cover:

• Recipe document structure, read/write sections, lifecycle hooks, variables, custom functions, shared controls, nested execution, batching/concurrency/matrix, and extension mechanisms.
• Every supported connector and each distinct read/write/run operation, including local files, databases, storage/transfers, HTTP, training/model content, notifications, and composition.
• Python recipe execution and CLI invocation, standalone public functions, DataFrame accessors, authentication/configuration, model discovery/content/update/training, and documented AI configuration/cache behavior.
• Active, deprecated, alias and unsupported/internal dispositions. Shared wrappers should link to a common behavioral explanation while retaining distinct callable contracts.

This public-API inventory needs explicit review; importing a helper or exposing a Python name is not sufficient justification to publish it as a supported capability.

4. Group pages and individual pages need different jobs

The current group generator repeats the complete content of each member after a generic introduction. For example, the generated Select group is 1,946 Markdown lines. Readers receive a long reference page before being helped to choose a wrangle.

Individual pages exist, but the generated sidebar includes only group pages. In the existing local preview, the individual extract.ai page had no group sidebar, used the ambiguous title “AI,” and put a long parameter table before its first example. The group entry headings link to local anchors rather than the individual canonical page.

The search implementation indexes route/id-derived text rather than Registry descriptions, aliases and tags. Searching for the exact executable key convert.case returned no results in the local preview.

Recommendation:

• Group pages: explain the task family, compare alternatives, show a representative industrial workflow, then provide a compact catalog linking to canonical entries. Preserve group browsing; make detailed expansion secondary if retained.
• Individual canonical pages: identify the exact callable/context and lead with purpose, prerequisites and a useful example; follow with behavior, parameter interactions, complete reference, edge cases and related capabilities.
• Make canonical pages reachable from group pages, sidebar, search, related links and breadcrumbs. Preserve routes or add redirects for any changes; assess canonical metadata for overlapping views.
• Validate keyboard navigation, narrow screens, table readability, code copying, input/output labels, and direct links during implementation.

Evidence: page and sidebar generation.

5. Human-readable improvements will not automatically reach Rai

Rai currently vendors Registry 0.2.0 and rejects other Registry versions. It also assumes all manifest entries are wrangle keys and that they equal the runtime manifest and full wrangle-schema key sets. Docs already emits 0.2.1, so a bundle update requires deliberate compatibility work.

Rai search uses key/title/aliases/tags/description. Its contract projection omits structured examples, access/capability metadata and relationships, and truncates body guidance to 2,000 characters. Seven current guidance bodies already exceed that limit. Appending multi-run examples deep in a Markdown page therefore does not ensure the agent receives them.

Structured fixtures and raw Markdown are published but currently omitted from the checksum-member bundle that Rai vendors. Examples needed for offline retrieval must be bundled with integrity metadata or embedded in a deliberately supported representation.

Documented capability coverage must remain independent from authoring eligibility and authorization. Some entries are deliberately ineligible, and Rai has additional restrictions. Adding connector documentation must not silently expand what Rai can author or execute.

Evidence: RegistryStore, CatalogView, artifact generation.

Proposed content standard

Apply this standard to every existing entry and every newly included public capability.

1. Description: normally 2–3 purposeful sentences covering what it does, how it works at a high level, and when/why to use it. Advanced capabilities such as extract.ai get a longer explanation where needed. Sentence count is a review aid, not a substitute for useful content.
2. Industrial domain: use small synthetic or sanitized product records: part_code, manufacturer, part_description, dimensions/units, material, quantity, pricing and attributes. Include relevant missing values, duplicate codes, punctuation/Unicode and invalid values. Avoid toy data and invented callable names.
3. Example breadth: demonstrate every key behavior branch and important parameter interaction. Where alternatives are comparable, invoke the same wrangle multiple times in one example against the same source columns, writing clearly named result columns so the reader can compare outcomes. Avoid accidental cascading transformations. Use separate examples for genuinely different input/output shapes, execution contexts, errors or access requirements.
4. Example explanation: state the question, input, complete recipe or clearly labeled fragment, the changed parameters, expected output, and why the output changes. Each example has a stable ID and declares its covered parameters/behaviors. Mark whether an output table shows the full result or selected columns.
5. Parameter contract: document accepted shapes and values, omission versus null/empty behavior, defaults and their source, dependencies/exclusions, required alternatives, input/output effects, and failure/fallback behavior. Cover nested structures, not only top-level types.
6. Verification: move executable examples into structured records and reusable minimal fixtures. Validate complete recipes and outputs; execute deterministic local examples offline. Service-backed examples need controlled live evidence with semantic assertions, not claims of deterministic prose. Clearly distinguish verified outputs from illustrative outputs and manual prerequisites.
7. Two audiences, one authority: write clear user explanations; derive concise agent selection guidance, exact constraints, structured examples and typed relationships from the same source. Make deeper material retrievable on demand rather than relying on unbounded prose or silent truncation.

For extract.ai specifically, cover recipe-defined versus saved definitions, input and output shapes, field versus whole-record examples, output formatting, null/default handling, mode-specific prerequisites, and applicable execution/error controls. Use relevant combinations of key arguments rather than an exhaustive Cartesian product.

Proposed type model and cross-references

Introduce a versioned shared metadata base plus type-specific schemas. Suggested starting taxonomy for alignment:

Type	Responsibility
wrangle	Callable recipe transformation/control with its runtime key
connector	Shared connector capability with distinct read/write/run operation contracts
recipe	Recipe document and section structure/composition
run	Recipe invocation through the Python runner and CLI
function	Supported standalone Python API/accessor contracts
concept	Shared explanations such as conditions, column selection and variables

Use a separate operation/context field for read, wrangles, write, lifecycle actions, Python API and CLI. In particular, a run lifecycle action is different from the Python API that runs an entire recipe. Preserve existing identities and wrangle keys; do not invent new executable names to fit the taxonomy.

Add typed references to stable entry/operation IDs, with a short explanation of the relationship: used-in, composes, alternative, requires, and replacement. Validate targets and compatible types, derive useful reciprocal links, and expose the same references to agents. Keep prerequisites beside the relevant parameter/example and alternatives near selection guidance.

Two initial shared guides should establish the pattern:

Conditions: if, where and where_params

The current public API uses where and where_params. “Where clause” is descriptive terminology, not a separate parameter named where_clause.

Teach whole-operation conditions versus row selection first; then predicate syntax and context-dependent behavior. Explain read filtering, per-destination write filtering, ordinary wrangle merge-back, structural exceptions and unsupported cases. Cover quoted identifiers versus string values, AND/OR grouping, IN/LIKE, numeric comparisons, NULL versus empty strings, positional/named binding, recipe variables, no matches, and new output columns.

Use one industrial dataset across contrasting read/wrangle/write examples and show retained versus transformed/exported rows. Link back from every applicable control definition. The live conditions page already contains useful quoting and binding guidance; retain useful material after reconciling it with runtime behavior. SQLite documentation should be a further reference after the Wrangles-specific explanation.

Nested recipes and dataflow

Create one central comparison, linked from the recipe wrangle, recipe connector operations, read/write guides and lifecycle documentation:

Context	Behavior to explain
read: recipe	Import the child recipe's returned DataFrame
wrangles: recipe	Transform parent data with the child; explain integration and ordinary child-write filtering, including preserved write.dataframe
write: recipe	Pass selected parent data into an export branch; child writes run, and its returned DataFrame does not replace the parent's result
run lifecycle: recipe	Run a child as an independent action without parent DataFrame transfer

Show industrial normalization, importing prepared product data, separate approved/rejected export branches, and an independent preparation action. State parent result versus child side effects explicitly; filtering ordinary child writes is not a guarantee that all possible child side effects are suppressed.

The live write page explains returns, multiple destinations and simple filtering, but not nested recipe branching. Verify the new explanations against recipe connector implementation and recipe wrangle implementation.

Proposed delivery sequence after alignment

1. Inventory and standards: produce an explicit coverage matrix, resolve the type/context vocabulary, finalize human/agent templates and verification requirements, and document source conflicts. Approve representative simple, advanced, connector and shared-guide examples before scaling the editorial pass.
2. Contract and tooling: add typed entry/operation support, full recipe-schema coverage, relationships, structured example validation/execution, and bounded agent retrieval. Coordinate only the necessary producer and Rai compatibility changes with ongoing development.
3. Content completion: revise all 98 entries in reviewable families; add every approved missing public capability; migrate useful legacy material; add the shared guides and relationship links. Track parameter/example coverage and unresolved dispositions.
4. Usability and readiness evidence: regenerate all projections, review group/canonical navigation and search, run offline and controlled live checks separately, verify Rai bundle loading/discovery/example retrieval, and report readiness to Complete Docs Registry migration and production cutover #27. Deployment remains a separate decision.

Acceptance criteria

• Eric confirms scope, page structure, type/context model, and delivery sequence before implementation starts.
• A reviewed inventory accounts for all 98 existing wrangle entries, all 76 identified read/write/run operations, and the supported public Python/CLI surfaces; every omission/deprecation/internal classification has a reason.
• Every in-scope entry meets the description/domain/example standard; key parameter interactions have explicit example coverage and important alternatives are compared in the same example where practical.
• All published executable examples validate against the generated schema; invalid instructional examples, if any, are explicitly marked and tested as expected failures.
• Local deterministic examples execute offline with expected result checks; external-service examples have separate controlled verification evidence or explicit unresolved status.
• Runtime, docstring, embedded-schema and legacy-doc conflicts have a recorded resolution/disposition; access requirements, UUIDs, lifecycle and provenance are accurate.
• Full recipe and operation schemas are generated from typed contracts, with no unexplained loss of required/alternative constraints or accepted parameter shapes.
• Group pages help readers choose; canonical pages provide the complete ordered reference; exact-key and task-intent search, sidebar navigation, links/anchors and responsive layouts pass review.
• Conditions and nested-recipe guides explain the verified context differences and are cross-linked wherever those capabilities apply.
• Rai can load the deliberately versioned bundle and retrieve relevant constraints, examples and relationships without silent loss; tests preserve explicit authoring eligibility and authorization boundaries.
• Source records remain the sole editorial authority; deterministic generation, schema/runtime parity, fixture validation and relationship checks fail on drift. Production provenance and remaining cutover work are reported to Complete Docs Registry migration and production cutover #27.
• No WranglesXL functionality is added to this workstream.

Current disposition: assessment complete; awaiting alignment. No Registry content, source code, branches, or deployments were changed for this issue.