Revise Registry design: catalog IDs, field ownership, and Rai model metadata

[ebhills]

Problem and outcome

The Registry currently combines documentation Markdown and WranglesPY docstrings/schema by copying substantial technical metadata into Markdown frontmatter. This makes authoring cumbersome and leaves overlapping facts in multiple editable places. The design must also account for API Core's PostgreSQL model catalog and versioned model content.

Produce an agreed design for one authoring owner per fact and a complete generated Registry for documentation, recipe schemas and Rai. Preserve the existing grouped reference presentation and compatibility with current model IDs, recipes and API consumers.

This is a separate design workstream. Content refinement is paused.

Related work and scope boundaries

• Upgrade Registry content, full WranglesPY coverage, and human/agent usability before docs cutover #30 remains the content quality, industrial examples and full capability coverage workstream.
• Add optional Extract guides alongside the grouped Registry reference #31 preserves optional guide drafts as a future enhancement. Guides must complement the grouped reference, not replace it.
• Complete Docs Registry migration and production cutover #27 tracks production cutover. Completing this design does not authorize deployment.
• The current Extract prototype is uncommitted local work on codex/extract-registry-prototype. Preserve it; do not fold it into the redesign or resume content rewriting.
• Temporary database transition scripts are not evidence of current workflows or authoring ownership. They may supply structural clues that must be independently verified.
• Do not add documentation or functionality for WranglesXL. Existing UI terminology may be consulted only to understand current customer-facing labels.
• Initial authorization is assessment and design. Implementation, live database migrations, consumer cutover and deployment require a reviewed concrete plan and subsequent authorization.

Confirmed decisions and requirements

1. Central catalog identity

Eric confirmed that the models table is intended to contain the complete wrangle catalog. Stock and Recipe Wrangles were manually added recently so they also have identities.

Add a dedicated catalog_id allocated by the authoritative PostgreSQL catalog:

• Use a database-generated 64-bit integer counter, not a UUID.
• IDs are unique, immutable and never reused; gaps are acceptable.
• Backfill existing records and assign IDs to new records.
• Preserve existing models.id, execution model_id, saved recipes, permissions and legacy API behavior.
• Carry catalog IDs unchanged into published Registry snapshots and other environments. Define import and environment allocation rules so independent counters cannot collide.
• Adapt the Registry's current UUID-only identity assumption through an explicit versioned compatibility plan.
• Decide a lossless JSON/wire representation for PostgreSQL BIGINT across Python and JavaScript consumers.

A catalog ID identifies a catalog entry. The execution binding must still identify the exact wrangle_key and, when needed, the existing saved-model model_id. A generic wrapper's catalog record must not supply its own identity as the selected saved-model ID.

2. Customer terminology and technical classification

Customer Type means Extract, Classify, Standardize, Lookup, Map, Recipe, etc.; this is primarily database purpose. The current UI maps schema to Map.

Customer Variant combines concepts currently spread across database type and variant:

• DIY corresponds to the customer “My …” / custom-model concept.
• Stock and Bespoke are established families.
• AI is a subtype that can overlap DIY, not necessarily a mutually exclusive family.
• Recipe Wrangle is an internal distinction whose actual stored representation must be verified.
• Values such as v2 carry technical implementation information, distinct from customer Variant.
• Registry record kind—wrangle, connector, recipe structure, runner, etc.—is another dimension.

Existing UI rules recognize DIY/Stock/Bespoke from database type, AI from variant=extract-ai or embedding, and Cloud from variant=cloud. Label/icon/filter behavior is not perfectly consistent, so inventory those rules before defining a single normalized projection.

The user-supplied sample verifies Bespoke/Classify records, including technical v2 values. It does not establish the raw values in newly added Stock/Recipe Wrangle records. Obtain representative sanitized records rather than guessing.

3. Derived metadata for saved custom models

Rai needs a small preview of each applicable saved custom model's training/reference data:

Proposed metadata	Meaning
data_preview.columns	Original column names, in order
data_preview.rows	Up to five representative rows
data_preview.total_rows	Total row count in the referenced data
data_preview.content_version	Exact saved-content version represented

API Core owns this derived metadata. Generate or refresh it when content changes, and return the preview for the relevant version. Define behavior for production-version changes, older version selection, deletion and replacement.

Specify a deterministic sampling rule, preservation of blanks and value types, bounds on payload size, and treatment of non-tabular or unsupported content. Distinguish an empty dataset from an unavailable preview.

Rows must follow the permission required to inspect their underlying model content. A model-listing permission must not automatically grant access to training data. Expose the authorized preview to Rai; do not publish private model previews in the public Registry.

Training/reference columns are distinct from the customer's recipe input/output columns. Rai must continue using the wrapper contract and customer dataset to construct a recipe.

Proposed ownership and normalized mapping — to finalize

The direction is to assemble the complete Registry from smaller authoritative sources. Exact field names and boundaries below need review before implementation.

Existing information	Proposed normalized representation	Authoring owner / rule
New central catalog counter	catalog_id	API Core/PostgreSQL
Existing model identity	Existing id / execution model_id	Preserve API Core behavior
Model display name and catalog tags	name, tags	API Core catalog
Database purpose	wrangle_type	API Core naming projection
Database type	variant.family	Explicit mapping for confirmed families
Database variant	variant.subtype and runtime.implementation_variant	Explicit mapping; preserve unknown technical tags without inventing customer categories
Registry type	kind	Shared catalog/Registry classification; evaluate API Core ownership for all entity kinds as proposed below
Callable key and accepted runtime arguments	wrangle_key, technical parameter contract	WranglesPY, versioned with implementation
Capability description, parameter help, relationships	Editorial content	Docs
Curated recipe examples and expected results	Examples and fixtures	Docs; generate rendered tables
Saved-model-specific notes	model_notes	API Core
Database Ready/Deleted/etc.	model_status	Operational model lifecycle
Registry active/deprecated/removed/etc.	lifecycle_status	Versioned capability lifecycle
Runtime requirements	Explicit mode-dependent prerequisites	Capability contract
Actual permissions, ownership and preferences	Authorized model metadata	API Core
Settings, routing and saved content	Existing model/configuration/content responsibilities	API Core and model stores; no indiscriminate merge or public export
Custom-model data preview	data_preview	Generated versioned API metadata

Review stock notes/descriptions and tags before deciding which existing values are duplicate publication copies and which carry distinct information. Do not silently discard authored information or enable two independent editors for the same fact.

Keep runtime package versions, Registry releases, saved-content versions and implementation tags such as v2 distinct.

Additional proposal: one catalog for all capability kinds

Eric proposed extending the existing models table to include non-wrangle entities such as connectors and run capabilities, given its role as the catalog and central catalog_id issuer. Evaluate this as the preferred direction for a unified catalog; physical table reuse and the final entity taxonomy remain design decisions, not authorization to insert rows.

The benefit is one identity namespace and discovery/relationship system across the documented capability scope. A catalog entry is not automatically a target for a saved-model invocation: connectors and runners have their own contracts, and concepts may have no execution binding.

The design should address:

• Add an explicit kind shared by API catalog metadata and Registry outputs. Keep it independent of customer wrangle Type, Variant and operational status. Do not force connector/run/concept records into legacy purpose or Variant values.
• Define the common fields needed by all entries and which fields apply only to specific kinds. Do not require invented training data, model readiness, batch sizes or model-call bindings for entries to which they do not apply. Check existing database constraints, including any legacy row ID required independently of catalog_id.
• Choose identity granularity deliberately. Show how a connector family and its read/write/run operations are represented when operations need separate documentation, contracts or references. Distinguish a recipe structure/concept, a recipe-executing callable and a saved recipe instance. Not every heading or paragraph needs a catalog record.
• Define typed relationships using catalog IDs for their endpoints, such as an operation belonging to a connector or a capability using a shared condition concept. Decide where the relationships are authored once and how API/Registry projections validate missing targets, versions and visibility.
• Backfill existing rows with verified kinds and deploy kind-aware behavior before inserting non-wrangle entries. Preserve existing model-list, detail, training and execution behavior through explicit kind eligibility and conditional version hydration. Provide a catalog-aware query for consumers intentionally requesting all kinds; general discovery must not cause non-model records to appear as trainable models or receive inappropriate model defaults.
• Keep public publication eligibility and private model access explicit for every kind. A shared table must not make an entry public or expose saved-model content by default.
• Assess whether the existing table can support this cleanly with a common catalog core and kind-specific fields. If its model-specific constraints make that cumbersome, compare a small catalog core table with linked model-specific data while retaining one central identity issuer. A table rename or broad storage rewrite is not a prerequisite.

Current read-only code evidence: API-Core/src/db/wrangles.py includes all type=stock records in the model search query without a kind filter, joins model version history, and supplies model defaults. src/routes/user.py interprets the legacy type query as model purpose, validates its existing model values and hydrates model versions. Adding non-wrangle rows alone would therefore be insufficient. Verify the deployed behavior and identify the compatibility changes before backfilling new kinds.

Technical contract and authoring design

Reconsider the earlier assumption that the full hand-authored Python schema should move into Markdown frontmatter.

Proposed direction:

• Keep one structured executable contract maintained with WranglesPY. Its physical format may be YAML/JSON or a structured declaration; decide after examining current producers and consumers.
• Derive mechanical signature facts automatically. Explicitly retain constraints that signatures cannot express: enums, nested shapes, conditional requirements, supported kwargs, output semantics and service-backed options.
• Keep concise Python documentation useful for Python callers. Avoid a second independently authored customer guide or schema copy.
• Docs should hold small identity references, explanatory text, parameter-help supplements, examples and fixture references.
• Import a pinned runtime contract and versioned approved public catalog metadata; generate complete reference pages, recipe schemas, machine contracts and discovery artifacts.
• Preserve shared definitions for common controls and repeated parameter shapes.
• Reconciliation must surface conflicts; do not silently choose a source simply because it is labeled canonical.
• Avoid a dependency cycle in which PY generates its contract from Docs while Docs simultaneously imports that PY contract.
• Inventory connectors, read/write/run operations and other approved public capabilities as well as recipe wrangles. Evaluate registering these in the same catalog table under the proposal below; do not insert records until the design and migration are approved.

Moving large frontmatter into another file alone does not solve duplication. The design should remove repeated authorship while allowing complete generated consumption artifacts.

Rai and API integration

Preserve the separation between a pinned public capability Registry and live, authorized saved-model information. The code already composes these, but current deployment and version compatibility need verification.

Specify:

• Catalog selection by catalog_id and explicit translation to an execution binding.
• A versioned metadata/authoring API response with normalized fields and authorized custom-model previews.
• Complete capability constraints and useful examples available to Rai without silent truncation or omission.
• Public snapshot versus private/dynamic metadata boundaries.
• Schema, RegistryStore, custom-catalog and tool compatibility during adoption of numeric catalog identities.
• Actual execution-version behavior; a metadata snapshot must not be described as pinning execution unless it does so.

The API Core compact authoring projection was observed in a local branch ahead of its recorded main. Local code evidence does not establish production deployment.

First deliverable: design review, then pause

1. Produce a sanitized field inventory based on current API Core code and representative approved model metadata/content. Record actual versus inferred mappings and missing evidence.
2. Complete the ownership/mapping table, identity rules, content-preview contract, and source-to-output examples for one Extract capability, a saved custom model, and one non-wrangle capability. Include the proposed shared catalog design and its legacy API compatibility rules.
3. Specify an incremental cross-repository migration and verification plan, including rollback/compatibility behavior and proposed implementation issue boundaries.
4. Present the concrete design for Eric's review before changing tracked implementation files, applying schema/data changes, or updating consumers.

Do not copy private routing paths, credentials or sensitive model data into this public planning issue. Read-only metadata inventory should report only the sanitized fields needed to settle the design.

Suggested implementation slices after design approval

1. API Core: additive catalog identity, approved entity-kind scope and compatibility filters, verified backfill/allocation plan, normalized projection and versioned preview metadata.
2. WranglesPY: structured contract producer and parity checks, including required operation coverage.
3. Docs: compact authoring sources and compiler integration; reproduce the existing grouped Extract reference and complete machine artifacts.
4. WranglesAgent/Rai: compatible catalog identity resolution, capability contracts and authorized preview consumption.
5. Separate migration/cutover gate, then resume content work under Upgrade Registry content, full WranglesPY coverage, and human/agent usability before docs cutover #30. Keep optional guides under Add optional Extract guides alongside the grouped Registry reference #31.

Each slice should be a focused change against main. Database migration and production rollout are distinct from code review and local validation.

Acceptance criteria for the design phase

• Existing fields, user terminology, technical classifications and ownership are mapped with evidence.
• Catalog IDs are centrally allocated integers with defined backfill, uniqueness, non-reuse, wire representation and environment/import rules.
• Existing model IDs and execution behavior are preserved, with explicit catalog-to-callable/model bindings.
• Stock, Recipe Wrangle, DIY/AI and relevant technical-variant mappings are verified or explicitly unresolved.
• The shared catalog proposal covers non-wrangle kinds, identity granularity, relationships, kind-specific fields and legacy model API compatibility; physical table design is explicitly decided.
• The custom-model preview contract defines content-version consistency, sampling, access and lifecycle behavior.
• A small authoring example demonstrates removal of duplicated technical metadata while preserving the familiar reference output.
• Public capability snapshots and live authorized model metadata have clear boundaries and version compatibility.
• Every repeated field has one declared authoring owner; generated projections cannot become accidental second masters.
• The cross-repository implementation sequence includes meaningful validation, compatibility and rollout gates.
• Eric reviews the design before implementation begins.

Current status: planning/design only. Content refinement remains paused; no database migration, broad property renaming, implementation or deployment is authorized by creation of this issue.