Skip to content

Exploration proposes an alignment per table #554

Description

@WaylandYang

Exploration today reads a schema and produces entities of two classes it inserted itself. 0035 decisions 1, 2 and 6: it produces instead an alignment proposal per table — the class the table is a table of, the attributes its columns carry, the relations its foreign keys are, and for each aligned column an expression that turns the column into the attribute's value — and a person adopts it.

What a proposal holds

For dw.dwd_ord_dtl on the wide corpus:

column is
amt_pay, qty, ord_st, dt_crt attributes of Order
buyer_id, shop_id relations Order → Customer, Order → Shop
shop_nm, prov, buyer_lvl attributes of Shop, Address, Customer, flattened in
ver, etl_dt, rmk, price_old nothing

Exploration finds existing classes and attributes first (the ontology, with a pack from 0008, usually has Order) and proposes new ones only where none fits — 0003's loop with a schema in place of a corpus. On a base with no ontology it proposes the class (decision 6); the difference from today is that a proposal waits and ensure_concept_types did not.

Where it goes

ontology_proposals already has the four sections this needs — entity_types, attribute_types, relation_types, map_to — and the attribute payload already carries datatype and unit. What it does not have is a way to say "these forty rows are one table's alignment, adopt or reject them together", and decide_proposal today only marks a row; creating the ontology rows is the separate adopt endpoints' job. So this needs:

  • an alignment table: table → class, column → attribute (or relation) with the conversion tree from A column becomes an attribute through the tree a rule computes with #555;
  • a proposal that groups a table's rows, adopted as one thing — adopting Order while rejecting every attribute of it is not a decision anyone means to make;
  • an adopt path that writes the ontology rows and the alignment rows in one transaction.

What exploration does not propose

Definitions. GMV, refund rate, valid orders — 0 of 18 on the wide corpus, and what it would propose is sum(amt_pay): runs, looks right, off by a hundred and includes the test orders. A definition is a rule a person writes over aligned attributes (decision 4, #556). #507 is the other way a rule can arrive — stated in a document, with provenance — and is complementary, not this.

Measured by

scripts/bench/mappings.mjs scores today's proposals by number. An alignment is scored differently: did amt_pay land on Order.paid_amount, did buyer_id become a relation, did ver stay out. The truth file for wide has keys, traps and dimensions already; it gains an alignment section.

Depends on #553 having landed, or every exploration on a fresh base re-creates the column-name entities it is trying to replace.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions