You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Exploration today reads a schema and produces entities of two classes it inserted itself. 0035 decisions 1, 2 and 6: it produces instead an alignment proposal per table — the class the table is a table of, the attributes its columns carry, the relations its foreign keys are, and for each aligned column an expression that turns the column into the attribute's value — and a person adopts it.
What a proposal holds
For dw.dwd_ord_dtl on the wide corpus:
column
is
amt_pay, qty, ord_st, dt_crt
attributes of Order
buyer_id, shop_id
relations Order → Customer, Order → Shop
shop_nm, prov, buyer_lvl
attributes of Shop, Address, Customer, flattened in
ver, etl_dt, rmk, price_old
nothing
Exploration finds existing classes and attributes first (the ontology, with a pack from 0008, usually has Order) and proposes new ones only where none fits — 0003's loop with a schema in place of a corpus. On a base with no ontology it proposes the class (decision 6); the difference from today is that a proposal waits and ensure_concept_types did not.
Where it goes
ontology_proposals already has the four sections this needs — entity_types, attribute_types, relation_types, map_to — and the attribute payload already carries datatype and unit. What it does not have is a way to say "these forty rows are one table's alignment, adopt or reject them together", and decide_proposal today only marks a row; creating the ontology rows is the separate adopt endpoints' job. So this needs:
a proposal that groups a table's rows, adopted as one thing — adopting Order while rejecting every attribute of it is not a decision anyone means to make;
an adopt path that writes the ontology rows and the alignment rows in one transaction.
What exploration does not propose
Definitions. GMV, refund rate, valid orders — 0 of 18 on the wide corpus, and what it would propose is sum(amt_pay): runs, looks right, off by a hundred and includes the test orders. A definition is a rule a person writes over aligned attributes (decision 4, #556). #507 is the other way a rule can arrive — stated in a document, with provenance — and is complementary, not this.
Measured by
scripts/bench/mappings.mjs scores today's proposals by number. An alignment is scored differently: did amt_pay land on Order.paid_amount, did buyer_id become a relation, did ver stay out. The truth file for wide has keys, traps and dimensions already; it gains an alignment section.
Depends on #553 having landed, or every exploration on a fresh base re-creates the column-name entities it is trying to replace.
Exploration today reads a schema and produces entities of two classes it inserted itself. 0035 decisions 1, 2 and 6: it produces instead an alignment proposal per table — the class the table is a table of, the attributes its columns carry, the relations its foreign keys are, and for each aligned column an expression that turns the column into the attribute's value — and a person adopts it.
What a proposal holds
For
dw.dwd_ord_dtlon the wide corpus:amt_pay,qty,ord_st,dt_crtOrderbuyer_id,shop_idOrder → Customer,Order → Shopshop_nm,prov,buyer_lvlShop,Address,Customer, flattened inver,etl_dt,rmk,price_oldExploration finds existing classes and attributes first (the ontology, with a pack from 0008, usually has
Order) and proposes new ones only where none fits — 0003's loop with a schema in place of a corpus. On a base with no ontology it proposes the class (decision 6); the difference from today is that a proposal waits andensure_concept_typesdid not.Where it goes
ontology_proposalsalready has the four sections this needs —entity_types,attribute_types,relation_types,map_to— and the attribute payload already carriesdatatypeandunit. What it does not have is a way to say "these forty rows are one table's alignment, adopt or reject them together", anddecide_proposaltoday only marks a row; creating the ontology rows is the separate adopt endpoints' job. So this needs:Orderwhile rejecting every attribute of it is not a decision anyone means to make;What exploration does not propose
Definitions. GMV, refund rate, valid orders — 0 of 18 on the wide corpus, and what it would propose is
sum(amt_pay): runs, looks right, off by a hundred and includes the test orders. A definition is a rule a person writes over aligned attributes (decision 4, #556). #507 is the other way a rule can arrive — stated in a document, with provenance — and is complementary, not this.Measured by
scripts/bench/mappings.mjsscores today's proposals by number. An alignment is scored differently: didamt_payland onOrder.paid_amount, didbuyer_idbecome a relation, didverstay out. The truth file forwidehaskeys,trapsanddimensionsalready; it gains analignmentsection.Depends on #553 having landed, or every exploration on a fresh base re-creates the column-name entities it is trying to replace.