Mounting a data source ingests its schema as a markdown document under the per-base "Data schemas" folder, so chat can search_chunks for a table. That document then goes through extraction like any other, and the extractor — shown the Metric / Dimension classes exploration had just created — files every column name as an entity.
Measured on the wide corpus of #501: forty concept entities in the base, twelve from exploration, twenty-eight column names (amt_pay as a Metric, buyer_id as a Dimension, dw.dim_shop as a Dimension with eight facts on it). Timestamps line up: 23:59:36 is the mount, 00:01:26 is exploration.
0035 decision 7: the schema document is a search corpus, not a source of facts. It keeps its place in retrieval and leaves extraction. This is the one decision in that record that is a bug fix and can land on its own.
Shape
A document that is retrieved from but not learned from is a property of the document, set by whoever ingests it. sync_schema_doc is the only such ingester today. pipeline::process_document queues extract_document after the chunk vectors are written; a document marked this way stops there, with a graph_status that says so rather than none, so the Library does not show it as waiting.
Not a source-kind rule ("folders named Data schemas never extract"): that is a name standing in for a type guarantee, the defect class 0009 warned about.
Touches the ingest path that #526 also wants to move (the queue waiting for ontology vectors); the two are separable — this one decides whether a document is extracted, #526 decides when.
Mounting a data source ingests its schema as a markdown document under the per-base "Data schemas" folder, so chat can
search_chunksfor a table. That document then goes through extraction like any other, and the extractor — shown theMetric/Dimensionclasses exploration had just created — files every column name as an entity.Measured on the wide corpus of #501: forty concept entities in the base, twelve from exploration, twenty-eight column names (
amt_payas a Metric,buyer_idas a Dimension,dw.dim_shopas a Dimension with eight facts on it). Timestamps line up: 23:59:36 is the mount, 00:01:26 is exploration.0035 decision 7: the schema document is a search corpus, not a source of facts. It keeps its place in retrieval and leaves extraction. This is the one decision in that record that is a bug fix and can land on its own.
Shape
A document that is retrieved from but not learned from is a property of the document, set by whoever ingests it.
sync_schema_docis the only such ingester today.pipeline::process_documentqueuesextract_documentafter the chunk vectors are written; a document marked this way stops there, with agraph_statusthat says so rather thannone, so the Library does not show it as waiting.Not a source-kind rule ("folders named Data schemas never extract"): that is a name standing in for a type guarantee, the defect class 0009 warned about.
Touches the ingest path that #526 also wants to move (the queue waiting for ontology vectors); the two are separable — this one decides whether a document is extracted, #526 decides when.