Skip to content

CONCEPT.adoc

Latest commit

 

History

History
338 lines (257 loc) · 10.8 KB

CONCEPT.adoc

File metadata and controls

338 lines (257 loc) · 10.8 KB
Raw

Metadata Grammar: Integrated Successor to VoID, Dublin Core, and DCAT

Table of Contents

Overview

Metadata Grammar is a formal, integrated metadata framework that succeeds and unifies:

  • VoID (Vocabulary of Interlinked Datasets) - technical dataset metadata

  • Dublin Core - descriptive resource metadata

  • DCAT (Data Catalog Vocabulary) - catalog and distribution metadata

Unlike these predecessors which overlap without clean integration, Metadata Grammar provides a compositional grammar - not just vocabulary, but syntax and rules for metadata composition, validation, and transformation.

Motivation

Current Pain Points

  1. Overlapping vocabularies: VoID, DCAT, Dublin Core overlap but don’t align perfectly

  2. Missing modern concerns: Data ethics, privacy, AI/ML metadata, sustainability

  3. Format fragmentation: RDF-centric, not friendly to JSON-LD/JSON Schema users

  4. Governance unclear: Multiple W3C/DCMI/community efforts

  5. No formal semantics: Vocabularies lack formal grammars and composition rules

Why "Grammar" not "Vocabulary"

  • Vocabulary = terms and their definitions

  • Grammar = vocabulary + syntax + composition rules + validation

A grammar defines: - How to compose metadata (syntax trees) - Validation rules (well-formedness) - Transformation rules (crosswalks, migrations) - Formal semantics (provable properties)

The Paradigm Shift: White Box in Darkness

Note
 See PHILOSOPHY.adoc for the complete philosophical framework.

The Inversion

For two decades, we’ve obsessed over the "black box problem"—how to interpret opaque ML models and complex systems. We’ve been looking in the wrong direction.

Our boxes are white. We built our databases, models, and systems. We can inspect them. They’re not mysterious black boxes—they’re transparent artifacts.

The blackness is outside. The noumenal database—reality itself—is what we cannot see. The unexplored territory is where the mystery lies.

From "Is the cat dead or alive?" to "Are there cats beyond?"

Schrödinger’s question (inside the box): What’s the state of this quantum system?

Our question (beyond the box): What exists in unexplored reality? What domains haven’t we measured? Where are the blank spots on our maps?

The Three Zones of Reality

Zone 1: The White Box (~15% coverage) - Our phenomenal databases—observed, measured, mapped - Transparent, inspectable, schema-defined - "Here be data"

Zone 2: Known Darkness (~25% coverage) - Domains we know exist but haven’t measured - Spatial/temporal gaps, sampling biases, known blind spots - "Here be dragons" (we know the gaps exist)

Zone 3: Unknown Darkness (~60% coverage) - Domains we don’t even know to look for - Phenomena we lack instruments to detect - Unknown unknowns, epistemic darkness - "Here be…​ we don’t know what"

The Cartographic Paradigm

verisim: The cartographic index (atlas) - Indexes phenomenal databases (map of maps) - Tracks spatiotemporal coverage boundaries - Maps relationships and lineage - Coordinates the federation of databases - Guides exploration into the darkness

Phenomenal databases: Autonomous observations (the maps) - Individual datasets distributed across institutions - Each a partial, perspective-bound view of reality - Registered with verisim, indexed by coverage - The federation we navigate and compose

metadata-grammar: The cartographic standard - How to create maps (describe phenomenal databases) - How to register with the atlas (join the federation) - How to express coverage and blind spots - How to track provenance and evolution - How to guide exploration of the unknown

We are cartographers of the digital age, progressively mapping reality by stepping into darkness.

Architecture

Layer 1: Core Metadata (replaces DC + VoID core)

# Descriptive metadata
mg:title, mg:description, mg:keywords, mg:themes

# Technical metadata
mg:format, mg:size, mg:structure, mg:accessMethods

# Administrative metadata
mg:license, mg:rights, mg:provenance, mg:governance

Layer 2: Domain Extensions (modular)

  • Scientific: methodology, instruments, variables

  • Cultural: GLAM metadata (Europeana, CIDOC-CRM)

  • Geospatial: GeoSPARQL, WGS84

  • AI/ML: training data, model metadata, bias documentation

  • Sustainability: carbon footprint, data retention policies

Layer 3: Interoperability Bridges

  • Schema.org alignment (for SEO/discoverability)

  • DCAT-AP (EU standards compliance)

  • Bioschemas (life sciences)

  • Crosswalks to legacy vocabularies (VoID, Dublin Core, DCAT)

Integration with verisim: The Cartographic Index

verisim serves as the atlas that coordinates exploration of reality.

verisim as Cartographic Infrastructure

Core capabilities: - Temporal versioning: Track how our maps evolve as we understand reality better - Immutable history: Preserve full provenance of observations - Spatiotemporal indexing: Map coverage boundaries and blind spots - Federation coordination: Enable discovery and composition of autonomous databases

Cartographic functions: - Index the white box: Catalog all phenomenal databases (what we’ve mapped) - Mark known darkness: Track identified blind spots and gaps (Zone 2) - Acknowledge unknown darkness: Epistemic humility about Zone 3 - Guide exploration: Prioritize which darkness to explore next - Track cartographic evolution: "Our 2025 map corrected errors from 2020"

Metadata Grammar + verisim Integration

How they work together:

  1. metadata-grammar defines the cartographic standard (how to create maps)

  2. verisim implements the atlas infrastructure (indexes the maps)

  3. Phenomenal databases register with verisim using metadata-grammar specs

  4. The federation grows as we explore and map new domains

@prefix mg: <https://hyperpolymath.org/ns/metadata-grammar#> .
@prefix vsim: <https://hyperpolymath.org/ns/verisim#> .

# The cartographic index (atlas)
:GlobalAtlas a vsim:VerisimDatabase, mg:CartographicIndex ;
    # Tracks exploration state
    mg:whiteBoxCoverage 0.15 ;
    mg:knownDarknessCoverage 0.25 ;
    mg:unknownDarknessCoverage 0.60 ;

    # Indexes the federation
    mg:federationSize 1500000 ;
    mg:indexesDatabases :ClimateDB, :EconomicDB, :GenomicDB, ... .

# A phenomenal database in the federation
:ClimateDB a mg:PhenomenalDatabase ;
    # What it observes (what territory it maps)
    mg:observes mg-domain:GlobalClimate ;
    mg:perspective "satellite-temperature-measurements" ;

    # Spatiotemporal coverage (edge of white box)
    mg:spatialCoverage :NorthernHemisphere ;
    mg:temporalCoverage "1980-2025" ;

    # Known blind spots (darkness we're aware of)
    mg:knownBlindSpots [
        mg:spatial :SouthernOceans ;
        mg:temporal "pre-1980"
    ] ;

    # Registration with atlas
    mg:registeredIn :GlobalAtlas ;
    vsim:indexedBy :GlobalAtlas ;
    vsim:version-history :ClimateDB-History ;

    # Validation
    mg:validates-against :MetadataGrammarShapes ;
    mg:validation-status mg:Valid .

# Cartographic evolution tracked by verisim
:ClimateDB-History a vsim:VersionHistory ;
    vsim:snapshots [
        :ClimateDB_v2020,  # Our understanding in 2020
        :ClimateDB_v2025   # Improved cartography in 2025
    ] .

:ClimateDB_v2025
    mg:supersedes :ClimateDB_v2020 ;
    mg:improvements [
        mg:blindSpotsFilled :ArcticRegion ;
        mg:uncertaintyReduced 0.10 ;
        mg:resolutionImproved "1km → 500m"
    ] .

Formal Specification

Idris2 Type System

Metadata Grammar uses Idris2 dependent types to prove metadata properties:

-- Metadata must have title (non-empty string)
record Dataset where
  constructor MkDataset
  title : String
  {auto 0 nonEmpty : So (length title > 0)}
  created : ISO8601Date
  modified : ISO8601Date
  {auto 0 modifiedAfterCreated : LTE created modified}

SHACL Validation

Runtime validation via SHACL shapes:

:DatasetShape a sh:NodeShape ;
    sh:targetClass mg:Dataset ;
    sh:property [
        sh:path mg:title ;
        sh:minCount 1 ;
        sh:datatype xsd:string ;
        sh:minLength 1 ;
    ] ;
    sh:property [
        sh:path mg:modified ;
        sh:maxCount 1 ;
        sh:datatype xsd:date ;
    ] .

Example: Integrated Metadata

@prefix mg: <https://hyperpolymath.org/ns/metadata-grammar#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix void: <http://rdfs.org/ns/void#> .
@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix prov: <http://www.w3.org/ns/prov#> .

# Native Metadata Grammar (preferred)
:MyDataset a mg:Dataset ;
    # Core descriptive
    mg:title "Example Linked Dataset" ;
    mg:description "Proof-of-concept integrated dataset" ;
    mg:creator :JonathanJewell ;
    mg:created "2026-01-31"^^xsd:date ;
    mg:modified "2026-01-31"^^xsd:date ;
    mg:license <https://spdx.org/licenses/MPL-2.0> ;

    # Technical metadata
    mg:tripleCount 1000000 ;
    mg:entityCount 50000 ;
    mg:sparqlEndpoint <https://example.org/sparql> ;
    mg:dataDump <https://example.org/dump.nt.gz> ;

    # Provenance
    mg:wasGeneratedBy :DataGenerationActivity ;
    mg:wasAttributedTo :JonathanJewell ;

    # Distribution
    mg:distribution :RDF-Distribution, :CSV-Distribution ;

    # Validation
    mg:validates-against :MetadataGrammarCoreShapes ;
    mg:validation-status mg:Valid ;
    mg:last-validated "2026-01-31T12:00:00Z"^^xsd:dateTime .

# Legacy compatibility (automatic crosswalk)
:MyDataset
    # Dublin Core mapping
    dcterms:title "Example Linked Dataset" ;
    dcterms:creator :JonathanJewell ;

    # VoID mapping
    void:triples 1000000 ;
    void:sparqlEndpoint <https://example.org/sparql> ;

    # DCAT mapping
    dcat:distribution :RDF-Distribution .

Implementation Plan

Phase 1: Core Specification (Proof of Concept)

  • ❏ Idris2 formal grammar definition

  • ❏ SHACL shapes for validation

  • ❏ JSON-LD context

  • ❏ OWL ontology

  • ❏ Crosswalks to VoID/DC/DCAT

Phase 2: Tooling

  • ❏ Zig FFI for validation (C ABI)

  • ❏ ReScript web UI for metadata creation

  • ❏ Julia batch conversion scripts

  • ❏ Rust CLI for metadata operations

Phase 3: Integration

  • ❏ verisim storage backend

  • ❏ SPARQL endpoint with temporal queries

  • ❏ GitHub Actions workflow for metadata validation

  • ❏ Documentation site (AsciiDoc + Jekyll)

License

This specification is licensed under the Palimpsest Meta-Philosophical License (MPL-2.0).

SPDX-License-Identifier: CC-BY-SA-4.0