Bundle-local registry for external OKF references

[joern-schlingensiepen]

Proposal: Bundle-local registry for external OKF references

Framing

This proposal is written based on my current reading of the OKF v0.1
draft SPEC and is offered as a contribution to the discussion,
not as a finished specification. It is entirely possible that I am
identifying a problem where none exists; the OKF use case as the
maintainers see it may differ from mine, or the gap I describe may
already be intended to be handled outside the SPEC.

My reading of OKF: it is meant to be a self-contained format for
providing information to AI agents, which typically consume that
information from files on disk. From that vantage point, the use
case that motivates this proposal is that such bundles will often
need to include information from other OKF bundles without
giving up self-containment.

If the maintainers' framing of OKF differs; for example, if bundles
are envisioned as strictly leaf artefacts and cross-bundle knowledge
is expected to be handled by the consuming service or agent; then
the mechanism proposed below is not needed and I welcome that
redirection.

The rest of this document assumes that reading is at least plausible
and proposes a minimal, backwards-compatible mechanism.

Motivation

The OKF v0.1 SPEC establishes bundles as self-contained
hierarchical collections of markdown documents (§3). Absolute links
start with / and resolve relative to the bundle root (§5.1). This
is intentional and gives OKF its clean, portable substrate.

In practice, however, a bundle almost always references knowledge
that lives in another bundle. A private organisational catalog may
want to cite entries from crypto_bitcoin or ga4 under
GoogleCloudPlatform/knowledge-catalog; a research bundle may want
to pull in concepts from a domain reference bundle it doesn't own.

The SPEC currently offers no mechanism for such references beyond
filesystem-relative paths, which break the self-contained property
and depend on how the consuming project happens to lay bundles out on
disk.

This proposal adds a minimal, backwards-compatible mechanism for
external references while preserving OKF's self-containment.

Design

1. Bundle-local registry file

A bundle MAY contain a file okf-registry.yaml at its root. Its
presence declares that the bundle imports one or more external OKF
bundles at specific mount paths within its own tree.

Filename convention: okf-registry.yaml (fixed; discoverable by
name).

Schema:

version: 1
mounts:
  <path/within/bundle>:
    source: <URL-like reference>
    ref: <optional git ref, tag, or version string>
    subpath: <optional path within the source>
    checksum: <optional integrity digest>

Example:

version: 1
mounts:
  external/ga4:
    source: git+https://github.com/GoogleCloudPlatform/knowledge-catalog.git
    subpath: okf/bundles/ga4
    ref: main
  external/bitcoin:
    source: git+https://github.com/GoogleCloudPlatform/knowledge-catalog.git
    subpath: okf/bundles/crypto_bitcoin
    ref: main
  vendor/legal-base:
    catalog: https://acme.internal/okf-catalog.yaml
    catalog_query: okf-catalog-v1
    bundle: base
    ref: v2.1.0

Mount paths are ordinary subpaths within the bundle. Bundle authors
MAY group them under a common prefix (e.g. external/) or place
them anywhere.

Multi-source (mirrors and fallbacks). Any URL-valued field
(source, catalog) MAY be given as either a single string or a
YAML sequence of strings. Tools SHOULD try entries in order and use
the first that resolves successfully:

mounts:
  external/ga4:
    source:
      - git+https://github.com/GoogleCloudPlatform/knowledge-catalog.git
      - git+https://mirror.acme.internal/knowledge-catalog.git
    subpath: okf/bundles/ga4
    ref: main

2. Provider types and catalog resolution

A mount's source (or catalog: reference) is resolved to actual
bundle content by the OKF manager tool. Two resolution paths are
proposed.

2.1 Direct source

source names a concrete location for the bundle:

Prefix	Meaning
git+https://<repo>.git	Git repository over HTTPS. ref = branch, tag, or commit; subpath = bundle path inside the repo.
git+ssh://<repo>.git	Same, over SSH.
https://<url>.zip (or .tar.gz)	Downloadable archive; checksum recommended for integrity.

Example:

mounts:
  external/ga4:
    source: git+https://github.com/GoogleCloudPlatform/knowledge-catalog.git
    subpath: okf/bundles/ga4
    ref: main

2.2 Catalog reference

Instead of a direct source, a mount MAY delegate resolution to an
external catalog. Real-world knowledge catalogs come in many
shapes: some are static YAML files, others are live services (e.g.
Google Knowledge Catalog / Dataplex, or private data-catalog
products with proprietary APIs). This proposal therefore
standardises the resolution mechanism but deliberately does
not fix a single catalog file format or protocol.

A catalog reference declares three things:

• catalog: — the URL of the catalog (or an array of URLs).
• catalog_query: — the name of the query adapter that knows
  how to interpret the catalog and look up a bundle by name.
• bundle: — the name or ID of the bundle to look up inside
  that catalog.

Example:

mounts:
  external/ga4:
    catalog: https://acme.internal/okf-catalog.yaml
    catalog_query: okf-catalog-v1
    bundle: ga4
    ref: main   # optional, overrides the catalog's default

The tool selects the query adapter identified by catalog_query,
uses it to fetch the catalog at catalog: and look up bundle:,
then resolves the result to a concrete source and materialises the
bundle at the mount path.

Suggested default adapter name: okf-catalog-v1. The concrete
schema (or service protocol) behind this name is intentionally left
open so that alignment can happen upstream,
ideally in coordination with the Knowledge Catalog project and any
other prospective catalog backends.

Consumers unaware of a specific query adapter SHOULD fail cleanly
("unsupported catalog query") rather than attempt best-effort
resolution.

2.3 Self-containment

Both resolution paths keep the bundle self-describing: everything
needed to resolve an external mount is inside okf-registry.yaml.
A direct source names the concrete location; a catalog reference
names the catalog URL, the query adapter to interpret it, and the
bundle key to look up. Nothing depends on tool-side configuration
or on well-known names outside the bundle.

Consumers unaware of a specific provider form or query adapter
SHOULD ignore the mount gracefully.

3. Linking to external content

Within the importing bundle, references to external content are
ordinary bundle-absolute markdown links (§5.1):

See [GA4 Events](/external/ga4/tables/events.md) for schema.

No new URI scheme, no new link form. The link is valid OKF because
the mount path is part of the bundle's tree from the consumer's
perspective.

4. Tool responsibilities

An OKF manager tool implementing this proposal MUST:

1. Read okf-registry.yaml at bundle load time.
2. Make the declared external bundles available at their mount paths
   before consumers traverse the bundle.
3. Ensure the mount targets are read-only (writes belong in the
   source, not the mount).

Phase 1 implementation: materialise each mount as a read-only
filesystem copy fetched from source at the specified ref. Cache
under the mount path.

Phase 2 (optional): mount via FUSE (or an equivalent user-space
filesystem) so external content is streamed on demand rather than
copied. The mount path stays the same; only the implementation
underneath changes.

5. Version-control hygiene

Mount targets SHOULD be excluded from the bundle's version control
(e.g. listed in .gitignore). The okf-registry.yaml (including
source, ref, and optional checksum) is the reproducible
declaration; the materialised bytes are a cache.

6. Self-containment is preserved

From an OKF consumer's perspective, the bundle is self-contained:
every referenced path resolves within the bundle tree.

The registry is a producer-side agreement about how the tree
gets populated. This is fully compatible with the existing v0.1
SPEC — consumers that ignore okf-registry.yaml still see a valid,
self-contained bundle tree once the manager tool has populated the
mounts.

Open questions

• Nested bundles: if a mounted bundle contains its own
  okf-registry.yaml, does the tool recurse? Proposed: yes, with
  cycle detection.
• Read-only enforcement: MUST or SHOULD? Proposed: SHOULD;
  producers may explicitly override for local development.
• Registry file format: YAML only, or should JSON be permitted?
  Proposed: YAML only for consistency with frontmatter conventions.
• Catalog format(s) behind catalog_query: the concrete schema
  or service protocol identified by adapter names such as
  okf-catalog-v1 is deliberately left open by this proposal. Real
  catalogs today span static YAML files and live services (e.g.
  Google Knowledge Catalog / Dataplex). The community should align
  on one or more canonical formats, each with its own adapter name.
  This proposal only fixes the mechanism by which a registry
  entry names an adapter and passes it a bundle key.
• Multi-source semantics: for array-valued URL fields, tools
  SHOULD try entries in order — but should they fail fast on the
  first success, race in parallel, or verify all yield equivalent
  content? Proposed: sequential try-in-order, first success wins;
  checksums (where present) SHOULD be verified to catch mirror
  divergence.

Backwards compatibility

Fully backwards-compatible. okf-registry.yaml is optional; bundles
without it behave exactly as before. Consumers unaware of the
proposal see mounts as ordinary subdirectories (once populated) and
process them as regular OKF content.

Reference implementation

Not yet available. This proposal is the minimum specification
needed before implementation begins.