diff --git a/changelog.d/current-agent-guides.changed.md b/changelog.d/current-agent-guides.changed.md new file mode 100644 index 000000000..260ca2afd --- /dev/null +++ b/changelog.d/current-agent-guides.changed.md @@ -0,0 +1 @@ +- Agent skills and deployment guides now describe the current 0.13 workflows, exact receipts and live configuration updates. Obsolete commands, historical workarounds and duplicated migration instructions have been removed. diff --git a/docs/user/clusters/index.md b/docs/user/clusters/index.md index 31babbc3b..381405432 100644 --- a/docs/user/clusters/index.md +++ b/docs/user/clusters/index.md @@ -1,12 +1,9 @@ # Operating a cluster -An OmniGraph cluster is a declarative bundle of graphs, schemas, stored queries -and authorization policies. Apply converges graph inventory, schemas, queries, -policies and provider/Blob settings. Submit to the running server to activate -changes without a restart. - -Use a cluster for a multi-graph server or shared operational configuration. For -one local graph, the [quickstart](../quickstart.md) is simpler. +A cluster is a declarative bundle of graphs, schemas, stored queries, policies +and provider/Blob settings. Apply the bundle to create or update the cluster. +Submit changes to its running server to activate them without a restart. +For one local graph, start with the [quickstart](../quickstart.md). ## Create a bundle @@ -37,18 +34,15 @@ policies: applies_to: [knowledge] ``` -Paths are relative to `cluster.yaml`. Its configuration version is independent -of the deployment ledger version. The [configuration reference](config.md) -covers storage roots, embedding providers, external Blob policy and limits. +Paths are relative to `cluster.yaml`. See the [configuration reference](config.md) +for storage, providers, Blob rules and limits. The configuration version is +independent of the storage and deployment-ledger versions. -For server-owned deployment, the applied policy must grant the operator -`config_manage` at cluster scope. Schema changes and graph deletions additionally -require `read` and `schema_apply` on the affected graphs. Direct deletion also -enforces an installed graph policy and requires an authorized `--as` actor. -Deployment status reveals management -metadata under `config_manage`; it does not require data access on unrelated -graphs. New graphs need suitable declared policies too. -The server derives the actor from its bearer token; `--as` is for direct access. +The applied cluster policy must grant the deploying actor `config_manage`. +Schema changes and graph deletion also require `read` and `schema_apply` on +those graphs. Proposed permissions cannot authorize their own installation. +The server resolves the actor from the bearer token; `--as` selects an actor +only for direct access. See [authorization](../operations/policy.md). ## Bootstrap a cluster @@ -58,14 +52,12 @@ omnigraph cluster plan --config ./company-brain omnigraph cluster apply --config ./company-brain --as act-alice --json ``` -Fresh apply creates the deployment ledger and declared graphs. It captures all -source bytes before execution, prints the original `Deployment-ID`, and records -exact outcomes. It does not load rows; use `load` or `mutate` for data changes. +Fresh apply creates the ledger and declared graphs, records exact outcomes and +prints the deployment and lock IDs. Use `load` or `mutate` to add rows afterward. -Direct apply retains its admission lock after completion. Establish that the -owner and its accepted I/O have settled, then follow -[ownership transfer](../deployment.md#writer-topology) using the exact printed -lock ID before starting the server: +Direct apply retains its writer lock. Establish that its process and accepted +storage I/O have settled, then follow [ownership transfer](../deployment.md#writer-topology) +with the exact lock ID before starting the server: ```bash omnigraph --cluster file:///srv/company-brain cluster force-unlock '' @@ -73,142 +65,91 @@ OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{"act-alice":"secret"}' \ omnigraph-server --cluster file:///srv/company-brain --bind 0.0.0.0:8080 ``` -Use the actual root printed by apply. A directory boot resolves its storage root -through `cluster.yaml`; a root URI boots directly from applied resources. Editing -local files alone never changes serving behavior. See -[HTTP server](../operations/server.md) for authentication and routes. +Use the actual root printed by apply. A root URI boots from applied resources; +a directory resolves its root through `cluster.yaml`. Editing source files alone +never changes serving behavior. Wait for `/readyz` before sending requests. ## Deploy without restarting -Edit and validate the bundle, then submit it to the running owner: +Edit the bundle, validate it, preview through the server, then apply: ```bash +export OMNIGRAPH_BEARER_TOKEN='secret' omnigraph cluster validate --config ./company-brain -OMNIGRAPH_BEARER_TOKEN='secret' omnigraph cluster apply \ - --server https://graph.example.com --config ./company-brain --json +omnigraph cluster plan --server https://graph.example.com \ + --config ./company-brain --json +omnigraph cluster apply --server https://graph.example.com \ + --config ./company-brain --timeout 1800 --json ``` -With `--server`, omitted `storage` binds the bundle to the selected server’s -canonical root. An explicit absolute `storage` must match that root; relative -storage paths refuse. The CLI reads only local source files and needs no local -mount or storage credentials for the server’s root. +With `--server`, omitted `storage` binds to that server's canonical root. +Explicit storage must be an absolute matching root; relative paths refuse. +The CLI needs the source files and bearer credential, not storage credentials. +Plan changes nothing and reserves nothing; apply rechecks the current state. +**Review graph removals carefully: apply permanently deletes their managed +storage and history.** See [deployment boundaries](#deployment-boundaries). -Preview the same local configuration through the server before applying: +The server closes affected graphs to new requests, drains admitted work, applies +changes and activates the matching configuration. Other graphs keep serving; +the PID, listener and writer ownership remain unchanged. Each graph publishes +atomically, but a multi-graph deployment is not one transaction. -```bash -omnigraph cluster plan --server production --config . --json -omnigraph cluster apply --server production --config . --timeout 1800 --json -``` - -The served plan lists resource changes and schema migrations, including the exact -managed root and history that a graph removal deletes. It writes nothing and keeps -serving admission open. Its ledger CAS and input digest identify an observation; -apply rechecks current authority and physical execution eligibility. - -The CLI prints the deployment ID before submission. The server keeps its PID, -listener and writer ownership. It closes admission on affected graphs, -finishes their admitted requests, publishes schema changes, and activates -matching schemas, queries and runtime permissions together. Unaffected graphs -keep serving. -Graph additions and removals use the same deployment; see -[deletion semantics](#deployment-boundaries) before removing a declaration. - -The response separates the durable deployment result from `active`, which means -that result's affected bindings are installed in this process. An unrelated -blocked graph does not invalidate that activation. Default apply waits for -convergence and activation by polling the original ID. `--no-wait` instead returns -after durable acceptance; drain and preparation may precede that acknowledgment. -`--timeout SECONDS` bounds caller waiting, including acceptance (default 300, -maximum 3600). Expiry exits 5 with the original ID and last observation; it does -not cancel execution or prove failure. Resume observation with: +Apply prints an ID before submission and waits for both durable completion and +activation. `--no-wait` returns after durable acceptance instead. `--timeout` +bounds caller waiting (default 300 seconds, maximum 3600); expiry exits 5 and +does not cancel execution or prove failure. Continue observing the original ID: ```bash -omnigraph cluster status --server production --deployment-id ID --wait --timeout 1800 --json +omnigraph cluster status --server https://graph.example.com \ + --deployment-id '' --wait --timeout 1800 --json ``` -The exact response contains `deployment`, `active` and `in_progress`. General -cluster status retains its aggregate `status` object. An authenticated submitter -can read its own durable receipt even after its deployment removes its management -permission; general status and later deployments still require current permission. -Lost submission responses are followed only by original-ID reads, never automatic -resubmission. While waiting, observation retries transient HTTP 429/503 responses -and interrupted response bodies within the same budget; malformed receipts fail immediately. -Expired receipts and unknown outcomes require investigation. - -Each graph publishes atomically; deployment -across multiple graphs is not one transaction. Query-only changes create no graph -commit and also work with multiple branches. Schema changes remain main-only -and require a single live branch. - -Policies, provider definitions and graph bindings, and external-Blob rules can -change on existing graphs. Current permissions authorize the deployment; proposed -permissions cannot authorize themselves. Provider changes do not re-embed stored -vectors. Roots, format and credential/trust configuration stay fixed. -A refusal before effects restores unchanged serving views, including after a -drain timeout. That timeout bounds draining admitted requests; once drained, -the server owns preparation, completion and activation through the existing -shutdown boundary. There is one deployment protocol and no legacy execution fallback. - -Upgrade the CLI, server and cluster tools together. Finish outstanding deployments -with the build that accepted them before upgrading: captured inputs are exact -and are not translated into a different request. Use the stopped-ledger upgrade -command to remove obsolete completed-result runtime fields; graph data and exact -achieved receipts are preserved. The graph storage format is unchanged. +Exact status returns `deployment`, `active` and `in_progress`. `active` means +this result's affected bindings are installed in the current process; an +unrelated blocked graph does not invalidate it. A completed receipt can precede +activation. The authenticated submitter retains access to its exact receipt +after losing management permission; aggregate status still requires permission. + +The CLI follows a lost submission response with original-ID reads, never an +automatic resubmission. Observation retries transient 429/503 responses and +interrupted bodies within its waiting budget. Malformed or expired receipts +require investigation. See [recovery](#inspect-and-recover-a-deployment). ## Direct deployments and conversion -Without `--server`, apply executes under its own exclusive admission and requires -the serving owner to have stopped and handed off the lock. Start the server after -settlement to activate the applied revision. Direct apply never takes over a live -server or writes around its lock. +Without `--server`, apply requires stopped serving and exclusive writer +ownership. Transfer ownership before running it, then settle and hand off its +lock before starting the server. Use this path for bootstrap and offline work. +A completed read-only preflight refusal releases a newly acquired lock; +accepted work and uncertain effects retain it. -Use explicit stopped-ledger conversion for a v1 ledger or a v2 ledger whose -completed receipts still contain obsolete runtime activation fields: +Upgrade the CLI, server and integrations together. Finish outstanding work with +the build that accepted it. If status reports `ledger_upgrade_required`, stop +writers, establish prior graph/control I/O quiescence and run: ```bash omnigraph --cluster file:///srv/company-brain \ cluster upgrade-ledger --writers-stopped --json ``` -Stop serving, writers and maintenance and establish prior graph/control I/O -quiescence first. Conversion preserves rows, graph identities, branches, history -and applied resources. It does not reset graphs, replay old work or convert graph -storage formats. There is no automatic migration or v1 execution fallback. -For v2, conversion removes only those obsolete runtime fields, preserving the -ledger identity, next deployment sequence and exact achieved receipts. Outstanding -deployments and unsupported receipt shapes refuse; finish accepted work with -its originating build before upgrading. -Normal reads report `ledger_upgrade_required` with the stopped-upgrade command -for a recognized prior receipt; malformed or unknown state remains an error. - -A completed read-only preflight refusal releases a newly acquired direct lock. -Accepted work, cancellation and uncertain effects retain it for reconciliation. -`--as` labels a storage-owning operator; installed graph policies still govern -schema effects. It is not remote authentication. +Explicit conversion accepts a supported older ledger or completed receipt +shape. It preserves graph identities, rows, history, resources and exact achieved +outcomes. Outstanding work and unknown shapes refuse. It does not upgrade graph +storage; use the [upgrade guide](../operations/upgrade.md) for format changes. ## Inspect and recover a deployment -A lost connection does not mean failure. Observe the original ID: - -```bash -OMNIGRAPH_BEARER_TOKEN='secret' omnigraph cluster status \ - --server https://graph.example.com --deployment-id '' --json -``` - -Status is read-only. `active` is false for an older result after a newer revision -activates. Only the current runtime can prove activation; a durable completion -receipt alone cannot. Repeating apply with the same ID and identical captured input -returns its recorded outcome; it never executes again. Different input refuses. - -If the server stopped with an outstanding deployment, inspect the storage root: +A lost connection does not mean failure. Use the original-ID server status +command above first. If the server stopped with outstanding work, inspect its +root directly: ```bash omnigraph --cluster file:///srv/company-brain \ cluster status --deployment-id '' --json ``` -Establish prior-owner and accepted-I/O quiescence, exclude concurrent admissions -and unlocks, then reconcile that exact ID: +After establishing prior-owner and accepted-I/O quiescence, exclude competing +admissions and unlocks, then reconcile the exact ID: ```bash omnigraph --cluster file:///srv/company-brain cluster force-unlock '' @@ -216,65 +157,47 @@ omnigraph --cluster file:///srv/company-brain \ cluster apply --deployment-id '' --writers-stopped --json ``` -Reconciliation uses captured input and exact publication evidence; it never -replays an uncertain schema or graph-creation invocation. Unstarted schema or -graph-creation work can be recorded as not attempted. A partial graph birth can -be abandoned only when its exact unpublished, empty artifacts belong to that -attempt. A foreign or committed graph is never reset. Unknown outcomes stay -outstanding and block new writes. A later creation may reuse an empty local -directory tree left by that cleanup; files, symlinks and cloud markers still refuse. +Recovery uses captured inputs and publication evidence. It does not replay an +uncertain schema or graph-creation call, reset foreign data, or roll back graphs +that committed. Interrupted deletion resumes removal of its recorded root. +Unknown outcomes remain outstanding and block new writes. Never allocate a new +ID to retry them; receipt eviction also does not authorize replay. -A settled partial result permits a corrective successor from achieved state; -it does not roll back graphs that committed. Recovery retains a new admission -lock, so perform ownership transfer before starting another owner. Never -allocate a new ID to retry an unknown outcome. Result eviction reports acceptance -and outcome as unknown; it never authorizes replay. See [limits](config.md#limits). +A settled partial result permits a corrective deployment from the achieved +state. Recovery retains its lock, so complete ownership transfer before starting +the next owner. Only a running server can report activation; an older receipt's +`active` becomes false when a newer deployment activates. ## Deployment boundaries -`cluster apply` creates graphs, deletes removed graphs, and updates schemas, -stored queries, policies, providers and Blob rules. Removing a graph declaration -and applying is destructive: it deletes that graph's managed storage, including -all branches, retained history and managed Blob bytes. Review the deletion in -`cluster plan` and keep any required backup before applying. There is no separate -delete flag or unregister mode. External Blob source objects are not deleted. -On object stores, deletion removes the active namespace; provider version history -and retention policies may keep older objects. This is not secure erasure. - -Live apply closes the affected graph to new requests and waits for its admitted -work and response bodies before deletion. Other graphs continue serving. If -deletion is interrupted, reconcile the original deployment ID; the recorded -operation resumes removal of the exact root. Editing the desired configuration -does not cancel accepted deletion, and absence during recovery does not create a -replacement graph. - -Importing an existing root, recreating a missing managed graph, accepting schema -drift and repairing catalog payloads are not apply modes. Restore missing or -damaged authoritative data from a verified backup before deploying changes to -that graph. - -Every present affected root is checked against its achieved schema identity, -even when source text changes. Removing an already-absent graph still records a -completed deletion; it never recreates the root. An out-of-band replacement or schema change refuses with -`applied_schema_drift`; matching text cannot authorize a different graph identity. - -Plan always observes without taking the cluster writer lock. It reports the -existing owner and runs the same effect-free preparation used by apply, with -the intended actor. Its migration steps come from that preparation, not a second -schema-file read. Apply prepares again under writer admission because a preview -reserves nothing. Unavailable affected graphs, schema drift and migration -restrictions are errors, except that deletion accepts an already-absent root. An unavailable unrelated graph does not block an -independent change. Live apply also validates provider secrets and serving -settings on the server. +| Change | Behavior | +|---|---| +| Add a graph | Creates a new managed root and activates its declared configuration. | +| Remove a graph | Deletes its managed root, branches, history and managed Blob bytes. | +| Schema | Publishes atomically; requires `main` to be the only live branch. | +| Stored queries | Activates the new registry; permits existing branches and creates no graph commit. | +| Policies | Current permissions authorize the change; later requests use the new rules. | +| Providers and Blob rules | Activates new bindings; provider changes do not re-embed existing vectors. | +| Roots, storage format or credential/trust configuration | Requires the corresponding offline or runtime procedure. | + +Back up anything needed before deleting a graph. Removal does not delete +external Blob source objects; object-store versioning or retention may preserve +older objects. Accepted deletion cannot be canceled by editing YAML. + +Apply refuses unexpected graph identities, schema drift and missing managed +roots. It cannot adopt, recreate or repair them. Restore damaged authoritative +data from a verified backup before updating that graph. Deleting an already +absent graph is allowed and does not recreate it. Unavailable unrelated graphs +do not block an independent change. ## Operational boundaries -- One mutation-capable process owns a cluster. Online deployment runs inside it; - direct maintenance requires an ownership handoff. -- Object-storage clusters may boot directly from `s3://bucket/prefix` or - `az://container/prefix`; source files are not needed for serving or recovery. -- Azure remains a qualification preview. Its running server retains the - mandatory [admission wrapper](../deployment.md#azure-blob-preview) during HTTP - deployment; submission does not acquire a second storage-writer lease. -- A schema drop removes data from the branch head without reclaiming storage. - Retained historical commits remain readable until cleanup removes them. +- One mutation-capable process owns the cluster; direct maintenance requires + [ownership transfer](../deployment.md#writer-topology). +- Source files are unnecessary for serving or recovery from an applied root. +- A refusal before effects restores unchanged serving views. The drain timeout + bounds waiting for admitted work; it does not expire a deployment after effects. +- Azure deployments retain the mandatory [admission wrapper](../deployment.md#azure-blob-preview) + and qualification-preview boundary. +- Dropping schema content removes it from the current head; retained historical + data remains until cleanup. This differs from deleting an entire graph. diff --git a/docs/user/deployment.md b/docs/user/deployment.md index a0f0b84d9..989c41875 100644 --- a/docs/user/deployment.md +++ b/docs/user/deployment.md @@ -31,94 +31,56 @@ OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{"act-service":"secret"}' \ --bind 0.0.0.0:8080 ``` -Use `GET /healthz` for process health and `GET /readyz` for readiness: -`/readyz` reports `loading` (startup is still pending), `serving`, -`degraded` (some graphs unavailable), `blocked` (none ready), or `draining`. -It includes the applied `config_digest` it booted from -(`booted_serving_digest`), the ledger revision, and `served_graph_count`, -`ready_graph_count`, `loading_graph_count` and `blocked_graph_count`. -`served_graph_count` counts the whole registry. Graphs closed for a transition also -contribute to `blocked_graph_count`. Readiness stays 503 while any graph is -loading. Once startup finishes, it returns 200 when at least one graph is ready -or the applied inventory is empty, and 503 when all graphs are unavailable or -shutdown has begun. The listener opens after configuration and -authorization validation, before graphs open. Healthy graphs become available -as they finish loading, while readiness holds aggregate traffic until loading -finishes. A nonempty cluster with no healthy graph exits with a -startup error after its opening attempts finish. With `--require-all-graphs`, -all graphs remain unavailable until every graph opens successfully; any failure -stops the server. Wait for readiness before sending data requests: the printed -listening address and `/healthz` confirm only that the listener is live. - -Readiness is unauthenticated and names no graph. Authorized `GET /graphs` -returns one `graphs` list, including blocked entries, with each graph's `state` -(`loading`, `ready`, `blocked`, `transitioning` or `stopping`), `read_available`, `write_available` and -`action`. Availability describes the runtime, not the caller's permissions. -Blocked entries include a sanitized `failure`: `invalid_configuration`, -`invalid_policy`, `invalid_external_blob_policy`, `open_failed` or -`invalid_stored_queries`. Details remain in server logs. -`apply_correction_or_restart` means fix a query, policy or provider configuration -through deployment, or restore damaged graph storage from a verified backup and -restart. It does not authorize adopting a different graph identity. -`wait_for_restart` describes shutdown. -`wait_for_startup` means the graph's one startup attempt has not completed -admission. It promises no retry time and does not trigger another open. -`wait_for_transition` means that graph's admissions are closed while its owner -finishes a transition. Wait for that owner, or restart if the transition was -abandoned or expired; there is no promised retry time. Other ready graphs can -continue serving and transitioning. Server-owned deployments activate schema and -stored-query changes through this transition; a proved pre-effect refusal restores -the unchanged views. See [live deployment](clusters/index.md#deploy-without-restarting). -Ready entries use `none`. There is no separate `quarantined` list or automatic startup retry. - -`GET /cluster/deployments/{id}` separates the durable achieved result from its -current `active` status. Active means this process has installed that exact -deployment and all its graph bindings are ready. Loading, blocked graphs, -partial convergence and shutdown keep it false. A normal restart verifies its -own bindings before reporting the retained current deployment active; successful -apply alone is not activation. Activation does not rewrite the durable receipt. - -Known loading, blocked or transitioning graphs return 503 (`graph_unavailable`) to callers authorized to -read `main` or list the management inventory; other callers cannot use this response to discover -them. Unknown graphs return 404. A 503 does not authorize replaying a write. -Add `--require-all-graphs` when any blocked graph should fail startup. -`omnigraph graphs list --server ` displays graph ID, state and URI; -`--json` preserves the full inventory fields. Minimal discovery remains IDs/names only. - -An admitted write continues if its client disconnects. The server keeps its -capacity reserved through the whole operation, including optional source deletion -after a merge. A lost response can still leave the caller unsure whether the -write committed; follow the [failure outcome](operations/troubleshooting.md#failed-data-write-commands) -before submitting it again. - -Once its complete request body is accepted, a read also keeps its capacity -reserved until execution finishes, even if the caller disconnects. MCP -cancellation and its 30-second response deadline stop waiting for the result; -an admitted read continues and retains its tool slot until execution finishes. - -Shutdown is bounded: at SIGTERM the server closes operation admission and stops -accepting connections. It waits for entered graph opens, admitted writes, executing reads, read bodies -and server stream producers for at most `--shutdown-grace-seconds` (else -`OMNIGRAPH_SHUTDOWN_GRACE_SECONDS`, else 25), then exits 2 with the unfinished -work logged. The deadline is kept by a thread, so a stalled request or a -blocked runtime cannot extend it. Signal handling begins after configuration -loading, before graph opening, and covers serving startup. Disconnected callers -do not remove their executing work from this wait. A graph finishing startup -after shutdown begins cannot become available. A panic or uncertain -completion in startup or an admitted write closes admission -for every graph in that process and starts the same shutdown path without renewing -an existing deadline. After HTTP connections and the other known logical owners -finish, the process exits 2 immediately; the watchdog remains the upper bound. -Unresolved reservations stay charged until exit. This is process containment, -not proof that native storage I/O settled or that the engine can be reused. -A proven pre-effect failure, such as a failed initial schema read or a native-tag -retirement refusal, releases its reservation and keeps admission open. Error -status or a transient storage category alone does not establish that proof. -A clean drain exits 0. Set the orchestrator's own -termination grace longer than this value; a cutoff is crash-equivalent for the -work it interrupts, and the next open recovers it as after any crash. -For a v2 cluster, even a clean exit retains its cluster lock. Follow the -[ownership-transfer procedure](#writer-topology) before starting the next owner. +Use `GET /healthz` for process liveness and `GET /readyz` for readiness. +The listener can accept health checks while graphs are still loading. + +| Readiness state | HTTP | Meaning | +|---|---|---| +| `loading` | 503 | At least one graph is still opening. | +| `serving` | 200 | All graphs are ready, or the applied inventory is empty. | +| `degraded` | 200 | Startup finished; some graphs are unavailable but others can serve. | +| `blocked` | 503 | No graph is ready. A nonempty cluster whose startup opens all fail exits. | +| `draining` | 503 | Shutdown has begun. | + +Readiness includes the booted configuration digest, ledger revision and +ready/loading/blocked graph counts. It is unauthenticated and names no graphs. +`--require-all-graphs` holds every graph unavailable until all open successfully +and makes any opening failure fatal. There are no automatic startup retries. + +Authorized `GET /graphs` and `omnigraph graphs list --server URL --json` expose +each graph's state, read/write availability, suggested action and sanitized +failure category. Availability is independent of the caller's permissions. +Loading or transitioning entries tell callers to wait for their existing owner; +blocked entries require configuration correction or storage recovery. Details +are in server logs. Minimal identity discovery returns IDs and names only. + +Known unavailable graphs return `503 graph_unavailable` only to callers allowed +to discover their availability; unknown graphs return 404. Neither response +authorizes replaying a write. Use exact deployment status to observe +[activation](clusters/index.md#deploy-without-restarting): durable completion +alone does not establish that the affected bindings are serving. + +An admitted write continues after client disconnect, including optional source +removal after a merge. Executing reads also retain their capacity until execution +finishes. A lost write response can leave the outcome unknown; follow +[failure outcomes](operations/troubleshooting.md#failed-data-write-commands) +before submitting again. + +On SIGTERM the server closes admission and drains entered graph opens, admitted +operations and response producers. `--shutdown-grace-seconds` overrides +`OMNIGRAPH_SHUTDOWN_GRACE_SECONDS`; the default is 25 seconds. A clean drain exits +0. Unfinished work at the deadline is logged and the process exits 2. Set the +orchestrator's termination grace longer than the server's. + +A panic or uncertain completion in startup or an admitted write closes admission +for **every graph in that process**. Once the other known operations finish, the +process exits 2; the grace deadline is the upper bound. A proven pre-effect +refusal keeps admission open. The error's HTTP status or storage category alone +does not prove whether effects started. + +Drain does not prove native storage I/O has settled. Even a clean exit retains +the cluster lock. Follow [ownership transfer](#writer-topology) before starting +the next owner. ## Admission limits @@ -140,52 +102,32 @@ refuses the corresponding admission lane. | `OMNIGRAPH_READ_INGRESS_BYTES_MAX` | 67108864 (64 MiB) | Reserved/retained read/MCP body bytes | | `OMNIGRAPH_BODY_TIMEOUT_SECONDS` | 30 | Time allowed to collect one complete request body | -The default ingress allowances total 320 MiB across the independent lanes; -configure both byte limits when setting an instance's input budget. -Body timeouts above 86400 seconds also warn and use the 30-second default. -Ingress reserves the route's maximum before reading, then reduces the reservation -to the actual body size. Body collection limits remain 1 MiB for ordinary JSON -requests, 32 MiB for bulk load and 64 KiB for MCP. Cluster deployment POSTs -allow 16 MiB plus 1 KiB for the request envelope; the captured bundle itself is -limited to 16 MiB. MCP uses the read-body -lane after authentication. Registered bulk routes alone -receive the larger limit; a stored query named `load` or `ingest` keeps the -ordinary JSON limit. Stored-query admission uses the registry's typed read/write -kind. Disconnect does not release input or -operation capacity already transferred to executing work. The body timeout ends -at complete collection; it does not cancel admitted execution. These input limits -do not bound total process RSS or all memory and I/O used by the engine. Size an -instance for the graph, workload -and configured concurrency as well as request bytes. - -The server allows 128 read-response observers and, independently, 64 write-response -observers. Executing reads, pending results, bodies, yielded bytes and their server -producers retain the slot until their final owner releases it. Bodyless reads -consume only a read observer; slow -reads cannot exhaust write-body or write-response capacity. Status routes bypass -ordinary admission so they remain callable during saturation. - -Export and change-baseline responses reserve 256 KiB each from a fixed 2 MiB -process allowance, permitting eight simultaneous reservations. Each response -allows three outstanding 64 KiB chunks plus one pending producer chunk; at most -two chunks wait in its queue. Yielded chunks, including retained clones and -slices, keep their reservation until released. A new response waits at most -250 ms for process capacity before HTTP 413. Once admitted, a producer waits -for chunk capacity or disconnect without a per-chunk deadline; ordinary queue -backpressure has no additional deadline. An interrupted baseline supplies no -usable terminal cursor. Consumers must release each received server buffer to -continue; an in-process consumer retaining every buffer can exhaust its lane. -An HTTP client may still collect its received response: those client buffers are -separate from the server's transport buffers. These limits cover transport -payload buffers, not engine row/Arrow encoding, native work or total process RSS. - -The engine separately limits retained keyed batches, keyed parse estimates -and removed-ID collections across each operation's tables; see -[mutation limits](mutations/index.md#limits-and-conflicts). These fixed limits -return HTTP 413 with structured `resource_limit` details before the operation's -data fragments or publication, and leave the server available for smaller work. -They do not undo an implicitly created load branch or earlier command effects, -and do not establish a process memory ceiling or automatic retry safety. +The independent write/read ingress allowances total 320 MiB by default. +Collection reserves the route maximum, then reduces it to the actual body size. +Body limits are 1 MiB for ordinary JSON, 32 MiB for registered bulk-load routes, +64 KiB for MCP, and 16 MiB plus 1 KiB envelope for deployment submission. +A stored query named `load` still uses the ordinary JSON limit. Body timeouts +above 86400 seconds use the 30-second default. + +The collection deadline ends when the body is complete; it does not cancel +execution. Disconnects do not release capacity held by executing work. +Read/MCP bodies have a separate lane from writes. Independently, 128 read-response +observers and 64 write-response observers retain capacity through execution and +response delivery. Status routes remain available during saturation. + +Export and change-baseline responses share a fixed 2 MiB transport allowance: +eight 256 KiB reservations, each holding at most three outstanding 64 KiB chunks +and one pending producer chunk. New responses wait at most 250 ms for capacity +before HTTP 413. Admitted streams use backpressure without a per-chunk deadline, +so slow clients do not lose data merely by pausing. Interrupted baselines have +no usable terminal cursor. In-process consumers must release yielded buffers +to let producers continue. + +These are input and transport bounds, **not a total process memory ceiling**. +Size instances for graph execution, native I/O and concurrency too. The engine's +separate [mutation limits](mutations/index.md#limits-and-conflicts) return +structured HTTP 413 refusals before data publication; they do not undo earlier +commands or an implicitly created load branch, and do not establish retry safety. ## Container @@ -229,7 +171,7 @@ Set `AWS_ALLOW_HTTP=true` only for a trusted local development endpoint. Do not put credentials in `cluster.yaml` or graph URIs. The same storage root must be visible to servers and out-of-band cluster or -maintenance jobs. Submit schema/query changes and graph additions to the running +maintenance jobs. Submit configuration changes to the running server with `cluster apply --server`; the root's applied revision remains the deployment artifact. Direct apply requires ownership transfer before serving. diff --git a/docs/user/operations/policy.md b/docs/user/operations/policy.md index ad4fbca31..19cf8ccb6 100644 --- a/docs/user/operations/policy.md +++ b/docs/user/operations/policy.md @@ -2,8 +2,9 @@ OmniGraph uses Cedar policy bundles to authorize graph, server, and cluster configuration actions. -Policies are declared in `cluster.yaml` and installed during cluster bootstrap -or graph creation. Servers load them at startup or when activating a new graph. +Declare policies in `cluster.yaml`. Apply them at bootstrap or update them on +running graphs with `cluster apply --server URL --config DIR`. The server +activates the new permissions after affected requests drain. ## Actions @@ -14,7 +15,7 @@ Graph-scoped actions: | `read` | Queries, snapshots, branches, and commits | | `export` | Snapshot export | | `change` | Mutations and loads | -| `schema_apply` | Schema changes | +| `schema_apply` | Schema changes and graph deletion (also requires `read`) | | `branch_create` | Branch creation | | `branch_delete` | Branch deletion | | `branch_merge` | Branch merge | diff --git a/docs/user/operations/server.md b/docs/user/operations/server.md index f089585b7..19922127d 100644 --- a/docs/user/operations/server.md +++ b/docs/user/operations/server.md @@ -207,6 +207,7 @@ keep their existing routes and do not expose MCP. | `GET /openapi.json` | Runtime copy of the OpenAPI document | | `GET /graphs` | Graph metadata catalog; requires `graph_list` policy | | `GET /graphs/discovery` | Graph IDs and display names only; requires an identity credential | +| `POST /cluster/plan` | Preview configuration changes and schema migrations without applying them | | `POST /cluster/deployments` | Submit an exact-ID deployment: graph creation/deletion or schema, query, policy, provider and Blob-binding changes | | `GET /cluster/deployments`, `GET /cluster/deployments/{id}` | Authorized deployment status and current-process activation observation | | `GET /.well-known/oauth-protected-resource` | Public OIDC resource metadata; only when OIDC trust is configured | diff --git a/docs/user/operations/upgrade.md b/docs/user/operations/upgrade.md index 0a204d42b..5d0da2919 100644 --- a/docs/user/operations/upgrade.md +++ b/docs/user/operations/upgrade.md @@ -1,11 +1,21 @@ # Upgrading OmniGraph -Normal open accepts storage format v14 and never migrates a graph. A -standalone graph at v8 or v9 (release 0.11.x) or at v13 is converted in place -by the offline [storage upgrade](#storage-upgrade). A graph at any other -format, and every cluster-managed graph below v14, is refused and takes the -[export/import rebuild](#rebuild) with the binary that wrote the graph. -Check the [release notes](../../releases/) for storage and index compatibility. +OmniGraph 0.13 opens storage format v14. Normal open never migrates a graph. +Upgrade the CLI, server and integrations together; the HTTP contract is exactly +0.13. Keep old executables and a verified whole-root backup until cutover passes. + +| Starting point | Route to 0.13 | +|---|---| +| 0.12 / format v14 | No graph-format conversion. Convert the cluster ledger only if it reports `ledger_upgrade_required`; see [ledger conversion](../clusters/index.md#direct-deployments-and-conversion). | +| Standalone v8/v9 (0.11.x) or v13 | Offline [storage upgrade](#storage-upgrade), preserving branches and retained history. | +| Standalone v6 (0.9/0.10) | First use 0.11's `upgrade --to-format 8`, then the 0.13 storage upgrade. Run each binary's `--check` first; keep all readers and writers stopped across both steps. | +| Cluster-managed graph below v14 | Export with its old binary and [rebuild](#rebuild) at a new root. | +| Other unsupported format | Use the refusal's version guidance and [export binary table](#choose-the-export-binary); rebuild rather than bypassing the format check. | + +An export/rebuild preserves the exported entities, not old commit IDs, snapshots +or shared branch history. In-place storage upgrade preserves retained history. +See [what a rebuild preserves](#what-an-exportimport-rebuild-preserves) before +choosing a route. Old full-text indexes may also need an [index rebuild](#full-text-index-upgrade). ## Storage upgrade @@ -111,23 +121,16 @@ refused: a cluster-managed graph is exported with the build that wrote it and ## v0.9 to v0.10 -Released v0.9.0 uses Lance 9; v0.10.0 uses Lance 11. Both use graph storage -format v6, so existing entities, branches, and retained history do not need an -export/import migration. Development builds using Lance 10 follow the same -full-text upgrade procedure below. - -The CLI/API vocabulary changes are not rolling-compatible. Update the CLI, -server, and client integrations together, with application traffic stopped. -Even without full-text indexes, do not run an old and new fleet against the -same graph. Keep the old executables and a verified whole-root backup until the -upgrade is proven. See the [v0.10 compatibility notes](../../releases/v0.10.0.md#compatibility). +This earlier transition changed Lance's full-text analyzer while retaining +storage format v6. For its historical release details, see +[v0.10 compatibility notes](../../releases/v0.10.0.md#compatibility). +For an upgrade to 0.13, use the route table above; v6 is not a current open format. ## Full-text index upgrade -The v0.9-to-v0.10 Lance 11 transition keeps graph storage format v6, entities, -branches, and history. It changes the English stemmer used by full-text search. -Old indexes cannot safely be searched by the new analyzer, so OmniGraph explicitly refuses -full-text queries until the selected indexes have been rebuilt. +After converting a supported graph, full-text queries can still refuse indexes +created with the older Lance 9/10 analyzer. Rebuild the selected indexes before +searching them; a graph-format upgrade alone does not certify their analyzer. 1. Stop application readers and writers. Using the old CLI, inventory the live branches and record which need full-text search, including `main`: @@ -196,17 +199,9 @@ Keep the pre-upgrade cluster deployment bundle, configuration, and state backup alongside the graph-root backups. A rollback restores that consistent set with the old executables; it does not point an old server at upgraded index files. -If v0.10 applied an `external_blobs` policy and the cluster state itself is not -being restored, v0.9 cannot read that new optional state field. While every -writer is stopped, remove `external_blobs` from each graph's configuration, -review `cluster plan`, and use the v0.10 `cluster apply` to converge the removal -before starting v0.9. Editing only the YAML is insufficient; do not hand-edit -the state ledger. This state-shape step does **not** replace restoring the -pre-upgrade graph backup after full-text rebuilding. - -A downgrade also removes v0.10's default-deny external-URI enforcement: v0.9 -writers admitted arbitrary supported sources, including `file://`. Do not roll -back to v0.9 if that ingress boundary is required. +Do not downgrade by editing configuration fields or internal ledger objects. +Restore the coherent pre-upgrade backup with its matching binaries and security +configuration. Keep external Blob sources available throughout verification. ## What an export/import rebuild preserves diff --git a/docs/user/search/embeddings.md b/docs/user/search/embeddings.md index e58fce1b4..8b2f7de16 100644 --- a/docs/user/search/embeddings.md +++ b/docs/user/search/embeddings.md @@ -54,8 +54,10 @@ graphs: embedding_provider: default ``` -Inline API keys are rejected. `${ENV_VAR}` references are resolved when the -server starts, not when the cluster configuration is planned or applied. +Inline API keys are rejected. The server resolves `${ENV_VAR}` references at +startup and while preparing a live configuration deployment. Set secrets in +the server environment before applying provider changes; local validation does +not load those credentials. Updating a provider does not re-embed stored rows. ## Schema annotation diff --git a/skills/omnigraph/SKILL.md b/skills/omnigraph/SKILL.md index b992d81ca..19fbe796a 100644 --- a/skills/omnigraph/SKILL.md +++ b/skills/omnigraph/SKILL.md @@ -1,400 +1,113 @@ --- name: omnigraph -description: Operate OmniGraph graphs and deployments. Use for `.pg` schemas, `.gq` queries, OmniGraph CLI commands, `file://`/`s3://`/`az://` graph URIs, `cluster.yaml`, operator config, bearer-authenticated servers, graph-backed knowledge or memory, Blob values, embeddings, branches, commits, and change feeds. Apply especially before schema changes, bulk loads, and retries after uncertain remote outcomes. +description: Operate OmniGraph graphs and deployments. Use for .pg schemas, .gq queries, CLI and HTTP operations, cluster.yaml, bulk loads, branches, Blob values, embeddings, and change feeds. Read before schema changes or retrying an uncertain remote write. license: MIT (see LICENSE at repo root) -compatibility: Covers the current OmniGraph CLI and server, using HTTP contract 0.13. Upgrade the CLI, server and HTTP integrations together; only storage format 14 is served, and a standalone v8, v9 or v13 graph is converted by the offline `omnigraph upgrade`, never on open. metadata: author: ModernRelay version: "0.13.0" repository: https://github.com/ModernRelay/omnigraph --- -# Operating Omnigraph Locally - -This skill captures the operational rules for working with a locally or remotely deployed Omnigraph. Follow them when authoring schema, writing queries, loading data, evolving schema, or automating graph operations. - -Check `omnigraph version` and the command's `--help` before using these -instructions. This skill tracks the current CLI/server contract, including -`Omnigraph-Http-Api: 0.13`; the released 0.12 contract is unsupported. The engine -reports `internal-schema 14 (serves v14 to v14)` and serves storage -format 14 only. New graphs use `__id`/`__src`/`__dst` system columns; a graph -that already spells them `id`/`src`/`dst` keeps its physical columns. A -standalone v8 or v9 graph (written by 0.11.x) or a v13 graph is converted in -place by the offline `omnigraph upgrade`; a v0.10 graph uses v6 and is first -taken to v9 with the 0.11 binary, or rebuilt. See -[upgrading past v0.11](references/migrations.md#upgrade-v011-to-storage-format-14). -Read [migration guidance](references/migrations.md) before replacing a -deployed binary. - -## GQ 2.1 traversal support - -0.12.0 compiles GQ 2.1, which adds typed edge selections to traversal. A v0.11 -binary cannot parse this syntax; do not send it to a deployment not yet on 0.12. -`$a (knows | likes) $b` selects named edge types; `$a * $b` selects compatible -types from the captured schema and requires declared endpoint node types. -Omitted bounds always mean `{1,1}`; recursive traversal needs finite bounds -such as `{1,3}`. `$a $e:(knows | likes) $b` binds one-hop concrete edges, exposing -`$e.@type` and `$e.@id`. Common properties must exist with compatible base types -on every member; nullability and enum domains widen. - -`set traversal_work_limit = 1000000;` controls shared traversal row work in -statements using selections. Full selected-table sizes count before scans, even -for selective neighborhoods. A result limit cannot replace this cap. Exhaustion -terminates the query with an error; received rows do not become partial success. -Historical wildcard targets are refused. Stored-query validation can refuse a -schema addition that would invalidate a wildcard common-property reference. -See [traversal](../../docs/user/queries/traversal.md) and -[session settings](../../docs/user/queries/settings.md). - -## The Seven Rules - -1. **Lint before commit** — `omnigraph lint --schema schema.pg --query queries/foo.gq` validates both sides against each other. No running repo required. -2. **Plan before apply** — never run `schema apply` without a successful `schema plan` first. Apply is destructive; plan is free. (Cluster mode has the same rule with different verbs: `cluster plan` before `cluster apply` — the plan embeds the engine's real migration steps.) -3. **Branches are for data; apply is for schema** — review bulk data loads on a feature branch then merge. Schema changes go straight to `main`: in cluster mode edit the `.pg` and run `cluster apply` (`--server ` deploys to the running server without a restart; without it serving must be stopped first; a direct `schema apply` **refuses** a cluster-managed graph); for the managed service commit, `cluster push --managed`, `cluster plan --managed --rev `, then `cluster apply --managed --plan `; `schema plan`/`apply` is for a non-cluster store. -4. **Pick the right write command** — `mutate` for edits (typechecked, parameterized); `load` for bulk JSONL, local **or** remote, with a **required** `--mode` (`merge` upsert · `append` strict-insert · `overwrite` replaces only the node/edge types represented in the batch). `load --from ` forks a review branch in one shot; bare `load` needs an existing target branch. -5. **Parameterize everything** — never string-interpolate values into `.gq` bodies or `--params`. Declare `$var: Type` and pass via `--params`. -6. **Expose agent reads as aliases** — aliases decouple a read operation name - from its stored-query implementation. Aliases are read-only; invoke a served - stored mutation with `omnigraph mutate --server ...`. -7. **Treat a lost remote response as unknown** — an effectful data mutation or load returns its exact commit, but a proxy can return 504 after publication. Branch statements have different outcome/receipt semantics. On timeout, reconcile the intended effect and relevant history before replay. See `references/remote-ops.md`. - -## Essentials: Queries, Mutations, Loads - -The patterns below cover the daily 80% — enough to write correct `.gq` and JSONL without leaving this file. The long tail (multi-hop, negation, aggregations, hybrid search, every decorator) is in [`references/queries.md`](references/queries.md) and [`references/schema.md`](references/schema.md). - -**Comments in `.pg` and `.gq` are `//`, never `#`** (the #1 parse error). - -### Read query (`.gq`) - -```gq -query get_signal($slug: String) { - match { - $s: Signal { slug: $slug } // inline property filter goes in the match block - $s formsPattern $p // edge FormsPattern declared PascalCase, traversed lowerCamelCase - } - return { $s.slug, $s.name, $p.slug } -} -``` - -- **Parameterize, never interpolate.** Declare `$var: Type` in the signature; pass via `--params '{"slug":"sig-foo"}'`. An empty signature still needs parens: `query foo() { ... }`. -- **Edge traversal is lowerCamelCase** even though the schema declares edges PascalCase (`FormsPattern` → `formsPattern`). -- **System identity uses `@`.** Read `$s.@id` and `$e.@src`/`$e.@dst`; - bare `id`, `src`, and `dst` name declared user properties. `return { $s }` - returns an object containing `@id` and its non-Blob/non-Vector properties. - `schema show --json` reports the graph's physical `system_columns`; do not - substitute physical `__id` names into GQ. -- **List/sort** by appending `order { $s.stagingTimestamp desc } limit 50` after `return`. Without `order`, `limit n` may return any `n` valid rows (the subset can change between releases); order for stable pages. -- **`nearest` and `rrf` require a trailing `limit N`** — omitting it is a compile error. `bm25` does not require a limit, but use one to keep ranked output bounded. Ranking operators lead `order { }`, and still restrict rows (`bm25` keeps only text matches; `nearest` skips null vectors). Scope with `match`/filters first, then rank (`order { nearest($d.embedding, $q) } limit 10`). Project a score by repeating the leading key: `return { $d.slug, nearest($d.embedding, $q) as distance }`. - -### Mutation (`.gq`) - -There is **no top-level `mutation { }`** — every block is a named `query`; the verb (`insert`/`update`/`delete`) makes it a write. Dispatch with `omnigraph mutate` (not `query`). (A source may instead hold exactly one branch statement — `branch create|delete|merge|list`; see [`references/queries.md`](references/queries.md#branch-statements).) - -```gq -query add_signal($slug: String, $name: String, $brief: String, $createdAt: DateTime) { - insert Signal { slug: $slug, name: $name, brief: $brief, - stagingTimestamp: $createdAt, createdAt: $createdAt, updatedAt: $createdAt } -} -query link($from: String, $to: String) { insert FormsPattern { from: $from, to: $to } } -query retitle($slug: String, $t: String) { update Signal set { name: $t } where slug = $slug } -query remove($slug: String) { delete Signal where slug = $slug } -``` - -- **Every non-nullable property must be supplied.** Lint normally reports T12 - when one is missing. A non-null Vector target carrying `@embed` - is a known exception: source-only insert can lint successfully even though - execution still requires the vector. Supply it explicitly; writes never - embed automatically. -- A single mutation is insert/update-only **or** delete-only — never both (D₂ rule, refused at execution before any effect; `lint` does not catch it); split them. -- Edge inserts give logical `from`/`to` endpoint IDs; a propertyless edge needs only those, and there is no nested `data` block in GQ. An unkeyed edge insert is strict (inserting twice creates two edges). An edge type may declare `@key(@src, @dst[, prop…])` in its body when created; inserts then upsert by the derived id, and identical inserts on two branches converge on merge. Edges cannot be `update`d (T16). - -### Bulk load (JSONL) - -```jsonl -{"type":"Signal","data":{"slug":"sig-foo","name":"Foo","brief":"…","stagingTimestamp":"2026-04-14T00:00:00Z","createdAt":"2026-04-14T00:00:00Z","updatedAt":"2026-04-14T00:00:00Z"}} -{"edge":"FormsPattern","from":"sig-foo","to":"pat-bar","data":{}} -``` - -```bash -omnigraph load --data seed.jsonl --mode merge $GRAPH # --mode is REQUIRED (no default) -omnigraph load --data delta.jsonl --from main --branch review --mode merge $GRAPH # fork a review branch in one shot -``` - -- `--mode`: `merge` (upsert by logical entity ID; keyed node and keyed edge IDs derive from their `@key` tuple — re-running a merge file duplicates unkeyed edges) · `append` (fails on ID collision) · `overwrite` (destructive, staged). `--from ` forks a missing `--branch`; bare `load` needs an existing branch. Works local **and** remote. -- **Date values**: use a calendar-day string (`YYYY-MM-DD`) for `Date` and an ISO timestamp to the millisecond for `DateTime` (`.123` or `.123000`; `.123456` is refused), in both `mutate --params` and JSONL. `load` also accepts integer epoch days for `Date`. - -### Dispatching +# OmniGraph + +Check `omnigraph version` and the relevant command's `--help`. These instructions +cover 0.13 only, serving storage format 14. HTTP clients send and verify `Omnigraph-Http-Api: 0.13`; changing +an older client's header alone does not make its routes and response handling +compatible. Before replacing a binary, read [upgrades](references/migrations.md). + +## Choose the operation + +| Task | Workflow | Reference | +|---|---|---| +| Read or edit graph data | `query`, `mutate`, `load --mode …` | [Queries](references/queries.md), [data](references/data.md) | +| Change a running cluster | Edit declared files, `cluster plan --server …`, `cluster apply --server …` | [Cluster](references/cluster.md) | +| Bootstrap or update a stopped cluster | Direct `cluster plan`/`apply`, then transfer the exact writer lock before serving | [Cluster](references/cluster.md) | +| Evolve a standalone store | `schema plan`, then `schema apply` | [Schema](references/schema.md) | +| Operate the managed service | `cluster --managed` | [Managed clusters](references/cluster.md#managed-clusters) | +| Investigate a lost write response | Inspect the intended effect and exact receipt/history before replay | [Remote operations](references/remote-ops.md) | +| Maintain or upgrade storage | Direct maintenance with overlapping writers stopped | [Commands](references/commands.md), [upgrades](references/migrations.md) | + +A server serves an applied cluster, never a standalone graph. Live cluster apply +supports schemas, stored queries, policies, provider and external-Blob bindings, +and graph creation/deletion. Removing a graph declaration permanently deletes +its managed storage and retained history. Schema changes require only `main` to +remain live; a merge without branch deletion does not satisfy that requirement. + +## Addressing and authorization + +- Served graph: `--server --graph `. Named servers live in + `~/.omnigraph/config.yaml`; tokens live separately in `~/.omnigraph/credentials` + via `omnigraph login `. The bearer token determines the actor; a served + write refuses `--as`. +- Direct graph: `--store `, or a positional storage URI + where the command allows it. Query/mutate use `--store` because their positional + argument is the query name. Direct access is a separate storage trust boundary. +- `--config ` selects source/context files for `cluster` and `use`, not a + data-command target. `--cluster --graph ` addresses applied cluster + state for commands such as policy inspection and maintenance. +- `--profile` and operator defaults can supply a target. A managed folder's + `.omnigraph/context` routes supported data commands to the service; global + `--direct` selects ordinary addressing. Competing ambient targets refuse. +- Azure writes require `omnigraph-azure-admission` and remain a qualification + preview. Storage credentials belong in the writer's environment, never in + committed configuration. + +One mutation-capable process owns a cluster. A server, direct apply or direct +cluster write leaves its writer lock after exit. Release the exact lock only +once the previous owner has stopped and its I/O has settled; use served apply +while the server holds that lock. See [cluster recovery](references/cluster.md#recovery). + +## Authoring and data workflow + +Use `//` comments in `.pg` and `.gq`. Lint schema and queries together: ```bash -omnigraph alias signal sig-foo # operator alias → its bound stored read query -omnigraph query get_signal --params '{"slug":"sig-foo"}' # served stored query by name (verb asserts read vs write) -omnigraph query -e 'query q() { match { $s: Signal } return { $s.slug } limit 5 }' # ad-hoc/inline (or: --query f.gq ) -omnigraph mutate add_signal --query mutations.gq --params '{"slug":"sig-foo","name":"Foo","brief":"Example","createdAt":"2026-04-14T00:00:00Z"}' # name positional; ad-hoc file source -omnigraph lint --schema schema.pg --query queries/foo.gq # after EVERY .gq/.pg edit (no server needed) +omnigraph lint --schema schema.pg --query queries/people.gq +omnigraph query get_person --server production --graph knowledge \ + --params '{"slug":"ada"}' --json +omnigraph mutate update_person --server production --graph knowledge \ + --params '{"slug":"ada","name":"Ada"}' --if-commit --json ``` -### `.gq` grammar - -The non-obvious facts that bite, then the full grammar: - -- **Scalar param types**: `String Bool I32 I64 U32 U64 F32 F64 DateTime Date Blob`. Modifiers: `T?` (optional), `[T]` (list), `Vector(N)`. There is **no `Int`** — use `I64`. -- **A read query needs `match` *and* `return`** (`order`/`limit` optional); a mutation has neither — only `insert`/`update`/`delete`. -- **`limit` takes an integer literal, not a param** — `limit 50`, never `limit $n`. -- **Variable-hop traversal**: `$p knows{1,3} $f` — bounds are **required to be finite** (`{1,}` is rejected: "unbounded traversal is disabled"). Hops are shortest-path distances; each `$_` is a distinct anonymous node; variables beginning `__` are reserved. -- **Undirected traversal**: `$p $f` matches the edge in either direction, deduplicated (a pair connected both ways appears once). Same-endpoint-type edges only (e.g. `Related: Issue -> Issue`) — asymmetric edges are rejected (T22). Composes with bounds (`$p {1,3} $f`) and correlated blocks (`not { }`, `exists { }`, `count { } > 2`). -- **Edge bindings**: an optional `$var:` prefix on the edge word — `$src $w:knows $dst`, undirected `$a $w: $b` — binds the matched edge row, so edge properties work in filters (`$w.confidence = "asserted"`), projections (`return { $w.role }`), aggregates, and ordering. A bound traversal returns one row per edge (parallel edges stay distinct); binding a `{min,max}` multi-hop, rebinding a taken name, or projecting bare `$w` is rejected (T23). -- **Literals & calls**: `now()`, `date("2026-04-29")`, `datetime("…T00:00:00Z")`, list `[…]`. -`starts_with`, `contains`, `in`, `>=`, `<=`, `!=`, `>`, `<`, `=`, `is null`, `is not null` - -Those are the complete **filter operators** (`$m.number in $numbers` tests membership in a list parameter or literal); conditions compose with `and`, `or`, `not`, parentheses and `is null` / `is not null`, and a filter keeps only rows whose expression is true (a comparison with a null operand is null). String predicates are exact and -case-sensitive. **Aggregates** are `count/sum/avg/min/max` (`count($f) as n`); `min`/`max` also accept String, Bool, Date, and DateTime. Alias aggregates and order by the alias. -- **Result column names must be distinct** (`T25`): alias projections that would collide. -- **Stored-query metadata**: `@description("…")` / `@instruction("…")` may follow the param list. -- **Casing**: type names uppercase-initial (`Signal`); idents/edges lowercase-initial (`formsPattern`); variables `$`-prefixed. `//` and `/* */` comments only. - -The compiler grammar in `crates/omnigraph-compiler/src/query/query.pest` is the -single source of truth. - -## CLI Reference (condensed) - -Notation: `` required · `[x]` optional · `` choice · `…` repeatable. - -**Global addressing flags**: `--as ` (direct-engine writes, `rebuild-full-text-indexes`, and `cluster apply`/`upgrade-ledger`; a served write **refuses** `--as` because the server resolves the actor from the bearer token, and read verbs reject it), `--server `, `--cluster ` (cluster-managed storage, primarily for maintenance), `--graph ` (selects within a `--server` or `--cluster` scope; required for managed queries, mutations, loads and commit reads), `--profile ` (`$OMNIGRAPH_PROFILE`), `--store `, `--direct` (ignore managed context for data commands). Commands with an open positional slot also accept `file://`, `s3://`, or preview `az://` directly. `--config ` selects source/context folders for `cluster` and `use`. Output: `--json`, or read queries take `--format ` (`arrow` is listed but always fails in 0.12.0). **Write guards:** `--yes` skips non-local confirmation for destructive writes; `--quiet` suppresses the resolved-target echo. - -**Data plane** — `any` (served via `--server`/`--profile`, or direct via `--store`/URI): -- `query` `` — a **served stored query** by name (via `--server`/`--profile`); or ad-hoc `[] (--query | -e '')` where `` picks which query in the source. `[--params | --params-file

] [--branch | --snapshot ] [--format | --json]`. No positional URI — address via `--server`/`--store`/`--profile`. -- `mutate` — same shape (served stored mutation by ``, or ad-hoc `--query`/`-e`); `[--params …] [--branch ] [--if-commit ] [--json]`. The verb asserts kind; a failed precondition has no effect: a served mutate exits 4, a direct (`--store`) one exits 1. -- `load --data --mode [--branch ] [--from ] [--json]` — `--mode` required; `--from` forks a missing `--branch`; overwrite replaces only represented types -- `blob ` — dedicated Blob-cell reads; `get` supports ranges/`--out`, `stat` returns metadata -- `snapshot [--branch ] [--json]` -- `export [--branch ] [--type …]` (streams JSONL) -- `branch [--from ] | list | delete | merge [--into ] [--delete-branch]> [--json]` (`--from`/`--into` default to `main`; also available as GQ statements via `mutate -e`/`query -e`) -- `commit ] | show | changes [filters…]> [--json]` -- `changes | --cursor ] | baseline --out > [filters…] [--json]` -- `schema show` (alias `get`) - -Every read query runs on engine v2, the planned execution route and the only -`engine` value. `query -e 'explain query q() { … }'` -returns the v2 logical, physical and available DataFusion trees without -executing the query. It uses the ordinary query target and parameter flags. - -**Served only** (needs `--server`/`--profile`): `graphs list [--json]` - -**Direct / storage** — reject `--server`. `init` requires its positional URI; -`schema plan`/`apply` use a positional URI or `--store`; lint and maintenance also -accept `--cluster --graph `: -- `init --schema [--force]` -- `schema --schema [--json]` — `apply` refuses a cluster-managed graph; evolve those via `cluster apply` -- `upgrade [--check] [--to-format 14] [--json]` — offline standalone - conversion of a v8, v9 or v13 graph to format 14, the only target. - `schema upgrade-system-columns [--check] [--json]` respells a - format-14 graph's legacy `id`/`src`/`dst` columns without changing the - format. Read [migration preconditions](references/migrations.md) first; - cluster-managed roots refuse. The 0.11 binary's `upgrade` took - `--to-format <7|8|9>` and defaulted to v9. -- `lint --query [--schema ] [] [--json]` — offline with `--schema`, graph-backed with a URI -- `optimize [--json]` · `repair [--json]` · `cleanup [--keep ] [--older-than <7d>] --confirm [--json]` (at least one retention option; both may be combined) -- `rebuild-full-text-indexes [--branch ] [--json]` — replace full-text indexes on one branch with default English analysis; custom tokenizer settings are replaced. Stop overlapping writers and retain a whole-store backup for upgrades. `--as` records attribution; direct access does not load server policy. See [maintenance commands](references/commands.md#rebuild-full-text-indexes--explicit-analyzer-upgrade). - -**Control plane**: -- `cluster [--config

] [--json]` — self-hosted unless `--managed` is explicit; folder context never selects the mode. `plan` and `observe` are read-only. `plan`, `apply`, and `status` accept `--server `. Served `apply` waits for activation; `--no-wait` returns after durable acceptance, and `--timeout <1..3600>` bounds caller waiting. Resume observation with `status --deployment-id --wait [--timeout ]`; timeout never cancels execution. Direct recovery uses the original `--deployment-id` with `--writers-stopped`. -- `cluster force-unlock [--config ] [--json]` · `--cluster cluster upgrade-ledger --writers-stopped [--json]` (explicit conversion of a supported legacy ledger, with serving and writers stopped) -- Managed service: `use --api ` · `cluster --managed` for `create --api `, `push --expected-revision --message `, `plan [--rev ]`, `apply --plan `, `status [RUN_ID]`, `operation [--wait]`, `history`, `cancel `, `token ([--ttl 1h] | --clear)`, `delete --incarnation `, or `undo-delete --incarnation --deletion-id `. The mode rejects ordinary target/actor flags and `--direct`; see [managed clusters](references/cluster.md#managed-clusters). -- `policy | explain --actor --action [--branch | --target-branch ]> --cluster [--graph ]` -- `queries --cluster [--graph ] [--json]` - -**Local** (no graph): -- `alias [args…]` — invoke an operator alias's bound stored read query; `[--params … | --params-file

] [--format | --json]` (server/graph/query come from the binding) -- `embed (--seed | --input --output --spec ) [--reembed-all | --clean] [--type …] [--select ":="]` -- `login [--token ]` (prefer piping the token on stdin) · `logout ` · `login --api ` / `logout --api ` (managed session) · `profile ]>` · `version` - -Managed folders can select an Intent API with `login --api` and `use`; implicit -`query`, `mutate`, `load`, `commit list`/`show` and `graphs list` then acquire -and cache a separate identity credential themselves (`cluster token --managed` remains -for credential administration). Global `--direct` -selects ordinary addressing. See [managed routing](references/cluster.md#managed-clusters) -before using ambient targets or credentials. - -Pre-0.7.0 spellings (`read`/`change`/`ingest`, `--target`, positional `http://`) → [`references/migrations.md`](references/migrations.md). - -## Five Ontology Design Criteria (Gruber 1993) - -Omnigraph schemas are ontologies. The canonical design criteria from Gruber's *Toward Principles for the Design of Ontologies Used for Knowledge Sharing* (Int. J. Human-Computer Studies 43:907–928) apply directly when authoring `.pg` files. - -1. **Clarity** — definitions should communicate intended meaning unambiguously and be independent of social or computational context. In Omnigraph: precise type names, narrow enums over `String`, `@check`/`@range` for stated invariants. A reviewer should understand the domain from the schema alone. -2. **Coherence** — inferences sanctioned by the schema must be consistent with the domain modeled. Gruber's trap: defining quantity as a `(magnitude, unit)` pair makes `6 feet ≠ 2 yards` even though they describe the same length. In Omnigraph: watch for `@card`, `@unique`, and edge directionality that let the schema distinguish things the domain treats as equal. -3. **Extendibility** — the schema should support specialization without revising existing definitions. In Omnigraph: prefer interfaces for shared shape, leave enums open where the domain genuinely admits more, model identifiers via mapping functions rather than baking units/formats into the entity. -4. **Minimal encoding bias** — representation choices made for notation or implementation convenience leak into the model. In Omnigraph: don't type dates as `String` because the source API returns strings; separate conceptual entities (a publication date, a person) from their surface encoding (a year integer, a name string) when both matter. -5. **Minimal ontological commitment** — make as few claims about the world as the use case requires. In Omnigraph: don't add required properties, closed enums, or `@card(1..1)` "in case". Weaker schemas leave consumers room to specialize — but tightening later is a rebuild, not an in-place `schema apply` (adding `@key`/`@unique`/`@range`/`@check`, changing cardinality, and `T?` → `T` are refused; only `@index` additions, nullable additions, enum widening, renames, and drops apply in place), so decide deliberately. - -The criteria trade off against each other — Clarity wants tight definitions while Minimal Commitment wants weak ones. Gruber's resolution: *having decided a distinction is worth making, give it the tightest possible definition*. Decide what to model conservatively; once modeled, constrain precisely. - -## Schema Authoring Principles - -Twelve practical rules for `.pg` authoring — full text and examples in the bundled [`references/schema.md`](references/schema.md). In short: schema-is-the-contract · explicit identity via `@key` · model meaning not tables · strong intentional types · deliberate optionality · shared shape in interfaces · schema-level constraints (`@unique`/`@index`/`@range`/`@check`/`@card`) · search as a schema decision · edge semantics matter · reviewable schemas · intentional migrations (`@rename_from`) · domain clarity over ORM habits. - -Design flow: entities → stable keys → relationships worth their own edge → enum candidates → uniqueness/bounds/cardinality → search needs → shared shape into interfaces → evolution plan. - -## Provenance Is Structural (Multi-Agent Source of Truth) - -When Omnigraph serves as canonical truth across multiple agents, every assertion must answer *who said it, when, based on what evidence*. This is the runtime guarantee Gruber's criteria don't cover — his agents shared vocabulary; ours additionally must share attribution. Provenance belongs in the schema, not in logs. - -Without structural provenance, agents cannot reconcile contradictory assertions, retract facts when a source is discredited, replay graph state at a past timestamp, or distinguish high-evidence facts from speculation. - -**In Omnigraph:** model provenance as a `Claim` node linked by typed edges to -the asserted fact, an `Actor`, and a `Source`. Keep scalar facts such as -`asserted_at: DateTime` and optional `confidence: F64` on `Claim`; properties -cannot be node-typed. Don't stash provenance into a free-text `source: String` -or a metadata dump—structural provenance is queryable and migratable; -free-form provenance is neither. - -## Storage & Credentials - -A graph's bytes live in one of three supported URI families: - -- **Local filesystem** — a path or `file://` URI. In cluster mode `storage:` defaults to the config directory, so local dev needs no object store. -- **S3-compatible object storage** — AWS, Railway, Tigris, etc. (`s3://bucket/prefix`). Authenticate with the standard `AWS_*` environment contract. -- **Azure Blob storage preview** — `az://container/prefix`. Reads and Azurite qualification are available, but Azure is not production-supported; every write or maintenance process must be launched through the root-scoped `omnigraph-azure-admission` wrapper. - -Keep development credentials in a git-ignored `.env.omni` and source it before CLI calls: - -```bash -set -a && source .env.omni && set +a -``` - -Direct `init`/`load` and **`cluster apply`** write storage without an HTTP server. A served `load` is different: the CLI sends it to `omnigraph-server`, which performs the graph write. A direct `cluster apply` reaches the cluster ledger and graph datasets directly, so its host needs storage credentials (`cluster apply --managed --plan` is executed by the Intent API). A serving process also needs read-write access for served data-plane writes. Validate with `curl http://127.0.0.1:8080/readyz` (HTTP 200, `ready: true`; `/healthz` only proves the process is alive), then `omnigraph snapshot --server --graph --json`. - -## Project Layout - -### Deployment & access - -- **Cluster deployment — the only way to serve.** A `cluster.yaml` declares the - whole deployment (graphs, schemas, stored queries, policies, optional S3/Azure - `storage:` root); `omnigraph cluster apply` converges it and - `omnigraph-server --cluster .` (or `--cluster s3://bucket/prefix`, - config-free) serves it. A direct apply keeps its admission lock: once its - writes have settled, release that exact lock with - `cluster force-unlock ` before starting the server, and send later - changes with `cluster apply --server `. - See `references/cluster.md`. -- **Direct / embedded access — no server.** Address a graph's storage directly - with `--store ` or a positional URI for one-off CLI ops. - There is **no single-graph server mode** — the server is cluster-only. - -### The two config surfaces - -Configuration has two single-owner homes (RFC-007/008), plus an -everything-explicit flag/env tier: - -| Surface | Owner | Location | Declares | -|---|---|---|---| -| **Cluster config** | the team, in the repo | `cluster.yaml` + the `.pg`/`.gq`/policy files it references | what the system **is**: graphs, schemas, queries, policies, storage | -| **Operator config** | one person | `~/.omnigraph/config.yaml` (`$OMNIGRAPH_HOME` relocates it) | who **I** am: identity, named servers, output defaults, personal aliases | -| Flags / env | per invocation | — | everything, explicitly | - -```yaml -# ~/.omnigraph/config.yaml — per operator, never committed -operator: - actor: act-andrew # default --as identity -servers: - intel-dev: - url: https://graph.example.com # no tokens here, ever -defaults: - output: table # read-format default - server: intel-dev # default served scope (or `store: file://…/g.omni` for a local default — mutually exclusive) - default_graph: spike # graph within a server/cluster scope -profiles: # optional named scope bundles — pick with --profile - staging: { server: intel-staging, default_graph: spike } -aliases: # personal bindings to TEAM stored queries (see references/aliases.md) - triage: { server: intel-dev, graph: spike, query: weekly_triage, args: [since] } -``` - -The operator config and credentials are **auto-discovered — no flag points at them**: the CLI reads `$OMNIGRAPH_HOME/config.yaml` (default `~/.omnigraph/config.yaml`), and an absent file is just an empty layer (zero-config). `$OMNIGRAPH_HOME` relocates the *directory* only, not a specific file. `--config` selects source/context folders for `cluster` and `use`. - -Credentials live outside config: `echo $TOKEN | omnigraph login intel-dev` -writes `~/.omnigraph/credentials` (`0600`); the matching token resolves via -`OMNIGRAPH_TOKEN_INTEL_DEV` or that file. - -**Addressing a graph**: `--store ` or a positional URI for -direct storage; `--server ` (+ `--graph `) for a served remote; -`--profile ` for a named bundle; else the operator `defaults`. A remote is -addressed with `--server` (a bare `http(s)://` URL is not a graph address). Run -data-plane commands from a graph's project folder so relative `queries/`, -`schema.pg`, and `.env.omni` paths resolve. - -### What to commit - -**Commit:** `schema.pg`, `queries/*.gq`, `cluster.yaml`, `seed.md`, `seed.jsonl`, and the project's `README.md` and `CLAUDE.md`. - -**Ignore:** `.env.omni` (credentials), `.claude/` (local agent state), `*.omni/` (local graph artifacts), `__cluster/` and `graphs/` (cluster state + derived graph roots). - -### Give agents a `CLAUDE.md` - -A per-project `CLAUDE.md` tells coding agents where files live and what conventions matter. Without it, agents re-discover the same things every session. - -## Common Gotchas - -These are the traps most likely to bite. Scan this table before debugging any parse or runtime error. - -| Trap | Symptom | Fix | -|------|---------|-----| -| `#` comments in `.pg` | `parse error: expected schema_file` | Use `//` | -| Standalone `enum Foo { ... }` block | `parse error: expected EOI or schema_decl` | Inline: `kind: enum(a, b)` | -| `[Category]` (list of enum) | compile error | Use `[String]`; lists must contain scalars | -| Assuming `@embed` must quote its source | unnecessary schema churn | `@embed(text)` and `@embed("text")` are both valid; quoted form is canonical | -| Bare `src`/`dst` in an edge constraint on a new graph | offline `lint` passes; `init`/`schema plan` fail `unknown property reference 'Knows.src'; the system field is '@src'` | Put `@unique(@src)` inside the edge body; `src` means a declared user property | -| `_`-prefixed property on a new graph | `property name '_x' is reserved for system columns (names starting with '_')` | Rename it; `from`/`to` are also reserved on edges (`is reserved on edge types`) | -| Expecting `@embed` to populate vectors during load | missing/stale vectors | `@embed` is metadata; run the offline `omnigraph embed ... --reembed-all` file pipeline, then load its output | -| `schema apply` with feature branches open | `schema apply requires a graph with only main` | Delete them (`merge --delete-branch` or `branch delete`); a merge alone leaves the branch live | -| `nearest(...)` / `rrf(...)` without `limit` | compile error | Add `limit N`; a BM25-only query may omit it, though bounded output is recommended | -| Adding non-nullable property without backfill | unsupported migration | Make optional → backfill; keep it optional (tightening `T?` → `T` is refused, OG-MF-106) | -| `omnigraph init --json` | `unexpected argument '--json' found` | `init` doesn't support `--json`; drop the flag | -| `omnigraph init` on an already-initialized URI | `graph already initialized or initialization metadata exists at ''` | Never overwrite it. `--force` only replaces orphan schema artifacts after proving there is no graph manifest | -| `schema apply` dropping a property/type | nothing is reclaimed at apply; older commits still read the dropped data | to reclaim the space, run `cleanup` with a retention that excludes the pre-drop commits; the data is then unrecoverable | -| Committing `.env.omni` | credential leak | Add `.env*` to `.gitignore` | -| Non-parameterized query values | typecheck surprise, injection risk | Declare `$param: Type` and pass via `--params` | -| Missing required field in `insert` | ``T12: insert for `X` must provide non-nullable property `Y` `` | Accept the param in the mutation signature | -| Mixing `delete` with `insert`/`update` in one query | lint passes; `mutate` fails "mixes inserts/updates and deletes" | Split into two mutations (D₂ rule) | -| `$p.id`, `{ id: $v }`, `where id =` from a v0.10 query | ``T6``/``T2``/``T11``: ``type `Person` has no property `id`; the system identity is `$p.@id` `` (a failing stored-query registry quarantines its graph) | Use `$p.@id`, `$e.@src`/`@dst`, `where @id =`; lint every `.gq` before restarting on v0.11 or later | -| Long-lived feature branches | merge conflicts, schema apply blocked | Merge promptly; delete when done | -| `mutation { ... }` wrapper in `.gq` | `parse error: expected query_file` at line 1 | Use `query (...) { insert T { ... } }`; there is no top-level `mutation` keyword | -| `--config` on a data/schema command | `unexpected argument '--config' found` | Use it for `cluster` and `use`; use `--server`/`--graph`, `--store`, or `--profile` elsewhere | -| `--as` on a served write | `` `--as` is not allowed on a served write `` | Drop it; the bearer token selects the actor | -| Reading a large schema via stdout-capped tool | Truncated, garbled, or duplicated output | `omnigraph schema show --server --graph > /tmp/schema.pg`, then read the file in chunks | -| `omnigraph load` without `--mode` | `the following required arguments were not provided: --mode ` | Pass `--mode merge\|append\|overwrite` — there is no default (overwrite is destructive, so it is never implicit). Address direct storage or a served graph | -| Blind retry after 504 | duplicate unkeyed nodes/edges or a repeated effect | compare the intended branch/entity state first; retry only after proving the attempt did not land | -| Review branch left after a timed-out `load --from` | uncertain load result | Inspect its content and history; matching `main`'s head alone is not proof of abandonment or authorization to delete it | -| `omnigraph schema apply` / `init` on a cluster-managed graph | refused — bypasses the cluster ledger | Evolve cluster graphs via `omnigraph cluster apply --config .`; `schema apply`/`init` are for a non-cluster store | -| Assuming Blob-bearing data cannot compact | unnecessary skipped maintenance | Lance 11 Blob compaction is supported; `optimize` preserves null/empty/non-empty values | -| `@unique`/`@index` on a Blob column | schema parse/validation rejection | Blob properties cannot be keys, unique, or indexed | -| Full-text search on indexes built before Lance 11 (v0.9 era), including after `omnigraph upgrade` | `FullTextIndexRebuildRequired` | Stop mixed-version access and run `rebuild-full-text-indexes --branch ` on every live branch that needs text search; `upgrade` does not rebuild them | -| Reopening a v0.10/v6 graph with v0.12 | `__manifest is stamped at internal schema v6, but this omnigraph reads only v14 to v14` | Stop the old fleet; take a standalone graph to v9 with the 0.11 `upgrade`, then to 14 with the 0.12 `upgrade`, or rebuild by export, `init` and `load --mode overwrite` | -| Default `omnigraph upgrade` of the 0.11 binary on a branched graph or one with `_` properties | `the route ends at v9, which requires a graph with only main` (or `reserves property names starting with '_'`) | Delete branches / `@rename_from` first, or pass `--to-format 8` to both check and execution; the 0.12 `upgrade` converts v8 as well as v9 | -| Opening a v8 or v9 graph with 0.12 | refused: 0.12 reads format 14 only and names `omnigraph upgrade` | Stop every process using the graph, back up the whole root, then run the 0.12 `upgrade --check` and `upgrade` on a standalone graph; a cluster-managed graph is exported with 0.11.x and rebuilt. See [upgrading past v0.11](references/migrations.md#upgrade-v011-to-storage-format-14) | -| Using `data.id` for identity on a new graph | `unknown input field 'id': move data.id to the top-level 'id' field` | Put identity in top-level JSONL `id`; `data` contains user properties | -| Re-running a `--mode merge` load with unkeyed edges | duplicate edges | Supply stable top-level edge `id`s, or declare `@key(@src, @dst)` when creating the edge type | -| Assuming every JSON result cell has a key | absent keys for null values | Query/export JSON omits null cells; change images retain explicit nulls | -| `--format arrow` or `defaults.output: arrow` | `has no text rendering` after the query runs | 0.12.0 has no Arrow output; use `json` or `jsonl` | -| `..` or a symlink in a `cluster.yaml` relative path | `config_path_escape` / `config_path_symlink` | Keep referenced files inside the config directory with plain relative paths | -| Managed folder plus `OMNIGRAPH_PROFILE` or an operator default server/store | `managed_target_ambiguous` | Pass an explicit `--server`/`--store`/`--profile`, or `--direct` | -| Policy commands with both a graph and a `cluster` bundle applied | `matches 2 policy bundles` | Known 0.12.0 limitation of `policy validate/test/explain --graph`; inspect each bundle file directly | - -## Deep Dives - -For anything beyond the basics, load the relevant reference file. Each is self-contained — load only what you need. - -| Reference | When to load | -|-----------|--------------| -| [`references/schema.md`](references/schema.md) | Editing `.pg` files, running `schema plan`/`apply`, renaming types, backfilling required fields | -| [`references/queries.md`](references/queries.md) | Writing or linting `.gq` files, search functions, aggregations, multi-hop patterns | -| [`references/data.md`](references/data.md) | Choosing between `mutate` and `load` (required `--mode`, `--from` to fork a review branch); branch review workflow; exact overwrite scope | -| [`references/blobs.md`](references/blobs.md) | Writing and reading managed/external Blob values, selectors/ranges, security and lifecycle boundaries | -| [`references/changes.md`](references/changes.md) | Exact read/write positions, conditional mutations, commit diffs, feed cursors, baselines, and retention gaps | -| [`references/remote-ops.md`](references/remote-ops.md) | Operating through `--server`: exact receipts, unknown 504 outcomes, conflict handling, and safe retry decisions | -| [`references/cluster.md`](references/cluster.md) | Cluster configuration, plan/apply, drift, and serving | -| [`references/search.md`](references/search.md) | Embeddings, `@embed`, vector/text ranking, scope-then-rank pattern | -| [`references/aliases.md`](references/aliases.md) | Defining aliases for agents, structured output, JSON args | -| [`references/stored-queries.md`](references/stored-queries.md) | Cluster stored-query registry: declaration, `queries validate/list --cluster`, served invocation, and `invoke_query` Cedar gating | -| [`references/server-policy.md`](references/server-policy.md) | Starting the HTTP server, routes, bearer auth, Cedar policy gating, multi-graph mode | -| [`references/commands.md`](references/commands.md) | Current command shapes, addressing, output, and maintenance | -| [`references/migrations.md`](references/migrations.md) | Upgrading v0.11 graphs to storage format 14; v0.11 storage/system-column upgrades and wire changes; older v0.9→v0.10 and pre-0.7 boundaries | +- Declare typed query parameters and pass JSON values; do not interpolate values + into query source. Every `.gq` block is `query name(...) { … }`, including + mutations. `mutate` selects a write; there is no `mutation` wrapper. +- GQ identity is `$node.@id`, `$edge.@src` and `$edge.@dst`. Bare `id`, `src` and + `dst` refer to user properties. Physical column spelling is not query syntax. +- One mutation contains inserts/updates or deletes, never both. Required fields + must be supplied. `@embed` records a vector's source/model; mutation and load + do not generate vectors. Use the offline `embed` pipeline when needed. +- `load` requires `--mode merge|append|overwrite`. Merge upserts by entity ID; + append refuses collisions; overwrite replaces only types present in the batch. + Unkeyed rows need stable top-level JSONL `id`s for repeatable loads. +- Review bulk data changes with `load --from main --branch review`, then inspect + and merge. Schema changes use the declared main schema, with supported + `@rename_from` migrations preserving identity. Nullable additions cannot later + be tightened to required in place. +- Give reads an explicit order when stable subsets matter. `nearest` and `rrf` + need `limit N`; variable-hop traversal needs finite bounds. + +Successful effectful mutation/load responses carry the exact published commit. +A no-op has no commit; branch operations have distinct outcomes. Retain the +receipt instead of fetching a later head. A timeout, disconnect or proxy error +can hide a committed write: do not replay automatically. For deployment work, +resume observation by the original deployment ID. For conditional data writes, +use the commit captured with the read and re-read after a precondition failure. + +## Read the relevant reference + +| Reference | Use for | +|---|---| +| [Schema](references/schema.md) | Types, decorators, constraints, renames, evolution limits | +| [Queries](references/queries.md) | Grammar, traversal, filters, aggregation, result values | +| [Data](references/data.md) | JSONL envelopes, load modes, branches, review and merge | +| [Blobs](references/blobs.md) | Managed/external values, selectors, ranges and retention | +| [Changes](references/changes.md) | Commit diffs, feed cursors, baselines and retention gaps | +| [Remote operations](references/remote-ops.md) | Exact receipts, typed errors and retry decisions | +| [Cluster](references/cluster.md) | Configuration, live deployment, managed service and recovery | +| [Search](references/search.md) | Offline embeddings, providers, vector/text ranking | +| [Aliases](references/aliases.md) | Personal bindings to served read queries | +| [Stored queries](references/stored-queries.md) | Registries, invocation and policy gates | +| [Server and policy](references/server-policy.md) | HTTP contract, readiness, credentials and Cedar | +| [Commands](references/commands.md) | Command syntax, maintenance and output formats | +| [Upgrades](references/migrations.md) | Current storage/ledger migration choices and client cutover | diff --git a/skills/omnigraph/references/aliases.md b/skills/omnigraph/references/aliases.md index 2de14c8d4..bd9cecd23 100644 --- a/skills/omnigraph/references/aliases.md +++ b/skills/omnigraph/references/aliases.md @@ -1,155 +1,68 @@ -# Aliases & Agent Automation +# Read Aliases -## Contents -- What an alias is -- Operator alias schema -- Args binding & JSON-first parsing -- Default to structured output -- Alias naming convention -- Secrets don't belong in aliases -- Example alias set -- Invocation patterns - -How to wire Omnigraph operations for agents and scripts. - -## What an alias is - -An **operator alias** decouples a stable **operation name** from its implementation, so an agent calling `omnigraph alias signal …` keeps working as the query evolves. Aliases live in `~/.omnigraph/config.yaml` and are personal *bindings* to a **stored query on a named server** — they carry no query content; the stored query in the cluster catalog is the team's contract. +An operator alias binds a personal name and default parameters to a served +stored query. The team's query stays in the applied cluster registry; aliases +contain no `.gq` source and cannot invoke mutations. ```yaml # ~/.omnigraph/config.yaml +servers: + production: { url: https://graph.example.com } aliases: - triage: - server: intel-dev # an entry under servers: - graph: spike # optional (multi-graph servers) - query: weekly_triage # the STORED query's name — never a file - args: [since] # positional args → params, in order - params: { limit: 20 } # fixed defaults; positionals/--params win - format: table -``` - -```bash -omnigraph alias triage 2026-06-01 -# → POST /graphs/spike/queries/weekly_triage with the keyed credential -``` - -> **Alias vs stored query.** The alias is *yours* (a personal name + defaults); the **stored query** it points at is the *team's* — declared in `cluster.yaml`, type-checked and served by the cluster (`GET /graphs//queries`, `POST /graphs//queries/`, gated by `invoke_query`). See [`stored-queries.md`](stored-queries.md). -## Operator Alias Schema - -```yaml -aliases: - : - server: # an entry under servers: in ~/.omnigraph/config.yaml - graph: # optional: for multi-graph servers - query: # the stored query's NAME (never a file path) - args: [, ] # positional CLI args → named params, in order - params: { : } # fixed default params; positionals / --params win - format: table|kv|csv|jsonl|json # optional: output format + person: + server: production + graph: knowledge + query: get_person + args: [slug] + format: json + recent-people: + server: production + graph: knowledge + query: recent_people + params: { since: "2026-10-01T00:00:00Z" } + format: json ``` -Dispatch with `omnigraph alias [args]`. Operator aliases are read-only: -the alias asserts a read, so the server refuses a bound stored mutation with -`400`. Invoke stored mutations with -`omnigraph mutate --server --graph `. Aliases live in their -own namespace, so one can never shadow or be shadowed by a built-in verb. - -### `args` bind to query parameters - -If `args: [slug, name, age]`, then: - ```bash -omnigraph alias foo sig-bar "Some Name" 29 -``` - -...maps to `{"slug":"sig-bar","name":"Some Name","age":29}`. - -### Args are JSON-first - -Each arg is parsed as JSON first, then falls back to string: -- `29` → integer -- `'"29"'` → string (shell-quote the JSON quotes; a bare `"29"` reaches the CLI as `29`) -- `true` → boolean -- `Alice` → string (JSON parse fails, falls back) -- `{"x":1}` → object - -Explicit `--params '{...}'` wins on key conflict. - -## Default to Structured Output - -For scripts and agents, prefer `jsonl` or `json`; `table` is for humans. Set a default in `~/.omnigraph/config.yaml`: - -```yaml -defaults: - output: jsonl +omnigraph alias person ada +omnigraph alias person --params '{"slug":"grace"}' +omnigraph alias recent-people --format jsonl ``` -Or per-alias (`format: jsonl`), or per-call (`--format jsonl`). +The server, graph and query name come from the binding. To choose a different +scope or branch, call `query` directly. The `alias` command accepts +`--params`/`--params-file` and `--format`/`--json`; it has its own namespace and +cannot shadow built-in commands. Credentials belong in `omnigraph login` +storage, never alias configuration. -### When to use which +## Parameters -- **`jsonl`** — line-oriented NDJSON, with metadata first; ideal for agents -- **`json`** — pretty-printed `ReadOutput` envelope (metadata, columns, and rows) -- **`kv`** — `key: value` per line; good for single-row lookups -- **`csv`** — for spreadsheets or line-count-heavy analysis -- **`table`** — default human view; don't use in automation +`args` maps positional values to parameter names in order. Each argument is +parsed as JSON first, then as a String: -`--format arrow` is accepted by the parser in 0.12.0 but always fails with "has -no text rendering"; never set it as an alias or `defaults.output` format. +| Shell argument | Bound value | +|---|---| +| `29` | Integer | +| `'"29"'` | String `29` | +| `true` | Boolean | +| `Ada` | String | +| `'{"x":1}'` | Object | -Query row spelling follows [`queries.md`](queries.md#system-fields-and-result-values): -null fields are omitted, bare node projections are objects, and DateTime -strings are UTC without a trailing `Z`. Use `json` when the consumer needs -the complete column list or the snapshot's `graph_commit_id` for a later -conditional mutation; the JSONL metadata line does not carry those fields. +Explicit `--params` wins over positional arguments, which win over configured +`params`. Parameter types must match the stored query. -## Alias Naming Convention +## Results and policy -Short, hyphenated, matches the conceptual operation: +Use JSON when a consumer needs columns or the read's `graph_commit_id`; JSONL +omits those metadata fields. Null property values are omitted from rows and +DateTime output is UTC without a trailing `Z`. See [result values](queries.md#system-fields-and-result-values). -- `signal`, `pattern`, `element` — single lookup (typical pair with `format: kv`) -- `signals`, `patterns`, `elements` — list -- `signal-patterns`, `pattern-signals` — traversals - -## Secrets Don't Belong in Aliases - -Credentials never live in an alias or any config file. For remote servers, `omnigraph login ` stores the bearer token in `~/.omnigraph/credentials` (`0600`); for S3-backed storage, AWS creds go in `.env.omni`. Aliases should only contain query names and parameter bindings — never tokens, passwords, or API keys. - -## Example Alias Set - -```yaml -# ~/.omnigraph/config.yaml -servers: - intel-dev: { url: https://graph.example.com } -aliases: - # Lookups (kv format for single-row readability) - signal: { server: intel-dev, graph: spike, query: get_signal, args: [slug], format: kv } - pattern: { server: intel-dev, graph: spike, query: get_pattern, args: [slug], format: kv } - # Lists - signals: { server: intel-dev, graph: spike, query: recent_signals } - # Traversals - pattern-signals: { server: intel-dev, graph: spike, query: pattern_signals, args: [slug] } -``` - -Each `query:` names a stored query the cluster serves — declare them in `cluster.yaml` and `cluster apply` first (see [`stored-queries.md`](stored-queries.md)). - -## Invocation Patterns +Alias invocation requires the stored-query `invoke_query` and `read` grants. +A stored mutation bound as an alias is refused. Invoke it explicitly instead: ```bash -# Invoke a read alias -omnigraph alias signal sig-kimi-k25 - -# Invoke a served stored mutation directly (aliases are read-only) -omnigraph mutate add_signal --server intel-dev --graph spike \ - --params '{"slug":"sig-new","name":"Name","brief":"Brief","stagingTimestamp":"2026-04-14T00:00:00Z","createdAt":"2026-04-14T00:00:00Z","updatedAt":"2026-04-14T00:00:00Z"}' - -# Override output format -omnigraph alias signals --format jsonl - -# Explicit --params (wins over positional args on key conflict) -omnigraph alias signal --params '{"slug":"sig-override"}' +omnigraph mutate update_person --server production --graph knowledge \ + --params '{"slug":"ada","name":"Ada"}' --json ``` -The `alias` subcommand carries `--params`/`--params-file` and -`--format`/`--json`; the server, graph, and stored-query name come from the -binding. For a different server/graph or a branch read, call `query`/`mutate` -directly. +See [stored queries](stored-queries.md) for registry declaration and live updates. diff --git a/skills/omnigraph/references/changes.md b/skills/omnigraph/references/changes.md index f044d2991..cdc2facdb 100644 --- a/skills/omnigraph/references/changes.md +++ b/skills/omnigraph/references/changes.md @@ -30,7 +30,7 @@ published, even when another writer has since advanced the target; an `already_up_to_date` merge publishes nothing and returns `commit: null`. `branch list` goes to canonical `POST /query` and returns sorted `name` rows; branch statements -are refused on deprecated query/mutation routes and the conditional-write route. +are refused on the conditional-write route. ## Inspect one commit @@ -100,5 +100,5 @@ On POSIX, CLI `--out` syncs and atomically replaces the snapshot file before it prints the handshake to JSON stdout. Baselines require Cedar `export`; commit changes and feed polling require `read`. -Canonical contracts: [change feeds](https://github.com/ModernRelay/omnigraph/blob/v0.12.0/docs/user/branching/changes.md) -and [conditional mutations](https://github.com/ModernRelay/omnigraph/blob/v0.12.0/docs/user/mutations/index.md#conditional-mutations). +Canonical contracts: [change feeds](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/branching/changes.md) +and [conditional mutations](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/mutations/index.md#conditional-mutations). diff --git a/skills/omnigraph/references/cluster.md b/skills/omnigraph/references/cluster.md index 383f5d3b1..4c7c00e87 100644 --- a/skills/omnigraph/references/cluster.md +++ b/skills/omnigraph/references/cluster.md @@ -1,291 +1,184 @@ -# Cluster Mode — Declarative Deployments +# Cluster Deployments -## Contents -- The model -- The loop (validate → plan → apply → unlock → serve) -- The config contract (`cluster.yaml` vs `~/.omnigraph/config.yaml`) -- Serving (`--cluster`, config-free bucket boot, readiness, shutdown) -- Managed clusters -- Recovery cheat-sheet +A `cluster.yaml` and its referenced schemas, queries and policies declare the +whole deployment. Apply creates and manages its graphs. The server serves that +applied state; there is no standalone-graph server mode. -The cluster control plane manages a whole deployment — -graphs, schemas, stored queries, Cedar policies — as **declared files in one -directory**, converged Terraform-style. It is the **only way to serve** a -graph (the server is cluster-only); the data-plane operations in the other -references work against the cluster's graphs unchanged. - -## The model - -``` -company-brain/ -├── cluster.yaml # the deployment: graphs, schemas, queries, policies -├── schema.pg -├── queries/*.gq -├── *.policy.yaml -├── graphs/.omni # DERIVED — created by apply, never by hand (gitignore) -└── __cluster/ # ledger + lock + catalog — local state (gitignore) -``` +## Configuration ```yaml -# cluster.yaml version: 1 -# storage: s3://my-bucket/clusters/company-brain # optional object-store root; -# preview Azure roots use az://container/prefix (default: this folder) +# storage: s3://my-bucket/clusters/company-brain # default: config directory state: { backend: cluster, lock: true } graphs: knowledge: schema: schema.pg - queries: queries/ # the .gq files ARE the declaration — every `query ` registers - external_blobs: # omitted means deny new external references + queries: queries/ + external_blobs: allow: - { base: s3://company-assets/knowledge/, scope: server_safe } ``` -`queries` also accepts a file list (`[a.gq, b.gq]`) or a fine-grained -`name: { file: ... }` map. Discovery is loud: unparseable files and duplicate -names across files fail validation. +`queries` accepts a directory, file list or `name: { file: ... }` map. Duplicate +names and unparseable files fail validation. Relative schema/query/policy paths +must stay inside the config directory: `..` and symlink components are refused. +Keep `cluster.yaml` and referenced source files in Git; ignore generated +`__cluster/` state and `graphs/` roots. Credentials belong outside the declaration. -Relative `schema`, `queries`, and policy `file` paths must stay inside the -config directory: a `..` segment fails with `config_path_escape`, and a -symbolic link anywhere on the path (including a discovered `.gq` file) fails -with `config_path_symlink`. Absolute paths are not checked; prefer relative -paths. +The operator's `~/.omnigraph/config.yaml` separately owns named servers, +profiles, actor defaults and aliases. A server never reads that operator file. +`--config` selects source files; `--cluster` selects applied cluster storage. -## The loop (memorize this) +## Update a running server ```bash -omnigraph cluster validate --config . # parse + typecheck everything -omnigraph cluster plan --config . # preview — REQUIRED reading before apply -omnigraph cluster apply --config . --as # converge; the first run creates ledger + graphs -omnigraph cluster force-unlock --config . # clear the lock apply printed, once its work has settled -omnigraph-server --cluster . --bind 127.0.0.1:8080 --unauthenticated # serve (local dev) +omnigraph cluster validate --config . +omnigraph cluster plan --server production --config . --json +omnigraph cluster apply --server production --config . --timeout 1800 --json ``` -- **`apply` creates graphs** at `graphs/.omni` — there is no separate - `omnigraph init` in cluster mode. -- **Schema changes**: edit the `.pg`, `plan` shows the engine's real migration - steps (`add_property`, `drop_property`, `unsupported: …`), `apply` - migrates the live graph. **Drops reclaim nothing at apply** — a drop removes - the data from the branch head; older commits still read it until - `omnigraph cleanup` stops retaining them, and only then is it gone for good. -- **Two ways to apply.** `cluster apply --server --config .` submits - the bundle to the running server, which activates schema and stored-query - changes and new graphs without a restart. It needs a bearer token whose - actor the applied policy grants `config_manage` at cluster scope, `read` on - disclosed graphs, and `schema_apply` on each graph whose schema changes. A - direct `cluster apply --config .` writes storage itself: stop the server - first and start it afterwards. -- **Every writer leaves the cluster lock behind.** A direct apply, a direct - `--store` write to a cluster graph, and a server (even after a clean exit) - keep `__cluster/lock.json`, and the next writer or server start refuses with - `state_lock_held`. Once the previous owner has stopped and its I/O has - settled, clear that exact id with `cluster force-unlock --config .`. - While the server owns the lock, preview with `cluster plan --server `. -- **`storage: s3://bucket/prefix`** (optional) puts the entire cluster — state - ledger, lock, content-addressed catalog, and the derived graph roots - (`/graphs/.omni`) — on - S3-compatible object storage. The ledger CAS uses S3 conditional writes and - the lock becomes genuinely cross-machine. Absent, a direct apply keeps - everything in the config directory, and `cluster apply --server` uses the - server's own root. Credentials - come from the standard `AWS_*` env contract, never `cluster.yaml`. -- **`storage: az://container/prefix` is preview-only.** Every writer, server, - apply job, and maintenance process for an Azure root must run through - `omnigraph-azure-admission`; the preview lease is external admission, not an - engine-level distributed-writer fence. -- **One mutation-capable process per cluster is the supported topology.** The - server, direct applies and direct CLI writes all take the same cluster lock, - so they exclude each other. The lock does not fence older binaries, raw - storage tools or embedded writers; keep those stopped. -- **External Blob ingress is default-deny.** `graphs..external_blobs.allow` - lists normalized URI bases. `scope: server_safe` can be installed by the - server; `embedded_only` is never installed by the server or direct-store CLI. - Cedar chooses who may write; this list chooses which source objects a writer - may cause the process to inspect. A base must lie outside the cluster storage - root (`storage`, or the config directory), which holds every graph and the - ledger: `cluster validate`, `plan` and `apply` refuse an overlapping base in - either scope with `external_blob_base_overlaps_storage_root`, and a server - quarantines a graph whose applied `server_safe` base overlaps. Use a sibling - prefix, e.g. `s3://company-assets/cluster-external/` beside - `storage: s3://company-assets/cluster`. A same-kind base against a storage - root spelled with an empty or percent-sign path component is refused with - `external_blob_storage_root_uncomparable`; moving the base does not clear it, - so re-spell the root or remove those bases. -- **`--as ` labels a direct `cluster apply` or `cluster upgrade-ledger`** - (deployment record and engine commits where applicable). It defaults from - operator config's `operator.actor`. `cluster apply --server` refuses it, - because the server takes the actor from the bearer token; the other cluster - subcommands reject this flag. -- **Deployment scope**: apply creates graphs and updates schemas, queries, - policies, provider definitions/bindings, and `external_blobs` rules. Removing - a graph declaration deletes its exact managed root, including retained - history, after affected admission closes and work drains. Current permissions - authorize changes; proposed permissions cannot authorize themselves. Shared - external Blob objects are not deleted, and provider retention may keep object - versions. Root changes, adoption and missing-root recreation refuse. -- **Drift**: apply refuses an out-of-band schema change with - `applied_schema_drift`; it never adopts it by matching schema text. To inspect - without effects, `cluster observe --config .` reports current ledger, catalog - and graph observations; `cluster plan --config .` previews desired changes. - Both are read-only, report `authority: "observed"` and the `state_cas` they - read, and take no writer lock. Apply revalidates current authority. -- **Data is NOT cluster's job**: rows flow through `omnigraph load / mutate` - against the served graph (`--server … --graph `), with branches as usual. - -For a running server, preview and submit the same local bundle: +Review the preview before applying, especially graph removals. Apply supports: + +- schema and stored-query updates; +- graph and cluster policy grants, revocations and binding changes; +- embedding provider definitions/bindings and external-Blob ingress rules; +- graph creation and deletion. + +Affected graphs close admission, drain their admitted work and activate the new +configuration without restarting the process. Unrelated healthy graphs keep +serving. Across graphs, deployment is not one transaction. Schema changes still +require only `main` to be live; merge and delete feature branches first. + +Removing a graph declaration authorizes permanent deletion of its exact managed +root and all retained history. Shared external Blob source objects are not +deleted; object-store retention can keep historical object versions. A schema +property/type drop is different: older graph commits retain its data until +explicit cleanup stops retaining them. + +The current applied policy must grant `config_manage` at cluster scope, `read` +on disclosed graphs and `schema_apply` on existing graphs whose schema changes +or whose declaration is removed. +Proposed permissions cannot authorize their own installation. Served apply +refuses `--as`; the bearer token supplies the actor. + +Root changes, adoption of existing roots, recreation of missing managed roots +and acceptance of out-of-band schema drift are unsupported. A refusal such as +`applied_schema_drift` needs investigation, not a force or correction flag. +Rows are managed with `mutate`/`load`, separately from configuration apply. + +### Observe the original deployment + +Apply prints its deployment ID before submission and normally polls it until +convergence and activation. `--no-wait` returns after durable acceptance. +`--timeout` bounds caller waiting, including acceptance (default 300 seconds, +range 1–3600); expiry exits 5 without cancelling server work. ```bash -omnigraph cluster plan --server production --config . --json -omnigraph cluster apply --server production --config . --timeout 1800 --json -omnigraph cluster status --server production --deployment-id ID --wait --timeout 1800 --json +omnigraph cluster apply --server production --config . --no-wait --json +omnigraph cluster status --server production --deployment-id ID --wait \ + --timeout 1800 --json ``` -Apply prints its deployment ID before submission and normally polls that ID -until convergence and activation. `--no-wait` returns after durable acceptance; -`--timeout` bounds caller waiting, including acceptance (default 300 seconds, -range 1–3600). Expiry exits 5 without cancelling work. Resume with the exact -`status --deployment-id ID --wait`; these wait flags require `--server`. -A lost submission response triggers original-ID reads, never automatic replay. -The exact response contains `deployment`, `active` and `in_progress`; `active` -means the result's affected bindings are installed in this process. An -unrelated blocked graph does not invalidate that activation. The authenticated -submitter can still read its own retained receipt after losing management -permission; aggregate status and new deployments require current permission. -Deployment across graphs is not one transaction. - -## The config contract (do not blur this) - -| File | Owns | Read by | -|---|---|---| -| `cluster.yaml` | the deployment: graph set, schemas, stored queries, policy bindings, storage | `cluster` commands; the `--cluster` server | -| `~/.omnigraph/config.yaml` | per-operator: identity (`operator.actor`), named `servers:`, output defaults, personal aliases | data-plane CLI commands (tokens live in `~/.omnigraph/credentials` via `omnigraph login`) | - -Direct cluster commands use the operator actor default when `--as` is omitted -(`--as` > `operator.actor`). `cluster` selects self-hosted deployment unless -`--managed` explicitly selects the service API described below. Folder context -never selects the mode. -A `--cluster` server never reads the operator file: its configuration comes from -applied cluster state. -Address a cluster-managed graph's data via `--server`/aliases against the -serving instance. A direct `--store /graphs/.omni` read takes no -lock. On a ledger-v2 cluster a direct write takes the cluster lock: it is -refused while a server or another owner holds it, and the lock stays held after -the command until an exact-ID `cluster force-unlock`. - -## Serving - -`omnigraph-server --cluster ` is the exclusive boot source (there -is no separate `--config` merge) and is always multi-graph -(`/graphs/{id}/...`). By default, graph-attributed recovery, query-registry, or -provider failures quarantine only the affected graph and healthy graphs still -serve. Cluster-global or unattributable failures are fatal, as are any graph -failures with `--require-all-graphs`. Every healthy graph's applied query is -exposed (`GET /graphs//queries`, `POST -/graphs//queries/`); Cedar bundles attach via `applies_to` -(`cluster` → server-level gate incl. `graph_list`; a graph id → that -graph's gate incl. `invoke_query`). Bearer tokens and bind stay process-level -(env/flags). - -`GET /readyz` reports the booted applied digest, ledger revision/CAS and -`served_graph_count` (the whole registry) with `ready_graph_count`, -`loading_graph_count` and `blocked_graph_count`; it is HTTP 503 while any graph -is still loading, when a nonempty inventory has no ready graph, and when -draining. An applied empty cluster can serve a ready zero-graph inventory. A -nonempty cluster with no healthy graphs still refuses startup. `GET /graphs` -requires `graph_list` and returns one `graphs` list whose entries carry `state` -(`loading`, `ready`, `transitioning`, `blocked`, `stopping`), `read_available`, -`write_available`, `action` and, when blocked, a sanitized `failure`; readiness -itself exposes counts only. A request to a loading, blocked or transitioning -graph from a caller allowed to read `main` or list graphs answers `503 -graph_unavailable`, which never authorizes replaying a write. - -**Bounded shutdown.** `--shutdown-grace-seconds` (else -`OMNIGRAPH_SHUTDOWN_GRACE_SECONDS`, default 25; `0` cuts off immediately) -bounds the drain: at SIGTERM readiness turns off and in-flight requests drain; -a clean drain exits 0, reaching the deadline exits 2. Set the orchestrator's -termination grace longer than this value. - -**Config-free serving.** `--cluster` also accepts a `file://`, `s3://`, or -preview `az://` storage-root URI -directly — `omnigraph-server --cluster s3://bucket/prefix` boots from the -applied revision on the bucket with **no checkout of the config repo**. The -ledger and catalog on the bucket are the whole deployment artifact; policy -bundles serve as digest-verified content from the catalog. The preferred -container shape is **bucket, no volume** (see the container section of the -omnigraph repo's `docs/user/deployment.md`). The container entrypoint reads -`OMNIGRAPH_CLUSTER` (a storage URI or a mounted config directory) and passes -it as `--cluster`; the server binary itself reads only `--cluster`. The image -ships the CLI for in-container `cluster apply`. +A lost reply triggers reads of the original ID, never automatic resubmission. +Exact status returns `deployment`, `active` and `in_progress`. `active` describes +that result's affected bindings; an unrelated blocked graph does not invalidate +it. The authenticated submitter can read its own retained receipt after losing +management permission. Aggregate status and new deployments require current +permissions. See [remote outcomes](remote-ops.md) for data writes, whose recovery +is not deployment-ID polling. + +## Bootstrap or update while stopped + +Direct apply writes storage itself. Stop serving and overlapping writers first: + +```bash +omnigraph cluster validate --config . +omnigraph cluster plan --config . --json +omnigraph cluster apply --config . --as operator --json +# After the apply owner has stopped and its I/O has settled: +omnigraph cluster force-unlock --config . +omnigraph-server --cluster . --bind 127.0.0.1:8080 --unauthenticated +``` + +The final flag is for trusted local development. For authenticated serving, +configure tokens and policy as described in [server and policy](server-policy.md). +There is no separate `init` step for a cluster graph. + +One mutation-capable process per cluster is supported. Direct apply, direct +cluster writes and the server share exclusive writer admission. They leave the +lock after exit, including a clean exit; a successful shutdown does not itself +prove all native I/O has settled. Clear only the exact retained lock after that +proof. Older binaries, raw storage tools and embedded writers are not fenced +by this lock and must remain stopped. + +`cluster plan`, `observe`, `status` and direct graph reads take no writer lock. +Plan/observe report `authority: "observed"` and the `state_cas` read; apply +revalidates authority. Use served plan while the running server owns the root. + +## Storage and serving + +- `storage: s3://bucket/prefix` places state and derived graph roots together; + the writer uses standard `AWS_*` credentials and conditional storage writes. +- Azure `az://` roots remain a qualification preview. Every write, apply, + server and maintenance process must use `omnigraph-azure-admission`. +- New external Blob references are denied unless allowed by the graph's + `external_blobs` configuration. Servers install `server_safe` bases only; + `embedded_only` is not installed by the server or direct CLI. Allowed bases + must be outside cluster storage, never inside another managed graph root. +- `omnigraph-server --cluster ` boots applied state + without needing a checkout of the source bundle. The container entrypoint + accepts `OMNIGRAPH_CLUSTER`; the server binary takes `--cluster`. +- Use `/readyz` for rollout readiness and authorized `/graphs` for per-graph + availability. Process auth, bind settings and signed-token trust remain + startup configuration; applied graph configuration is live. See + [server and policy](server-policy.md). ## Managed clusters -An Intent API can own the control plane while the same CLI operates it: +Select the managed service explicitly with `--managed`: ```bash omnigraph login --api https://control.example omnigraph use CLUSTER_ID --api https://control.example --config . -omnigraph cluster plan --managed --config . --json -omnigraph cluster apply --managed --plan PLAN_RUN_ID --config . --json +omnigraph cluster push --managed --expected-revision REV --message "Update configuration" +omnigraph cluster plan --managed --rev NEW_REV --json +omnigraph cluster apply --managed --plan PLAN_RUN_ID --json omnigraph query find_person --graph knowledge --params '{"name":"Alice"}' --json ``` -`use` writes `.omnigraph/context` in the selected directory. API sessions and -data credentials are separate OS-keychain entries, never plaintext config. -Managed `query`, `mutate`, `load`, `commit list`/`show` and `graphs list` read -context only in the current directory (all but `graphs list` require -`--graph`) and use the cached data endpoint/credential, acquiring a missing or -expired identity credential first. They can operate during -a control-API outage until that credential expires. Other data commands keep -ordinary addressing. Explicit `--server`/`--profile`/`--store`/`--cluster` selects -ordinary routing; global `--direct` selects ordinary ambient defaults. Missing -or malformed managed authority refuses without fallback, and competing ambient -targets require an explicit choice. - -Managed creation, config upload, deletion and undo use `cluster create`, `push`, -`delete` and `undo-delete`, each with `--managed`. -`cluster status --managed [RUN_ID]` reads cluster projections or one run; -`cluster operation --managed ID [--wait]` observes a lifecycle operation. Durable -operation records bind uncertain submissions to their exact identity. Reconcile -the existing operation before issuing another. - -- **Schema/config changes**: commit, `cluster push --managed --expected-revision - --message …`, `cluster plan --managed --rev `, then `cluster apply --managed --plan `. - The Intent API executes the apply; there is no local `--as` or approval step. -- **Explicit mode**: `--managed` requires context, except creation and - explicit-origin operation lookup. Without it, `cluster` ignores context and needs no - `--direct` escape. Managed operations reject `--direct`, `--as`, `--server`, - `--profile`, `--graph`, `--store`, and `--cluster`; failures never fall back to - local deployment. Service-only verbs require `--managed`; `validate`, - `observe`, `force-unlock`, and `upgrade-ledger` reject it. -- **Sessions** from `login --api` hold an access credential of at most 15 - minutes that renews silently for up to eight hours after sign-in; then log - in again. For unattended runs, set `OMNIGRAPH_CONTROL_API` and - `OMNIGRAPH_CONTROL_TOKEN` together; reuse the same `--idempotency-key` after - an uncertain response. -- **Data credentials**: graph commands acquire an identity credential on - their own; `cluster token --managed [--ttl 1h]` issues one explicitly (TTL 60s–24h, - default 1h) and `--clear` forgets the local copy without revoking it. Only - identity credentials are supported: they carry no graph or action grants, - and the applied Cedar policy must permit the authenticated actor. -- **Exit codes** for plan/apply/lifecycle runs: 0 converged, 1 failed or - transport error, 2 refused or blocked, 3 partially converged, 4 recovery - required, 5 stalled or wait deadline, 6 cancelled. -See the authoritative [managed command reference](https://github.com/ModernRelay/omnigraph/blob/v0.12.0/docs/user/cli/reference.md#managed-cluster-commands), -[lifecycle](https://github.com/ModernRelay/omnigraph/blob/v0.12.0/docs/user/cli/managed-lifecycle.md) and -[data-access guide](https://github.com/ModernRelay/omnigraph/blob/v0.12.0/docs/user/cli/managed-data.md) for flags and limits. - -## Recovery cheat-sheet - -| Symptom | Fix | +`use` writes `.omnigraph/context`. API sessions and data credentials are separate +OS-keychain entries. Supported data commands (`query`, `mutate`, `load`, +`commit list`/`show`, `graphs list`) use that folder's endpoint and acquire/cache +an identity credential; all but `graphs list` need `--graph`. A cached credential +can work during a control-API outage until expiry. Other data commands retain +ordinary addressing. Explicit `--server`/`--profile`/`--store`/`--cluster`, or +`--direct` for ordinary ambient defaults, selects ordinary routing. + +Without `--managed`, cluster commands ignore folder context. Managed commands +reject ordinary target flags, `--direct` and `--as`; errors do not fall back to +local deployment. Lifecycle commands include `create`, `delete`, `undo-delete` +and `operation`; `status` observes a run, and `history`/`cancel` manage runs. +Use the same idempotency key to reconcile an uncertain managed submission. + +`cluster token --managed` issues an identity credential explicitly; it contains +no graph/action grants. Applied Cedar policy decides access. `--clear` forgets +the local credential without revoking it. For unattended control access, set +`OMNIGRAPH_CONTROL_API` and `OMNIGRAPH_CONTROL_TOKEN` together. + +See the [managed command reference](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/cli/reference.md#managed-cluster-commands), +[lifecycle](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/cli/managed-lifecycle.md) +and [data access](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/cli/managed-data.md) +for exact flags, session limits and exit codes. + +## Recovery + +| Situation | Action | |---|---| -| Served apply timed out or lost its response | continue original-ID observation with `cluster status --server --deployment-id --wait --json`; timeout does not cancel the owner. Never resubmit under a new ID | -| Direct apply interrupted, or accepted work needs stopped recovery | never retry under a new deployment id. Read the original deployment with `omnigraph --cluster cluster status --deployment-id --json` (apply prints the id and root). Stop the prior owner, prove its I/O has settled, clear the held lock with `omnigraph --cluster cluster force-unlock `, then reconcile that same id: `omnigraph --cluster cluster apply --deployment-id --writers-stopped` | -| Held lock (`state_lock_held`) | expected after any apply, direct write or server run. `cluster observe` shows state and the holder without refusing. First stop the owner and prove its I/O has settled; then use `cluster status` and clear that exact id with `cluster force-unlock --config .` | -| `config_path_escape` / `config_path_symlink` | move the referenced file inside the config directory and reference it by a plain relative path | -| Missing `state.json` | new cluster: `cluster apply` creates the ledger. Bootstrap never adopts graph roots that already exist (`graph_already_exists`); for those, restore a trusted cluster-state backup | -| Corrupt `state.json` | restore a trusted cluster-state backup or follow the diagnostic | -| Server refuses to boot | the error names its code and remedy; common ones: `state_lock_held` (see Held lock above), `ledger_upgrade_required` (stop serving and writers, then `cluster upgrade-ledger --writers-stopped`), `cluster_deployment_outstanding` (reconcile that exact id with `cluster apply --deployment-id --writers-stopped`) | -| `ledger_upgrade_required` (supported legacy ledger or obsolete completed-result fields) | stop every server, writer and maintenance job, then `omnigraph --cluster cluster upgrade-ledger --writers-stopped`; it converts the ledger only and keeps rows, branches and history | - -Full reference: the omnigraph repo's `docs/user/clusters/index.md` (operator guide) -and `docs/user/clusters/config.md` (every key, flag, and diagnostic). +| Served apply timed out or lost its response | Continue `cluster status --server … --deployment-id ID --wait`; do not submit a new ID. | +| Direct apply interrupted or accepted work needs stopped recovery | Read `omnigraph --cluster ROOT cluster status --deployment-id ID --json`. Stop the prior owner, prove its I/O settled, clear its exact lock, then use `omnigraph --cluster ROOT cluster apply --deployment-id ID --writers-stopped`. | +| `state_lock_held` | `cluster observe` shows the holder. Stop it and prove I/O settlement before `cluster force-unlock LOCK_ID`; a lock's age is not proof. | +| `ledger_upgrade_required` | With all writers stopped, run `omnigraph --cluster ROOT cluster upgrade-ledger --writers-stopped`. Conversion preserves graph data and history. | +| Missing/corrupt ledger or unexpected graph root | Restore a trusted consistent backup or follow the diagnostic. Bootstrap never adopts an existing root; do not recreate state by hand. | + +Canonical guide: [clusters](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/clusters/index.md) +and [configuration](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/clusters/config.md). diff --git a/skills/omnigraph/references/commands.md b/skills/omnigraph/references/commands.md index 9b622a2cd..023a7701c 100644 --- a/skills/omnigraph/references/commands.md +++ b/skills/omnigraph/references/commands.md @@ -1,423 +1,195 @@ -# Reference Commands +# Command Reference -## Contents -- Inspect state (snapshot, export) -- Branches · commits · changes · graphs -- Blob reads -- Schema · offline storage upgrade · lint · embed · init -- Load (bulk JSONL) -- Query / mutate -- Maintenance (optimize, full-text rebuild, cleanup) -- Stored queries -- Operator config & credentials -- Addressing a graph -- Output formats and positions · health check -- Cluster control plane (direct and managed) +Use `omnigraph --help` for the complete flags. Examples below use +`graph.omni` for a standalone store and `production` for a configured server. +Read the linked workflow before an effectful operation. -Commands you'll reach for but don't need best-practice rules around. Quick syntax reference. +## Addressing and credentials -## Inspect State +| Scope | Selector | +|---|---| +| Served graph | `--server --graph ` | +| Direct graph | `--store `, or a positional storage URI where supported | +| Applied cluster | `--cluster --graph ` for commands that support it | +| Operator profile | `--profile ` or `OMNIGRAPH_PROFILE` | +| Source bundle / managed context | `--config

` on `cluster` and `use` | -### `snapshot` — node/edge types and entity counts +`--store`, a positional storage URI and `--server` are mutually exclusive. +Query/mutate reserve their positional argument for the query name. An HTTP URL +is addressed with `--server`, never as a positional graph URI. Operator defaults +supply otherwise unspecified targets; explicit scopes override them. ```bash -omnigraph snapshot $REPO --branch main --json +echo "$TOKEN" | omnigraph login production +omnigraph logout production +omnigraph login --api https://control.example +omnigraph logout --api https://control.example ``` -Returns every node/edge type with entity counts and published dataset versions. -Use it to verify a load or inspect the graph shape. +`OMNIGRAPH_HOME` relocates the auto-discovered operator directory (normally +`~/.omnigraph`). Named server tokens are separate from `config.yaml`; managed +API sessions/data credentials use the OS keychain. Managed folder routing and +`--direct` are described in [cluster](cluster.md#managed-clusters). -### `export` — full JSONL dump +## Query and mutate ```bash -omnigraph export $REPO --branch main > graph.jsonl +omnigraph query get_person --server production --graph knowledge \ + --params '{"slug":"ada"}' --json +omnigraph query get_person --query queries/people.gq --store graph.omni \ + --params '{"slug":"ada"}' --json +omnigraph mutate update_person --server production --graph knowledge \ + --params '{"slug":"ada","name":"Ada"}' --if-commit COMMIT_ID --json +omnigraph query -e 'query q() { match { $p: Person } return { $p.@id } limit 5 }' \ + --store graph.omni --json ``` -Streams all nodes and edges as JSONL. The right tool for large-snapshot inspection. Don't try to page through the whole graph with read queries. +Omit `--query`/`-e` only for a served stored query. `--params-file` can replace +inline JSON. Reads select `--branch` or `--snapshot`; writes select `--branch`. +`--as` attributes direct writes; served writes refuse it. For grammar and +conditional-write semantics, see [queries](queries.md) and [changes](changes.md). -Filter by type: +## Inspect state ```bash -omnigraph export $REPO --branch main --type Signal > signals.jsonl +omnigraph snapshot --store graph.omni --branch main --json +omnigraph schema show --store graph.omni --json +omnigraph export --store graph.omni --branch main --type Person > people.jsonl +omnigraph graphs list --server production --json ``` -Use repeatable `--type` to filter node or edge types. +Export streams JSONL; repeat `--type` to include several types, or omit it for +all entities. It is not query-result JSON. `/graphs` includes availability and +requires `graph_list`; `graphs list --discovery` uses an identity credential and +returns only IDs/names. See [server and policy](server-policy.md). -Entity identity is top-level `id`; `data` holds user properties. Query/export -JSON omits null cells and renders dates as strings. See [data envelopes](data.md). - -## Blob Reads - -```bash -omnigraph blob stat node Document manual content --store "$REPO" --json -omnigraph blob get node Document manual content --store "$REPO" --out manual.bin -omnigraph blob get node Document manual content --store "$REPO" --offset 0 --length 4096 -``` - -The selector is ` `. Choose a branch or -snapshot; `get` streams managed bytes and `stat` inspects metadata. See -[`blobs.md`](blobs.md) before handling external references or ranged reads. - -## Branches - -```bash -omnigraph branch create --from main --store $REPO -omnigraph branch list --store $REPO -omnigraph branch merge --into main --delete-branch --store $REPO -omnigraph branch delete --store $REPO -``` - -All support `--json`. `--from` and `--into` default to `main`. -`--delete-branch` removes the source only after a successful merge publication. -A served `branch delete --json` needs `--yes`. - -The same operations are GQ branch statements, useful when a client already -sends `.gq` source: - -```bash -omnigraph mutate -e 'branch create "review/x" from main' --server --graph -omnigraph mutate -e 'branch merge "review/x" into main' --server --graph -omnigraph mutate -e 'branch delete "review/x"' --server --graph -omnigraph query -e 'branch list' --server --graph -``` - -Quote names containing `/`, `-`, `.`, or a leading uppercase letter or digit. -Statements take no name, `--params`, `--branch`, `--snapshot`, or -`--if-commit`. Their `outcome` and optional `commit` have different meanings -from data-mutation receipts; see [branch statements](changes.md#branch-statements). - -## Commits (History) - -```bash -omnigraph commit list --store "$REPO" --branch main -omnigraph commit show --store "$REPO" -omnigraph commit changes --store "$REPO" --json -``` - -`commit changes` compares one commit with its first parent and can filter with -repeatable `--kind`, `--type`, and `--op`. For a durable branch feed and -baseline recovery, see [`changes.md`](changes.md). - -## Graphs (multi-graph servers) +## Branches, commits and changes ```bash -omnigraph graphs list --server --json +omnigraph branch create review --from main --store graph.omni --json +omnigraph branch list --store graph.omni --json +omnigraph branch merge review --into main --delete-branch --store graph.omni --json +omnigraph branch delete review --store graph.omni --json +omnigraph commit list --store graph.omni --branch main --json +omnigraph commit show COMMIT_ID --store graph.omni --json +omnigraph commit changes COMMIT_ID --store graph.omni --json +omnigraph changes poll --start now --store graph.omni --json +omnigraph changes poll --cursor CURSOR --store graph.omni --json +omnigraph changes baseline --out snapshot.jsonl --store graph.omni --json ``` -Lists the graphs a multi-graph server serves, with each graph's `state`. Remote -servers only (rejects local URIs); a cluster-scoped policy bundle must grant -`graph_list`. `--discovery` (implied in a managed folder) returns only graph -ids and names to an identity credential and needs no `graph_list`. See -[`server-policy.md`](server-policy.md). +Merge receipts name the exact published commit. Feed cursors differ from page +tokens; save a baseline cursor only after consuming the complete snapshot. +See [data review](data.md) and [change feeds](changes.md). -## Schema +## Blob reads ```bash -omnigraph schema plan --schema next.pg $REPO --json -omnigraph schema apply --schema next.pg $REPO +omnigraph blob stat node Document manual content --store graph.omni --json +omnigraph blob get node Document manual content --store graph.omni --out manual.bin +omnigraph blob get node Document manual content --store graph.omni --offset 0 --length 4096 ``` -Both commands require direct storage and refuse `--server`. Cluster-managed -graphs evolve through `cluster apply`. See `references/schema.md` for the full -workflow. +The selector is ` `, with either `--branch` or +`--snapshot`. Read [Blobs](blobs.md) for ranges and external-reference handling. -### Offline storage upgrade - -```bash -omnigraph upgrade "$REPO" --check --json -omnigraph upgrade "$REPO" --json -omnigraph schema upgrade-system-columns "$REPO" --check --json -``` - -`omnigraph upgrade` converts a standalone v8 or v9 graph (written by 0.11) or -a v13 graph to storage format 14 in place, keeping branches, commit ids and -commit history; `--to-format` accepts 14 only. `schema upgrade-system-columns` -is a separate step on a graph already at 14: it respells the legacy -`id`/`src`/`dst` system columns and requires only `main` and no user property -starting with `_`. Both are standalone, offline operations: stop every process -using the graph and keep a verified whole-root backup; cluster-managed roots -refuse. Read -[upgrading from v0.11](migrations.md#upgrade-v011-to-storage-format-14) and -the other [migration preconditions](migrations.md) before executing. - -With the 0.11 binary the default target was v9, and `--to-format 8` kept -legacy system spellings and could preserve live branches. - -## Lint - -```bash -omnigraph lint --schema schema.pg --query queries/foo.gq --json -# or against a live repo: -omnigraph lint --query queries/foo.gq $REPO --json -``` - -`lint` is the single query-validation command. See `references/queries.md`. - -## Embed +## Schema, load and embeddings ```bash +omnigraph lint --schema schema.pg --query queries/people.gq --json +omnigraph schema plan --schema next.pg --store graph.omni --json +omnigraph schema apply --schema next.pg --store graph.omni +omnigraph init --schema schema.pg graph.omni +omnigraph load --data seed.jsonl --mode merge --store graph.omni --json +omnigraph load --data delta.jsonl --mode merge --from main --branch review \ + --server production --graph knowledge --json omnigraph embed --input raw.jsonl --output embedded.jsonl --spec embeddings.json -omnigraph embed --input raw.jsonl --output embedded.jsonl --spec embeddings.json --reembed-all -omnigraph embed --seed embed-config.yaml --clean -omnigraph embed --seed embed-config.yaml --select "Type:field=value" -``` - -See `references/search.md`. - -## Init - -```bash -omnigraph init --schema schema.pg $REPO ``` -Creates a new graph at `$REPO` with the given schema. Declare the deployment in a `cluster.yaml` (see `references/cluster.md`). - -**Strict by default:** `init` refuses any initialized graph. `--force` only -replaces orphan schema artifacts after proving there is no `__manifest`; it -never overwrites an initialized graph or purges Lance datasets. - -**Note:** `init` does not accept `--json`. Drop the flag if you see `unexpected argument --json`. - -## Load (bulk JSONL) - -```bash -# bare load: operates on an existing branch (default main); --mode is required -omnigraph load --data seed.jsonl --mode merge $REPO - -# --from forks a missing branch from , then loads onto it (one-shot review branch) -omnigraph load --data delta.jsonl --branch feature-x --from main --mode merge $REPO -``` - -`--mode` is **required** (no default): `merge`, `append`, or `overwrite`. -Address either direct storage or a served graph. See `references/data.md`. - -## Query / Mutate - -```bash -omnigraph query get_signal --query queries/signals.gq --params '{"slug":"sig-foo"}' # ad-hoc file; is positional -omnigraph query get_signal --server intel-dev --params '{"slug":"sig-foo"}' # served stored query by name -omnigraph mutate add_signal --query queries/mutations.gq --params '{"slug":"sig-foo","name":"Foo","brief":"Example","createdAt":"2026-04-14T00:00:00Z"}' -``` - -With a read alias: - -```bash -omnigraph alias signal sig-foo -``` - -Aliases are read-only. Invoke a served stored mutation with `omnigraph mutate - --server ...`. - -> `query` and `mutate` also accept inline source via `-e/--query-string ''` instead of `--query `. +`schema plan`/`apply` and `init` are for standalone stores; cluster-managed +schemas use cluster apply. `init` has no `--json`; `--force` only replaces orphan +schema artifacts and never overwrites an initialized graph. Load requires an +explicit `--mode`; embeddings are a separate file transformation. See +[schema](schema.md), [data](data.md) and [search](search.md). ## Maintenance -### `optimize` — compaction and index reconciliation - -```bash -omnigraph optimize $REPO --json -``` - -Compacts fragments and reconciles declared scalar/vector index coverage without -deleting retained versions. Lance 11 supports compaction of Blob-bearing data; -null, -valid-empty, and non-empty Blob values remain distinct. Existing full-text -indexes are preserved and any uncovered tail is scanned; use the explicit -rebuild command when full-text coverage or analyzer compatibility must change. -`optimize` also persists the derived traversal adjacency used to speed up -traversal startup; it does not collect unused forks (that is `cleanup`). +Run direct maintenance with overlapping writers stopped. For a cluster, use +`--cluster ROOT --graph ID`, stop the server and transfer the exact writer lock +as described in [cluster](cluster.md#recovery). -### `rebuild-full-text-indexes` — explicit analyzer upgrade +### Optimize ```bash -omnigraph rebuild-full-text-indexes "$REPO" --branch main --json +omnigraph optimize --store graph.omni --json ``` -Direct storage only; `--cluster --graph ` also works. Stop overlapping -writers, preserve a verified whole-store backup, and rebuild every live branch -that needs search after a Lance 11 upgrade. Do not mix old and new serving -binaries. Rebuilds use default English analysis, replacing custom tokenizer -settings; JSON `warnings` reports this for actual work. Check the selected -`branch`, `graph_commit_id`, and `rebuilt_indexes`; an empty list/null commit is -a no-op, not a migration of other branches or historical snapshots. `--as` is -actor attribution and does not install server policy on direct access. +Compacts fragments, including Blob-bearing data, and reconciles declared +scalar/vector indexes and derived traversal state. It preserves retained +versions. Existing full-text indexes remain usable; explicit rebuild owns +analyzer compatibility and full-text coverage changes. -### `cleanup` — destructive version GC +### Rebuild full-text indexes — explicit analyzer upgrade ```bash -omnigraph cleanup $REPO --keep 5 --older-than 7d --confirm +omnigraph rebuild-full-text-indexes --store graph.omni --branch main --json ``` -Deletes table versions that no retained graph commit pins, dropping -time-travel reachability for anything pruned. **Destructive** — requires -`--confirm`. At least one of `--keep` and `--older-than` is required; with -both, a graph commit must be outside both windows. Duration units: `s`, `m`, -`h`, `d`, `w`. +Preserve a verified backup and run on every live branch that needs incompatible +full-text indexes rebuilt. The default English analyzer replaces custom tokenizer +settings. Inspect `branch`, `graph_commit_id`, `rebuilt_indexes` and `warnings`; +a no-op does not migrate other branches or old snapshots. See [upgrades](migrations.md). -`--keep N` retains the newest `N` graph commits of every live branch and every -table version they pin; it counts graph commits, not Lance versions of a -dataset. Each live branch's head, selected merge bases and natively tagged -snapshots are always retained. Cleanup also reclaims table forks left by -earlier builds and retired branch refs once nothing needs them. Branch -deletion and `optimize` leave this reclamation to `cleanup`. - -## Stored Queries +### Cleanup ```bash -omnigraph queries validate --cluster . # type-check every applied registry -omnigraph queries list --cluster . --graph knowledge # list names and typed params +omnigraph cleanup --store graph.omni --keep 5 --older-than 7d --confirm ``` -`queries` operates on applied cluster state, not a graph URI. `validate` -checks every registry against its applied schema; `list` may select one graph. -Distinct from `lint`, which validates authoring source. See -[`stored-queries.md`](stored-queries.md). +Prunes history and table versions no retained commit needs. At least one of +`--keep`/`--older-than` is required; with both, a commit must be outside both +windows. Units are `s`, `m`, `h`, `d`, `w`. Retention always protects live heads, +required merge bases and tagged snapshots. Quiesce readers that depend on +history about to be removed. Without `--confirm`, cleanup previews its work. -## Operator Config & Credentials +## Applied registries and policy ```bash -echo "$TOKEN" | omnigraph login # store a bearer token in ~/.omnigraph/credentials (0600) -omnigraph logout # remove it (idempotent) -omnigraph login --api # managed Intent API session (OS keychain) -omnigraph logout --api # revoke that session and remove its keychain entry +omnigraph queries validate --cluster . +omnigraph queries list --cluster . --graph knowledge +omnigraph policy validate --cluster . --graph knowledge +omnigraph policy test --cluster . --graph knowledge --tests policy.tests.yaml ``` -The operator config and `~/.omnigraph/credentials` are **auto-discovered — there is no flag to point at them.** `$OMNIGRAPH_HOME` relocates the `~/.omnigraph` directory, and an absent file is an empty layer. `--config` selects source/context folders for `cluster` and `use`. - -## Addressing a Graph +These inspect applied state; validate desired files with `cluster validate`. +See [stored queries](stored-queries.md) and [policy](server-policy.md). -How the CLI resolves which graph a data command (`query`, `mutate`, `load`, `branch`, …) runs against. A remote is addressed with `--server` (a bare `http(s)://` URL is not a graph address). - -Precedence (highest first): - -1. **`--store `** or a **positional `file://`/`s3://`/`az://` URI** — direct storage access (bypasses any server; no catalog, so stored-query *names* don't resolve). `--store` is exclusive with a positional URI and with `--server`. Azure is a qualification preview and writes require `omnigraph-azure-admission`. -2. **`--server `** (+ `--graph ` for a multi-graph server) — served/remote. A name resolves from `servers:` in `~/.omnigraph/config.yaml`; a literal `http(s)://` URL also works. -3. **`--profile `** (or `$OMNIGRAPH_PROFILE`) — a named scope bundle from `profiles:` in the operator config (binds one of server/cluster/store + a default graph). -4. **Operator defaults** — `defaults.server` + `defaults.default_graph`, or `defaults.store` for a zero-flag local scope (mutually exclusive with `defaults.server`). - -**Managed folders.** In a directory with `.omnigraph/context` (written by -`omnigraph use`) and no explicit `--server`/`--profile`/`--store`/`--cluster`, -`query`, `mutate`, `load`, `commit list`/`show` and `graphs list` go to the -managed data endpoint with an identity credential the CLI acquires and caches -itself; all but `graphs list` require `--graph`. If `OMNIGRAPH_PROFILE` or an -operator default server/store competes, they refuse with -`managed_target_ambiguous`. Global `--direct` restores the ordinary resolution -above. Other data commands are unaffected. - -`cluster` uses `--config ` for source/context folders; policy -and stored-query control-plane commands use `--cluster `. Maintenance against a -cluster-managed graph uses `--cluster --graph `. -Each command declares a **capability** — `any` / `served` / `direct` / -`control` / `local` — shown in `omnigraph --help`; mis-addressing fails loudly. - -For query source (`query`/`mutate`): - -1. **`--query `** or **`-e/--query-string ''`** — at most one; omit both to invoke a served stored query or mutation by its positional ``. Either source may instead hold one branch statement. (Operator aliases are invoked via the separate `alias` subcommand.) -2. Relative `--query` paths resolve from the current working directory - -For params: - -1. **Explicit `--params '{...}'`** wins on key conflict -2. **Positional alias args** map to alias `args` list - -## Output Formats and Positions - -`--format ` on read queries and aliases: - -- `table` (default) — human-readable -- `kv` — `key: value` per line; good for single rows -- `csv` — comma-separated -- `jsonl` — NDJSON, one per line, with metadata line first -- `json` — pretty `ReadOutput` envelope (metadata, columns, and rows); the `rows` array is printed compact and verbatim - -`--format arrow` appears in `--help` but 0.12.0 has no Arrow output: it always -fails with "has no text rendering". Do not use it, or set it as -`defaults.output`. - -Mutations do not take `--format`; use `--json`. Successful `mutate --json` and -`load --json` include the exact published `commit` (`null` for a no-op -mutation). `query --json` includes `graph_commit_id` when the read snapshot has -an effective graph head (a fresh pre-commit graph can omit it). When returned, -use that position with `mutate --if-commit`; see [`changes.md`](changes.md). - -For admin commands that advertise it (branch, commit, schema): use `--json` for -structured output, otherwise human text. Policy subcommands do not offer JSON. - -## Health Check +## Cluster commands ```bash -curl http://127.0.0.1:8080/healthz +omnigraph cluster validate --config . +omnigraph cluster plan --server production --config . --json +omnigraph cluster apply --server production --config . --no-wait --json +omnigraph cluster status --server production --deployment-id ID --wait --json +omnigraph cluster observe --config . --json ``` -`/healthz` is process health. Use `GET /readyz` for rollout readiness: it reports -its `status`, the booted applied digest, ledger revision/CAS, the registered, -ready, loading and blocked graph counts and shutdown grace. It returns `503` -while a graph is still loading, when a nonempty inventory has no ready graph, -and once draining starts. Readiness is not proof that every configured graph -is healthy; authorized `graphs list --server --json` returns one -`graphs` list with each graph's `state` and `action`, blocked graphs included. +Direct bootstrap, stopped recovery, ledger conversion and managed service +commands are owned by [cluster](cluster.md). Use its procedures rather than +substituting direct storage access for served apply. -## Cluster Control Plane +## Output -```bash -omnigraph cluster validate --config # parse + typecheck the declaration -omnigraph cluster plan --config [--json] # preview (schema changes show migration steps) -omnigraph cluster apply --config --as # direct (serving stopped): bootstrap or update -omnigraph cluster plan --server --config --json # read-only served preview -omnigraph cluster apply --server --config [--no-wait] [--timeout 1800] --json -omnigraph cluster status --server --deployment-id --wait [--timeout 1800] --json -omnigraph cluster status --config [--json] # read the ledger (read-only) -omnigraph cluster observe --config [--json] # current observations: no lock, no write -omnigraph cluster force-unlock --config # remove an exact lock once its owner is quiescent -omnigraph --cluster cluster upgrade-ledger --writers-stopped # convert a supported stopped legacy ledger -``` - -`plan` and `observe` are read-only and report `authority: "observed"` with the -ledger `state_cas` they read. A plan is an observation; apply revalidates it. -Served apply normally polls its original ID until convergence and activation. -`--no-wait` returns after durable acceptance; `--timeout` bounds caller waiting -(default 300 seconds, range 1–3600), including acceptance, and never cancels work. -A wait timeout exits 5. Exact served status returns `deployment`, `active`, and -`in_progress`; use `--wait` with `--deployment-id` to resume observation. These -wait flags require `--server`. Lost responses trigger original-ID reads, never -automatic resubmission. See [deployment receipts](cluster.md#the-loop-memorize-this). - -The 0.11 verbs `import`, `refresh` and `approve` are removed. A fresh direct -`apply` creates the ledger, and a ledger written by 0.11 is converted once with -`upgrade-ledger` while serving and writers are stopped. Direct `apply` keeps -its admission lock after it completes: once the owner and its I/O have settled, -release that exact lock id with `force-unlock` before starting the server. - -`cluster` commands select the managed service only with `--managed`; otherwise -they ignore managed context (see [`cluster.md`](cluster.md#managed-clusters)): +Reads accept `--json` or `--format table|kv|csv|jsonl|json`. JSON contains +metadata, columns and rows; JSONL has a metadata line followed by rows, but does +not carry the full column list or read commit position. Use JSON for conditional +write workflows. The parser lists `arrow`, but it has no text renderer; do not +set it as an output default. -```bash -omnigraph use --api [--config ] -omnigraph cluster create --managed --api -omnigraph cluster push --managed --expected-revision --message -omnigraph cluster plan --managed [--rev ] -omnigraph cluster apply --managed --plan -omnigraph cluster status --managed [RUN_ID] -omnigraph cluster operation --managed [--api ] [--wait] [--timeout 1800] -omnigraph cluster history --managed [--limit N] [--since ] -omnigraph cluster cancel --managed -omnigraph cluster token --managed ([--ttl 1h] | --clear) # identity only; applied Cedar policy decides access -omnigraph cluster delete --managed --incarnation [--retention-seconds 86400] -omnigraph cluster undo-delete --managed --incarnation --deletion-id -``` +Mutation/load and other commands advertising it use `--json`, not `--format`. +Store the exact receipt with downstream state. For large schemas or exports, +redirect to a file and check command success before consuming it. -Managed plan, apply, create, delete, and undo-delete take `--no-wait`, -`--timeout <1..3600>`, and `--idempotency-key` (reuse the same key after an uncertain response). Managed -plan/apply/lifecycle exit codes: 0 converged, 1 failed or transport error, 2 -refused or blocked, 3 partially converged, 4 recovery required, 5 stalled or -wait deadline, 6 cancelled. - -Topology rule: `omnigraph schema apply` and `omnigraph init` **refuse a -cluster-managed graph** — in a cluster their jobs belong to `cluster apply`. -Data commands (`load`, `mutate`, branches) normally go through the running -server (`--server --graph `). A direct write or maintenance run at -the derived root (`/graphs/.omni`, or `/graphs/.omni` for -an S3-backed cluster) takes the cluster's exclusive writer admission and keeps -the lock after it succeeds: stop the server first, then release that exact -lock id with `cluster force-unlock` before another owner starts. See -`references/cluster.md`. +For non-local destructive commands, noninteractive execution needs explicit +`--yes` where advertised (`cleanup` additionally uses `--confirm`). Use `/readyz` +for rollout readiness and `/healthz` for process health. diff --git a/skills/omnigraph/references/data.md b/skills/omnigraph/references/data.md index 8935a21b0..74423f883 100644 --- a/skills/omnigraph/references/data.md +++ b/skills/omnigraph/references/data.md @@ -5,8 +5,6 @@ - `mutate` — single edits - `load` — bulk JSONL (`--mode`, `--from`) - Branches: review before merge -- Destructive ops go through a branch -- Branch commands - Inspecting state after changes How to modify data safely in Omnigraph. @@ -48,7 +46,7 @@ edits. > (writing storage directly) *and* an `omnigraph-server` endpoint (the > server orchestrates the write and publishes one atomic commit). See > [`references/remote-ops.md`](remote-ops.md) for remote-specific concerns -> (504 handling, write-verification ritual). +> (lost responses and safe retry decisions). ## `mutate` — Single Edits @@ -146,12 +144,6 @@ branch operation had no effect. See [branch outcome details](changes.md#branch-s vectors in the JSONL, or run the offline `omnigraph embed` file transformation and load its output. See [`search.md`](search.md). -### `overwrite` is scoped but destructive - -It removes existing entities of every type represented in the batch. Use it -for a complete type replacement, preferably on a review branch. Do not assume -that it clears unrelated types or the whole branch. - ## Branches: Review Before Merge Branches exist for **data review**, not schema changes. Schema goes straight to `main` via `plan` + `apply`. @@ -189,8 +181,8 @@ happens after a successful merge publication. Deleting a parent branch is supported while descendants remain. Logical deletion retains the native history descendants need; only explicit `cleanup` reclaims -unretained table versions, retired refs, and table forks left by branches -created under 0.11 or earlier. `optimize` does not perform that collection. +unretained table versions and retired refs. `optimize` does not perform that +collection. ### Merge conflicts @@ -209,32 +201,6 @@ keeps the history that base needs. `omnigraph schema apply` rejects the request if any non-main branches exist. Delete them first (`branch merge … --delete-branch` or `branch delete`); a merge alone leaves the source branch live. This is enforced — it's not just a guideline. -## Destructive Ops Go Through a Branch - -For any bulk load that could disrupt downstream queries (overwriting a -heavily-referenced node type, removing edges en masse, or reseeding a core -type), use a feature branch: - -```bash -omnigraph load --data risky.jsonl --branch recovery-2026-04-14 \ - --from main --mode overwrite $REPO -# inspect, diff, verify reads -omnigraph branch merge recovery-2026-04-14 --into main --delete-branch --store $REPO -``` - -## Branch Commands (quick reference) - -```bash -omnigraph branch create --from main --store $REPO -omnigraph branch list --store $REPO -omnigraph branch merge --into main --delete-branch --store $REPO -omnigraph branch delete --store $REPO -``` - -All support `--json` for automation-friendly output. Address the graph with a -positional `file://`/`s3://`/preview `az://` URI (shown), `--store `, or -`--server `. - ## Inspecting State After Changes ```bash @@ -246,7 +212,6 @@ omnigraph commit list $REPO --branch main --json # history `export` is the right tool for large-snapshot inspection — don't try to page through the whole graph with read queries. > **Cluster note:** everything in this file applies unchanged in cluster -> deployments — the control plane owns schema/queries/policies; rows, loads, -> and branches stay on the data plane against the derived graph roots -> (`/graphs/.omni`, or `/graphs/.omni` for an S3-backed -> cluster). +> deployments. Use `--server --graph ` for rows, loads and +> branches while the server owns writer admission; live configuration goes +> through cluster apply. diff --git a/skills/omnigraph/references/migrations.md b/skills/omnigraph/references/migrations.md index 3717f8c81..710cb3f36 100644 --- a/skills/omnigraph/references/migrations.md +++ b/skills/omnigraph/references/migrations.md @@ -1,82 +1,34 @@ -# Migration and Retired Vocabulary +# Upgrading to 0.13 -The rest of this skill targets the current CLI and HTTP contract. The versioned -procedures below use their named predecessor binaries. Read this page before replacing -an older binary, rebuilding a graph, or translating earlier API examples. +Upgrade the CLI, server and integrations together. The only HTTP contract is +`Omnigraph-Http-Api: 0.13`; the only served storage format is 14. Normal open +never converts storage. Preserve the old executable and a verified whole-root +backup until the upgrade and cutover are proven. -Current HTTP integrations must use exactly `Omnigraph-Http-Api: 0.13`, including -the current routes, shapes and credentials. A `0.12` peer is refused before data -dispatch or graph access; changing an old integration's header alone is not an -upgrade. See [server policy](server-policy.md). +## Choose the storage path -## Upgrade v0.10 to v0.11 - -This section is the 0.11 procedure and runs with the 0.11 binary. 0.12.0 -cannot run it: it serves storage format 14 only, its `upgrade` accepts -`--to-format 14` only, and it reports `unsupported_source` for a v6 graph. - -v0.10 wrote storage format v6; v0.11 created v9 and served v8 and v9 without -migrating them on open. Keep application traffic stopped while coordinating the -CLI, server, queries, loaders and client bindings. Matching package version -numbers alone do not establish client compatibility. - -For a qualified **standalone** graph, stop every writer, server and maintenance -process, retain the source-compatible executable, verify a whole-root backup, -and run the 0.11 binary: - -```bash -omnigraph upgrade ./graph.omni --check --json -omnigraph upgrade ./graph.omni --json -``` - -The default route is v6→v7→v8→v9 (or its remaining suffix). It preserves data, -identity, retained commits and snapshots, but ending at v9 requires only `main` -and no user property beginning with `_`. v0.11 cannot open a v6 graph, so -either delete non-main branches and `@rename_from` offending properties with -the v0.10 binary first, or convert with `--to-format 8`, fix both with v0.11 -(`branch delete`, `schema apply`), then run `schema upgrade-system-columns`. -A merge alone leaves a branch live. To preserve live branches and legacy -system spellings, use `--to-format 8` on both check and execution; v0.11 -serves the result. Explicit target 7 is an intermediate format -that v0.11 refuses on ordinary open. Checks are advisory and execution validates -again; inspect findings, deferred checks and any required recovery action. - -On 0.11, already-v8 graphs can use `omnigraph schema upgrade-system-columns -./graph.omni --check --json` and then omit `--check` for the v9 step. This is -irreversible; retain the backup and stop overlapping processes. The stored -schema's edge constraints are respelled automatically, but local `.pg` files -still saying `@unique(src, dst)` must become `@unique(@src, @dst)` before the -next `schema plan`/`apply`. An interrupted 0.11 run is completed by the next -0.11 read-write open. Follow an interruption's recovery instructions without -deleting protocol markers. Rollback restores the entire pre-upgrade backup with -the old executable. - -Cluster-managed roots and unqualified sources refuse in-place upgrade. Rebuild -with a source-compatible export into a newly applied graph or parallel cluster, -then verify and cut over. Before the new `init`, edit the old `schema show` -output (`src`/`dst` → `@src`/`@dst` in edge constraints; rename `_`-prefixed, -`_distance`, and `_score` properties in both schema and JSONL `data`) and -relocate export identity: +| Source | Action | +|---|---| +| Format 14, including release 0.12 | No graph-format migration. Update clients and check whether cluster ledger conversion is required. | +| Standalone format 8 or 9 (0.11.x), or 13 | Offline `omnigraph upgrade --check`, then `omnigraph upgrade`; preserves data, branches, commit IDs and retained history. | +| Cluster-managed graph below format 14 | Export with the source-compatible binary; rebuild in a new graph ID or parallel cluster. | +| Other storage formats | Rebuild with a source-compatible export, subject to the predecessor option below. | -```bash -jq -c 'if has("id") then . else .id = .data.id | del(.data.id) end' \ - graph.jsonl > graph-current.jsonl -``` +A standalone format-6 graph from 0.9/0.10 can first use the **0.11 binary's** +qualified upgrade to format 8, then use 0.13's upgrade to 14. Pass +`--to-format 8` to both the 0.11 check and execution to keep live branches and +legacy system-column spellings; its default target 9 has additional restrictions. +Read the [0.11 procedure](https://github.com/ModernRelay/omnigraph/blob/v0.11.0/docs/user/operations/upgrade.md) +before using that intermediate executable. Do not run those older flags against +0.13, whose only upgrade target is 14. -Do not point `init --force` at the old root. The v0.11.0 -[upgrade guide](https://github.com/ModernRelay/omnigraph/blob/v0.11.0/docs/user/operations/upgrade.md) owns qualification, -interruption handling, external Blob caveats and cluster cutover details. +For current qualification, limits, interrupted attempts and recovery findings, +use the [storage upgrade guide](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/operations/upgrade.md#storage-upgrade). -## Upgrade v0.11 to storage format 14 +## Offline standalone conversion -OmniGraph 0.12.0 serves storage format 14 only: -it refuses a v8 or v9 graph on open and names `omnigraph upgrade`. Its -`upgrade` converts a standalone v8, v9 or v13 graph to 14 in place and keeps -branches, commit ids and commit history; no table row is rewritten. -`--to-format` accepts 14 only. The upgrade is offline and cannot verify that -itself: stop every server, reader, writer and maintenance process, take and -verify a backup of the whole graph root (rollback is that backup with the -0.11 binary), then with the 0.12.0 binary: +Stop every reader, writer, server and maintenance process using the graph; +keep them stopped until completion. With the 0.13 binary: ```bash omnigraph upgrade ./graph.omni --check --json @@ -84,184 +36,66 @@ omnigraph upgrade ./graph.omni --json omnigraph commit list ./graph.omni --json ``` -`--check` writes nothing. A run without `--check` marks the graph as pending, -which every normal open refuses, and a run that stops is resumed by the same -command with the same executable. Never delete the marker or anything under -`__history/`. What a 0.11 graph brings with it: - -- Its schema is read from `_schema.pg`, `_schema.ir.json` and - `__schema_state.json` at the graph root, which must be present, consistent - and match the tables' columns. The upgrade leaves them in place; format 14 - never reads them. Every pre-upgrade commit is recorded under the one schema - live at the upgrade, as 0.11 kept only the latest schema. -- A `.staging` schema object or a recovery file left by 0.11 is refused until - a 0.11 read-write open resolves it (`omnigraph snapshot ./graph.omni` with - the 0.11 binary is such an open; a property-only `.staging` 0.11 refuses - too and names the manual choice). A `__schema_apply_lock__` branch left - alone by a killed 0.11 schema apply is handled by the upgrade. -- `cleanup` deletes no commit of a live branch, so a commit can outlive the - table versions it names: on a root where 0.11 `cleanup` or - `schema apply --allow-data-loss` ran, and after any narrower earlier - `cleanup`. Such a commit stays listed, but a read at it is refused. A - `cleanup` whose policy retains one refuses its tables - (`is absent from the listing`) and still exits 0. A branch base, merge base - or tag is retained under every policy: while one is such a commit, every - `cleanup` refuses, and deleting that branch or tag releases it. Otherwise - `--older-than D` passes once those commits are older than `D`, `--keep N` - once they fall outside the newest `N`, and `cleanup --keep 1 --confirm` - passes. -- A v8 graph keeps its legacy `id`/`src`/`dst` system columns through the - upgrade. In 0.12.0, - `schema upgrade-system-columns ./graph.omni [--check] [--json]` respells a - format 14 graph to `__id`/`__src`/`__dst` in one graph commit and stays at - format 14. It still needs only `main` and no property beginning with `_`, - refuses a cluster-managed graph and has no reverse; a run interrupted - before it publishes changes nothing and is rerun. -- A cluster-managed graph is not converted: export with 0.11, then `init` and - `load --mode overwrite` with 0.12.0, as in the section above. -- A 0.9/0.10 graph (v6) is first taken to v9 with the 0.11 `upgrade` above, - then to 14; v7 and v10 to v12 are rebuilt. - -The -[upgrade guide](https://github.com/ModernRelay/omnigraph/blob/v0.12.0/docs/user/operations/upgrade.md#storage-upgrade) -owns the findings table, the limits and the recovery actions. The storage -upgrade is not the whole move to 0.12.0: upgrade CLI, server and HTTP -integrations together, since every protected request needs the -`Omnigraph-Http-Api: 0.12` header, and read the -[v0.12.0 release notes](https://github.com/ModernRelay/omnigraph/releases/tag/v0.12.0) -for the other wire and CLI changes. +A pending upgrade refuses ordinary opens. Resume an interrupted conversion with +the same command and executable; never delete its markers or history objects. +Rollback restores the whole backup with the source-compatible binary. -## Identity and response changes +Physical `id`/`src`/`dst` spellings survive this conversion. GQ always uses +`@id`/`@src`/`@dst`. The optional `schema upgrade-system-columns` step requires a +standalone format-14 graph with only `main` and no `_`-prefixed user properties; +it preserves data/history and stays at format 14. -These changes arrived with v0.11, at the v0.10 to v0.11 boundary. +## Rebuild and cutover -- GQ system fields are `$p.@id`, `$e.@src`, and `$e.@dst`. Bare `id`, `src` - and `dst` are user properties. New schemas reserve leading `_`; new edge - endpoint constraints use `@unique(@src, @dst)`. `GET /schema` and - `schema show --json` report each graph's `system_columns`. -- **Rewrite existing `.gq` before restarting on v0.11**, including every - stored-query registry: `$x.id`/`.src`/`.dst` → `$x.@id`/`.@src`/`.@dst`, - `{ id: $v }` match filters → `$x.@id = $v`, and `where id = …` → - `where @id = …`. Unless the type declares a user property named `id`, a bare - `id` is an unknown property (`T6` in projections, `T2` in match filters, - `T11` in mutation predicates) on either vintage, and a registry that fails typecheck leaves its graph `blocked` at startup. - Run `lint` or `cluster validate` first. Unaliased result columns change name - (`p.id` → `p.@id`). -- Inputs that v0.10 accepted can now fail: a `Date` string with a time of day, - whole-number float or boolean date values, and out-of-range stored date - counts (reads/exports of that column fail until corrected). -- JSONL identity is top-level `id`, beside `type` or `edge`. On a graph whose - `system_columns` are `__id`/`__src`/`__dst`, `data.id` is a declared user - property and `data.__id` is refused. A graph that keeps the legacy - `id`/`src`/`dst` spellings also accepts `data.id` when top-level `id` is - absent. Move predecessor export identity out - of `data` before loading it into a new graph; preserve exports already using - top-level `id` and any user property named `id`. -- Bare-node projection returns an object with `@id` and non-Blob/non-Vector - properties. Query and export JSON omit null-valued property keys; change - images keep explicit nulls because absence means outside that commit's schema. - `Date` is a calendar date; `DateTime` output has no trailing `Z`. F32 values - use 32-bit rendering, and integers of every width are JSON numbers: JavaScript - consumers must account for precision above their safe integer range. -- `/readyz` reports the replica's applied revision and `served_graph_count`, - `ready_graph_count`, `loading_graph_count` and `blocked_graph_count`; 0.12.0 - removed the 0.11 `quarantined` fields, and `GET /graphs` returns one `graphs` - list with each graph's `state`. Branch statements can use canonical `/query` - and `/mutate`; inspect - their `outcome` instead of treating zero affected counts or `commit: null` - as a data no-op. See [server routes](server-policy.md) and - [write positions](changes.md). +Export with the binary that can read the source. Preserve top-level JSONL `id`s; +for predecessor exports whose identity lived in `data.id`, follow the canonical +[rebuild procedure](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/operations/upgrade.md#rebuild) +rather than rewriting every `id` property indiscriminately. Update old GQ system +field references and lint all schemas and stored queries before deployment. -Full-text compatibility is independent of the storage conversion. Existing -Lance-11-compatible indexes do not need a new rebuild solely for v0.11. Indexes -from the older Lance transition still need the explicit procedure below; -`omnigraph upgrade` does not rebuild them. +For a cluster, create the replacement with `cluster apply`: graph roots are +managed by the declaration, so direct `init` is not the rebuild entry point. +Load and verify the replacement before moving clients. Export/import preserves +entities and their values, including vectors and managed Blobs, but does not +reconstruct old commit IDs, snapshots or shared branch history. Export each +needed branch separately. External Blob references need an applied allow-list +and a policy-aware load through the server. -Released v0.10 graphs need no action for the withdrawn interim actor-provenance -feature. Graphs or cluster ledgers written by a development build with it -(schema IR v3, `actor_provenance` config, `--actor-provenance`) are refused. +Removing the old graph from the declaration and applying deletes its entire +managed root and retained history. Keep the source until the replacement is +verified and that deletion is intended. See the +[cluster cutover guide](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/operations/upgrade.md#cluster-cutover). -## Upgrade v0.9 to v0.10 +## Cluster ledger -v0.10 moves from Lance 9 to Lance 11 but retains graph storage format v6, so -existing entities, branches, and retained history do not need entity -export/import. The interface boundary is not rolling-compatible: stop traffic -and upgrade CLI, server, and client bindings together. Do not mix Lance 9/10 and -Lance 11 readers or writers on one graph root. - -Preserve a verified backup of the whole graph root and the cluster deployment -state. Then rebuild every full-text index on every live branch that needs -full-text search: +An explicit `ledger_upgrade_required` refusal names a supported ledger +conversion. With every server, writer and maintenance process stopped: ```bash -omnigraph branch list --store graph.omni --json -omnigraph rebuild-full-text-indexes --store graph.omni \ - --branch main --as operator --json +omnigraph --cluster cluster upgrade-ledger --writers-stopped ``` -The rebuild uses the default English analyzer and replaces custom tokenizer -settings. It does not rewrite historical snapshots. Ordinary traversal and -vector search remain available without it; full-text search refuses indexes -whose analyzer compatibility cannot be proved. An unknown physical index kind -is a fail-closed migration case—use compatible old tooling for a controlled -export/import rather than editing index metadata. - -Before reopening traffic, verify representative full-text searches and entity -counts on every rebuilt branch, then check the upgraded CLI/API integrations -against the new response fields. Resume only with the new fleet. - -Rollback means restoring the whole pre-upgrade graph and cluster-state backup -with the old fleet. v0.9 cannot read applied `external_blobs` state and also -lacks v0.10's default-deny external ingress, so do not downgrade when that -boundary is required. - -v0.10 also removes ambiguous client vocabulary such as `table_key`, `row_id`, -`manifest_version`, `rows_loaded`, and `export --table`. Use node/edge, type, -entity, property, graph-manifest, and published-dataset terms plus -`export --type`. See the [v0.10.0 upgrade procedure](https://github.com/ModernRelay/omnigraph/blob/v0.10.0/docs/user/operations/upgrade.md). - -## Pre-0.7 configuration - -| Before | Current | -|---|---| -| `omnigraph.yaml` | `cluster.yaml` for team deployment plus `~/.omnigraph/config.yaml` for operator settings | -| `cli.actor` | `operator.actor` | -| `cli.graph` / `server.graph` | `defaults.default_graph` plus optional `defaults.server` | -| `targets:` / `target:` | `graphs:` / `graph:` | - -`omnigraph.yaml` is removed and there is no automatic config migration. Move -schema/query/policy declarations into `cluster.yaml`; move identity, named -servers, output defaults, profiles, and aliases into the operator file. - -## Retired addressing and verbs - -| Before | Current | -|---|---| -| `--target ` | `--server`, `--store`, `--cluster`, or `--profile`, as the command permits | -| positional HTTP URL | `--server ` | -| `--cluster-graph ` | `--cluster --graph ` | -| query `--name ` | positional query name plus `--query`/`-e` for ad-hoc source | -| `ingest` | `load --mode ` | -| `read` / `change` | `query` / `mutate` | -| `query lint` / `query check` | `lint` | -| query/mutate `--alias` | dedicated `alias ` command | - -The server is cluster-only: start it with `omnigraph-server --cluster -`. Data-plane commands do not take the cluster -control-plane `--config` flag. `policy` and the stored-query registry use -`--cluster` plus optional `--graph`. - -Direct `schema apply` remains available for a non-cluster store. A cluster-only -server rejects the legacy schema-apply route with `409`; edit the declared `.pg` -and submit it with `cluster apply --server --config `. - -## HTTP compatibility aliases - -Canonical routes are `/query`, `/mutate`, and `/load`. `/read`, `/change`, and -`/ingest` remain deprecated compatibility aliases and identify their successor -in response headers. Per-graph routes are nested below `/graphs/{id}`; old flat -single-graph routes are gone. - -The pre-v0.4 transactional Run state machine, `/runs`, and the `run_publish` / -`run_abort` policy actions are removed. Writes publish directly; use exact write -receipts, commit history, and the `change` action. +This converts the ledger, not graph storage; rows, branches and history remain. +Use [cluster recovery](cluster.md#recovery) for retained writer locks and an +outstanding deployment. Do not edit the ledger by hand or submit a new deployment +ID to replace an uncertain attempt. + +## Integration cutover + +Use current `query`, `mutate`, `load` and `lint` CLI commands and +`cluster --managed` for the service API. HTTP uses `/query`, `/mutate` +and `/load` under `/graphs/{id}`; schema/config changes go through cluster +plan/deployment routes. Removed route aliases and the graph schema-apply route +are unavailable. Signed data credentials are version-2 identities; applied +Cedar policy supplies all permissions. + +Before switching traffic, verify contract discovery, representative query/load +results, stored-query invocation, policy grants and refusals, complete exports +and deployment observation by exact ID. Readiness is `/readyz`; `/healthz` only +proves the process is alive. See [server contract](server-policy.md) and +[remote outcomes](remote-ops.md). + +A storage upgrade does not rebuild incompatible full-text indexes. If a query +reports `FullTextIndexRebuildRequired`, follow the +[full-text upgrade procedure](https://github.com/ModernRelay/omnigraph/blob/v0.13.0/docs/user/operations/upgrade.md#full-text-index-upgrade) +on each live branch that needs search. diff --git a/skills/omnigraph/references/queries.md b/skills/omnigraph/references/queries.md index 3495aefd3..5c292c62f 100644 --- a/skills/omnigraph/references/queries.md +++ b/skills/omnigraph/references/queries.md @@ -11,17 +11,14 @@ - Filter operators - Mutations - Branch statements -- Naming convention -- Aliases over raw queries Writing `.gq` query files in Omnigraph. ## File Organization -- One `.gq` file per primary node type (`signals.gq`, `patterns.gq`, `elements.gq`) -- One `mutations.gq` file for all insert/update/delete queries -- Put query files in `queries/`, then declare `graphs..queries: queries/` in - `cluster.yaml`; cluster mode does not scan an undeclared directory. +Group related query declarations in `.gq` files and declare them under +`graphs..queries` in `cluster.yaml`. The server does not scan undeclared +directories. See [stored queries](stored-queries.md) for deployment. ## Linting @@ -197,6 +194,21 @@ query busy_signals($least: I64) { } ``` +### Typed edge selections + +GQ 2.1 supports `$a (knows | likes) $b` to select named edge types and `$a * $b` +to select compatible types from the captured schema. Wildcards require declared +endpoint node types. Omitted bounds mean `{1,1}`; recursion needs finite bounds +such as `{1,3}`. `$a $e:(knows | likes) $b` binds one-hop concrete edges and +exposes `$e.@type` and `$e.@id`. Common properties must exist with compatible +base types on every member; nullability and enum domains widen. + +`set traversal_work_limit = 1000000;` sets the shared traversal row-work cap for +statements using selections. Selected-table sizes count before scans; a result +limit does not replace this cap. Exhaustion fails the query, never partial +success. Historical wildcard targets are refused. Stored-query validation can +refuse a schema addition that invalidates wildcard common-property references. + ## System Fields and Result Values Use `$p.@id` for a node's identity and `$w.@id`, `$w.@src`, `$w.@dst` for a @@ -231,61 +243,10 @@ range fails the read instead of printing. ## Search Functions -### Text search - -```gq -match { - $d: Doc - search($d.title, $q) // full-text on @index'd String -} -``` - -```gq -match { - $d: Doc - fuzzy($d.title, $q, 2) // fuzzy match, max 2 edits -} -``` - -```gq -match { - $d: Doc - match_text($d.body, $q) // regular full-text match (not phrase search) -} -``` - -### Vector/ranking - -```gq -query vector_search($q: Vector(3072)) { - match { $d: Doc } - return { $d.slug, $d.title } - order { nearest($d.embedding, $q) } - limit 10 -} -``` - -`nearest`, `bm25`, and `rrf` belong in `order`, but they also restrict rows: a -`bm25` ordering returns only text matches, `nearest` skips rows whose vector is -null, and `rrf` returns the union of its arms' candidates. `nearest` and `rrf` -require `limit N`; BM25 alone does not, though a limit is recommended for -bounded output. - -`nearest(...)` and `bm25(...)` can also be projected as scores when the -expression exactly repeats the leading `order` key; see -[`search.md`](search.md#projecting-scores). `rrf(...)` and search predicates -cannot be projected. - -### Hybrid (reciprocal rank fusion) - -```gq -query hybrid_search($vq: Vector(3072), $tq: String) { - match { $d: Doc } - return { $d.slug, $d.title } - order { rrf(nearest($d.embedding, $vq), bm25($d.title, $tq)) } - limit 10 -} -``` +`search`, `fuzzy` and `match_text` filter text matches. `nearest`, `bm25` and +`rrf` lead the `order` clause and also restrict rows; `nearest` and `rrf` need a +finite `limit N`. Score projection repeats the leading ranking expression. +Read [search](search.md) for query examples, scoping, providers and exact limits. ## Aggregations @@ -458,14 +419,3 @@ A statement takes no name, `--params`, `--branch`, `--snapshot`, or `branch delete`. `lint` and stored-query registries reject statement files. Outcome and receipt semantics differ from data mutations; see [`changes.md`](changes.md#branch-statements). - -## Naming Convention - -`verb_object`: -- `get_signal`, `recent_signals`, `search_signals` -- `signal_patterns`, `signal_elements` (traversal queries) -- `add_signal`, `link_signal_forms_pattern` (mutations) - -## Aliases Over Raw Queries - -For anything an agent or script will call repeatedly, define an operator alias. See `references/aliases.md`. diff --git a/skills/omnigraph/references/remote-ops.md b/skills/omnigraph/references/remote-ops.md index be26299a2..08267f1d3 100644 --- a/skills/omnigraph/references/remote-ops.md +++ b/skills/omnigraph/references/remote-ops.md @@ -80,8 +80,7 @@ Over HTTP, beside the `Omnigraph-Http-Api: 0.13` header every graph request needs, send the raw id in the `Omnigraph-If-Graph-Commit` header to `POST /graphs/{id}/mutate/if-graph-commit` or `POST /graphs/{id}/queries/{name}/if-graph-commit`. The plain routes reject that -header; an older server answers `404`, so never fall back to the unconditional -route. +header; never fall back to the unconditional route after a refusal. ## Typed failures and recovery @@ -95,14 +94,13 @@ route. - `key_conflict`: an append or strict insert found an existing id. Decide whether that entity is the intended one; do not silently turn the operation into an upsert. -- `recovery_required` (HTTP `503` with `recovery_required.operation_id`): no - 0.12 mutation, load or branch merge returns it, because a write that fails - before its single publication leaves the graph unchanged. It remains for a - graph that still holds a `__recovery/` sidecar from a pre-0.12 build (open - it read-write with that build first) and for a schema apply or - system-column upgrade that fails after its commit was published; the named - commit already contains the change. Follow the error message's remedy and - do not replay. +- `recovery_required` (HTTP `503` with `recovery_required.operation_id`): + follow the named operation's remedy. This is a completion requirement, not + proof that the attempted write had no effect. Do not replay it or clear + storage/locks to bypass the refusal. +- `graph_unavailable` (HTTP `503`): inspect graph availability and the active + deployment. A loading, transitioning or blocked graph does not authorize + replay of an earlier uncertain write. The effect-free conflict details are distinct from a lost response or a recovery requirement. Neither HTTP `409`/`503` nor the CLI's generic failure exit `1` is diff --git a/skills/omnigraph/references/schema.md b/skills/omnigraph/references/schema.md index 51f7568c3..625c667b7 100644 --- a/skills/omnigraph/references/schema.md +++ b/skills/omnigraph/references/schema.md @@ -100,7 +100,7 @@ This validates the schema **and** the queries against it. No running repo requir ## Evolution (schema plan/apply) -### Plan before apply — always +### Plan before apply ```bash omnigraph schema plan --schema next.pg s3://bucket/repo --json @@ -117,10 +117,9 @@ a code is present, match it rather than the free-form message text. the current schema; older commits still read the dropped data until `omnigraph cleanup` stops retaining them, and only then is it gone for good. No flag makes a drop destructive at apply: to reclaim the space, run `cleanup` -with a retention that excludes the commits before the drop. A cluster-only -server rejects -`POST /graphs/{id}/schema/apply` with `409`; evolve a served graph through -`cluster plan` and `cluster apply`. +with a retention that excludes the commits before the drop. A served graph +evolves through `cluster plan --server …` and `cluster apply --server …`; +there is no graph schema-apply HTTP endpoint. ### Apply is main-only @@ -148,7 +147,7 @@ unsupported. Pattern: 3. Backfill via a `mutate` or `load --mode merge` 4. Keep it optional: tightening `T?` -> `T` is currently refused by the planner (a property-type change, OG-MF-106). Enforce presence at write time by - convention until required-tightening ships as a migration step. + convention, or rebuild with the stricter schema when required. ### Enum widening is a supported apply @@ -170,9 +169,12 @@ endpoints are refused as unsupported. In-place migrations are additions of nullable properties and types, `@index` additions, enum widening, renames, and drops; tightening a constraint means a rebuild. -### `schema apply` blocks writes while running +### Availability during apply -No concurrent mutations during an apply. Plan for a short read-only window. +Standalone schema apply serializes with writes. Served cluster apply closes +affected graph admission and drains admitted work before changing its schema; +new requests to that graph can receive `503 graph_unavailable` during the +transition. Unrelated ready graphs keep serving. ## Supported Types @@ -209,7 +211,7 @@ No concurrent mutations during an apply. Plan for a short read-only window. ## Interfaces -Supported but rarely used. Declare shared property contracts and node types implement them: +Declare a shared property contract and have node types implement it: ```pg interface Searchable { @@ -223,8 +225,6 @@ node Doc implements Searchable { } ``` -Most schemas are fine without interfaces. Reach for them only when 3+ node types need to share a property contract. - ## Design Principles (brief) - **Identity is explicit** — use `@key` on a semantic slug, not internal row IDs @@ -240,7 +240,7 @@ schema is declared (`graphs..schema:` in `cluster.yaml`) and converged: ```bash $EDITOR schema.pg -omnigraph cluster plan --config . --observe # migration steps; takes no cluster lock +omnigraph cluster plan --server --config . --json omnigraph cluster apply --server --config . # the running server publishes and activates the new shape; no restart ``` @@ -251,7 +251,7 @@ root itself: stop the server and transfer its cluster lock first (see Differences from direct `schema apply` (on a non-cluster store): out-of-band schema changes on the live graph are *drift* — the next `cluster apply` -refuses with `applied_schema_drift`, and `--schema-correction` accepts a -reviewed observed schema. +refuses with `applied_schema_drift`. There is no correction/adoption flag that +bypasses this check; investigate the mismatched graph authority. Everything else in this file (`@rename_from`, backfills, linting, enum discipline) applies unchanged to the `.pg` you edit. diff --git a/skills/omnigraph/references/search.md b/skills/omnigraph/references/search.md index 226187635..0e5a9af7b 100644 --- a/skills/omnigraph/references/search.md +++ b/skills/omnigraph/references/search.md @@ -213,9 +213,12 @@ are `OMNIGRAPH_EMBED_DEADLINE_MS`, `OMNIGRAPH_EMBED_TIMEOUT_MS`, For a served graph, declare a named provider under `providers.embedding` in `cluster.yaml` and bind it with `graphs..embedding_provider`. API keys must -be `${ENV_VAR}` references and are resolved by the server at startup. Generated -vectors are finite, nonzero, and L2-normalized. - -Full-text indexes built before Lance 11 (the v0.9 era) can still require -`rebuild-full-text-indexes` on each live branch before full-text queries run. Ordinary reads and vector -search do not depend on that rebuild; see [`commands.md`](commands.md#rebuild-full-text-indexes--explicit-analyzer-upgrade). +be `${ENV_VAR}` references. The server resolves them when building a provider +at startup or live deployment; source validation does not expose secret values. +Update definitions/bindings through `cluster apply --server …`; changing the +provider never regenerates stored vectors. Generated vectors are finite, +nonzero, and L2-normalized. + +For `FullTextIndexRebuildRequired`, rebuild the affected live branches explicitly; +ordinary reads and vector search do not depend on it. See +[`commands.md`](commands.md#rebuild-full-text-indexes--explicit-analyzer-upgrade). diff --git a/skills/omnigraph/references/server-policy.md b/skills/omnigraph/references/server-policy.md index 67493bfc0..13fbc0584 100644 --- a/skills/omnigraph/references/server-policy.md +++ b/skills/omnigraph/references/server-policy.md @@ -12,9 +12,9 @@ omnigraph-server --cluster s3://bucket/prefix --bind 0.0.0.0:8080 ``` The server loads the cluster's applied revision and serves each graph under -`/graphs/{id}`. `cluster apply --server` activates schema and stored-query -changes and new graphs without a restart; a direct `cluster apply` needs the -server stopped. There is no single-graph server mode. A stopped server, even +`/graphs/{id}`. `cluster apply --server` activates schema, stored-query, policy, +provider and external-Blob changes, plus graph creation/deletion, without a +restart. Direct apply requires serving to be stopped. There is no single-graph server mode. A stopped server, even after a clean exit, leaves the cluster lock: once its writes have settled, clear it with `cluster force-unlock ` before the next start (see [cluster mode](cluster.md)). @@ -32,7 +32,8 @@ termination grace. Check rollout readiness with `GET /readyz`, not `/healthz`. | `GET /healthz`, `/openapi.json` | Process metadata | | `GET /readyz` | Replica readiness, applied revision and served/ready/loading/blocked counts | | `GET /graphs` | List graphs with their availability (`graph_list`) | -| `POST /cluster/deployments`, `GET /cluster/deployments[/{id}]` | Submit and inspect a `cluster apply --server` deployment (`config_manage`) | +| `POST /cluster/plan` | Preview the submitted configuration without effects (`config_manage`) | +| `POST /cluster/deployments`, `GET /cluster/deployments[/{id}]` | Submit once and inspect a deployment; exact-ID observation survives caller disconnect | | `/graphs/{id}/query`, `/mutate` | Inline GQ reads and writes | | `/graphs/{id}/mutate/if-graph-commit` | Conditional inline mutation (header `Omnigraph-If-Graph-Commit`) | | `/graphs/{id}/queries` | List/invoke stored queries; `/queries/{name}/if-graph-commit` for conditional writes | @@ -51,16 +52,15 @@ refused with `400 api_contract_mismatch` before graph access. Responses carry the same header. `/healthz`, `/readyz` and `/openapi.json` need none. Upgrade the CLI, server and HTTP clients together. -`/read`, `/change`, and `/ingest` are deprecated compatibility routes. A -cluster-only server retains `POST /graphs/{id}/schema/apply` for wire -compatibility but rejects it with `409`; use `cluster plan`/`cluster apply`. +Graph schema changes use cluster plan/apply. There is no graph schema-apply +endpoint and no alias routes for older HTTP contracts. Discover the contract +with `/healthz` before sending protected requests; redirects and mismatched +responses are not permission to replay an effectful request. Canonical `/query` JSON includes the graph commit pinned with its rows when the read snapshot has an effective graph head; a fresh pre-commit graph can omit it. Exact write receipts and conditional semantics are summarized in [commit changes and feeds](changes.md). Blob delivery is in [Blob values](blobs.md). -The deprecated `/read` response does not carry that commit position; consumers -that need conditional writes must use `/query`. Canonical `/query` accepts `branch list`; `/mutate` accepts `branch create`, `branch delete` and `branch merge` with no request target, name or params. These @@ -69,7 +69,7 @@ return branch outcomes, not ordinary data-mutation receipts; see physical `system_columns`; GQ uses `@id`/`@src`/`@dst` on either storage vintage. `/readyz` is unauthenticated and contains no graph names. It identifies the -applied revision this process booted from, reports `ready: false`/HTTP 503 while +booted applied revision, reports `ready: false`/HTTP 503 while a graph is loading, when a nonempty inventory has no ready graph, and while draining, and includes `served_graph_count`, `ready_graph_count`, `loading_graph_count` and `blocked_graph_count`. Authorized `GET /graphs` @@ -100,7 +100,7 @@ actions remain denied. Managed signed data credentials are a separate token source, enabled with `--data-token-trust `. The trust file binds keys to the exact deployment; tokens select an immutable principal actor. An identity credential carries no -permissions; a legacy restricted one also narrows current Cedar grants. +permissions; only version-2 identity credentials are accepted. That actor (`principal:`) must itself be permitted by the applied Cedar policy, or every graph request is denied; an identity credential can still list graph ids through `GET /graphs/discovery`. Trust changes need a @@ -117,14 +117,15 @@ Graph-scoped actions are: | `read` | Queries, snapshots, branches, commits, Blob reads, and change polling | | `export` | Export and change-feed baseline capture | | `change` | Mutations and loads | -| `schema_apply` | Schema changes: `cluster apply --server` needs it on each existing graph whose schema changes | +| `schema_apply` | Served schema changes and graph deletion: required on each affected existing graph | | `branch_create`, `branch_delete`, `branch_merge` | Corresponding branch operation | | `invoke_query` | Entry to a stored query | | `admin` | Reserved; no current public operation | `graph_list` is cluster-scoped and controls `GET /graphs`. `config_manage` is -cluster-scoped too and authorizes `cluster apply --server`; put it in a rule -of its own. A stored read needs +cluster-scoped too and authorizes served cluster planning and deployment; put +it in a rule of its own. Current policy authorizes configuration changes; a +proposed policy cannot authorize its own installation. A stored read needs `invoke_query` plus `read`; a stored mutation needs `invoke_query` plus `change`. A denied and unknown stored-query name both appear as `404` to a caller lacking `invoke_query`. A load with `from` also needs `branch_create` @@ -178,14 +179,16 @@ omnigraph policy explain --cluster . --graph knowledge \ --actor act-alice --action read --branch main ``` -The `policy` commands evaluate the cluster's **applied** bundles, not draft -files: run `cluster validate` on the draft, then `cluster apply`. A bundle is -installed when the cluster is first applied or when its graph is created; a -later apply that edits an existing bundle, or changes its bindings other than -by adding a newly created graph, is refused with `deployment_scope`. -`--graph ` must match exactly one applied -bundle; with both a graph bundle and a `cluster` bundle applied (as in the -example above), the command refuses with "matches 2 policy bundles". +The `policy` commands evaluate applied bundles, not draft files. Validate the +source with `cluster validate`, preview with `cluster plan --server …`, then +apply. Live policy changes drain admitted work before later requests use the +new HTTP and engine permissions. Process token sources and signed-token trust +remain startup settings. + +The CLI policy selector currently requires exactly one matching bundle. With +both a graph bundle and a `cluster` bundle applied, `--graph ` refuses with +"matches 2 policy bundles"; this inspection limitation does not prevent the +server from applying or enforcing those bundles. ## Direct access is a separate trust boundary diff --git a/skills/omnigraph/references/stored-queries.md b/skills/omnigraph/references/stored-queries.md index c418d7741..f27838459 100644 --- a/skills/omnigraph/references/stored-queries.md +++ b/skills/omnigraph/references/stored-queries.md @@ -1,6 +1,9 @@ # Stored-Query Registries -A **stored query** is a `.gq` query that the *server* loads, type-checks at startup, and exposes by name — without ever accepting ad-hoc query source from the client. It's how you publish a vetted, typed query surface to remote callers and MCP tools. +A stored query is a named `.gq` declaration published through cluster apply. +The server type-checks it at startup or live activation and exposes invocation +by name, with typed parameters and policy gates. The invocation request contains +parameters, not query source. It is distinct from CLI `aliases:` (see [`aliases.md`](aliases.md)): an alias is local client ergonomics; a stored query is a server-published,