diff --git a/docs/en/docs/how-to/ingest-text-files-with-opendal.md b/docs/en/docs/how-to/ingest-text-files-with-opendal.md new file mode 100644 index 000000000..7c349cf1e --- /dev/null +++ b/docs/en/docs/how-to/ingest-text-files-with-opendal.md @@ -0,0 +1,95 @@ +--- +title: Ingest text files with OpenDAL +description: Capture UTF-8 files as typed Sources with an independent OpenDAL Connector worker. +--- + +# Ingest text files with OpenDAL + +`powercontext-connector-opendal` is deployed independently from PowerContext Server. It owns OpenDAL credentials, +provider configuration, the executable Source Definition, and file reads. The Server stores only a declarative +Definition manifest, materialized Source observations, named projections, and opaque checkpoints. + +## Before you begin + +The integration requires Python 3.12 or later. Start PowerContext Server, then install the worker from a checkout: + +```bash +uv tool install ./integrations/opendal +``` + +Choose a stable `source_namespace` that distinguishes storage authorities. Do not put credentials in the namespace, +Source payload, or checkpoint. If Server authentication is enabled, provide its bearer token through the +`POWERCONTEXT_TOKEN` environment variable. + +## Run a binding + +This independent process scans `/absolute/path/to/project/docs`. The `binding_id` identifies checkpoint continuity; +the `scope_id` determines which Scope owns accepted Sources: + +```bash +powercontext-connector-opendal \ + --base-url http://127.0.0.1:8765 \ + --scope-id project:example \ + --binding-id project-docs \ + --service fs \ + --storage-option root=/absolute/path/to/project \ + --root docs \ + --source-namespace project-docs +``` + +For remote storage, replace the OpenDAL service and pass its `--storage-option KEY=VALUE` arguments. These options +remain inside the worker process and are never sent through the ingestion API. + +On every run, the worker idempotently registers the `text-file-snapshot` Definition manifest, reads the binding +checkpoint, submits changed Source observations, and compare-and-swaps the checkpoint after every durable receipt. +Use cron, a Kubernetes Job, or another external scheduler to run the command periodically. + +## Embed the lifecycle in a worker + +Use the generic remote lifecycle when a deployment needs custom supervision or schedules multiple bindings: + +```python +from powercontext.client import PowerContextClient, RemoteConnectorWorker +from powercontext.sources import ConnectorBinding, SourceDefinitionRegistry +from powercontext_connector_opendal import ( + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + OpenDALTextFileConnector, +) + +connector = OpenDALTextFileConnector.from_service( + "fs", + source_namespace="project-docs", + root="docs", + storage_options={"root": "/absolute/path/to/project"}, +) +binding = ConnectorBinding( + scope_id="project:example", + binding_id="project-docs", + connector_name=connector.name, + connector_version=connector.version, +) +registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + +async with PowerContextClient("http://127.0.0.1:8765") as client: + result = await RemoteConnectorWorker(client=client, registry=registry).run(connector, binding) +``` + +## Runtime semantics + +Each item outcome is `accepted`, `replayed`, `rejected`, or `failed`. The checkpoint advances only when the run +completes without rejected or failed items. Otherwise the prior checkpoint remains, and the next run safely retries +from it. Files whose digest matches the committed checkpoint are skipped. + +Accepted Sources enter the target Scope's Source journal. The worker also computes the standard +`powercontext.text-evidence` projection, so Memory consumers need not understand the native `text-file-snapshot` +schema. A Connector run does not create Memory directly; the normal source-window flush or schedule still does that. + +## Limits + +- The default patterns select Markdown, text, reStructuredText, and AsciiDoc files. +- A run selects at most 10,000 files and reads at most 2 MiB per file by default. +- Only UTF-8 content is accepted. +- Changed content creates a new exact snapshot Source; earlier snapshots remain available. +- A full scan removes missing paths from the next checkpoint but does not delete Sources or claim authoritative + deletion. +- The Connector does not provide a change feed; an external scheduler must run the worker again to observe changes. diff --git a/docs/en/rfcs/0000_source_definition_and_observation_model.md b/docs/en/rfcs/0000_source_definition_and_observation_model.md new file mode 100644 index 000000000..f721ef57a --- /dev/null +++ b/docs/en/rfcs/0000_source_definition_and_observation_model.md @@ -0,0 +1,663 @@ +- Proposal Name: `source_definition_and_observation_model` +- Start Date: 2026-08-27 +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md), [RFC 0014](0014_memory_layer_design.md), + [RFC 0019](0019_local_source_memory_runtime.md), [RFC 0048](0048_handoff_artifact.md) + +# Summary + +This RFC defines the standard Source model and the contract for defining additional Source types. + +A Source belongs to exactly one Scope. Within that Scope, a `SourceKey` identifies one logical source and a +`SourceRef` identifies one immutable observation of that source. Advancing the current observation, observing a +deletion, changing an external locator, or disconnecting a Connector does not alter an earlier observation or move +it to another Scope. + +A Source Definition gives one stable Source type its value schema, provenance schema, identity rules, observation +rules, materialization contract, canonicalization, and compatibility policy. Definitions are registered explicitly +and remain fixed for the lifetime of a composed Runtime. Persistence, transport, and Artifact consumers route by the +stable definition name and version rather than by a concrete Python class. + +A Definition may advertise named projection capabilities for consumers that do not understand its native value. +Each projection has an independently versioned schema and deterministic meaning over one exact observation. A +consumer selects a projection by capability name and version, never by inspecting a concrete Source class. + +A Connector lifecycle binds provider acquisition to a Scope, resolves definition-native inputs in its worker, +submits materialized observations, records per-item outcomes, and advances an opaque checkpoint only after accepted +observations are durable. Connector runs distinguish complete discovery from incomplete discovery so that absence +is not silently converted into deletion. + +Materialization identifies the authority used to resolve an exact observation. A captured observation is resolved +from the canonical value retained by PowerContext. A referenced observation is resolved from an immutable external +revision. An external locator, modification time, ETag, or current-provider read does not by itself satisfy the +referenced contract. + +`ContentSource` remains a simple captured-text Source. Its caller-stable identity and immutable-payload conflict rule +make it useful for one-shot content capture, but it is not the general external integration model. + +This RFC defines Source, projection, Connector lifecycle, and the remote ingestion boundary between a worker and the +PowerContext Server. It does not define plugin discovery, a scheduler, credential transport, a concrete Source +family, or a Connector implementation. + +# Motivation + +`ContentSource` and `POST /v1/sources/content` provide captured-text ingestion. The caller +chooses one `source_id`; replaying an identical payload is idempotent, while reusing that identity with a different +payload is a conflict. This gives exact evidence only when the caller treats the identity as immutable. + +External systems usually expose a different lifecycle. A wiki page, issue, object, message, or file has one logical +identity but may produce several values over time. The external object can be renamed, revised, deleted, restored, +or become temporarily unreadable. Artifacts that used an earlier value must continue to cite that exact evidence. +The two-part `(source_type, source_id)` Source reference cannot express both the stable logical object and its immutable observation +without making every integration invent a composite `source_id`. + +For example, using only the provider object ID makes the second value conflict with or replace the first. Using only +a value digest keeps both values but loses the fact that they describe the same continuing object: + +```text +provider object 42 + | + +-- value v1 ----> exact observation 1 + `-- value v2 ----> exact observation 2 + ^ + | + same logical Source +``` + +The model therefore keeps logical identity and exact evidence separate. Consumers can follow the continuing Source +when they need current state while Artifacts keep citing the observation they actually used. + +The extension boundary is also incomplete. A Source adapter binds a native input class to a concrete +Source class and a read result, while the built-in Runtime and relational persistence assemble a fixed adapter set. +This does not state the durable rules an independently defined Source type must follow across identity, persistence, +transport, and Artifact evidence. + +The standard model must answer six questions without assigning them to one identifier: + +1. Which Scope owns this evidence? +2. Which logical external or internal source does it describe? +3. Which exact observed value did an Artifact use? +4. Where does PowerContext read that exact value from? +5. Which definition gives the value and provenance their meaning? +6. Which declared view may a consumer use without understanding the native value? + +Connector concerns are adjacent but distinct. Discovery, credentials, filtering, checkpoints, retries, provider +change handling, and deletion detection decide which observations are submitted. They do not define Source identity, +weaken exact evidence, or change Scope ownership. + +# Guide-level explanation + +## Domain model + +Read the model by establishing ownership first, then logical identity, exact observation, materialization authority, +and type semantics: + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Where does the Source belong? | +| Logical identity | `SourceKey` | Which continuing source is this? | +| Exact evidence | `SourceRef` | Which immutable observation is cited? | +| Read authority | materialization | Where is that exact value resolved? | +| Type semantics | Source Definition | How are value, provenance, and identity interpreted? | +| Consumer view | named projection | Which declared representation may a consumer use? | +| Acquisition | Connector or direct caller | How are new observations found and submitted? | + +These responsibilities form one direction of dependency: + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +A Connector can use one Source Definition, several Connectors can use the same Definition, and a direct caller can +submit a Source without a Connector. Connector identity therefore does not become Source type identity. + +## Scope ownership + +Every SourceKey and observation belongs to exactly one Scope. Scope ownership is not inferred from an external +workspace, path, repository, provider account, Connector instance, or Source locator. Those values may contribute to +binding or provenance, but they do not allocate or replace `scope_id`. + +The fully qualified logical identity is: + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +The fully qualified exact identity is: + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +A scope-bound operation may obtain `scope_id` from its fixed request binding instead of accepting it as an arbitrary +argument. The durable resolved reference still retains the owner Scope so that evidence remains unambiguous after +publication, reporting, or export. + +Changing a Scope Parent, Context References, an Agent binding, or an observation selection changes no SourceKey or +SourceRef. Publishing an Artifact across Scopes preserves the original Scope and exact SourceRef in provenance. It +does not move or implicitly copy the Source history. + +## Logical Source and immutable observation + +`source_id` names a logical source within one `(scope_id, source_type)` namespace. Its meaning is defined by the +Source Definition. It may correspond to a provider object ID, a stable import identity, or another normalized key. +It must not silently change when a new value is observed. + +`observation_id` names one immutable observation under a SourceKey. It is opaque to generic PowerContext components. +It may be derived from a provider revision, a canonical value digest, or a definition-specific combination. It does +not imply an integer sequence, timestamp order, or ancestry. + +The following invariants apply: + +- one `(SourceKey, observation_id)` identifies one canonical observation forever; +- re-observing the same canonical observation is idempotent; +- a different canonical observation cannot reuse an observation ID; +- one SourceKey may have several observations with the same value digest when their identity-bearing provenance is + different; +- observations with the same value digest are not automatically the same logical Source; and +- an Artifact cites an exact SourceRef, never a moving SourceKey or `latest` observation. + +For example, updating one logical Source retains its SourceKey and produces another SourceRef: + +```text +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" +``` + +An Artifact derived from `observation-1` continues to cite it after `observation-2` becomes current. + +## Source Definition + +A Source Definition is the durable semantic contract for one `source_type`. It declares: + +- a stable definition name and version; +- the Source value and typed provenance shapes; +- Source ID normalization and equality; +- observation ID normalization and equality; +- identity-bearing fields and non-identifying annotations; +- canonical bytes and the value digest algorithm; +- supported materialization modes and exact-read requirements; +- limits and validation failures; and +- compatibility rules for older definition versions. + +A Definition resolves definition-native input into a canonical observation and reads the definition-owned value from +an exact persisted observation. Resolution does not select a Scope, mutate a catalog, advance a head, or discover +external objects. Reading does not resolve `latest` or substitute another observation. + +Definitions are explicit and typed. A new integration must not simulate a new Source type by placing an undocumented +schema inside `ContentSource.metadata`. Provider-specific provenance may extend a Definition's declared schema, but +fields that affect identity, exactness, or compatibility must be named by the Definition. + +## Named projection capabilities + +A named projection is an optional, Definition-owned view of one exact observation. It allows an Artifact family or +another consumer to use a declared representation without knowing the native Source value or concrete Python class. + +A projection is selected by a stable name and version. Its Definition declares the output schema, canonicalization, +digest rules, and failures. The projection is evaluated against an exact SourceRef and cannot resolve a head, +`latest`, or a current provider value. For the same Definition version, projection version, and exact observation, it +returns the same canonical result. + +Projection capability is explicit. A consumer that requires a projection rejects a Source that does not advertise a +compatible capability; it does not infer content from metadata or fall back to a similarly shaped Source class. A +projection can be cached or persisted as a derivative, but its authority remains the exact Source observation and +its lineage retains that SourceRef. + +This contract does not prescribe a catalog of standard projection names or payload schemas. A projection becomes a +shared standard only after interoperating definitions and consumers demonstrate that its semantics are stable. Until +then, a Definition may expose namespaced projections without making them mandatory for other Source types. + +## Materialization authority + +Materialization answers where the value returned for an exact SourceRef comes from: + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | Canonical value retained by PowerContext | The retained value matches the observation digest | +| `referenced` | Immutable external revision | Re-reading the reference returns the same canonical value and digest | + +A captured Source may retain an external locator, provider revision, and digest as provenance. It remains captured +because the retained value is the read authority. This covers the useful part of a hybrid design without creating a +third mode with ambiguous fallback semantics. + +A Definition can use referenced materialization only when the external system and its reader can address immutable +historical values. Reading the current value at a path, page ID, issue ID, or URL is not sufficient. Modification +times and ETags may contribute to provenance or conflict detection, but a Definition must state whether the provider +guarantees that they address an immutable value. + +When the referenced value is unavailable or its digest differs, exact resolution fails. PowerContext does not return +the current provider value, a stale cache entry, or another observation. A provider that cannot satisfy this rule +must use captured materialization or reject the observation. + +## Current head and deletion + +A Source history is immutable; its current head is a mutable catalog selection. The head can select one exact +SourceRef or record that the logical Source was positively observed as deleted. The head is useful for current-state +queries and later acquisition, but it is not evidence and cannot appear in an Artifact citation. + +Advancing or deleting a head changes no observation. A timeout, permission failure, incomplete listing, unavailable +Connector, or disconnect is not positive deletion evidence and does not change the head. A Source Definition may +define a tombstone value only when deletion itself is meaningful Source evidence; a generic head deletion does not +fabricate one. + +## ContentSource + +`ContentSource` remains the neutral captured-text path defined by RFC 0019. Its caller chooses an identity that can be +committed once with one canonical payload. The persistence conflict rule makes an accepted ContentSource exact, but +it does not provide a separate logical Source lifecycle. + +The standard model treats this as a valid single-observation Source implementation: + +- the existing identity remains immutable; +- an identical replay remains idempotent; +- different content under the same identity remains a conflict; +- references that resolve ContentSource remain exact and unchanged; and +- no mutable head or multi-observation behavior is inferred from metadata. + +ContentSource is suitable for prompts, explicit text capture, import records, and other cases where the caller +already owns an immutable identity. Integrations that observe one logical object over time should define or reuse a +multi-observation Source type instead. + +The two ingestion paths differ at acquisition time but converge on Scope-owned Source history: + +| Concern | `ContentSource` capture | Source Definition and Connector ingestion | +| --- | --- | --- | +| Typical input | Text already held by the caller | Objects discovered in an external system | +| Identity | One caller-stable immutable identity | One logical identity with exact observations | +| Type contract | Built-in captured text and metadata | Definition-owned value, provenance, and projections | +| Synchronization | One request, with no checkpoint | Discovery, per-item outcomes, replay, and checkpoint comparison | +| Downstream use | Built-in text evidence | A named projection understood by the consumer | + +`ContentSource` is the shorter path when the caller already has final text and an immutable identity. The remote +ingestion APIs do not replace it. They add the lifecycle needed when a worker must discover, normalize, and +re-observe external objects without loading provider code or credentials into the Server. + +## Source participation in Scope flows + +Durable acceptance appends an exact observation to the Source journal of its owner Scope. Acceptance does not create +Memory or another Artifact by itself. A Scope-local processor later selects a bounded Source window, asks for a +projection it understands, and may produce a new Artifact revision that cites the exact SourceRef: + +```text +Connector or direct caller + | + | bind Scope A + v +Source observation + | + v +Scope A Source journal ----> Scope-local processor + | + named projection + | + v + Scope A Memory revision + cites exact SourceRef +``` + +A new observation can cause later processing, but it does not rewrite an earlier Artifact revision: + +```text +SourceKey(scope-a, record, provider-object-42) +|-- observation-1 ----> Memory revision 3 +`-- observation-2 ----> Memory revision 4 + +Memory revision 3 continues to cite observation-1. +``` + +A consumer uses a Source only through its native Definition or a compatible named projection. For example, a Memory +extractor that requires text evidence can consume any Source Definition that advertises the matching text +projection. It does not need to know whether the Source began as a file, page, issue, or `ContentSource`. A missing +capability remains explicit; the consumer does not infer text from metadata. + +Cross-Scope use depends on the intended ownership and delivery behavior: + +```text +Scope A Source history + | + +-- Context Reference from Scope B + | `-- later Prepare Context may read eligible Scope A material + | + +-- publish exact Artifact revision + | `-- Scope B receives one selected result with origin provenance + | + `-- deliberate capture into Scope B + `-- Scope B owns a new Source and runs its own downstream flow +``` + +Use a Context Reference for continuing read access and exact Artifact publication for a selected result. If Scope B +must own and independently process the external value, capture it into Scope B as a new Scope-owned observation and +retain the origin reference in provenance when applicable. None of these operations moves the original Source or +makes Parent imply read access. + +# Reference-level explanation + +## Source identity contract + +`scope_id` is the ownership boundary defined by the Scope organization design. `source_type` is the stable Source +Definition name. `source_id` is a non-empty, normalized identifier whose equality and bounds are declared by that +Definition. + +Source identity is Scope-local. Two Scopes may contain equivalent external material without sharing ownership or +identity. A Definition may include a stable external instance or connection discriminator in its `source_id` rules +when required to prevent collisions, but the discriminator does not replace `scope_id`. + +Renames are definition-specific. A provider object ID may preserve SourceKey across locator changes. A path-derived +identity normally treats a rename as one logical deletion and one creation. A Definition must not claim rename-stable +identity when its provider and acquisition path cannot prove it. + +## Observation contract + +An observation contains these standard fields: + +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` + +`value_digest` uses SHA-256 over the canonical bytes declared by the Definition and is encoded as +`sha256:`. For structured values, the Definition specifies a deterministic canonicalization. The +digest verifies value equality; it does not replace SourceKey or observation identity. + +The canonical observation contains every field that the Definition says affects identity or exact meaning. +Operational facts such as a retry count, last scan time, or processing status are not Source value and do not change +observation identity. If a timestamp or provider attribute affects provenance meaning, the Definition must classify +and canonicalize it explicitly. + +## Source reference contract + +A SourceRef identifies an exact observation and includes its owner Scope. It never accepts an absent observation ID, +`latest`, a head version, or a current provider locator. + +Within a scope-bound operation, a compact local representation may omit a repeated `scope_id` only while the current +Scope is fixed and the resolved durable value restores it. Any reference that crosses a Scope boundary, leaves the +Runtime, or enters durable cross-Scope provenance carries the owner Scope explicitly. + +Reference resolution verifies all four identity components and the stored observation's definition version and +digest. Failure to resolve the exact observation is distinct from the logical Source being deleted, the head having +advanced, or the Connector being unavailable. + +An accepted compatibility reference without `observation_id` still denotes one immutable observation. Resolution +must not treat it as a SourceKey, a current head, or `latest`. A compatibility layer may restore the full SourceRef at +its boundary, but it cannot redirect the evidence. + +## Definition registration contract + +Executable Definitions belong to the worker that resolves definition-native inputs, canonicalizes Source values, +and computes named projections. The Server does not import Connector or Definition packages and does not execute +their Python classes. + +Before submitting an observation, the worker registers an immutable declarative manifest containing the stable +Definition name and version, the canonical Source JSON Schema, every projection key and output JSON Schema, and a +fingerprint over the complete declaration. The fingerprint is SHA-256 over RFC 8785 canonical JSON. Registration is +idempotent for an identical manifest and rejects a different declaration for an existing `(source_type, +definition_version)`. + +The Server validates the manifest's schemas and any named projection it recognizes as a shared standard. A manifest +does not transfer executable identity rules, canonicalization code, read behavior, credentials, or provider +configuration. Those remain worker-owned. The registered manifest is sufficient for the Server to validate and +retain an opaque canonical observation without loading plugin code. + +Definition discovery and registration are separate. A package entry point or another discovery mechanism may report +available Definitions, but installation does not imply activation. This RFC does not select entry points, a central +settings format, pluggy, or a Connector marketplace. + +## Remote worker ingestion contract + +A Connector runs in an independent worker process. The worker owns provider access and all executable Definition +behavior. The Server owns durable Source history, Artifact consumption, and checkpoint comparison. Their data-plane +interaction consists of four generic operations: + +1. register an immutable Source Definition manifest; +2. read the opaque checkpoint for one Connector binding; +3. submit a worker-materialized Source observation with all declared projections; and +4. compare-and-swap the binding checkpoint from the value read at run start. + +The normal sequence is: + +```text +Connector worker PowerContext Server + | | + |-- register Definition manifest -------------->| + |<---------------- exact registered manifest ---| + | | + |-- get binding checkpoint -------------------->| + |<-------------------------- checkpoint C0 -----| + | | + |-- submit observation 1 ---------------------->| + |<---------------- durable SourceRef receipt ---| + |-- submit observation 2 ---------------------->| + |<---------------- durable SourceRef receipt ---| + | | + |-- commit checkpoint expected=C0, next=C1 ---->| + |<-------------------------- committed C1 ------| +``` + +If the worker stops after a durable receipt but before the checkpoint commit, the next run starts from the earlier +checkpoint and may submit the observation again. Identical submission is idempotent. Checkpoint comparison prevents +two runs of the same binding from silently replacing each other's progress. + +The observation envelope carries the Definition name, version and fingerprint, canonical Source payload, and one +value for every projection declared by the manifest. The Server validates envelope identity, payload schema, +projection-key equality, projection schemas, and standard projection invariants before durable acceptance. Provider +names, storage services, paths, credentials, or other Connector-specific configuration do not appear in this API +unless a Definition deliberately includes them in its canonical Source schema. + +The Server returns a durable Source receipt before the worker may commit a checkpoint. The checkpoint operation uses +optimistic comparison so concurrent runs of the same binding cannot silently overwrite each other. Submission is +idempotent for an identical Source identity and payload; conflicting content for an accepted identity is rejected. + +## Definition compatibility contract + +The Definition name remains stable across compatible schema evolution. Each persisted observation records the +Definition version used to validate and canonicalize it. A newer Definition version must either declare how it reads +an older observation without changing its canonical meaning or coexist with a reader for the older version. + +A Definition change is incompatible when it changes SourceKey equality, observation equality, canonical value bytes, +provenance meaning, or materialization guarantees for an accepted observation. Such a change requires a new +Definition version and cannot rewrite existing SourceRefs. + +A projection change is incompatible when it changes the output schema, canonical bytes, or meaning for an accepted +observation. Such a change requires a new projection version. It does not require a new Source Definition version +when the Source value and observation semantics remain unchanged. + +Renaming a Definition creates a new `source_type`. Reclassifying an existing observation under another Definition is +an explicit derivation with provenance, not an in-place migration of identity. + +## Connector lifecycle contract + +A Connector owns provider interaction: discovery, credentials, filtering, checkpoints, retries, rate limits, +provider change handling, and positive deletion detection. It submits definition-native inputs against a Scope +binding and receives exact accepted SourceRefs. + +A Source Definition owns semantic normalization: logical identity, observation identity, canonical value, +provenance, materialization validity, and exact read. A Connector cannot override those rules. If the intersection of +provider capabilities, Connector behavior, and Definition requirements cannot satisfy a selected materialization, +the observation is rejected or captured under a valid mode. + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +A Connector type declares a stable name and version, its configuration schema, the Source Definitions it can submit, +and the acquisition capabilities it provides. Capabilities are optional and explicit. Typical capabilities include a +complete snapshot, a change feed, checkpoint resume, and authoritative deletion events. A Connector cannot advertise +a capability that its provider and acquisition path cannot enforce. + +A Connector binding activates one Connector configuration for exactly one Scope. The binding has a stable identity +for checkpoint and provider-namespace continuity, but it does not own Sources and does not replace `scope_id` or +`source_type`. Credentials are resolved by the hosting environment and do not become Source value or provenance. + +A Connector run begins from an opaque binding checkpoint, resolves zero or more definition-native inputs inside the +worker, and submits their materialized observations. It records an outcome for every item. An accepted or +idempotently replayed observation returns its exact SourceRef. A rejected or failed item remains visible in the run +outcome and cannot be hidden by advancing the checkpoint past work that is not safely replayable. + +A run finishes as complete or incomplete. A complete snapshot may produce positive deletion evidence for previously +known provider objects that are absent. An incomplete listing, timeout, permission failure, cancellation, or lost +connection produces no absence-based deletion evidence. An authoritative provider deletion event may produce +positive deletion evidence independently of snapshot completeness when its binding and object identity are verified. + +The completed checkpoint advances only after its accepted observations and deletion evidence are durable. Retrying +from an earlier checkpoint is valid because Source observation submission is idempotent. Connector checkpoint, +health, retry, and run-status records are operational state rather than Source observations or Artifact evidence. + +Installation, discovery, activation, and execution are separate concerns. Installing a Connector package does not +activate a binding. A Connector package executes outside the PowerContext Server and uses the remote worker +ingestion contract; scheduling and process supervision belong to the deployment environment. + +## Artifact evidence and cross-Scope delivery + +An Artifact revision records exact SourceRefs used directly by its computation. Advancing a Source head does not +change existing Artifact lineage. Recalculation against a newer observation produces a new Artifact revision rather +than rewriting prior evidence. + +An observation referenced by a durable Artifact revision is protected from ordinary retention and garbage +collection. Advancing or deleting a Source head does not authorize removing that observation. An explicit deletion +policy may make cited evidence unavailable, but it must preserve the SourceRef in lineage and report the +unavailability rather than resolve the reference to another observation. + +Sources remain in their producing Scope. A Context Reference may expand a read selection according to the Scope +organization contract, but it does not change Source ownership. Exact Artifact publication across Scopes retains the +origin Scope and exact SourceRef in lineage. Publishing an Artifact does not publish every Source in its origin Scope. + +If an application deliberately captures the same external value into another Scope, the target receives a new +Scope-owned Source observation. Its provenance may cite the origin Scoped SourceRef, but the original Source is not +moved and the two SourceKeys are not made identical. + +## Conformance + +A Source Definition can be supported only after its mandatory contract passes conformance scenarios for: + +- identity normalization and collision rejection; +- identical observation replay; +- conflicting payload rejection for one observation ID; +- several immutable observations under one SourceKey; +- exact old-observation reads after head advancement and deletion; +- digest verification for captured and referenced values; +- referenced-value unavailability and mutation; +- Scope isolation and explicit owner preservation; +- Definition version compatibility and unavailable-definition behavior; and +- explicit registration conflict handling. + +A named projection can be advertised only after conformance verifies deterministic output for exact observations, +schema and version conflict handling, exact SourceRef lineage, and explicit failure when the capability is absent. + +A Connector capability can be advertised only after conformance verifies checkpoint replay, per-item outcome +visibility, durable checkpoint ordering, complete-versus-incomplete run behavior, and the claimed deletion evidence. +Provider-specific behavior is established by its implementation evidence rather than generalized into the standard +contract. + +A remote worker path additionally verifies manifest fingerprint and conflict handling, rejection of unregistered or +schema-invalid observations, exact projection-set validation, durable receipt ordering, and stale checkpoint CAS +rejection across a Server restart. + +# Drawbacks + +- Separating SourceKey, SourceRef, Source head, and Definition version introduces more concepts than one immutable + `(source_type, source_id)` pair. +- Exact SourceRefs retain owner Scope and observation identity, increasing lineage payload size. +- Definition authors must specify canonicalization, provenance, and compatibility instead of relying on arbitrary + metadata. +- Named projections and Connector lifecycle state add contracts that must evolve independently from Source values. +- Referenced Sources are unavailable for providers that expose only current values, so some integrations must retain + captured data. +- Explicit registration requires deployment coordination before a custom Source observation can be accepted. + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +Adding provider fields to ContentSource would preserve the `POST /v1/sources/content` capture API, but it would keep logical identity, +observation identity, and provenance inside caller conventions. Different integrations would encode incompatible +schemas in metadata, and non-text Source values would still need another model. ContentSource remains a useful +single-observation implementation instead. + +## Use one opaque Source envelope + +A universal JSON payload would make persistence and transport uniform, but would move schema validation and +compatibility into runtime conventions. Definition-owned typed values and provenance make the extension boundary +reviewable and allow consumers to reject unsupported Source types before interpretation. + +## Put an observation digest inside source_id + +An integration can preserve the two-part SourceRef shape by composing logical identity and digest into `source_id`. This +makes immutable capture possible but hides the continuing logical Source from the catalog. Updates, current-head +selection, deletion, and provider identity then become integration-private conventions. The standard model represents +both identities directly. + +## Make SourceRef logical and add a separate ObservationRef + +Two public reference types would make SourceRef logical, but Artifact evidence would need +to reject SourceRef and accept only ObservationRef. Defining SourceRef itself as exact follows the existing ArtifactRef +principle that durable lineage references immutable state. + +## Add hybrid materialization + +A third mode that sometimes reads externally and sometimes falls back to captured data obscures which value is +authoritative and which failures are visible. A captured observation can retain a complete external reference as +provenance. A referenced observation either resolves exactly or fails. + +## Let Parent or Connector identity own Sources + +Scope Parent is organization, and Connector identity is acquisition provenance. Neither is a durable ownership +boundary. Using either would conflict with the Scope organization contract and would make reorganization or +Connector replacement change Source identity. + +# Prior art + +- The [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) separates + Scope ownership, read sharing, organization, delivery, and observation. This RFC applies the same separation to + Source ownership, identity, exact evidence, and acquisition. +- DataHub stateful ingestion separates connector checkpoints and stale-entity detection from emitted metadata + identity. Airbyte treats connector state as an opaque recovery boundary rather than record identity. +- OpenMetadata separates the Source that emits records from connection checks, workflow status, and the sink. +- Nowledge Mem's TiddlyWiki importer uses stable logical IDs, canonical payload digests, source revalidation, and + per-item outcomes. Those behaviors inform the separation between Source observations and Connector run state. +- [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory/tree/5299c00aaf65481703c180fd69df066d11254eb7) + uses a SourceFetcher registry for provider acquisition, provider revisions and content hashes for change detection, + and separate synchronization and audit state. Those patterns belong to Connector acquisition. They do not replace + immutable Source observations because an Artifact citation must retain the value it used after the provider's + current state changes. + +# Unresolved questions + +- Must every durable SourceRef carry `scope_id` directly, or may a canonical scoped envelope contain a local exact + SourceRef while preserving the same fully qualified identity? +- Which Source Definition versions must a Runtime retain simultaneously before a Definition can be considered + supported? +- Should Source head deletion be one common catalog state, or should the standard contract expose only an + active exact head and leave deletion entirely to Connector state? +- Which projection names and schemas have enough implementation evidence to become shared standards rather than + namespaced capabilities? + +# Future possibilities + +Explicit plugin discovery and deployment policy may build on Definition and Connector registration without making +package installation equivalent to activation. + +Retention policies may reclaim captured values only after defining how exact Artifact evidence reports unavailable +content and how legal or user-requested deletion interacts with immutable lineage. A Source head deletion alone does +not authorize evidence removal. diff --git a/docs/zh/docs/how-to/ingest-text-files-with-opendal.md b/docs/zh/docs/how-to/ingest-text-files-with-opendal.md new file mode 100644 index 000000000..74412ac70 --- /dev/null +++ b/docs/zh/docs/how-to/ingest-text-files-with-opendal.md @@ -0,0 +1,93 @@ +--- +title: 使用 OpenDAL 采集文本文件 +description: 用独立 OpenDAL Connector worker 把 UTF-8 文件捕获为类型化 Source。 +--- + +# 使用 OpenDAL 采集文本文件 + +`powercontext-connector-opendal` 是独立于 PowerContext Server 部署的 worker。它拥有 OpenDAL credential、 +provider configuration、可执行 Source Definition 和文件读取逻辑。Server 只保存声明式 Definition manifest、 +已经物化的 Source observation、named projection 与 opaque checkpoint。 + +## 前置条件 + +该集成要求 Python 3.12 或更高版本。先启动 PowerContext Server，再从 checkout 安装 worker： + +```bash +uv tool install ./integrations/opendal +``` + +选择稳定的 `source_namespace` 来区分不同 storage authority。不要把 credential 写进 namespace、Source payload 或 +checkpoint。Server 启用 authentication 时，通过 `POWERCONTEXT_TOKEN` 环境变量提供 bearer token。 + +## 运行一个 binding + +下面的独立进程扫描 `/absolute/path/to/project/docs`。`binding_id` 标识 checkpoint continuity，`scope_id` 决定 +接受后的 Source 属于哪个 Scope： + +```bash +powercontext-connector-opendal \ + --base-url http://127.0.0.1:8765 \ + --scope-id project:example \ + --binding-id project-docs \ + --service fs \ + --storage-option root=/absolute/path/to/project \ + --root docs \ + --source-namespace project-docs +``` + +访问远端存储时，替换 OpenDAL service 与对应的 `--storage-option KEY=VALUE`。这些 option 只存在于 worker 进程， +不会通过摄取 API 发送给 Server。 + +Worker 每次运行都会幂等注册 `text-file-snapshot` Definition manifest，读取 binding checkpoint，提交本轮变化的 +Source observation，并在所有 durable receipt 返回后 compare-and-swap checkpoint。可以由 cron、Kubernetes Job +或其他外部 scheduler 周期执行该命令。 + +## 嵌入自定义 worker + +需要自定义进程监管或多 binding 调度时，可直接使用通用远程 lifecycle： + +```python +from powercontext.client import PowerContextClient, RemoteConnectorWorker +from powercontext.sources import ConnectorBinding, SourceDefinitionRegistry +from powercontext_connector_opendal import ( + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + OpenDALTextFileConnector, +) + +connector = OpenDALTextFileConnector.from_service( + "fs", + source_namespace="project-docs", + root="docs", + storage_options={"root": "/absolute/path/to/project"}, +) +binding = ConnectorBinding( + scope_id="project:example", + binding_id="project-docs", + connector_name=connector.name, + connector_version=connector.version, +) +registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + +async with PowerContextClient("http://127.0.0.1:8765") as client: + result = await RemoteConnectorWorker(client=client, registry=registry).run(connector, binding) +``` + +## 运行语义 + +每个 item outcome 是 `accepted`、`replayed`、`rejected` 或 `failed`。只有本轮完整结束并且没有 rejected 或 failed +item 时 checkpoint 才会前移。否则保留旧 checkpoint，下一轮从同一位置安全重试。与已提交 checkpoint 中 digest +相同的文件会被跳过。 + +接受的 Source 进入目标 Scope 的 Source journal。Worker 同时计算标准 `powercontext.text-evidence` projection， +因此不了解 `text-file-snapshot` native schema 的 Memory consumer 仍可消费文本。Connector run 不直接创建 Memory； +Memory 仍由常规 source-window flush 或调度任务生成。 + +## 限制 + +- 默认选择 Markdown、纯文本、reStructuredText 与 AsciiDoc 文件。 +- 默认每轮最多选择 10,000 个文件，每个文件最多读取 2 MiB。 +- 只接受 UTF-8 内容。 +- 内容变化会生成新的精确 snapshot Source，旧 snapshot 继续保留。 +- 全量扫描会从下一 checkpoint 移除已消失 path，但不会删除 Source，也不声明 authoritative deletion。 +- Connector 不提供 change feed；后续变化依赖外部 scheduler 再次运行 worker。 diff --git a/docs/zh/rfcs/0000_source_definition_and_observation_model.md b/docs/zh/rfcs/0000_source_definition_and_observation_model.md new file mode 100644 index 000000000..bfca3ab92 --- /dev/null +++ b/docs/zh/rfcs/0000_source_definition_and_observation_model.md @@ -0,0 +1,619 @@ +- Proposal Name: `source_definition_and_observation_model` +- Start Date: 2026-08-27 +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md)、[RFC 0014](0014_memory_layer_design.md)、 + [RFC 0019](0019_local_source_memory_runtime.md)、[RFC 0048](0048_handoff_artifact.md) + +# Summary + +本 RFC 定义标准 Source 模型，以及新增 Source 类型时必须遵守的契约。 + +每个 Source 只属于一个 Scope。在该 Scope 内，`SourceKey` 标识一个逻辑 Source，`SourceRef` 标识这个 +Source 的一次不可变观察。推进当前观察、观察到删除、修改外部 locator 或断开 Connector，都不会改变 +已经接受的观察，也不会将其移动到另一个 Scope。 + +Source Definition 为一个稳定的 Source 类型定义 value schema、provenance schema、身份规则、观察规则、 +materialization 契约、canonicalization 与兼容策略。Definition 显式注册，并在组合完成的 Runtime 生命周期内 +保持不变。持久化、传输与 Artifact consumer 按稳定的 Definition 名称和版本路由，而不是按具体 Python 类路由。 + +Definition 可以为无法理解 native value 的 consumer 声明 named projection capability。每个 projection 拥有独立 +版本的 schema，并对一个精确 observation 具有确定语义。Consumer 按 capability name 与 version 选择 projection， +而不是检查具体 Source class。 + +Connector lifecycle 将 provider acquisition 绑定到 Scope，在 worker 内解析 definition-native input，提交 +materialized observation，记录 per-item outcome，并且只在接受的 observation 已持久化后推进 opaque checkpoint。 +Connector run 区分 complete discovery 与 incomplete discovery，避免把缺失对象静默转换为删除。 + +Materialization 表达解析某个精确观察时所依赖的权威来源。Captured observation 从 PowerContext 保留的 +canonical value 解析；referenced observation 从外部不可变 revision 解析。仅有外部 locator、修改时间、 +ETag 或 provider 当前值读取，并不能满足 referenced 契约。 + +`ContentSource` 继续作为简单的 captured-text Source。调用方提供稳定身份，加上 immutable-payload 冲突规则， +适合一次性内容捕获，但它不是通用的外部集成模型。 + +本 RFC 定义 Source、projection、Connector lifecycle，以及 worker 与 PowerContext Server 之间的远程摄取边界。 +它不定义插件发现、scheduler、credential transport、具体 Source family 或 Connector 实现。 + +# Motivation + +`ContentSource` 与 `POST /v1/sources/content` 提供 captured-text ingestion。调用方选择一个 +`source_id`；使用完全相同的 payload 重放具有幂等性，而用不同 payload 复用该身份会产生冲突。只有调用方把 +这个身份当作不可变身份时，它才能表达精确证据。 + +外部系统通常具有不同的生命周期。Wiki 页面、issue、object、message 或 file 拥有一个逻辑身份，但会随时间 +产生多个值。外部对象可能被重命名、修订、删除、恢复或暂时无法读取。使用过旧值的 Artifact 必须继续引用当时的 +精确证据。二元 `(source_type, source_id)` Source reference 无法同时表达稳定的逻辑对象和不可变观察，只能迫使每个集成自行发明复合 +`source_id`。 + +例如，只使用 provider object ID 时，第二个 value 会与第一个冲突或替换它。只使用 value digest 虽然能保留 +两个 value，却无法表达它们来自同一个持续存在的对象： + +```text +provider object 42 + | + +-- value v1 ----> exact observation 1 + `-- value v2 ----> exact observation 2 + ^ + | + same logical Source +``` + +因此，本模型分别保存 logical identity 与 exact evidence。需要 current state 的 consumer 可以沿着同一个 +logical Source 读取，而 Artifact 继续引用它实际使用的 observation。 + +扩展边界也不完整。Source adapter 将 native input class 绑定到具体 Source class 和读取结果，而内置 +Runtime 与关系型持久化会组装固定 adapter 集合。它没有说明独立定义的 Source 类型在身份、持久化、传输与 +Artifact evidence 上必须长期满足哪些规则。 + +标准模型必须回答六个问题，且不能把它们压进同一个 identifier： + +1. 哪个 Scope 拥有这份证据？ +2. 它描述哪个逻辑上的外部或内部 Source？ +3. Artifact 使用的是哪个精确观察值？ +4. PowerContext 从哪里读取该精确值？ +5. 哪个 Definition 赋予 value 与 provenance 语义？ +6. Consumer 可以使用哪个 declared view，而不必理解 native value？ + +Connector concerns 与此相邻但不同。Discovery、credentials、filtering、checkpoints、retries、provider +change handling 与 deletion detection 决定提交哪些观察；它们不定义 Source identity，不能削弱精确证据， +也不能改变 Scope ownership。 + +# Guide-level explanation + +## Domain model + +理解该模型时，依次确定 ownership、logical identity、exact observation、materialization authority 与 type +semantics： + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Source 属于哪里？ | +| Logical identity | `SourceKey` | 这是哪个持续存在的 Source？ | +| Exact evidence | `SourceRef` | 引用的是哪个不可变观察？ | +| Read authority | materialization | 从哪里解析该精确值？ | +| Type semantics | Source Definition | 如何解释 value、provenance 与 identity？ | +| Consumer view | named projection | Consumer 可以使用哪个 declared representation？ | +| Acquisition | Connector or direct caller | 如何发现并提交新观察？ | + +这些职责形成单向依赖： + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +一个 Connector 可以使用一个 Source Definition，多个 Connector 可以共用同一个 Definition，直接调用方也可以 +在没有 Connector 的情况下提交 Source。因此 Connector identity 不会成为 Source type identity。 + +## Scope ownership + +每个 SourceKey 与 observation 都只属于一个 Scope。Scope ownership 不从外部 workspace、path、repository、 +provider account、Connector instance 或 Source locator 推导。这些值可以参与 binding 或 provenance，但不能 +分配或替代 `scope_id`。 + +完整限定的逻辑身份为： + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +完整限定的精确身份为： + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +Scope-bound operation 可以从固定 request binding 获得 `scope_id`，而不把它作为任意参数接收。持久化的解析结果 +仍然保留 owner Scope，使证据在 publication、reporting 或 export 后仍无歧义。 + +修改 Scope Parent、Context References、Agent binding 或 observation selection，都不会改变 SourceKey 或 +SourceRef。跨 Scope 发布 Artifact 时，provenance 保留原始 Scope 与精确 SourceRef；不会移动或隐式复制 +Source history。 + +## Logical Source and immutable observation + +`source_id` 在一个 `(scope_id, source_type)` namespace 内命名逻辑 Source，其含义由 Source Definition +规定。它可以对应 provider object ID、稳定 import identity 或其他 normalized key。观察到新值时，它不能静默改变。 + +`observation_id` 在一个 SourceKey 下命名一次不可变观察。对通用 PowerContext component 而言它是不透明的； +可以派生自 provider revision、canonical value digest 或 Definition 特有组合。它不隐含整数序列、时间顺序或祖先关系。 + +适用以下不变量： + +- 一个 `(SourceKey, observation_id)` 永远标识同一个 canonical observation； +- 再次观察相同 canonical observation 具有幂等性； +- 不同 canonical observation 不能复用 observation ID； +- 如果 identity-bearing provenance 不同，同一个 SourceKey 下可以有 value digest 相同的多个 observation； +- value digest 相同的 observation 不会自动成为同一个逻辑 Source； +- Artifact 只引用精确 SourceRef，绝不引用移动的 SourceKey 或 `latest` observation。 + +例如，更新一个逻辑 Source 会保留其 SourceKey，并产生新的 SourceRef： + +```text +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" +``` + +即使 `observation-2` 已成为 current，派生自 `observation-1` 的 Artifact 仍然引用后者。 + +## Source Definition + +Source Definition 是一个 `source_type` 的持久语义契约。它声明： + +- 稳定的 Definition name 与 version； +- Source value 与 typed provenance 的结构； +- Source ID normalization 与 equality； +- observation ID normalization 与 equality； +- identity-bearing fields 与 non-identifying annotations； +- canonical bytes 与 value digest algorithm； +- 支持的 materialization modes 与 exact-read requirements； +- limits 与 validation failures； +- older Definition versions 的 compatibility rules。 + +Definition 将 definition-native input 解析为 canonical observation，并从精确的 persisted observation 读取 +Definition 拥有的 value。解析不会选择 Scope、修改 catalog、推进 head 或发现外部 object；读取不会解析 +`latest`，也不会替换为另一个 observation。 + +Definition 必须显式且类型化。新的集成不能通过在 `ContentSource.metadata` 中放置未声明 schema 来模拟新 +Source 类型。Provider-specific provenance 可以扩展 Definition 声明的 schema，但影响 identity、exactness +或 compatibility 的字段必须由 Definition 命名。 + +## Named projection capabilities + +Named projection 是一个 exact observation 的可选 Definition-owned view。它让 Artifact family 或其他 consumer +无需理解 native Source value 或具体 Python class，就能使用声明过的 representation。 + +Projection 通过稳定的 name 与 version 选择。其 Definition 声明 output schema、canonicalization、digest rules +与 failures。Projection 针对精确 SourceRef 求值，不能解析 head、`latest` 或 provider current value。对于相同的 +Definition version、projection version 与 exact observation，它必须返回相同的 canonical result。 + +Projection capability 必须显式声明。需要某个 projection 的 consumer 会拒绝未声明兼容 capability 的 Source， +而不会从 metadata 推断 content，也不会回退到形态相似的 Source class。Projection 可以作为 derivative 被缓存或 +持久化，但其 authority 仍是 exact Source observation，lineage 保留对应 SourceRef。 + +本契约不规定标准 projection name 或 payload schema 的目录。只有当多个 Definition 与 consumer 的互操作证明其 +语义稳定后，projection 才成为 shared standard。在此之前，Definition 可以暴露 namespaced projection，但不会让 +它成为其他 Source type 的 mandatory capability。 + +## Materialization authority + +Materialization 回答精确 SourceRef 的返回值来自哪里： + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | PowerContext 保留的 canonical value | 保留值与 observation digest 一致 | +| `referenced` | Immutable external revision | 重读 reference 得到相同 canonical value 与 digest | + +Captured Source 可以把 external locator、provider revision 与 digest 保留为 provenance。因为读取权威仍是 +保留值，所以它依然是 captured。这覆盖了 hybrid design 中有价值的部分，而不引入 fallback 语义含糊的第三种模式。 + +只有当外部系统及其 reader 能够寻址不可变历史值时，Definition 才能使用 referenced materialization。读取 +path、page ID、issue ID 或 URL 的当前值并不足够。Modification time 与 ETag 可以参与 provenance 或 conflict +detection，但 Definition 必须说明 provider 是否保证它们指向不可变值。 + +Referenced value 不可用或 digest 不同时，精确解析失败。PowerContext 不返回 provider 当前值、stale cache +entry 或其他 observation。不能满足该规则的 provider 必须使用 captured materialization，或者拒绝该 observation。 + +## Current head and deletion + +Source history 不可变；current head 是可变的 catalog selection。Head 可以选择一个精确 SourceRef，或记录已 +明确观察到逻辑 Source 被删除。Head 可用于 current-state query 与后续 acquisition，但它不是 evidence，不能 +出现在 Artifact citation 中。 + +推进或删除 head 不改变任何 observation。Timeout、permission failure、incomplete listing、Connector +unavailable 或 disconnect 都不是明确的 deletion evidence，不能改变 head。只有当 deletion 本身是有意义的 +Source evidence 时，Source Definition 才可以定义 tombstone value；通用 head deletion 不会伪造这种值。 + +## ContentSource + +`ContentSource` 继续作为 RFC 0019 定义的 neutral captured-text path。调用方选择一个只能与一个 canonical +payload 一起提交的身份。Persistence conflict rule 使接受后的 ContentSource 可作为精确证据，但它不提供独立的 +logical Source lifecycle。 + +标准模型把它视为有效的 single-observation Source： + +- 现有 identity 保持不可变； +- 相同内容重放继续保持幂等； +- 同一 identity 下的不同内容继续发生冲突； +- 解析 ContentSource 的 reference 保持精确且不变； +- 不从 metadata 推导 mutable head 或 multi-observation behavior。 + +ContentSource 适合 prompt、显式文本捕获、import record，以及调用方已经拥有不可变身份的其他场景。持续观察同一 +逻辑对象的集成应定义或复用 multi-observation Source type。 + +两条 ingestion 路径的 acquisition 方式不同，但最终都进入 Scope-owned Source history： + +| Concern | `ContentSource` capture | Source Definition 与 Connector ingestion | +| --- | --- | --- | +| Typical input | 调用方已经持有的文本 | 从外部系统发现的对象 | +| Identity | 一个由调用方保持稳定的不可变身份 | 一个 logical identity 及其 exact observations | +| Type contract | 内置 captured text 与 metadata | Definition-owned value、provenance 与 projections | +| Synchronization | 单次请求，没有 checkpoint | Discovery、per-item outcomes、replay 与 checkpoint comparison | +| Downstream use | 内置 text evidence | Consumer 能理解的 named projection | + +当调用方已经持有最终文本和不可变身份时，`ContentSource` 是更短的路径。Remote ingestion API 不替代它； +这组 API 用于 worker 需要发现、规范化并重复观察外部对象的场景，同时避免把 provider code 或 credentials +加载进 Server。 + +## Source 如何参与 Scope 流程 + +Server 持久化接受 exact observation 后，会将它追加到 owner Scope 的 Source journal。接受 observation +本身不会创建 Memory 或其他 Artifact。Scope-local processor 随后选择一个有界 Source window，请求它能理解的 +projection，并可能产生一个引用 exact SourceRef 的新 Artifact revision： + +```text +Connector or direct caller + | + | bind Scope A + v +Source observation + | + v +Scope A Source journal ----> Scope-local processor + | + named projection + | + v + Scope A Memory revision + cites exact SourceRef +``` + +新的 observation 可以触发后续处理，但不会重写旧 Artifact revision： + +```text +SourceKey(scope-a, record, provider-object-42) +|-- observation-1 ----> Memory revision 3 +`-- observation-2 ----> Memory revision 4 + +Memory revision 3 continues to cite observation-1. +``` + +Consumer 只能通过 native Definition 或兼容的 named projection 使用 Source。例如，需要 text evidence 的 +Memory extractor 可以处理任何声明了对应 text projection 的 Source Definition，不需要知道 Source 最初来自 +file、page、issue 还是 `ContentSource`。缺失的 capability 必须保持显式，consumer 不会从 metadata 推断文本。 + +跨 Scope 使用 Source 时，需要先确定预期的 ownership 与 delivery 行为： + +```text +Scope A Source history + | + +-- Context Reference from Scope B + | `-- later Prepare Context may read eligible Scope A material + | + +-- publish exact Artifact revision + | `-- Scope B receives one selected result with origin provenance + | + `-- deliberate capture into Scope B + `-- Scope B owns a new Source and runs its own downstream flow +``` + +持续读取使用 Context Reference；交付一个选定结果时，发布 exact Artifact revision。如果 Scope B 必须拥有并 +独立处理这个外部值，应在 Scope B 中显式 capture，形成新的 Scope-owned observation，并在适用时把 origin +reference 保留到 provenance。以上操作都不会移动原始 Source，Parent 也不会因此获得 read access。 + +# Reference-level explanation + +## Source identity contract + +`scope_id` 是 Scope organization design 定义的 ownership boundary。`source_type` 是稳定的 Source Definition +name。`source_id` 是非空的 normalized identifier，其 equality 与 bounds 由 Definition 声明。 + +Source identity 以 Scope 为本地边界。两个 Scope 可以包含等价的外部材料，但不共享 ownership 或 identity。 +需要避免碰撞时，Definition 可以在 `source_id` 规则中包含稳定的 external instance 或 connection discriminator， +但 discriminator 不替代 `scope_id`。 + +Rename 行为由 Definition 决定。Provider object ID 可以在 locator 变化时保留 SourceKey；path-derived identity +通常把 rename 视为一次逻辑 deletion 与一次 creation。当 provider 与 acquisition path 无法证明 rename-stable +identity 时，Definition 不能宣称支持它。 + +## Observation contract + +Observation 包含以下标准字段： + +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` + +`value_digest` 对 Definition 声明的 canonical bytes 使用 SHA-256，并编码为 `sha256:`。对结构化 +value，Definition 指定 deterministic canonicalization。Digest 用于验证 value equality，不替代 SourceKey +或 observation identity。 + +Canonical observation 包含所有被 Definition 认定会影响 identity 或 exact meaning 的字段。Retry count、 +last scan time 或 processing status 等 operational facts 不是 Source value，不改变 observation identity。 +如果 timestamp 或 provider attribute 会影响 provenance meaning，Definition 必须显式分类并 canonicalize。 + +## Source reference contract + +SourceRef 标识精确 observation，并包含 owner Scope。它不接受缺失 observation ID、`latest`、head version 或 +current provider locator。 + +在 scope-bound operation 内，只有当 current Scope 固定且解析出的 durable value 会恢复 `scope_id` 时，紧凑的 +local representation 才可以省略重复的 `scope_id`。跨越 Scope boundary、离开 Runtime 或进入 durable +cross-Scope provenance 的 reference 必须显式携带 owner Scope。 + +Reference resolution 会验证全部四个 identity components，以及 stored observation 的 Definition version 与 +digest。无法解析精确 observation，不等同于 logical Source 已删除、head 已推进或 Connector 不可用。 + +不带 `observation_id` 的已接受兼容引用仍然标识一个不可变 observation。解析时不得将其视为 SourceKey、current +head 或 `latest`。compatibility layer 可以在边界恢复完整 SourceRef，但不得重定向该 evidence。 + +## Definition registration contract + +Executable Definition 属于 worker。Worker 用它解析 definition-native input、canonicalize Source value，并计算 +named projection。Server 不导入 Connector 或 Definition package，也不执行其中的 Python class。 + +提交 observation 前，worker 注册不可变的声明式 manifest。Manifest 包含稳定的 Definition name 与 version、 +canonical Source JSON Schema、每个 projection key 与 output JSON Schema，以及覆盖完整声明的 fingerprint。 +Fingerprint 是 RFC 8785 canonical JSON 的 SHA-256。相同 manifest 的注册是幂等的；同一个 +`(source_type, definition_version)` 对应不同声明时必须拒绝。 + +Server 验证 manifest schema，以及其识别为 shared standard 的 named projection。Manifest 不传输可执行的 identity +rule、canonicalization code、read behavior、credential 或 provider configuration；这些仍由 worker 持有。 +注册后的 manifest 足以让 Server 在不加载 plugin code 的情况下验证并保存 opaque canonical observation。 + +Definition discovery 与 registration 相互独立。Package entry point 或其他 discovery mechanism 可以报告 +可用 Definition，但安装不意味着激活。本 RFC 不选择 entry points、central settings format、pluggy 或 +Connector marketplace。 + +## Remote worker ingestion contract + +Connector 在独立 worker 进程中运行。Worker 拥有 provider access 与所有 executable Definition behavior；Server +拥有 durable Source history、Artifact consumption 与 checkpoint comparison。双方的数据面交互只有四个通用操作： + +1. 注册不可变的 Source Definition manifest； +2. 读取一个 Connector binding 的 opaque checkpoint； +3. 提交 worker 已物化的 Source observation 及其全部声明 projection； +4. 从 run 开始时读到的值 compare-and-swap binding checkpoint。 + +正常时序如下： + +```text +Connector worker PowerContext Server + | | + |-- register Definition manifest -------------->| + |<---------------- exact registered manifest ---| + | | + |-- get binding checkpoint -------------------->| + |<-------------------------- checkpoint C0 -----| + | | + |-- submit observation 1 ---------------------->| + |<---------------- durable SourceRef receipt ---| + |-- submit observation 2 ---------------------->| + |<---------------- durable SourceRef receipt ---| + | | + |-- commit checkpoint expected=C0, next=C1 ---->| + |<-------------------------- committed C1 ------| +``` + +如果 worker 在收到 durable receipt 后、提交 checkpoint 前停止，下次 run 会从较早的 checkpoint 开始，并可能 +再次提交同一个 observation。相同提交具有幂等性。Checkpoint comparison 会阻止同一 binding 的两个 run +静默覆盖彼此的进度。 + +Observation envelope 携带 Definition name、version 与 fingerprint、canonical Source payload，以及 manifest 声明的 +每个 projection value。Server 在 durable acceptance 前验证 envelope identity、payload schema、projection key +集合相等、projection schema 与标准 projection invariant。Provider name、storage service、path、credential 或其他 +Connector-specific configuration 不出现在该 API 中；只有 Definition 刻意将其声明为 canonical Source schema 的 +一部分时才例外。 + +Server 必须先返回 durable Source receipt，worker 才能提交 checkpoint。Checkpoint operation 使用 optimistic +comparison，防止同一 binding 的并发 run 静默覆盖。相同 Source identity 与 payload 的提交是幂等的；已接受 identity +对应不同内容时必须拒绝。 + +## Definition compatibility contract + +Definition name 在兼容 schema 演进中保持稳定。每个 persisted observation 记录验证和 canonicalize 它时使用的 +Definition version。新的 Definition version 必须声明如何在不改变 canonical meaning 的前提下读取旧 observation， +或与旧版本 reader 共存。 + +如果 Definition change 会改变已接受 observation 的 SourceKey equality、observation equality、canonical value +bytes、provenance meaning 或 materialization guarantee，它就是不兼容变更。此类变更需要新的 Definition version， +且不能重写已有 SourceRef。 + +如果 projection change 会改变已接受 observation 的 output schema、canonical bytes 或 meaning，它就是不兼容 +变更，需要新的 projection version。如果 Source value 与 observation semantics 保持不变，则不要求新的 Source +Definition version。 + +重命名 Definition 会产生新的 `source_type`。把已有 observation 重新分类到另一个 Definition 是带 provenance +的显式 derivation，不是 identity 的原地 migration。 + +## Connector lifecycle contract + +Connector 负责 provider interaction：discovery、credentials、filtering、checkpoints、retries、rate limits、 +provider change handling 与 positive deletion detection。它依据 Scope binding 提交 definition-native input， +并接收接受后的精确 SourceRef。 + +Source Definition 负责 semantic normalization：logical identity、observation identity、canonical value、 +provenance、materialization validity 与 exact read。Connector 不能覆盖这些规则。如果 provider capabilities、 +Connector behavior 与 Definition requirements 的交集无法满足选定 materialization，则拒绝 observation，或在 +合法模式下 captured。 + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +Connector type 声明稳定的 name 与 version、configuration schema、可提交的 Source Definition，以及它提供的 +acquisition capability。Capability 是可选且显式的，通常包括 complete snapshot、change feed、checkpoint resume +和 authoritative deletion event。Connector 不能声明 provider 与 acquisition path 无法兑现的 capability。 + +Connector binding 为一个 Scope 激活一份 Connector configuration。Binding 拥有用于 checkpoint 与 provider +namespace continuity 的稳定 identity，但不拥有 Source，也不替代 `scope_id` 或 `source_type`。Credential 由 +hosting environment 解析，不会成为 Source value 或 provenance。 + +Connector run 从 opaque binding checkpoint 开始，在 worker 内解析零个或多个 definition-native input，再提交其 +materialized observation，并记录每个 item 的 outcome。Accepted 或 idempotently replayed observation 返回精确 +SourceRef。Rejected 或 failed item 会保留在 run outcome 中；如果尚不能安全重放，checkpoint 不能越过这些工作。 + +Run 以 complete 或 incomplete 结束。Complete snapshot 可以为之前已知但本次缺失的 provider object 产生 positive +deletion evidence。Incomplete listing、timeout、permission failure、cancellation 或 lost connection 不会产生 +absence-based deletion evidence。当 binding 与 object identity 均已验证时，authoritative provider deletion event +可以独立于 snapshot completeness 产生 positive deletion evidence。 + +Completed checkpoint 只有在 accepted observation 与 deletion evidence 均已持久化后才能推进。由于 Source +observation submission 具有幂等性，从更早 checkpoint 重试是合法行为。Connector checkpoint、health、retry 与 +run-status record 是 operational state，而不是 Source observation 或 Artifact evidence。 + +Installation、discovery、activation 与 execution 相互独立。安装 Connector package 不会激活 binding。Connector +package 在 PowerContext Server 之外执行，并使用 remote worker ingestion contract；调度与进程监管属于部署环境。 + +## Artifact evidence and cross-Scope delivery + +Artifact revision 记录其计算直接使用的精确 SourceRef。推进 Source head 不改变现有 Artifact lineage。针对较新 +observation 的重新计算会产生新的 Artifact revision，而不是重写旧 evidence。 + +被 durable Artifact revision 引用的 observation 受普通 retention 与 garbage collection 保护。推进或删除 Source +head 不会授权删除该 observation。显式 deletion policy 可以使被引用的 evidence 不再可用，但必须在 lineage 中 +保留 SourceRef 并报告不可用状态，不能把该引用解析到另一个 observation。 + +Source 保留在 producing Scope。Context Reference 可以按照 Scope organization contract 扩展 read selection， +但不会改变 Source ownership。跨 Scope 的精确 Artifact publication 在 lineage 中保留 origin Scope 与精确 +SourceRef。发布 Artifact 不会发布其 origin Scope 中的所有 Source。 + +如果 application 刻意把同一个外部值 captured 到另一个 Scope，target 会得到由该 Scope 拥有的新 Source +observation。其 provenance 可以引用 origin scoped SourceRef，但原始 Source 不会移动，两个 SourceKey 也不会 +因此变成同一 identity。 + +## Conformance + +Source Definition 只有在以下 mandatory contract 的 conformance scenario 通过后才能被支持： + +- identity normalization 与 collision rejection； +- identical observation replay； +- 同一 observation ID 的 conflicting payload rejection； +- 一个 SourceKey 下的多个 immutable observation； +- head advancement 与 deletion 后仍能精确读取旧 observation； +- captured 与 referenced value 的 digest verification； +- referenced-value unavailability 与 mutation； +- Scope isolation 与显式 owner preservation； +- Definition version compatibility 与 unavailable-definition behavior； +- explicit registration conflict handling。 + +Named projection 只有在 conformance 验证 exact observation 的 deterministic output、schema 与 version conflict +handling、exact SourceRef lineage，以及 capability 缺失时显式失败之后才能被声明。 + +Connector capability 只有在 conformance 验证 checkpoint replay、per-item outcome visibility、durable checkpoint +ordering、complete-versus-incomplete run behavior，以及其声明的 deletion evidence 后才能被声明。Provider-specific +behavior 由对应实现证据确定，不会被直接推广为标准契约。 + +Remote worker path 还必须验证 manifest fingerprint 与 conflict handling、拒绝未注册或 schema-invalid observation、 +projection set 精确校验、durable receipt ordering，以及跨 Server restart 的 stale checkpoint CAS rejection。 + +# Drawbacks + +- 分离 SourceKey、SourceRef、Source head 与 Definition version，比一个不可变的 `(source_type, source_id)` pair + 引入更多概念。 +- 精确 SourceRef 保留 owner Scope 与 observation identity，会增加 lineage payload 大小。 +- Definition author 必须声明 canonicalization、provenance 与 compatibility，而不能依赖任意 metadata。 +- Named projection 与 Connector lifecycle state 增加了需要独立于 Source value 演进的契约。 +- 只暴露当前值的 provider 无法使用 Referenced Source，因此部分集成必须保留 captured data。 +- Custom Source observation 被接受之前，显式 registration 需要部署协调。 + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +向 ContentSource 添加 provider fields 可以保留 `POST /v1/sources/content` capture API，但仍会把 logical identity、observation identity +与 provenance 留在调用方约定中。不同集成会在 metadata 中编码不兼容 schema，non-text Source value 仍需要另一 +套模型。因此 ContentSource 继续作为有用的 single-observation implementation。 + +## Use one opaque Source envelope + +通用 JSON payload 可以统一 persistence 与 transport，但会把 schema validation 和 compatibility 推给 runtime +convention。Definition-owned typed value 与 provenance 让扩展边界可审查，并允许 consumer 在解释前拒绝不支持的 +Source type。 + +## Put an observation digest inside source_id + +集成可以把 logical identity 与 digest 组合进 `source_id`，从而维持二元 SourceRef 形态。这可以表达 immutable +capture，却会在 catalog 中隐藏持续存在的 logical Source。Update、current-head selection、deletion 与 provider +identity 都会变成 integration-private convention。标准模型直接表达两类 identity。 + +## Make SourceRef logical and add a separate ObservationRef + +两个 public reference type 可以让 SourceRef 表示逻辑身份，但 Artifact evidence 必须拒绝 SourceRef，只接受 +ObservationRef。让 SourceRef 本身保持精确，符合现有 ArtifactRef 原则：durable lineage 引用 immutable state。 + +## Add hybrid materialization + +增加一种有时从外部读取、有时回退到 captured data 的第三种模式，会掩盖哪个 value 才是 authoritative，以及哪些 +failure 应对外可见。Captured observation 可以把完整 external reference 保留为 provenance；referenced +observation 要么被精确解析，要么失败。 + +## Let Parent or Connector identity own Sources + +Scope Parent 用于 organization，Connector identity 是 acquisition provenance，二者都不是持久 ownership +boundary。使用其中任意一个都会与 Scope organization contract 冲突，并让 reorganization 或 Connector +replacement 改变 Source identity。 + +# Prior art + +- [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) 分离 + Scope ownership、read sharing、organization、delivery 与 observation。本 RFC 对 Source ownership、identity、 + exact evidence 与 acquisition 应用同样的分离原则。 +- DataHub stateful ingestion 把 connector checkpoint 与 stale-entity detection 同 emitted metadata identity + 分离。Airbyte 把 connector state 当作 opaque recovery boundary，而不是 record identity。 +- OpenMetadata 把负责生成 record 的 Source 与 connection check、workflow status、sink 分离。 +- Nowledge Mem 的 TiddlyWiki importer 使用 stable logical ID、canonical payload digest、source revalidation 与 + per-item outcome。这些行为为 Source observation 与 Connector run state 的分离提供依据。 +- [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory/tree/5299c00aaf65481703c180fd69df066d11254eb7) + 使用 SourceFetcher registry 获取 provider value，以 provider revision 与 content hash 检测变化，并单独维护 + synchronization 和 audit state。这些模式属于 Connector acquisition，不能替代 immutable Source observation， + 因为 Artifact citation 必须在 provider current state 变化后仍然保留它使用过的值。 + +# Unresolved questions + +- 每个 durable SourceRef 是否必须直接携带 `scope_id`，还是可以由 canonical scoped envelope 包含 local exact + SourceRef，同时保留相同的 fully qualified identity？ +- Runtime 必须同时保留哪些 Source Definition version，才能宣称某个 Definition 受支持？ +- Source head deletion 应是通用 catalog state，还是标准契约只暴露 active exact head，并把 deletion 完全留给 + Connector state？ +- 哪些 projection name 与 schema 已有足够实现证据，可以成为 shared standard 而不是 namespaced capability？ + +# Future possibilities + +显式 plugin discovery 与 deployment policy 可以建立在 Definition 和 Connector registration 之上，但不会让 +package installation 等同于 activation。 + +Retention policy 只有在定义精确 Artifact evidence 如何报告 unavailable content，以及 legal/user-requested +deletion 如何与 immutable lineage 交互之后，才能回收 captured value。Source head deletion 本身不授权删除证据。 diff --git a/e2e/bub/uv.lock b/e2e/bub/uv.lock index 3a361263c..6e3268e99 100644 --- a/e2e/bub/uv.lock +++ b/e2e/bub/uv.lock @@ -1627,6 +1627,9 @@ requires-dist = [ { name = "httpx", extras = ["socks"], marker = "extra == 'client'", specifier = ">=0.28,<1" }, { name = "inquirerpy", marker = "extra == 'cli'", specifier = ">=0.3,<1" }, { name = "jinja2", marker = "extra == 'server'", specifier = ">=3.1,<4" }, + { name = "jsonschema", marker = "extra == 'builtin'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'seekdb'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'server'", specifier = ">=4.23,<5" }, { name = "opentelemetry-api", marker = "extra == 'cli'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'client'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'server'", specifier = ">=1.30,<2" }, @@ -1697,7 +1700,7 @@ dependencies = [ requires-dist = [ { name = "bub", specifier = ">=0.4.0,<0.5.0" }, { name = "httpx", specifier = ">=0.28,<1" }, - { name = "powercontext", extras = ["client"], specifier = ">=0.0.3" }, + { name = "powercontext", extras = ["client"] }, { name = "pydantic-settings", specifier = ">=2.7,<3" }, ] diff --git a/integrations/bub/pyproject.toml b/integrations/bub/pyproject.toml index 4a3f36128..8063d100f 100644 --- a/integrations/bub/pyproject.toml +++ b/integrations/bub/pyproject.toml @@ -20,7 +20,7 @@ requires-python = ">=3.12,<4.0" dependencies = [ "bub>=0.4.0,<0.5.0", "httpx>=0.28,<1", - "powercontext[client]>=0.0.3", + "powercontext[client]", "pydantic-settings>=2.7,<3", ] diff --git a/integrations/dsh/plugins/powercontext/lib/index.js b/integrations/dsh/plugins/powercontext/lib/index.js index 9c977a528..e2a59576f 100644 --- a/integrations/dsh/plugins/powercontext/lib/index.js +++ b/integrations/dsh/plugins/powercontext/lib/index.js @@ -105,6 +105,30 @@ const OPERATIONS = { location: "body", scope: true }, + register_source_definition: { + method: "POST", + path: "/v1/source-definitions/register", + location: "body", + scope: false + }, + get_connector_checkpoint: { + method: "POST", + path: "/v1/connector-checkpoints/get", + location: "body", + scope: false + }, + submit_source_observation: { + method: "POST", + path: "/v1/source-observations", + location: "body", + scope: false + }, + commit_connector_checkpoint: { + method: "POST", + path: "/v1/connector-checkpoints/commit", + location: "body", + scope: false + }, prepare_context: { method: "POST", path: "/v1/context/prepare", diff --git a/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml b/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml index 2c8681f99..bed6472a9 100644 --- a/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml +++ b/integrations/dsh/plugins/powercontext/openapi/powercontext.yaml @@ -111,6 +111,107 @@ paths: $ref: "#/components/responses/Unavailable" "500": $ref: "#/components/responses/InternalError" + /v1/source-definitions/register: + post: + tags: [source-ingestion] + summary: Register a worker-owned Source Definition manifest + description: Registers an immutable declarative manifest without loading worker plugin code. + operationId: register_source_definition + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/RegisterSourceDefinitionRequest" + responses: + "200": + description: The exact manifest is registered or was already registered identically. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceDefinitionManifest" + "409": + $ref: "#/components/responses/Conflict" + "401": + $ref: "#/components/responses/Unauthorized" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/get: + post: + tags: [source-ingestion] + summary: Read a Connector binding checkpoint + operationId: get_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/GetConnectorCheckpointRequest" + responses: + "200": + description: The current opaque checkpoint, including a normal null initial value. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/source-observations: + post: + tags: [source-ingestion] + summary: Submit a worker-materialized Source observation + description: Validates the observation against its registered manifest and durably appends it before receipt. + operationId: submit_source_observation + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/SubmitSourceObservationRequest" + responses: + "202": + description: The observation is durably accepted and can be referenced exactly. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceObservationReceipt" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "404": + $ref: "#/components/responses/NotFound" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/commit: + post: + tags: [source-ingestion] + summary: Commit a Connector binding checkpoint + description: Replaces the checkpoint only when its expected starting value still matches. + operationId: commit_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/CommitConnectorCheckpointRequest" + responses: + "200": + description: The new opaque checkpoint is durable. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" /v1/context/prepare: post: tags: [context] @@ -2663,6 +2764,178 @@ components: position: type: integer minimum: 1 + SourceProjectionKey: + type: object + additionalProperties: false + required: [name, version] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + SourceProjectionManifest: + type: object + additionalProperties: false + required: [key, schema] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + schema: + type: object + additionalProperties: true + SourceDefinitionManifest: + type: object + additionalProperties: false + required: [name, version, fingerprint, source_schema, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + source_schema: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionManifest" + RegisterSourceDefinitionRequest: + type: object + additionalProperties: false + required: [manifest] + properties: + manifest: + $ref: "#/components/schemas/SourceDefinitionManifest" + ConnectorBinding: + type: object + additionalProperties: false + required: [scope_id, binding_id, connector_name, connector_version] + properties: + scope_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + binding_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + connector_name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + connector_version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + GetConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + ConnectorCheckpointState: + type: object + additionalProperties: false + required: [binding, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + checkpoint: + nullable: true + SourceProjectionValue: + type: object + additionalProperties: false + required: [key, value] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + value: {} + ProjectedSource: + type: object + additionalProperties: false + required: + [name, definition_version, materialization, source_type, definition_fingerprint, payload, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 256 + definition_version: + type: string + minLength: 1 + maxLength: 128 + materialization: + type: string + enum: [captured, referenced] + description: + type: string + nullable: true + source_type: + type: string + minLength: 1 + maxLength: 128 + definition_fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + payload: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionValue" + SubmitSourceObservationRequest: + type: object + additionalProperties: false + required: [binding, source] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + source: + $ref: "#/components/schemas/ProjectedSource" + SourceObservationReceipt: + type: object + additionalProperties: false + required: [source, position] + properties: + source: + $ref: "#/components/schemas/SourceReference" + position: + type: integer + minimum: 1 + CommitConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding, expected, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + expected: + nullable: true + checkpoint: + nullable: true CommitHandoffRequest: type: object additionalProperties: false diff --git a/integrations/dsh/plugins/powercontext/src/operations.generated.ts b/integrations/dsh/plugins/powercontext/src/operations.generated.ts index 092c7108a..06178ba3b 100644 --- a/integrations/dsh/plugins/powercontext/src/operations.generated.ts +++ b/integrations/dsh/plugins/powercontext/src/operations.generated.ts @@ -21,6 +21,10 @@ export const OPERATIONS = { get_readiness: { method: 'GET', path: '/health/ready', location: null, scope: false }, get_capabilities: { method: 'GET', path: '/v1/capabilities', location: null, scope: false }, capture_content_source: { method: 'POST', path: '/v1/sources/content', location: "body", scope: true }, + register_source_definition: { method: 'POST', path: '/v1/source-definitions/register', location: "body", scope: false }, + get_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/get', location: "body", scope: false }, + submit_source_observation: { method: 'POST', path: '/v1/source-observations', location: "body", scope: false }, + commit_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/commit', location: "body", scope: false }, prepare_context: { method: 'POST', path: '/v1/context/prepare', location: "body", scope: true }, create_work_contract: { method: 'POST', path: '/v1/work/contracts/create', location: "body", scope: true }, handoff_current_work: { method: 'POST', path: '/v1/work/handoffs/prepare-current', location: "body", scope: true }, diff --git a/integrations/opencode/plugins/powercontext/src/operations.generated.ts b/integrations/opencode/plugins/powercontext/src/operations.generated.ts index 092c7108a..06178ba3b 100644 --- a/integrations/opencode/plugins/powercontext/src/operations.generated.ts +++ b/integrations/opencode/plugins/powercontext/src/operations.generated.ts @@ -21,6 +21,10 @@ export const OPERATIONS = { get_readiness: { method: 'GET', path: '/health/ready', location: null, scope: false }, get_capabilities: { method: 'GET', path: '/v1/capabilities', location: null, scope: false }, capture_content_source: { method: 'POST', path: '/v1/sources/content', location: "body", scope: true }, + register_source_definition: { method: 'POST', path: '/v1/source-definitions/register', location: "body", scope: false }, + get_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/get', location: "body", scope: false }, + submit_source_observation: { method: 'POST', path: '/v1/source-observations', location: "body", scope: false }, + commit_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/commit', location: "body", scope: false }, prepare_context: { method: 'POST', path: '/v1/context/prepare', location: "body", scope: true }, create_work_contract: { method: 'POST', path: '/v1/work/contracts/create', location: "body", scope: true }, handoff_current_work: { method: 'POST', path: '/v1/work/handoffs/prepare-current', location: "body", scope: true }, diff --git a/integrations/opendal/README.md b/integrations/opendal/README.md new file mode 100644 index 000000000..e23e26ab3 --- /dev/null +++ b/integrations/opendal/README.md @@ -0,0 +1,27 @@ +# OpenDAL Connector worker + +`powercontext-connector-opendal` is an independently deployed Connector worker. It owns the executable text-file +Source Definition and uses OpenDAL through `opendalfs` to acquire files. PowerContext Server only receives the +Definition manifest, projected Source observations, and opaque checkpoint comparisons. + +Install from a checkout: + +```bash +uv tool install ./integrations/opendal +``` + +Run one bounded scan against a filesystem backend: + +```bash +powercontext-connector-opendal \ + --base-url http://127.0.0.1:8765 \ + --scope-id project-a \ + --binding-id workspace-documents \ + --service fs \ + --storage-option root=/path/to/workspace \ + --source-namespace workspace-a +``` + +The process registers its immutable Definition manifest before each run. It advances the binding checkpoint only +after every accepted Source observation has a durable Server receipt and the scan completes without rejected or +failed items. Set `POWERCONTEXT_TOKEN` when the Server requires bearer authentication. diff --git a/integrations/opendal/pyproject.toml b/integrations/opendal/pyproject.toml new file mode 100644 index 000000000..79f795c1b --- /dev/null +++ b/integrations/opendal/pyproject.toml @@ -0,0 +1,35 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[project] +name = "powercontext-connector-opendal" +version = "0.0.1" +description = "OpenDAL file connector worker for PowerContext." +readme = "README.md" +license = "Apache-2.0" +requires-python = ">=3.12,<4.0" +dependencies = [ + "opendalfs>=0.1,<0.2", + "powercontext[client]>=0.0.3,<1", +] + +[project.scripts] +powercontext-connector-opendal = "powercontext_connector_opendal.cli:main" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/powercontext_connector_opendal"] diff --git a/integrations/opendal/src/powercontext_connector_opendal/__init__.py b/integrations/opendal/src/powercontext_connector_opendal/__init__.py new file mode 100644 index 000000000..634863194 --- /dev/null +++ b/integrations/opendal/src/powercontext_connector_opendal/__init__.py @@ -0,0 +1,43 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""OpenDAL Connector worker for PowerContext.""" + +from powercontext_connector_opendal.connector import ( + OPENDAL_TEXT_FILE_CONNECTOR_NAME, + OpenDALTextFileCheckpoint, + OpenDALTextFileConnector, +) +from powercontext_connector_opendal.source import ( + TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER, + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + TextFileEvidenceProjection, + TextFileSnapshotCapture, + TextFileSnapshotSource, + TextFileSnapshotSourceAdapter, +) + +__all__ = [ + "OPENDAL_TEXT_FILE_CONNECTOR_NAME", + "TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER", + "TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION", + "TEXT_FILE_SNAPSHOT_SOURCE_NAME", + "OpenDALTextFileCheckpoint", + "OpenDALTextFileConnector", + "TextFileEvidenceProjection", + "TextFileSnapshotCapture", + "TextFileSnapshotSource", + "TextFileSnapshotSourceAdapter", +] diff --git a/integrations/opendal/src/powercontext_connector_opendal/cli.py b/integrations/opendal/src/powercontext_connector_opendal/cli.py new file mode 100644 index 000000000..ca5f1bf94 --- /dev/null +++ b/integrations/opendal/src/powercontext_connector_opendal/cli.py @@ -0,0 +1,95 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Command-line entry point for one independently scheduled Connector run.""" + +from __future__ import annotations + +import argparse +import asyncio +import os +from collections.abc import Sequence + +from powercontext.client import PowerContextClient, RemoteConnectorWorker +from powercontext.sources import ConnectorBinding, ConnectorRunStatus, SourceDefinitionRegistry +from powercontext_connector_opendal.connector import OPENDAL_TEXT_FILE_CONNECTOR_NAME, OpenDALTextFileConnector +from powercontext_connector_opendal.source import TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION + + +def main() -> None: + """Run one scan and return a process status suitable for an external scheduler.""" + + raise SystemExit(asyncio.run(_run(_parser().parse_args()))) + + +async def _run(args: argparse.Namespace) -> int: + options = _storage_options(args.storage_option) + if args.pattern: + connector = OpenDALTextFileConnector.from_service( + args.service, + source_namespace=args.source_namespace, + root=args.root, + storage_options=options, + patterns=tuple(args.pattern), + max_files=args.max_files, + max_file_size=args.max_file_size, + ) + else: + connector = OpenDALTextFileConnector.from_service( + args.service, + source_namespace=args.source_namespace, + root=args.root, + storage_options=options, + max_files=args.max_files, + max_file_size=args.max_file_size, + ) + binding = ConnectorBinding( + scope_id=args.scope_id, + binding_id=args.binding_id, + connector_name=OPENDAL_TEXT_FILE_CONNECTOR_NAME, + connector_version=connector.version, + ) + registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + async with PowerContextClient(args.base_url, token=os.environ.get("POWERCONTEXT_TOKEN")) as client: + result = await RemoteConnectorWorker(client=client, registry=registry).run(connector, binding) + return 0 if result.status is ConnectorRunStatus.COMPLETE else 1 + + +def _storage_options(values: Sequence[str]) -> dict[str, str]: + options: dict[str, str] = {} + for value in values: + key, separator, option = value.partition("=") + if not separator or not key or key.strip() != key: + raise ValueError("storage options must use KEY=VALUE") # noqa: TRY003 + options[key] = option + return options + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--base-url", required=True) + parser.add_argument("--scope-id", required=True) + parser.add_argument("--binding-id", required=True) + parser.add_argument("--service", required=True) + parser.add_argument("--source-namespace", required=True) + parser.add_argument("--root", default="") + parser.add_argument("--storage-option", action="append", default=[]) + parser.add_argument("--pattern", action="append") + parser.add_argument("--max-files", type=int, default=10_000) + parser.add_argument("--max-file-size", type=int, default=2 * 1024 * 1024) + return parser + + +if __name__ == "__main__": + main() diff --git a/integrations/opendal/src/powercontext_connector_opendal/connector.py b/integrations/opendal/src/powercontext_connector_opendal/connector.py new file mode 100644 index 000000000..6d8b95da3 --- /dev/null +++ b/integrations/opendal/src/powercontext_connector_opendal/connector.py @@ -0,0 +1,296 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Capture UTF-8 text files through the OpenDAL fsspec implementation.""" + +from __future__ import annotations + +import asyncio +import fnmatch +import hashlib +import mimetypes +import posixpath +from collections.abc import Mapping, Sequence +from datetime import UTC, datetime +from importlib import import_module +from typing import Any, Literal, Protocol + +from pydantic import BaseModel, ConfigDict, Field, JsonValue, ValidationError, field_validator + +from powercontext.errors import InvalidConnectorRunError +from powercontext.sources import ( + ConnectorCapability, + ConnectorRunCompletion, + ConnectorRunSession, + ConnectorRunStatus, +) +from powercontext_connector_opendal.source import ( + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + TextFileSnapshotCapture, +) + +OPENDAL_TEXT_FILE_CONNECTOR_NAME = "opendal-text-files" +_DEFAULT_PATTERNS = ("**/*.md", "**/*.markdown", "**/*.txt", "**/*.rst", "**/*.adoc") + + +class _FsspecFileSystem(Protocol): + def find(self, path: str, *, detail: bool) -> Mapping[str, Mapping[str, object]]: ... + + def cat_file(self, path: str) -> bytes: ... + + +class OpenDALTextFileCheckpoint(BaseModel): + """Opaque content-digest checkpoint for one Connector binding.""" + + model_config = ConfigDict(frozen=True) + + schema_version: Literal["1"] = "1" + files: dict[str, str] = Field(default_factory=dict) + + @field_validator("files") + @classmethod + def validate_files(cls, value: dict[str, str]) -> dict[str, str]: + for path, digest in value.items(): + _validate_relative_path(path) + if not digest.startswith("sha256:") or len(digest) != 71: + raise ValueError("checkpoint digests must use sha256:") # noqa: TRY003 + try: + int(digest.removeprefix("sha256:"), 16) + except ValueError as error: + raise ValueError("checkpoint digest must contain lowercase hexadecimal") from error # noqa: TRY003 + if digest != digest.lower(): + raise ValueError("checkpoint digest must contain lowercase hexadecimal") # noqa: TRY003 + return value + + +class OpenDALTextFileConnector: + """Perform bounded full scans through an OpenDAL-backed fsspec filesystem.""" + + name = OPENDAL_TEXT_FILE_CONNECTOR_NAME + version = "1" + source_definitions = frozenset({TEXT_FILE_SNAPSHOT_SOURCE_NAME}) + capabilities = frozenset({ConnectorCapability.CHECKPOINT_RESUME}) + + def __init__( + self, + filesystem: _FsspecFileSystem, + *, + source_namespace: str, + root: str = "", + patterns: Sequence[str] = _DEFAULT_PATTERNS, + max_files: int = 10_000, + max_file_size: int = 2 * 1024 * 1024, + ) -> None: + if not source_namespace or source_namespace.strip() != source_namespace: + raise ValueError("source_namespace must be a non-empty trimmed string") # noqa: TRY003 + if max_files < 1: + raise ValueError("max_files must be positive") # noqa: TRY003 + if max_file_size < 1: + raise ValueError("max_file_size must be positive") # noqa: TRY003 + if isinstance(patterns, str): + raise TypeError("patterns must be a sequence of glob patterns") # noqa: TRY003 + normalized_patterns = tuple(patterns) + if not normalized_patterns or any(not pattern or pattern.strip() != pattern for pattern in normalized_patterns): + raise ValueError("patterns must contain non-empty trimmed values") # noqa: TRY003 + self._filesystem = filesystem + self._source_namespace = source_namespace + self._root = _normalize_root(root) + self._patterns = normalized_patterns + self._max_files = max_files + self._max_file_size = max_file_size + + @classmethod + def from_service( + cls, + service: str, + *, + source_namespace: str, + root: str = "", + storage_options: Mapping[str, object] | None = None, + patterns: Sequence[str] = _DEFAULT_PATTERNS, + max_files: int = 10_000, + max_file_size: int = 2 * 1024 * 1024, + ) -> OpenDALTextFileConnector: + """Create a Connector from one OpenDAL service and its runtime-only options.""" + + try: + opendalfs = import_module("opendalfs") + except ImportError as error: + raise ImportError( # noqa: TRY003 + "OpenDALTextFileConnector.from_service requires powercontext-connector-opendal on Python 3.12+" + ) from error + backend_options: dict[str, Any] = dict(storage_options or {}) + filesystem = opendalfs.OpendalFileSystem( + scheme=service, + asynchronous=False, + skip_instance_cache=True, + **backend_options, + ) + return cls( + filesystem, + source_namespace=source_namespace, + root=root, + patterns=patterns, + max_files=max_files, + max_file_size=max_file_size, + ) + + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + previous = _checkpoint(session.checkpoint) + entries = await asyncio.to_thread(self._filesystem.find, self._root, detail=True) + files = self._selected_files(entries) + if len(files) > self._max_files: + raise InvalidConnectorRunError( + "file-limit", + f"scan selected {len(files)} files, maximum is {self._max_files}", + ) + + current_files: dict[str, str] = {} + for relative_path, storage_path, info in files: + size = _non_negative_int(info.get("size")) + if size is not None and size > self._max_file_size: + session.reject( + relative_path, + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + f"file exceeds {self._max_file_size} bytes", + ) + continue + try: + content_bytes = await asyncio.to_thread(self._filesystem.cat_file, storage_path) + except Exception as error: + session.fail(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, type(error).__name__) + continue + if not isinstance(content_bytes, bytes): + session.fail(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, "filesystem returned non-bytes content") + continue + if len(content_bytes) > self._max_file_size: + session.reject( + relative_path, + TEXT_FILE_SNAPSHOT_SOURCE_NAME, + f"file exceeds {self._max_file_size} bytes", + ) + continue + + content_digest = f"sha256:{hashlib.sha256(content_bytes).hexdigest()}" + current_files[relative_path] = content_digest + if previous.files.get(relative_path) == content_digest: + continue + try: + content = content_bytes.decode("utf-8") + except UnicodeDecodeError: + session.reject(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, "file is not valid UTF-8") + continue + capture = TextFileSnapshotCapture( + namespace=self._source_namespace, + path=relative_path, + content=content, + media_type=mimetypes.guess_type(relative_path)[0] or "text/plain", + etag=_optional_string(info.get("etag")), + provider_version=_optional_string(info.get("version")), + modified_at=_optional_datetime(info.get("mtime")), + ) + await session.submit(relative_path, TEXT_FILE_SNAPSHOT_SOURCE_NAME, capture) + + checkpoint = OpenDALTextFileCheckpoint(files=current_files) + return ConnectorRunCompletion( + status=ConnectorRunStatus.COMPLETE, + checkpoint=checkpoint.model_dump(mode="json"), + ) + + def _selected_files( + self, + entries: Mapping[str, Mapping[str, object]], + ) -> tuple[tuple[str, str, Mapping[str, object]], ...]: + selected: list[tuple[str, str, Mapping[str, object]]] = [] + for storage_path, info in entries.items(): + if info.get("type") != "file": + continue + relative_path = _relative_path(storage_path, self._root) + if not _matches(relative_path, self._patterns): + continue + selected.append((relative_path, storage_path, info)) + selected.sort(key=lambda item: item[0]) + return tuple(selected) + + +def _checkpoint(value: JsonValue | None) -> OpenDALTextFileCheckpoint: + if value is None: + return OpenDALTextFileCheckpoint() + try: + return OpenDALTextFileCheckpoint.model_validate(value) + except ValidationError as error: + raise InvalidConnectorRunError("checkpoint", "does not match OpenDALTextFileCheckpoint") from error + + +def _normalize_root(value: str) -> str: + if value != value.strip() or "\\" in value: + raise ValueError("root must be a normalized POSIX path") # noqa: TRY003 + normalized = posixpath.normpath(value).strip("/") + if normalized in {"", "."}: + return "" + _validate_relative_path(normalized) + return normalized + + +def _relative_path(storage_path: str, root: str) -> str: + normalized = posixpath.normpath(storage_path).strip("/") + relative = posixpath.relpath(normalized, root) if root else normalized + _validate_relative_path(relative) + return relative + + +def _validate_relative_path(value: str) -> None: + if not value or value.startswith("/") or "\\" in value: + raise ValueError("file path must be a relative POSIX path") # noqa: TRY003 + normalized = posixpath.normpath(value) + if normalized != value or normalized == ".." or normalized.startswith("../"): + raise ValueError("file path escapes the configured root") # noqa: TRY003 + + +def _matches(path: str, patterns: tuple[str, ...]) -> bool: + return any( + fnmatch.fnmatchcase(path, pattern) + or (pattern.startswith("**/") and fnmatch.fnmatchcase(path, pattern.removeprefix("**/"))) + for pattern in patterns + ) + + +def _non_negative_int(value: object) -> int | None: + if not isinstance(value, int) or isinstance(value, bool) or value < 0: + return None + return value + + +def _optional_string(value: object) -> str | None: + return value if isinstance(value, str) and value and value.strip() == value else None + + +def _optional_datetime(value: object) -> datetime | None: + if isinstance(value, datetime): + return value + if isinstance(value, int | float) and not isinstance(value, bool): + return datetime.fromtimestamp(value, tz=UTC) + if not isinstance(value, str): + return None + try: + return datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return None + + +__all__ = [ + "OPENDAL_TEXT_FILE_CONNECTOR_NAME", + "OpenDALTextFileCheckpoint", + "OpenDALTextFileConnector", +] diff --git a/integrations/opendal/src/powercontext_connector_opendal/source.py b/integrations/opendal/src/powercontext_connector_opendal/source.py new file mode 100644 index 000000000..99f2ddecc --- /dev/null +++ b/integrations/opendal/src/powercontext_connector_opendal/source.py @@ -0,0 +1,179 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Typed captured text-file snapshots for the OpenDAL Connector.""" + +from __future__ import annotations + +import hashlib +from datetime import datetime +from pathlib import PurePosixPath +from typing import Literal + +from pydantic import BaseModel, JsonValue, field_validator + +from powercontext.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + AdapterSourceDefinition, + Source, + SourceMaterialization, + TextEvidence, +) + +TEXT_FILE_SNAPSHOT_SOURCE_NAME = "text-file-snapshot" + + +class TextFileSnapshotCapture(BaseModel): + """One UTF-8 file value captured with non-authoritative provider annotations.""" + + namespace: str + path: str + content: str + media_type: str = "text/plain" + encoding: Literal["utf-8"] = "utf-8" + etag: str | None = None + provider_version: str | None = None + modified_at: datetime | None = None + + @field_validator("namespace", "media_type") + @classmethod + def validate_trimmed_text(cls, value: str) -> str: + if not value or value.strip() != value: + raise ValueError("value must be a non-empty trimmed string") # noqa: TRY003 + return value + + @field_validator("path") + @classmethod + def validate_path(cls, value: str) -> str: + if not value or value.strip() != value or "\\" in value: + raise ValueError("path must be a non-empty normalized POSIX path") # noqa: TRY003 + path = PurePosixPath(value) + if path.is_absolute() or value != path.as_posix() or any(part in {"", ".", ".."} for part in path.parts): + raise ValueError("path must be a relative normalized POSIX path") # noqa: TRY003 + return value + + @field_validator("etag", "provider_version") + @classmethod + def validate_optional_annotation(cls, value: str | None) -> str | None: + if value is not None and (not value or value.strip() != value): + raise ValueError("annotation must be a non-empty trimmed string") # noqa: TRY003 + return value + + +class TextFileSnapshotSource(Source): + """Captured text-file bytes with explicit filesystem provenance.""" + + namespace: str + path: str + content: str + media_type: str + encoding: Literal["utf-8"] + content_digest: str + size: int + etag: str | None = None + provider_version: str | None = None + modified_at: datetime | None = None + + +class TextFileSnapshotSourceAdapter: + """Canonicalize UTF-8 file captures into immutable snapshot Sources.""" + + input_class = TextFileSnapshotCapture + name = TEXT_FILE_SNAPSHOT_SOURCE_NAME + source_class = TextFileSnapshotSource + + async def resolve(self, value: TextFileSnapshotCapture, /) -> TextFileSnapshotSource: + content_bytes = value.content.encode(value.encoding) + content_digest = f"sha256:{hashlib.sha256(content_bytes).hexdigest()}" + source_id = _snapshot_id(value.namespace, value.path, content_digest) + return TextFileSnapshotSource( + name=source_id, + materialization=SourceMaterialization.CAPTURED, + description=f"Captured text file {value.path}", + namespace=value.namespace, + path=value.path, + content=value.content, + media_type=value.media_type, + encoding=value.encoding, + content_digest=content_digest, + size=len(content_bytes), + etag=value.etag, + provider_version=value.provider_version, + modified_at=value.modified_at, + ) + + async def read(self, source: TextFileSnapshotSource, /) -> TextFileSnapshotCapture: + return TextFileSnapshotCapture( + namespace=source.namespace, + path=source.path, + content=source.content, + media_type=source.media_type, + encoding=source.encoding, + etag=source.etag, + provider_version=source.provider_version, + modified_at=source.modified_at, + ) + + +class TextFileEvidenceProjection: + """Expose one file snapshot through the shared text-evidence capability.""" + + name = TEXT_EVIDENCE_PROJECTION_KEY.name + version = TEXT_EVIDENCE_PROJECTION_KEY.version + source_class = TextFileSnapshotSource + output_class: type[BaseModel] = TextEvidence + + def project(self, source: TextFileSnapshotSource, /) -> TextEvidence: + metadata: dict[str, JsonValue] = { + "namespace": source.namespace, + "path": source.path, + "media_type": source.media_type, + "encoding": source.encoding, + "content_digest": source.content_digest, + "size": source.size, + } + if source.etag is not None: + metadata["etag"] = source.etag + if source.provider_version is not None: + metadata["provider_version"] = source.provider_version + if source.modified_at is not None: + metadata["modified_at"] = source.modified_at.isoformat() + return TextEvidence( + source_type=TEXT_FILE_SNAPSHOT_SOURCE_NAME, + source_id=source.name, + content=source.content, + metadata=metadata, + ) + + +def _snapshot_id(namespace: str, path: str, content_digest: str) -> str: + identity = "\0".join((namespace, path, content_digest)) + return f"text_file_{hashlib.sha256(identity.encode()).hexdigest()}" + + +TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER = TextFileSnapshotSourceAdapter() +TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION = AdapterSourceDefinition( + TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER, + projections=(TextFileEvidenceProjection(),), +) + +__all__ = [ + "TEXT_FILE_SNAPSHOT_SOURCE_ADAPTER", + "TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION", + "TEXT_FILE_SNAPSHOT_SOURCE_NAME", + "TextFileEvidenceProjection", + "TextFileSnapshotCapture", + "TextFileSnapshotSource", + "TextFileSnapshotSourceAdapter", +] diff --git a/integrations/pi/plugins/powercontext/src/operations.generated.ts b/integrations/pi/plugins/powercontext/src/operations.generated.ts index 092c7108a..06178ba3b 100644 --- a/integrations/pi/plugins/powercontext/src/operations.generated.ts +++ b/integrations/pi/plugins/powercontext/src/operations.generated.ts @@ -21,6 +21,10 @@ export const OPERATIONS = { get_readiness: { method: 'GET', path: '/health/ready', location: null, scope: false }, get_capabilities: { method: 'GET', path: '/v1/capabilities', location: null, scope: false }, capture_content_source: { method: 'POST', path: '/v1/sources/content', location: "body", scope: true }, + register_source_definition: { method: 'POST', path: '/v1/source-definitions/register', location: "body", scope: false }, + get_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/get', location: "body", scope: false }, + submit_source_observation: { method: 'POST', path: '/v1/source-observations', location: "body", scope: false }, + commit_connector_checkpoint: { method: 'POST', path: '/v1/connector-checkpoints/commit', location: "body", scope: false }, prepare_context: { method: 'POST', path: '/v1/context/prepare', location: "body", scope: true }, create_work_contract: { method: 'POST', path: '/v1/work/contracts/create', location: "body", scope: true }, handoff_current_work: { method: 'POST', path: '/v1/work/handoffs/prepare-current', location: "body", scope: true }, diff --git a/openapi/powercontext.yaml b/openapi/powercontext.yaml index 2c8681f99..bed6472a9 100644 --- a/openapi/powercontext.yaml +++ b/openapi/powercontext.yaml @@ -111,6 +111,107 @@ paths: $ref: "#/components/responses/Unavailable" "500": $ref: "#/components/responses/InternalError" + /v1/source-definitions/register: + post: + tags: [source-ingestion] + summary: Register a worker-owned Source Definition manifest + description: Registers an immutable declarative manifest without loading worker plugin code. + operationId: register_source_definition + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/RegisterSourceDefinitionRequest" + responses: + "200": + description: The exact manifest is registered or was already registered identically. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceDefinitionManifest" + "409": + $ref: "#/components/responses/Conflict" + "401": + $ref: "#/components/responses/Unauthorized" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/get: + post: + tags: [source-ingestion] + summary: Read a Connector binding checkpoint + operationId: get_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/GetConnectorCheckpointRequest" + responses: + "200": + description: The current opaque checkpoint, including a normal null initial value. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/source-observations: + post: + tags: [source-ingestion] + summary: Submit a worker-materialized Source observation + description: Validates the observation against its registered manifest and durably appends it before receipt. + operationId: submit_source_observation + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/SubmitSourceObservationRequest" + responses: + "202": + description: The observation is durably accepted and can be referenced exactly. + content: + application/json: + schema: + $ref: "#/components/schemas/SourceObservationReceipt" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "404": + $ref: "#/components/responses/NotFound" + "422": + $ref: "#/components/responses/InvalidRequest" + /v1/connector-checkpoints/commit: + post: + tags: [source-ingestion] + summary: Commit a Connector binding checkpoint + description: Replaces the checkpoint only when its expected starting value still matches. + operationId: commit_connector_checkpoint + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/CommitConnectorCheckpointRequest" + responses: + "200": + description: The new opaque checkpoint is durable. + content: + application/json: + schema: + $ref: "#/components/schemas/ConnectorCheckpointState" + "401": + $ref: "#/components/responses/Unauthorized" + "409": + $ref: "#/components/responses/Conflict" + "422": + $ref: "#/components/responses/InvalidRequest" /v1/context/prepare: post: tags: [context] @@ -2663,6 +2764,178 @@ components: position: type: integer minimum: 1 + SourceProjectionKey: + type: object + additionalProperties: false + required: [name, version] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + SourceProjectionManifest: + type: object + additionalProperties: false + required: [key, schema] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + schema: + type: object + additionalProperties: true + SourceDefinitionManifest: + type: object + additionalProperties: false + required: [name, version, fingerprint, source_schema, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + source_schema: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionManifest" + RegisterSourceDefinitionRequest: + type: object + additionalProperties: false + required: [manifest] + properties: + manifest: + $ref: "#/components/schemas/SourceDefinitionManifest" + ConnectorBinding: + type: object + additionalProperties: false + required: [scope_id, binding_id, connector_name, connector_version] + properties: + scope_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + binding_id: + type: string + minLength: 1 + maxLength: 256 + pattern: '.*\S.*' + connector_name: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + connector_version: + type: string + minLength: 1 + maxLength: 128 + pattern: '.*\S.*' + GetConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + ConnectorCheckpointState: + type: object + additionalProperties: false + required: [binding, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + checkpoint: + nullable: true + SourceProjectionValue: + type: object + additionalProperties: false + required: [key, value] + properties: + key: + $ref: "#/components/schemas/SourceProjectionKey" + value: {} + ProjectedSource: + type: object + additionalProperties: false + required: + [name, definition_version, materialization, source_type, definition_fingerprint, payload, projections] + properties: + name: + type: string + minLength: 1 + maxLength: 256 + definition_version: + type: string + minLength: 1 + maxLength: 128 + materialization: + type: string + enum: [captured, referenced] + description: + type: string + nullable: true + source_type: + type: string + minLength: 1 + maxLength: 128 + definition_fingerprint: + type: string + pattern: '^sha256:[0-9a-f]{64}$' + payload: + type: object + additionalProperties: true + projections: + type: array + maxItems: 16 + items: + $ref: "#/components/schemas/SourceProjectionValue" + SubmitSourceObservationRequest: + type: object + additionalProperties: false + required: [binding, source] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + source: + $ref: "#/components/schemas/ProjectedSource" + SourceObservationReceipt: + type: object + additionalProperties: false + required: [source, position] + properties: + source: + $ref: "#/components/schemas/SourceReference" + position: + type: integer + minimum: 1 + CommitConnectorCheckpointRequest: + type: object + additionalProperties: false + required: [binding, expected, checkpoint] + properties: + binding: + $ref: "#/components/schemas/ConnectorBinding" + expected: + nullable: true + checkpoint: + nullable: true CommitHandoffRequest: type: object additionalProperties: false diff --git a/pyproject.toml b/pyproject.toml index 81bcb5874..dcf24d2db 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,6 +40,7 @@ classifiers = [ builtin = [ "aiosqlite>=0.22,<1", "apscheduler>=3.11,<4", + "jsonschema>=4.23,<5", "pydantic-ai-slim[anthropic,openai]>=2.27.1,<3", "pydantic-settings>=2.7,<3", "pyobvector>=0.2.28,<0.3", @@ -135,6 +136,7 @@ extra-paths = [ "./integrations/workbuddy/plugins/powercontext", "./integrations/workbuddy/plugins/powercontext/hooks", "./integrations/pydantic-ai/src", + "./integrations/opendal/src", ] [tool.ty.src] @@ -165,7 +167,7 @@ missing-override-decorator = "ignore" [tool.pytest.ini_options] testpaths = ["tests"] -pythonpath = ["integrations/pydantic-ai/src"] +pythonpath = ["integrations/pydantic-ai/src", "integrations/opendal/src"] markers = [ "real_e2e: uses real Codex, external model providers, and the configured database", ] diff --git a/src/powercontext/__init__.py b/src/powercontext/__init__.py index 9ae96033b..5a4abf7ff 100644 --- a/src/powercontext/__init__.py +++ b/src/powercontext/__init__.py @@ -27,30 +27,59 @@ ArtifactError, ArtifactFamilyMismatchError, ArtifactNotFoundError, + ConnectorError, InvalidArtifactReferenceError, + InvalidConnectorError, + InvalidConnectorRunError, InvalidSourceAdapterError, + InvalidSourceDefinitionError, InvalidSourceEntryError, + InvalidSourceObservationError, + InvalidSourceProjectionError, InvalidSourceReferenceError, InvalidSourceResultError, PowerContextError, RevisionConflictError, SourceAdapterNotFoundError, SourceConflictError, + SourceDefinitionNotFoundError, SourceError, SourceNotFoundError, + SourceProjectionNotFoundError, ) from powercontext.sources import ( + AdapterSourceDefinition, + CatalogConnectorSourceSink, + Connector, + ConnectorBinding, + ConnectorCapability, + ConnectorCheckpointStore, + ConnectorItemOutcome, + ConnectorLifecycle, + ConnectorRunCompletion, + ConnectorRunResult, + ConnectorRunSession, + ConnectorRunStatus, + ConnectorSourceSink, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, Source, SourceAdapter, SourceCatalog, SourceCatalogBackend, + SourceDefinition, + SourceDefinitionRegistry, SourceMaterialization, + SourceProjection, + SourceProjectionKey, SourceRef, SourceStore, + validate_connector, ) from powercontext.triggers import PolicyTransition, Trigger __all__ = [ + "AdapterSourceDefinition", "Artifact", "ArtifactCatalog", "ArtifactDraft", @@ -61,9 +90,29 @@ "ArtifactRef", "ArtifactStore", "Artifacts", + "CatalogConnectorSourceSink", + "Connector", + "ConnectorBinding", + "ConnectorCapability", + "ConnectorCheckpointStore", + "ConnectorError", + "ConnectorItemOutcome", + "ConnectorLifecycle", + "ConnectorRunCompletion", + "ConnectorRunResult", + "ConnectorRunSession", + "ConnectorRunStatus", + "ConnectorSourceSink", + "ConnectorSubmissionResult", + "ConnectorSubmissionStatus", "InvalidArtifactReferenceError", + "InvalidConnectorError", + "InvalidConnectorRunError", "InvalidSourceAdapterError", + "InvalidSourceDefinitionError", "InvalidSourceEntryError", + "InvalidSourceObservationError", + "InvalidSourceProjectionError", "InvalidSourceReferenceError", "InvalidSourceResultError", "PolicyTransition", @@ -76,11 +125,18 @@ "SourceCatalog", "SourceCatalogBackend", "SourceConflictError", + "SourceDefinition", + "SourceDefinitionNotFoundError", + "SourceDefinitionRegistry", "SourceError", "SourceMaterialization", "SourceNotFoundError", + "SourceProjection", + "SourceProjectionKey", + "SourceProjectionNotFoundError", "SourceRef", "SourceStore", "Sources", "Trigger", + "validate_connector", ] diff --git a/src/powercontext/builtin/persistence/__init__.py b/src/powercontext/builtin/persistence/__init__.py index 9f75b5f30..703f204f4 100644 --- a/src/powercontext/builtin/persistence/__init__.py +++ b/src/powercontext/builtin/persistence/__init__.py @@ -15,6 +15,11 @@ """SQLAlchemy-backed relational persistence building blocks.""" from powercontext.builtin.persistence.candidates import CandidateRepository +from powercontext.builtin.persistence.connectors import ( + ConnectorCheckpointRepository, + RelationalConnectorCheckpointStore, + StoredConnectorCheckpoint, +) from powercontext.builtin.persistence.database import AsyncDatabase from powercontext.builtin.persistence.errors import ( DatabaseClosedError, @@ -29,6 +34,10 @@ StoredPayloadConflictError, ) from powercontext.builtin.persistence.external_skills import ExternalSkillRepository +from powercontext.builtin.persistence.source_definitions import ( + SourceDefinitionManifestRepository, + StoredSourceDefinitionManifest, +) from powercontext.builtin.persistence.statistics import ( StatisticsRepository, StoredInventoryCounts, @@ -39,6 +48,7 @@ __all__ = ( "AsyncDatabase", "CandidateRepository", + "ConnectorCheckpointRepository", "DatabaseClosedError", "ExternalSkillRepository", "GenerationConflictError", @@ -47,11 +57,15 @@ "InvalidStoredColumnError", "InvalidStoredPayloadError", "PersistenceError", + "RelationalConnectorCheckpointStore", "RepositoryError", "RepositoryNotFoundError", + "SourceDefinitionManifestRepository", "StatisticsRepository", + "StoredConnectorCheckpoint", "StoredInventoryCounts", "StoredModelUsage", "StoredPayloadConflictError", "StoredRecallTokenUsage", + "StoredSourceDefinitionManifest", ) diff --git a/src/powercontext/builtin/persistence/connectors.py b/src/powercontext/builtin/persistence/connectors.py new file mode 100644 index 000000000..edb901617 --- /dev/null +++ b/src/powercontext/builtin/persistence/connectors.py @@ -0,0 +1,180 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Durable Connector checkpoints and their runtime store adapter.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + +from pydantic import BaseModel, ConfigDict, JsonValue +from sqlalchemy import insert, select, update +from sqlalchemy.exc import IntegrityError +from sqlalchemy.ext.asyncio import AsyncConnection + +from powercontext.builtin.persistence.codec import dump_model, load_model, stored_bytes +from powercontext.builtin.persistence.database import AsyncDatabase +from powercontext.builtin.persistence.tables import CONNECTOR_CHECKPOINTS_TABLE +from powercontext.errors import InvalidConnectorRunError +from powercontext.sources import ConnectorBinding + + +class _CheckpointPayload(BaseModel): + model_config = ConfigDict(frozen=True) + + value: JsonValue | None + + +class StoredConnectorCheckpoint(BaseModel): + """One decoded checkpoint bound to an exact Connector identity.""" + + model_config = ConfigDict(frozen=True) + + binding: ConnectorBinding + checkpoint: JsonValue | None + + +class ConnectorCheckpointRepository: + """Persist opaque Connector checkpoints with value-based comparison.""" + + async def load( + self, + connection: AsyncConnection, + binding: ConnectorBinding, + /, + *, + for_update: bool = False, + ) -> StoredConnectorCheckpoint | None: + statement = select(CONNECTOR_CHECKPOINTS_TABLE).where( + CONNECTOR_CHECKPOINTS_TABLE.c.scope_id == binding.scope_id, + CONNECTOR_CHECKPOINTS_TABLE.c.binding_id == binding.binding_id, + ) + if for_update: + statement = statement.with_for_update() + row = (await connection.execute(statement)).mappings().one_or_none() + if row is None: + return None + stored = _decode_row(row) + if stored.binding != binding: + raise InvalidConnectorRunError( + "binding-conflict", + f"checkpoint {binding.binding_id!r} belongs to a different Connector identity", + ) + return stored + + async def save( + self, + connection: AsyncConnection, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> StoredConnectorCheckpoint: + existing = await self.load(connection, binding, for_update=True) + actual = None if existing is None else existing.checkpoint + if actual != expected: + raise _checkpoint_conflict(binding) + + payload = _dump_checkpoint(binding, checkpoint) + if existing is None: + try: + async with connection.begin_nested(): + await connection.execute( + insert(CONNECTOR_CHECKPOINTS_TABLE).values( + scope_id=binding.scope_id, + binding_id=binding.binding_id, + connector_name=binding.connector_name, + connector_version=binding.connector_version, + checkpoint=payload, + ) + ) + except IntegrityError: + raise _checkpoint_conflict(binding) from None + else: + result = await connection.execute( + update(CONNECTOR_CHECKPOINTS_TABLE) + .where( + CONNECTOR_CHECKPOINTS_TABLE.c.scope_id == binding.scope_id, + CONNECTOR_CHECKPOINTS_TABLE.c.binding_id == binding.binding_id, + CONNECTOR_CHECKPOINTS_TABLE.c.checkpoint == _dump_checkpoint(binding, expected), + ) + .values(checkpoint=payload) + ) + if result.rowcount != 1: + raise _checkpoint_conflict(binding) + return StoredConnectorCheckpoint(binding=binding, checkpoint=checkpoint) + + +class RelationalConnectorCheckpointStore: + """Adapt the Connector checkpoint protocol to an ``AsyncDatabase``.""" + + def __init__(self, database: AsyncDatabase, repository: ConnectorCheckpointRepository, /) -> None: + self._database = database + self._repository = repository + + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: + async with self._database.transaction() as connection: + stored = await self._repository.load(connection, binding) + return None if stored is None else stored.checkpoint + + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: + async with self._database.transaction() as connection: + await self._repository.save(connection, binding, checkpoint, expected=expected) + + +def _dump_checkpoint(binding: ConnectorBinding, checkpoint: JsonValue | None) -> bytes: + return dump_model( + _CheckpointPayload(value=checkpoint), + kind="connector-checkpoint", + name=binding.binding_id, + ) + + +def _checkpoint_conflict(binding: ConnectorBinding) -> InvalidConnectorRunError: + return InvalidConnectorRunError( + "checkpoint-conflict", + f"binding {binding.binding_id!r} changed during the run", + ) + + +def _decode_row(row: Mapping[Any, Any]) -> StoredConnectorCheckpoint: + binding = ConnectorBinding( + scope_id=str(row["scope_id"]), + binding_id=str(row["binding_id"]), + connector_name=str(row["connector_name"]), + connector_version=str(row["connector_version"]), + ) + payload = load_model( + _CheckpointPayload, + stored_bytes(row["checkpoint"], column="checkpoint"), + kind="connector-checkpoint", + name=binding.binding_id, + ) + return StoredConnectorCheckpoint(binding=binding, checkpoint=payload.value) + + +__all__ = [ + "ConnectorCheckpointRepository", + "RelationalConnectorCheckpointStore", + "StoredConnectorCheckpoint", +] diff --git a/src/powercontext/builtin/persistence/source_definitions.py b/src/powercontext/builtin/persistence/source_definitions.py new file mode 100644 index 000000000..eae333ab1 --- /dev/null +++ b/src/powercontext/builtin/persistence/source_definitions.py @@ -0,0 +1,130 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Persistence for worker-owned Source Definition manifests.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + +from pydantic import BaseModel, ConfigDict +from sqlalchemy import insert, select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.ext.asyncio import AsyncConnection + +from powercontext.builtin.persistence.codec import dump_model, load_model, stored_bytes +from powercontext.builtin.persistence.errors import ( + IdentityMismatchError, + RepositoryNotFoundError, + StoredPayloadConflictError, +) +from powercontext.builtin.persistence.tables import SOURCE_DEFINITION_MANIFESTS_TABLE +from powercontext.sources import SourceDefinitionManifest + + +class StoredSourceDefinitionManifest(BaseModel): + """One exact declarative Source Definition registration.""" + + model_config = ConfigDict(frozen=True) + + manifest: SourceDefinitionManifest + + +class SourceDefinitionManifestRepository: + """Register immutable worker-owned Definition manifests by name and version.""" + + async def register( + self, + connection: AsyncConnection, + manifest: SourceDefinitionManifest, + /, + ) -> StoredSourceDefinitionManifest: + payload = dump_model(manifest, kind="source-definition-manifest", name=manifest.name) + existing = await self.find(connection, manifest.name, manifest.version) + if existing is not None: + if existing.manifest != manifest: + raise StoredPayloadConflictError("source-definition-manifest", (manifest.name, manifest.version)) + return existing + try: + await connection.execute( + insert(SOURCE_DEFINITION_MANIFESTS_TABLE).values( + definition_name=manifest.name, + definition_version=manifest.version, + fingerprint=manifest.fingerprint, + manifest=payload, + ) + ) + except IntegrityError: + existing = await self.find(connection, manifest.name, manifest.version) + if existing is None or existing.manifest != manifest: + raise StoredPayloadConflictError( + "source-definition-manifest", + (manifest.name, manifest.version), + ) from None + return existing + return StoredSourceDefinitionManifest(manifest=manifest) + + async def get( + self, + connection: AsyncConnection, + name: str, + version: str, + /, + ) -> StoredSourceDefinitionManifest: + stored = await self.find(connection, name, version) + if stored is None: + raise RepositoryNotFoundError("source-definition-manifest", (name, version)) + return stored + + async def find( + self, + connection: AsyncConnection, + name: str, + version: str, + /, + ) -> StoredSourceDefinitionManifest | None: + row = ( + ( + await connection.execute( + select(SOURCE_DEFINITION_MANIFESTS_TABLE).where( + SOURCE_DEFINITION_MANIFESTS_TABLE.c.definition_name == name, + SOURCE_DEFINITION_MANIFESTS_TABLE.c.definition_version == version, + ) + ) + ) + .mappings() + .one_or_none() + ) + return None if row is None else _decode_row(row) + + +def _decode_row(row: Mapping[Any, Any]) -> StoredSourceDefinitionManifest: + name = str(row["definition_name"]) + version = str(row["definition_version"]) + fingerprint = str(row["fingerprint"]) + manifest = load_model( + SourceDefinitionManifest, + stored_bytes(row["manifest"], column="manifest"), + kind="source-definition-manifest", + name=name, + ) + indexed = (name, version, fingerprint) + decoded = (manifest.name, manifest.version, manifest.fingerprint) + if indexed != decoded: + raise IdentityMismatchError("source-definition-manifest", indexed, decoded) + return StoredSourceDefinitionManifest(manifest=manifest) + + +__all__ = ["SourceDefinitionManifestRepository", "StoredSourceDefinitionManifest"] diff --git a/src/powercontext/builtin/persistence/sources.py b/src/powercontext/builtin/persistence/sources.py index 7a9425702..9a09b3006 100644 --- a/src/powercontext/builtin/persistence/sources.py +++ b/src/powercontext/builtin/persistence/sources.py @@ -34,9 +34,9 @@ StoredPayloadConflictError, ) from powercontext.builtin.persistence.tables import SOURCE_JOURNAL_HEADS_TABLE, SOURCES_TABLE -from powercontext.errors import SourceAdapterNotFoundError, SourceConflictError +from powercontext.errors import SourceDefinitionNotFoundError from powercontext.limits import MAX_SCOPE_ID_LENGTH -from powercontext.sources import Source, SourceAdapter, SourceRef +from powercontext.sources import ProjectedSource, Source, SourceAdapter, SourceDefinitionRegistry, SourceRef _AnySourceAdapter = SourceAdapter[Any, Any, Any] @@ -50,18 +50,18 @@ class StoredSource(BaseModel): class SourceRepository: - """Persist Sources using their concrete adapter routes.""" - - def __init__(self, adapters: Iterable[_AnySourceAdapter], /) -> None: - self._by_name: dict[str, _AnySourceAdapter] = {} - self._by_source: dict[type[Source], _AnySourceAdapter] = {} - for adapter in adapters: - if adapter.name in self._by_name: - raise SourceConflictError("name", adapter.name) - if adapter.source_class in self._by_source: - raise SourceConflictError("source_class", adapter.source_class) - self._by_name[adapter.name] = adapter - self._by_source[adapter.source_class] = adapter + """Persist Sources using the Runtime's fixed Source Definition registry.""" + + def __init__( + self, + definitions: SourceDefinitionRegistry | Iterable[_AnySourceAdapter], + /, + ) -> None: + self._registry = ( + definitions + if isinstance(definitions, SourceDefinitionRegistry) + else SourceDefinitionRegistry.from_adapters(definitions) + ) async def add( self, @@ -73,9 +73,12 @@ async def add( """Add one stable Source or return an identical existing capture.""" _require_identity("scope_id", scope_id, MAX_SCOPE_ID_LENGTH) - adapter = self._adapter_for_value(source) - ref = SourceRef(source_type=adapter.name, source_id=source.name) - payload = dump_model(source, kind="source", name=adapter.name) + if isinstance(source, ProjectedSource): + ref = SourceRef(source_type=source.source_type, source_id=source.name) + else: + definition = self._registry.definition_for_source(source) + ref = SourceRef(source_type=definition.name, source_id=source.name) + payload = dump_model(source, kind="source", name=ref.source_type) await _lock_journal_head(connection, scope_id) existing = await self._find_row(connection, scope_id, ref) if existing is not None: @@ -163,17 +166,11 @@ async def journal_position(self, connection: AsyncConnection, scope_id: str, /) raise InvalidStoredColumnError("journal_position", "an integer") return int(value) - def _adapter_for_value(self, source: Source) -> _AnySourceAdapter: - try: - return self._by_source[type(source)] - except KeyError: - raise SourceAdapterNotFoundError("source", type(source)) from None - - def _adapter_by_name(self, name: str) -> _AnySourceAdapter: + def _definition_by_name(self, name: str) -> _AnySourceAdapter: try: - return self._by_name[name] - except KeyError: - raise RepositoryNotFoundError("source-adapter", name) from None + return self._registry.definition_for_name(name) + except SourceDefinitionNotFoundError: + raise RepositoryNotFoundError("source-definition", name) from None async def _find_row( self, @@ -198,15 +195,26 @@ async def _find_row( def _decode_row(self, row: Mapping[Any, Any]) -> StoredSource: source_type = str(row["source_type"]) source_id = str(row["source_id"]) - adapter = self._adapter_by_name(source_type) - source = load_model( - adapter.source_class, - stored_bytes(row["payload"], column="payload"), - kind="source", - name=source_type, - ) + try: + definition = self._definition_by_name(source_type) + except RepositoryNotFoundError: + source = load_model( + ProjectedSource, + stored_bytes(row["payload"], column="payload"), + kind="projected-source", + name=source_type, + ) + decoded = SourceRef(source_type=source.source_type, source_id=source.name) + else: + source = load_model( + definition.source_class, + stored_bytes(row["payload"], column="payload"), + kind="source", + name=source_type, + ) + self._registry.definition_for_source(source) + decoded = SourceRef(source_type=definition.name, source_id=source.name) indexed = SourceRef(source_type=source_type, source_id=source_id) - decoded = SourceRef(source_type=adapter.name, source_id=source.name) if indexed != decoded: raise IdentityMismatchError("source", indexed, decoded) return StoredSource( diff --git a/src/powercontext/builtin/persistence/tables.py b/src/powercontext/builtin/persistence/tables.py index 99e1013a6..f81ed8b18 100644 --- a/src/powercontext/builtin/persistence/tables.py +++ b/src/powercontext/builtin/persistence/tables.py @@ -274,6 +274,26 @@ def _entry_text_type(): CheckConstraint("generation >= 0", name="ck_pc_source_cursors_generation_nonnegative"), ) +CONNECTOR_CHECKPOINTS_TABLE = Table( + "pc_connector_checkpoints", + SHARED_METADATA, + Column("scope_id", identity_string(MAX_SCOPE_ID_LENGTH), primary_key=True), + Column("binding_id", identity_string(MAX_SOURCE_ID_LENGTH), primary_key=True), + Column("connector_name", identity_string(MAX_SOURCE_TYPE_LENGTH), nullable=False), + Column("connector_version", identity_string(MAX_SOURCE_TYPE_LENGTH), nullable=False), + Column("checkpoint", _canonical_payload_type(), nullable=False), +) + +SOURCE_DEFINITION_MANIFESTS_TABLE = Table( + "pc_source_definition_manifests", + SHARED_METADATA, + Column("definition_name", identity_string(MAX_SOURCE_TYPE_LENGTH), primary_key=True), + Column("definition_version", identity_string(MAX_SOURCE_TYPE_LENGTH), primary_key=True), + Column("fingerprint", identity_string(71), nullable=False), + Column("manifest", _canonical_payload_type(), nullable=False), + UniqueConstraint("definition_name", "fingerprint", name="uq_pc_source_definition_manifest_fingerprint"), +) + EXTERNAL_SKILL_REGISTRATIONS_TABLE = Table( "pc_external_skill_registrations", SHARED_METADATA, @@ -357,6 +377,8 @@ def _entry_text_type(): ARTIFACT_CANDIDATE_VERSIONS_TABLE, ARTIFACT_CANDIDATE_HEADS_TABLE, SOURCE_CURSORS_TABLE, + CONNECTOR_CHECKPOINTS_TABLE, + SOURCE_DEFINITION_MANIFESTS_TABLE, EXTERNAL_SKILL_REGISTRATIONS_TABLE, ) diff --git a/src/powercontext/builtin/runtime/__init__.py b/src/powercontext/builtin/runtime/__init__.py index ef7eba4e3..d965d64cc 100644 --- a/src/powercontext/builtin/runtime/__init__.py +++ b/src/powercontext/builtin/runtime/__init__.py @@ -45,6 +45,7 @@ ExternalSkillApplication, HandoffApplication, MemoryApplication, + RemoteIngestionApplication, ReviewApplication, ScheduledExperienceProcessor, ScheduledSourceProcessor, @@ -79,6 +80,8 @@ from powercontext.builtin.runtime.models import ( ApproveArtifactCandidateRequest, CaptureSource, + CommitConnectorCheckpoint, + ConnectorCheckpointState, ExperienceCandidate, ExperienceCandidatePage, ExperienceIncubationResult, @@ -120,8 +123,9 @@ SearchMemoryRequest, SkillCandidate, SourceReceipt, + SubmitSourceObservation, ) -from powercontext.builtin.runtime.protocols import PowerContextProvider +from powercontext.builtin.runtime.protocols import PowerContextProvider, RemoteIngestion from powercontext.builtin.runtime.readiness import ( CachedReadinessProbe, ReadinessCheckStatus, @@ -168,6 +172,8 @@ "CandidateFamilyCount", "CandidateInventoryStatistics", "CaptureSource", + "CommitConnectorCheckpoint", + "ConnectorCheckpointState", "DatabaseConfig", "ExperienceApplication", "ExperienceCandidate", @@ -245,6 +251,8 @@ "RecallTokenValue", "RejectArtifactCandidateRequest", "RememberMemoryRequest", + "RemoteIngestion", + "RemoteIngestionApplication", "ResolveExternalSkillRequest", "ResolvedUsagePeriod", "RetireMemoryEntryRequest", @@ -279,6 +287,7 @@ "Statistics", "StatisticsApplication", "StatisticsPeriod", + "SubmitSourceObservation", "UsageStatistics", "WorkApplication", "dependency_readiness_probe", diff --git a/src/powercontext/builtin/runtime/application.py b/src/powercontext/builtin/runtime/application.py index 750b7d20e..ddb28f89c 100644 --- a/src/powercontext/builtin/runtime/application.py +++ b/src/powercontext/builtin/runtime/application.py @@ -83,6 +83,8 @@ from powercontext.builtin.runtime.models import ( ApproveArtifactCandidateRequest, CaptureSource, + CommitConnectorCheckpoint, + ConnectorCheckpointState, ExperienceCandidate, ExperienceIncubationResult, ExternalSkillList, @@ -118,11 +120,13 @@ SearchMemoryRequest, SkillCandidate, SourceReceipt, + SubmitSourceObservation, ) from powercontext.builtin.runtime.prepared_context import PreparedContextBuild, PreparedContextBuilder from powercontext.builtin.runtime.protocols import ( BuiltinTriggers, PowerContextProvider, + RemoteIngestion, RuntimeSpan, RuntimeTracing, TraceAttribute, @@ -170,7 +174,7 @@ ) from powercontext.context import PowerContext from powercontext.errors import ArtifactNotFoundError, RevisionConflictError -from powercontext.sources import SourceRef +from powercontext.sources import ConnectorBinding, SourceDefinitionManifest, SourceRef if TYPE_CHECKING: from apscheduler.schedulers.asyncio import AsyncIOScheduler @@ -214,6 +218,7 @@ def __init__(self, code: str) -> None: "empty-write": "explicit Memory write did not produce a Memory", "experience-incubation": "Experience incubation is not configured", "external-skill-registry": "External Skill Registry is not configured", + "remote-ingestion": "Remote Source ingestion is not configured", "review": "Candidate Review services are not configured", "scheduler": "Built-in Runtime scheduler is already started", "statistics": "Statistics services are not configured", @@ -250,6 +255,35 @@ def for_scope(self, scope_id: str, /) -> ScopedSourceApplication: return ScopedSourceApplication(self._runtime, scope_id) +class RemoteIngestionApplication: + """Expose worker-owned Definition and observation operations.""" + + def __init__(self, runtime: BuiltinRuntime, service: RemoteIngestion | None) -> None: + self._runtime = runtime + self._service = service + + def _require_service(self) -> RemoteIngestion: + if self._service is None: + raise _RuntimeStateError("remote-ingestion") + return self._service + + async def register(self, manifest: SourceDefinitionManifest, /) -> SourceDefinitionManifest: + async with self._runtime._operation(): + return await self._require_service().register_source_definition(manifest) + + async def checkpoint(self, binding: ConnectorBinding, /) -> ConnectorCheckpointState: + async with self._runtime._operation(): + return await self._require_service().connector_checkpoint(binding) + + async def submit(self, request: SubmitSourceObservation, /) -> SourceReceipt: + async with self._runtime._operation(): + return await self._require_service().submit_source_observation(request) + + async def commit(self, request: CommitConnectorCheckpoint, /) -> ConnectorCheckpointState: + async with self._runtime._operation(): + return await self._require_service().commit_connector_checkpoint(request) + + class ScopedStatisticsApplication: """Read product statistics and record model usage for one scope.""" @@ -1230,6 +1264,7 @@ def __init__( readiness: RuntimeReadinessChecks | None = None, clock: Clock | None = None, tracing: RuntimeTracing | None = None, + remote_ingestion: RemoteIngestion | None = None, ) -> None: if source_window_limit < 1: raise _RuntimeConfigurationError("source_window_limit") @@ -1264,6 +1299,7 @@ def __init__( self._scheduler: AsyncIOScheduler | None = None self._scheduler_runtime_key: str | None = None self.sources = SourceApplication(self) + self.ingestion = RemoteIngestionApplication(self, remote_ingestion) self.context = ContextApplication(self) self.experience = ExperienceApplication(self) self.external_skills = ExternalSkillApplication(self) diff --git a/src/powercontext/builtin/runtime/composition.py b/src/powercontext/builtin/runtime/composition.py index b74668587..9a781af60 100644 --- a/src/powercontext/builtin/runtime/composition.py +++ b/src/powercontext/builtin/runtime/composition.py @@ -72,8 +72,9 @@ dependency_readiness_probe, ) from powercontext.builtin.runtime.relational import RelationalContexts -from powercontext.builtin.sources import CONTENT_SOURCE_NAME, ContentSource -from powercontext.sources import Source +from powercontext.builtin.sources import BUILTIN_SOURCE_REGISTRY, TEXT_EVIDENCE_PROJECTION_KEY +from powercontext.errors import SourceProjectionNotFoundError +from powercontext.sources import Source, SourceDefinitionRegistry, SourceProjectionKey if TYPE_CHECKING: from pydantic_ai.models.instrumented import InstrumentationSettings @@ -96,30 +97,30 @@ def __init__(self, issue: str) -> None: super().__init__(messages[issue]) -class _ContentEvidenceProjector(DefaultMemoryEvidenceProjector): +class _DefinitionEvidenceProjector(DefaultMemoryEvidenceProjector): + def __init__(self, definitions: SourceDefinitionRegistry, projection: SourceProjectionKey) -> None: + self._definitions = definitions + self._projection = projection + @override def project_source(self, source: Source, /) -> JsonValue: - if isinstance(source, ContentSource): - return { - "source_type": CONTENT_SOURCE_NAME, - "source_id": source.name, - "content": source.content, - "metadata": source.model_dump(mode="json")["metadata"], - } - return super().project_source(source) + try: + return self._definitions.project(source, self._projection) + except SourceProjectionNotFoundError: + return super().project_source(source) + +class _DefinitionHandoffEvidenceProjector(DefaultHandoffEvidenceProjector): + def __init__(self, definitions: SourceDefinitionRegistry, projection: SourceProjectionKey) -> None: + self._definitions = definitions + self._projection = projection -class _ContentHandoffEvidenceProjector(DefaultHandoffEvidenceProjector): @override def project_source(self, source: Source, /) -> JsonValue: - if isinstance(source, ContentSource): - return { - "source_type": CONTENT_SOURCE_NAME, - "source_id": source.name, - "content": source.content, - "metadata": source.model_dump(mode="json")["metadata"], - } - return super().project_source(source) + try: + return self._definitions.project(source, self._projection) + except SourceProjectionNotFoundError: + return super().project_source(source) class _TracingMemoryReranker: @@ -170,10 +171,12 @@ async def open_builtin_runtime( instrumentation: InstrumentationSettings | None = None, scope_cache_observer: ScopeCacheObserver | None = None, tracing: RuntimeTracing | None = None, + source_registry: SourceDefinitionRegistry | None = None, ) -> AsyncIterator[BuiltinRuntime]: """Open the selected database, inference adapters, and built-in runtime.""" async with AsyncExitStack() as resources: + configured_source_registry = source_registry or BUILTIN_SOURCE_REGISTRY ( generated_memory, generated_incubation, @@ -183,7 +186,13 @@ async def open_builtin_runtime( generated_reranker, generation_readiness, ) = ( - await _generation_pipelines(config.inference, config.runtime, resources, instrumentation) + await _generation_pipelines( + config.inference, + config.runtime, + resources, + instrumentation, + configured_source_registry, + ) if ( candidate_pipeline is None or experience_pipeline is None @@ -237,6 +246,7 @@ async def open_builtin_runtime( embedding_model=configured_embedding, token_estimator=token_estimator, memory_reranker=configured_reranker, + source_registry=configured_source_registry, ) ) readiness_probes: dict[str, ReadinessProbeDefinition] = { @@ -281,6 +291,7 @@ async def open_builtin_runtime( recall_token_estimator=contexts.estimate_recall_tokens, readiness=RuntimeReadinessChecks(readiness_probes), tracing=tracing, + remote_ingestion=contexts, ) ) if config.handoff_report.enabled: @@ -318,6 +329,7 @@ async def open_builtin_contexts( embedding_model: EmbeddingModel | None = None, token_estimator: TokenEstimator | None = None, memory_reranker: MemoryReranker | None = None, + source_registry: SourceDefinitionRegistry | None = None, ) -> AsyncIterator[RelationalContexts]: """Open the selected database and expose scope-bound PowerContext providers.""" @@ -352,6 +364,7 @@ async def open_builtin_contexts( token_estimator=configured_token_estimator, memory_reranker=memory_reranker, memory_rerank_candidate_limit=config.runtime.memory_rerank_candidate_limit, + source_registry=source_registry, ) return experience_index = OceanBaseExperienceFTSIndex() @@ -384,6 +397,7 @@ async def open_builtin_contexts( token_estimator=configured_token_estimator, memory_reranker=memory_reranker, memory_rerank_candidate_limit=config.runtime.memory_rerank_candidate_limit, + source_registry=source_registry, ) @@ -392,6 +406,7 @@ async def _generation_pipelines( runtime: RuntimeConfig, resources: AsyncExitStack, instrumentation: InstrumentationSettings | None, + source_registry: SourceDefinitionRegistry, ) -> tuple[ CandidatePipeline | None, ExperienceCandidatePipeline | None, @@ -512,14 +527,14 @@ async def probe_generation() -> None: return ( LLMMemoryCandidatePipeline( UsageReportingStructuredGenerator(memory_generator), - evidence_projector=_ContentEvidenceProjector(), + evidence_projector=_DefinitionEvidenceProjector(source_registry, TEXT_EVIDENCE_PROJECTION_KEY), ), LLMExperienceCandidatePipeline(UsageReportingStructuredGenerator(experience_generator)), LLMExperienceGenerator(UsageReportingStructuredGenerator(explicit_experience_generator)), LLMSkillGenerator(UsageReportingStructuredGenerator(skill_generator)), LLMHandoffGenerationPipeline( UsageReportingStructuredGenerator(handoff_generator), - evidence_projector=_ContentHandoffEvidenceProjector(), + evidence_projector=_DefinitionHandoffEvidenceProjector(source_registry, TEXT_EVIDENCE_PROJECTION_KEY), ), (None if rerank_generator is None else LLMMemoryReranker(UsageReportingStructuredGenerator(rerank_generator))), CachedReadinessProbe(dependency_readiness_probe(probe_generation)), diff --git a/src/powercontext/builtin/runtime/models.py b/src/powercontext/builtin/runtime/models.py index 95e8c4cf1..42dd8decf 100644 --- a/src/powercontext/builtin/runtime/models.py +++ b/src/powercontext/builtin/runtime/models.py @@ -49,7 +49,7 @@ ) from powercontext.builtin.review.generation import SkillGenerationOrigin from powercontext.builtin.sources import ExternalSkillImportMode -from powercontext.sources import SourceRef +from powercontext.sources import ConnectorBinding, ProjectedSource, SourceDefinitionManifest, SourceRef PreparedContextSchema: TypeAlias = Literal["powercontext.prepared-context.v1"] PreparedContextStatus: TypeAlias = Literal["ready", "empty"] @@ -77,6 +77,34 @@ class SourceReceipt(BaseModel): sequence: int +class RegisterSourceDefinition(BaseModel): + """Register one immutable worker-owned Source Definition manifest.""" + + manifest: SourceDefinitionManifest + + +class SubmitSourceObservation(BaseModel): + """Submit one worker-materialized observation for durable acceptance.""" + + binding: ConnectorBinding + source: ProjectedSource + + +class ConnectorCheckpointState(BaseModel): + """Current opaque checkpoint for one exact Connector binding.""" + + binding: ConnectorBinding + checkpoint: JsonValue | None + + +class CommitConnectorCheckpoint(BaseModel): + """Compare and replace one binding checkpoint after durable submissions.""" + + binding: ConnectorBinding + expected: JsonValue | None + checkpoint: JsonValue | None + + class RuntimeCapabilities(BaseModel): """Behavior available from the assembled Source-to-Memory Runtime.""" diff --git a/src/powercontext/builtin/runtime/protocols.py b/src/powercontext/builtin/runtime/protocols.py index 480b019a1..a2b71be08 100644 --- a/src/powercontext/builtin/runtime/protocols.py +++ b/src/powercontext/builtin/runtime/protocols.py @@ -21,9 +21,16 @@ from typing import Protocol, TypeVar from powercontext.builtin.artifacts.handoff import ActivateHandoff, HandoffActivation -from powercontext.builtin.runtime.models import MemoryFlushResult +from powercontext.builtin.runtime.models import ( + CommitConnectorCheckpoint, + ConnectorCheckpointState, + MemoryFlushResult, + SourceReceipt, + SubmitSourceObservation, +) from powercontext.builtin.sources import SourceCursor from powercontext.context import PowerContext +from powercontext.sources import ConnectorBinding, SourceDefinitionManifest SourcesT = TypeVar("SourcesT", covariant=True) ArtifactsT = TypeVar("ArtifactsT", covariant=True) @@ -64,6 +71,26 @@ class PowerContextProvider(Protocol[SourcesT, ArtifactsT, TriggersT]): async def get(self, scope_id: str, /) -> PowerContext[SourcesT, ArtifactsT, TriggersT]: ... +class RemoteIngestion(Protocol): + """Server-side authority used by independent Connector workers.""" + + async def register_source_definition( + self, + manifest: SourceDefinitionManifest, + /, + ) -> SourceDefinitionManifest: ... + + async def connector_checkpoint(self, binding: ConnectorBinding, /) -> ConnectorCheckpointState: ... + + async def submit_source_observation(self, request: SubmitSourceObservation, /) -> SourceReceipt: ... + + async def commit_connector_checkpoint( + self, + request: CommitConnectorCheckpoint, + /, + ) -> ConnectorCheckpointState: ... + + class BuiltinTriggers(Protocol): """Atomically execute the built-in Trigger policies for one scope.""" diff --git a/src/powercontext/builtin/runtime/relational.py b/src/powercontext/builtin/runtime/relational.py index 534a828f5..7cb187b4f 100644 --- a/src/powercontext/builtin/runtime/relational.py +++ b/src/powercontext/builtin/runtime/relational.py @@ -17,11 +17,15 @@ from __future__ import annotations import asyncio -from collections.abc import Callable +from collections.abc import Callable, Mapping from dataclasses import dataclass from typing import Any, cast from uuid import uuid4 +from jsonschema import Draft202012Validator +from jsonschema.exceptions import SchemaError +from jsonschema.exceptions import ValidationError as JsonSchemaValidationError +from jsonschema.protocols import Validator from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncConnection @@ -63,6 +67,10 @@ from powercontext.builtin.inference import EmbeddingModel, InvalidInferenceOutputError, TokenEstimator from powercontext.builtin.persistence.artifacts import ArtifactRepository from powercontext.builtin.persistence.candidates import CandidateRepository +from powercontext.builtin.persistence.connectors import ( + ConnectorCheckpointRepository, + RelationalConnectorCheckpointStore, +) from powercontext.builtin.persistence.cursors import SourceCursorRepository from powercontext.builtin.persistence.database import AsyncDatabase from powercontext.builtin.persistence.errors import RepositoryNotFoundError, StoredPayloadConflictError @@ -74,6 +82,7 @@ ) from powercontext.builtin.persistence.memory import RelationalMemoryBackend from powercontext.builtin.persistence.memory_index import MemoryIndex, NoMemoryIndex +from powercontext.builtin.persistence.source_definitions import SourceDefinitionManifestRepository from powercontext.builtin.persistence.sources import SourceRepository, StoredSource from powercontext.builtin.persistence.statistics import StatisticsRepository from powercontext.builtin.persistence.tables import ARTIFACT_HEADS_TABLE, SOURCE_JOURNAL_HEADS_TABLE @@ -84,13 +93,20 @@ SkillGenerationOrigin, ) from powercontext.builtin.review.service import ReviewService -from powercontext.builtin.runtime.models import ExperienceIncubationResult, MemoryFlushResult +from powercontext.builtin.runtime.models import ( + CommitConnectorCheckpoint, + ConnectorCheckpointState, + ExperienceIncubationResult, + MemoryFlushResult, + SourceReceipt, + SubmitSourceObservation, +) from powercontext.builtin.runtime.prepared_context import PreparedContextBuild from powercontext.builtin.runtime.protocols import BuiltinTriggers from powercontext.builtin.runtime.recall import RelationalRecallTokenEstimator from powercontext.builtin.runtime.statistics import RelationalScopedStatistics from powercontext.builtin.sources import ( - CONTENT_SOURCE_ADAPTER, + BUILTIN_SOURCE_REGISTRY, EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER, ExternalSkillImportMode, ExternalSkillSnapshotCapture, @@ -109,19 +125,27 @@ SourceWindowTrigger, ) from powercontext.context import PowerContext -from powercontext.errors import ArtifactNotFoundError, SourceConflictError, SourceNotFoundError +from powercontext.errors import ( + ArtifactNotFoundError, + InvalidSourceDefinitionError, + InvalidSourceObservationError, + SourceConflictError, + SourceDefinitionNotFoundError, + SourceNotFoundError, +) from powercontext.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + ConnectorBinding, + ProjectedSource, Source, - SourceAdapter, SourceCatalog, + SourceDefinitionManifest, + SourceDefinitionRegistry, SourceRef, + TextEvidence, ) IdFactory = Callable[[str], str] -_SOURCE_ADAPTERS: tuple[SourceAdapter[Any, Any, Any], ...] = ( - CONTENT_SOURCE_ADAPTER, - EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER, -) @dataclass(frozen=True, slots=True) @@ -131,6 +155,8 @@ class _Repositories: sources: SourceRepository artifacts: ArtifactRepository candidates: CandidateRepository + connector_checkpoints: ConnectorCheckpointRepository + source_definitions: SourceDefinitionManifestRepository cursors: SourceCursorRepository external_skills: ExternalSkillRepository statistics: StatisticsRepository @@ -158,6 +184,7 @@ class _ScopedServices: memory_artifact_id: str source_lock: asyncio.Lock token_estimator: TokenEstimator | None + source_registry: SourceDefinitionRegistry def sources( self, @@ -166,12 +193,12 @@ def sources( backend = _RelationalSources( database=self.database, scope_id=self.scope_id, - adapters=_SOURCE_ADAPTERS, + registry=self.source_registry, repository=self.repositories.sources, write_lock=self.source_lock, connection=connection, ) - return backend, SourceCatalog(backend=backend, adapters=_SOURCE_ADAPTERS) + return backend, SourceCatalog(backend=backend, registry=self.source_registry) def memory( self, @@ -303,17 +330,21 @@ def __init__( id_factory: IdFactory | None = None, handoff_artifact_id: str = "handoff", memory_artifact_id: str = "memory", + source_registry: SourceDefinitionRegistry | None = None, ) -> None: self.database = database + self.source_registry = source_registry or BUILTIN_SOURCE_REGISTRY self.index = NoMemoryIndex() if index is None else index self.experience_index = NoExperienceIndex() if experience_index is None else experience_index self.repositories = _Repositories( - sources=SourceRepository(_SOURCE_ADAPTERS), + sources=SourceRepository(self.source_registry), artifacts=ArtifactRepository((Handoff, Memory, Experience, Skill)), candidates=CandidateRepository({ Experience.family: ExperienceContent, Skill.family: SkillContent, }), + connector_checkpoints=ConnectorCheckpointRepository(), + source_definitions=SourceDefinitionManifestRepository(), cursors=SourceCursorRepository(), external_skills=ExternalSkillRepository(), statistics=StatisticsRepository(), @@ -369,6 +400,70 @@ def statistics(self, scope_id: str, /) -> RelationalScopedStatistics: return self._services_for(scope_id).statistics() + async def register_source_definition( + self, + manifest: SourceDefinitionManifest, + /, + ) -> SourceDefinitionManifest: + """Register one immutable declarative Definition supplied by a worker.""" + + _validate_source_definition_manifest(manifest) + try: + async with self.database.transaction() as connection: + stored = await self.repositories.source_definitions.register(connection, manifest) + except StoredPayloadConflictError as error: + raise SourceConflictError("definition-manifest", error.identity) from None + return stored.manifest + + async def connector_checkpoint(self, binding: ConnectorBinding, /) -> ConnectorCheckpointState: + """Read the checkpoint owned by one remote Connector binding.""" + + checkpoint = await RelationalConnectorCheckpointStore( + self.database, + self.repositories.connector_checkpoints, + ).load(binding) + return ConnectorCheckpointState(binding=binding, checkpoint=checkpoint) + + async def submit_source_observation( + self, + request: SubmitSourceObservation, + /, + ) -> SourceReceipt: + """Validate and durably append one worker-materialized Source observation.""" + + source = request.source + try: + async with self.database.transaction() as connection: + stored_manifest = await self.repositories.source_definitions.get( + connection, + source.source_type, + source.definition_version, + ) + except RepositoryNotFoundError: + raise SourceDefinitionNotFoundError(source.source_type, source.definition_version) from None + _validate_projected_source(source, stored_manifest.manifest) + services = self._services_for(request.binding.scope_id) + source_store, source_catalog = services.sources() + stored = await source_store.add(source) + return SourceReceipt( + source_ref=source_catalog.as_ref(stored), + sequence=await source_store.position(stored), + ) + + async def commit_connector_checkpoint( + self, + request: CommitConnectorCheckpoint, + /, + ) -> ConnectorCheckpointState: + """Commit one worker checkpoint only when its starting value still matches.""" + + store = RelationalConnectorCheckpointStore( + self.database, + self.repositories.connector_checkpoints, + ) + await store.save(request.binding, request.checkpoint, expected=request.expected) + return ConnectorCheckpointState(binding=request.binding, checkpoint=request.checkpoint) + async def estimate_recall_tokens( self, scope_id: str, @@ -522,6 +617,7 @@ def _services_for(self, scope_id: str) -> _ScopedServices: memory_artifact_id=self._memory_artifact_id, source_lock=self._source_locks.setdefault(scope, asyncio.Lock()), token_estimator=self._token_estimator, + source_registry=self.source_registry, ) @@ -531,14 +627,14 @@ def __init__( *, database: AsyncDatabase, scope_id: str, - adapters: tuple[SourceAdapter[Any, Any, Any], ...], + registry: SourceDefinitionRegistry, repository: SourceRepository, write_lock: asyncio.Lock, connection: AsyncConnection | None = None, ) -> None: self._database = database self._scope_id = scope_id - self._source_names = {adapter.source_class: adapter.name for adapter in adapters} + self._registry = registry self._repository = repository self._write_lock = write_lock self._bound_connection = connection @@ -585,7 +681,10 @@ async def entries(self) -> tuple[SourceJournalEntry, ...]: ) def _as_ref(self, source: Source) -> SourceRef: - return SourceRef(source_type=self._source_names[type(source)], source_id=source.name) + if isinstance(source, ProjectedSource): + return SourceRef(source_type=source.source_type, source_id=source.name) + definition = self._registry.definition_for_source(source) + return SourceRef(source_type=definition.name, source_id=source.name) class _RelationalArtifactResolver: @@ -850,6 +949,67 @@ def _validate_experience_plans( ) +def _validate_source_definition_manifest(manifest: SourceDefinitionManifest) -> None: + if len(manifest.model_dump_json(by_alias=True).encode()) > 64 * 1024: + raise InvalidSourceDefinitionError(type(manifest), "manifest", "must not exceed 64 KiB") + try: + BUILTIN_SOURCE_REGISTRY.definition_for_name(manifest.name) + except SourceDefinitionNotFoundError: + pass + else: + raise InvalidSourceDefinitionError(type(manifest), "name", "must not replace a built-in Source Definition") + _json_schema_validator(manifest.name, manifest.source_schema) + standard_text_schema = TextEvidence.model_json_schema() + for projection in manifest.projections: + _json_schema_validator(projection.key.name, projection.schema_) + if projection.key == TEXT_EVIDENCE_PROJECTION_KEY and projection.schema_ != standard_text_schema: + raise InvalidSourceDefinitionError( + type(manifest), + "projection", + f"{projection.key.name!r} must use the standard schema", + ) + + +def _validate_projected_source(source: ProjectedSource, manifest: SourceDefinitionManifest) -> None: + if source.source_type != manifest.name or source.definition_version != manifest.version: + raise InvalidSourceObservationError("definition", "does not match the registered manifest identity") + if source.definition_fingerprint != manifest.fingerprint: + raise InvalidSourceObservationError("fingerprint", "does not match the registered manifest") + if len(source.model_dump_json().encode()) > 4 * 1024 * 1024: + raise InvalidSourceObservationError("size", "must not exceed 4 MiB") + _validate_schema_value(manifest.name, manifest.source_schema, source.payload) + + declarations = {projection.key: projection for projection in manifest.projections} + supplied = {projection.key: projection.value for projection in source.projections} + if declarations.keys() != supplied.keys(): + raise InvalidSourceObservationError("projections", "must exactly match the registered manifest") + for key, declaration in declarations.items(): + value = supplied[key] + _validate_schema_value(key.name, declaration.schema_, value) + if key == TEXT_EVIDENCE_PROJECTION_KEY: + evidence = TextEvidence.model_validate(value) + if evidence.source_type != source.source_type or evidence.source_id != source.name: + raise InvalidSourceObservationError( + "text-evidence", + "source identity does not match the observation envelope", + ) + + +def _json_schema_validator(name: str, schema: Mapping[str, Any]) -> Validator: + try: + Draft202012Validator.check_schema(schema) + return Draft202012Validator(schema) + except SchemaError as error: + raise InvalidSourceDefinitionError(type(schema), "schema", f"{name!r} is not valid JSON Schema") from error + + +def _validate_schema_value(name: str, schema: Mapping[str, Any], value: object) -> None: + try: + _json_schema_validator(name, schema).validate(value) + except JsonSchemaValidationError as error: + raise InvalidSourceObservationError("schema", f"value does not match {name!r}") from error + + def _scoped_id_factory(memory_artifact_id: str, delegate: IdFactory | None) -> IdFactory: def new_id(kind: str) -> str: if kind == "memory": diff --git a/src/powercontext/builtin/sources/__init__.py b/src/powercontext/builtin/sources/__init__.py index d596a4588..66a50d50e 100644 --- a/src/powercontext/builtin/sources/__init__.py +++ b/src/powercontext/builtin/sources/__init__.py @@ -16,13 +16,16 @@ from powercontext.builtin.sources.content import ( CONTENT_SOURCE_ADAPTER, + CONTENT_SOURCE_DEFINITION, CONTENT_SOURCE_NAME, ContentCapture, ContentSource, ContentSourceAdapter, + ContentTextEvidenceProjection, ) from powercontext.builtin.sources.external_skill import ( EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER, + EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION, EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME, ExternalSkillImportMode, ExternalSkillSnapshotCapture, @@ -35,15 +38,26 @@ SourceJournalEntry, validate_scope_id, ) +from powercontext.sources import TEXT_EVIDENCE_PROJECTION_KEY, SourceDefinitionRegistry, TextEvidence + +BUILTIN_SOURCE_REGISTRY = SourceDefinitionRegistry(( + CONTENT_SOURCE_DEFINITION, + EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION, +)) __all__ = [ + "BUILTIN_SOURCE_REGISTRY", "CONTENT_SOURCE_ADAPTER", + "CONTENT_SOURCE_DEFINITION", "CONTENT_SOURCE_NAME", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER", + "EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME", + "TEXT_EVIDENCE_PROJECTION_KEY", "ContentCapture", "ContentSource", "ContentSourceAdapter", + "ContentTextEvidenceProjection", "ExternalSkillImportMode", "ExternalSkillSnapshotCapture", "ExternalSkillSnapshotSource", @@ -51,5 +65,6 @@ "SourceCursor", "SourceJournal", "SourceJournalEntry", + "TextEvidence", "validate_scope_id", ] diff --git a/src/powercontext/builtin/sources/content.py b/src/powercontext/builtin/sources/content.py index d516f0471..34a7c0157 100644 --- a/src/powercontext/builtin/sources/content.py +++ b/src/powercontext/builtin/sources/content.py @@ -20,6 +20,7 @@ from pydantic import BaseModel, Field, JsonValue, field_validator +from powercontext.sources import TEXT_EVIDENCE_PROJECTION_KEY, AdapterSourceDefinition, TextEvidence from powercontext.sources.models import Source, SourceMaterialization CONTENT_SOURCE_NAME = "content" @@ -71,4 +72,25 @@ async def read(self, source: ContentSource, /) -> ContentCapture: ) +class ContentTextEvidenceProjection: + """Expose captured text without coupling consumers to ``ContentSource``.""" + + name = TEXT_EVIDENCE_PROJECTION_KEY.name + version = TEXT_EVIDENCE_PROJECTION_KEY.version + source_class = ContentSource + output_class: type[BaseModel] = TextEvidence + + def project(self, source: ContentSource, /) -> TextEvidence: + return TextEvidence( + source_type=CONTENT_SOURCE_NAME, + source_id=source.name, + content=source.content, + metadata=source.metadata, + ) + + CONTENT_SOURCE_ADAPTER = ContentSourceAdapter() +CONTENT_SOURCE_DEFINITION = AdapterSourceDefinition( + CONTENT_SOURCE_ADAPTER, + projections=(ContentTextEvidenceProjection(),), +) diff --git a/src/powercontext/builtin/sources/external_skill.py b/src/powercontext/builtin/sources/external_skill.py index 2aa9d6ad4..f49ccbb84 100644 --- a/src/powercontext/builtin/sources/external_skill.py +++ b/src/powercontext/builtin/sources/external_skill.py @@ -22,7 +22,7 @@ from pydantic import BaseModel from powercontext.builtin.artifacts.skill import ExternalSkillSnapshot -from powercontext.sources import Source, SourceMaterialization +from powercontext.sources import AdapterSourceDefinition, Source, SourceMaterialization EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME = "external-skill-snapshot" @@ -82,9 +82,11 @@ def _snapshot_id(value: ExternalSkillSnapshotCapture) -> str: EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER = ExternalSkillSnapshotSourceAdapter() +EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION = AdapterSourceDefinition(EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER) __all__ = [ "EXTERNAL_SKILL_SNAPSHOT_SOURCE_ADAPTER", + "EXTERNAL_SKILL_SNAPSHOT_SOURCE_DEFINITION", "EXTERNAL_SKILL_SNAPSHOT_SOURCE_NAME", "ExternalSkillImportMode", "ExternalSkillSnapshotCapture", diff --git a/src/powercontext/client/__init__.py b/src/powercontext/client/__init__.py index 55ca5bd9e..673ffb321 100644 --- a/src/powercontext/client/__init__.py +++ b/src/powercontext/client/__init__.py @@ -16,11 +16,19 @@ from powercontext.client.client import PowerContextClient from powercontext.client.errors import ClientError, InvalidResponseError, ServerResponseError, TransportError +from powercontext.client.ingestion import ( + RemoteConnectorCheckpointStore, + RemoteConnectorSourceSink, + RemoteConnectorWorker, +) __all__ = [ "ClientError", "InvalidResponseError", "PowerContextClient", + "RemoteConnectorCheckpointStore", + "RemoteConnectorSourceSink", + "RemoteConnectorWorker", "ServerResponseError", "TransportError", ] diff --git a/src/powercontext/client/client.py b/src/powercontext/client/client.py index 2c1618f2f..47d60929e 100644 --- a/src/powercontext/client/client.py +++ b/src/powercontext/client/client.py @@ -35,8 +35,10 @@ Capabilities, CaptureContentSourceRequest, CaptureContentSourceResponse, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -51,6 +53,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -97,6 +100,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ResolveExternalSkillRequest, @@ -109,7 +113,10 @@ SearchMemoryRequest, SearchMemoryResponse, SkillArtifact, + SourceDefinitionManifest, + SourceObservationReceipt, StoredHandoffReportActivity, + SubmitSourceObservationRequest, UpdateHandoffReportProjectRequest, UpdateHandoffReportWorkstreamRequest, WorkSourceReceipt, @@ -122,6 +129,7 @@ APPROVE_ARTIFACT_CANDIDATE, ATTACH_HANDOFF_REPORT_WORKSPACE, CAPTURE_CONTENT_SOURCE, + COMMIT_CONNECTOR_CHECKPOINT, COMMIT_HANDOFF, CONTINUE_HANDOFF, CREATE_HANDOFF_REPORT_PROJECT, @@ -133,6 +141,7 @@ GENERATE_SKILL, GET_ARTIFACT_CANDIDATE, GET_CAPABILITIES, + GET_CONNECTOR_CHECKPOINT, GET_EXPERIENCE, GET_HANDOFF_REPORT, GET_HANDOFF_REPORT_PROJECT, @@ -160,6 +169,7 @@ RECORD_HANDOFF_REPORT_ACTIVITY, RECORD_TASK_OUTCOME, REGISTER_HANDOFF_REPORT_WORKSTREAM, + REGISTER_SOURCE_DEFINITION, REJECT_ARTIFACT_CANDIDATE, REMEMBER_MEMORY, RESOLVE_EXTERNAL_SKILL, @@ -168,6 +178,7 @@ REVISE_MEMORY_ENTRY, SCAN_EXTERNAL_SKILLS, SEARCH_MEMORY, + SUBMIT_SOURCE_OBSERVATION, UPDATE_HANDOFF_REPORT_PROJECT, UPDATE_HANDOFF_REPORT_WORKSTREAM, Operation, @@ -423,6 +434,29 @@ async def capture_content_source(self, request: CaptureContentSourceRequest) -> return await self._request(CAPTURE_CONTENT_SOURCE, request) + async def register_source_definition(self, request: RegisterSourceDefinitionRequest) -> SourceDefinitionManifest: + """Register one immutable worker-owned Source Definition manifest.""" + + return await self._request(REGISTER_SOURCE_DEFINITION, request) + + async def get_connector_checkpoint(self, request: GetConnectorCheckpointRequest) -> ConnectorCheckpointState: + """Read the current opaque checkpoint for one Connector binding.""" + + return await self._request(GET_CONNECTOR_CHECKPOINT, request) + + async def submit_source_observation(self, request: SubmitSourceObservationRequest) -> SourceObservationReceipt: + """Submit one worker-materialized Source observation.""" + + return await self._request(SUBMIT_SOURCE_OBSERVATION, request) + + async def commit_connector_checkpoint( + self, + request: CommitConnectorCheckpointRequest, + ) -> ConnectorCheckpointState: + """Commit a binding checkpoint using optimistic comparison.""" + + return await self._request(COMMIT_CONNECTOR_CHECKPOINT, request) + async def create_work_contract(self, request: CreateWorkContractRequest) -> WorkSourceReceipt: """Create one grounded delegation baseline as durable Source evidence.""" diff --git a/src/powercontext/client/ingestion.py b/src/powercontext/client/ingestion.py new file mode 100644 index 000000000..eb049050f --- /dev/null +++ b/src/powercontext/client/ingestion.py @@ -0,0 +1,176 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Run worker-owned Connectors against the remote ingestion contract.""" + +from __future__ import annotations + +from pydantic import JsonValue +from typing_extensions import override + +from powercontext.client.client import PowerContextClient +from powercontext.errors import InvalidConnectorRunError +from powercontext.http import ( + CommitConnectorCheckpointRequest, + GetConnectorCheckpointRequest, + RegisterSourceDefinitionRequest, + SubmitSourceObservationRequest, +) +from powercontext.http import ( + ConnectorBinding as HttpConnectorBinding, +) +from powercontext.http import ( + ProjectedSource as HttpProjectedSource, +) +from powercontext.http import ( + SourceDefinitionManifest as HttpSourceDefinitionManifest, +) +from powercontext.sources import ( + Connector, + ConnectorBinding, + ConnectorCheckpointStore, + ConnectorLifecycle, + ConnectorRunResult, + ConnectorSourceSink, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, + SourceDefinitionRegistry, + SourceRef, + manifest_for_definition, + project_source_for_transport, + validate_connector, +) + + +class RemoteConnectorSourceSink(ConnectorSourceSink): + """Resolve and project Definition-native values inside the worker.""" + + def __init__(self, *, client: PowerContextClient, registry: SourceDefinitionRegistry) -> None: + self._client = client + self._registry = registry + + @override + async def submit( + self, + binding: ConnectorBinding, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: + del item_id + self._registry.definition_for_name(definition_name) + source = await self._registry.resolve(value) + projected = project_source_for_transport(self._registry, source) + if projected.source_type != definition_name: + raise InvalidConnectorRunError( + "definition-mismatch", + f"input resolved as {projected.source_type!r}, expected {definition_name!r}", + ) + receipt = await self._client.submit_source_observation( + SubmitSourceObservationRequest( + binding=_http_binding(binding), + source=HttpProjectedSource.model_validate(projected.model_dump(mode="json")), + ) + ) + source_ref = SourceRef(source_type=receipt.source.name, source_id=receipt.source.source_id) + expected_ref = SourceRef(source_type=projected.source_type, source_id=projected.name) + if source_ref != expected_ref: + raise InvalidConnectorRunError("identity-mismatch", "Server receipt changed the accepted Source identity") + return ConnectorSubmissionResult(status=ConnectorSubmissionStatus.ACCEPTED, source_ref=source_ref) + + +class RemoteConnectorCheckpointStore(ConnectorCheckpointStore): + """Load and compare-and-swap opaque checkpoints through the Server API.""" + + def __init__(self, client: PowerContextClient) -> None: + self._client = client + + @override + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: + state = await self._client.get_connector_checkpoint( + GetConnectorCheckpointRequest(binding=_http_binding(binding)) + ) + _validate_checkpoint_binding(binding, state.binding) + return state.checkpoint + + @override + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: + state = await self._client.commit_connector_checkpoint( + CommitConnectorCheckpointRequest( + binding=_http_binding(binding), + expected=expected, + checkpoint=checkpoint, + ) + ) + _validate_checkpoint_binding(binding, state.binding) + if state.checkpoint != checkpoint: + raise InvalidConnectorRunError("checkpoint-mismatch", "Server returned a different Connector checkpoint") + + +class RemoteConnectorWorker: + """Register worker-owned Definitions and execute one Connector binding.""" + + def __init__(self, *, client: PowerContextClient, registry: SourceDefinitionRegistry) -> None: + self._client = client + self._registry = registry + self._lifecycle = ConnectorLifecycle( + sink=RemoteConnectorSourceSink(client=client, registry=registry), + checkpoints=RemoteConnectorCheckpointStore(client), + ) + + async def run(self, connector: Connector, binding: ConnectorBinding, /) -> ConnectorRunResult: + source_definitions, _ = validate_connector(connector, binding) + for definition_name in sorted(source_definitions): + definition = self._registry.definition_for_name(definition_name) + manifest = manifest_for_definition(definition) + registered = await self._client.register_source_definition( + RegisterSourceDefinitionRequest( + manifest=HttpSourceDefinitionManifest.model_validate( + manifest.model_dump(mode="json", by_alias=True) + ) + ) + ) + if registered.model_dump(mode="json", by_alias=True) != manifest.model_dump(mode="json", by_alias=True): + raise InvalidConnectorRunError( + "manifest-mismatch", + f"Server returned a different manifest for {definition.name!r}", + ) + return await self._lifecycle.run(connector, binding) + + +def _http_binding(binding: ConnectorBinding) -> HttpConnectorBinding: + return HttpConnectorBinding.model_validate(binding.model_dump(mode="json")) + + +def _validate_checkpoint_binding( + expected_binding: ConnectorBinding, + actual_binding: HttpConnectorBinding, +) -> None: + if actual_binding.model_dump(mode="json") != expected_binding.model_dump(mode="json"): + raise InvalidConnectorRunError("binding-mismatch", "Server returned a different Connector binding") + + +__all__ = [ + "RemoteConnectorCheckpointStore", + "RemoteConnectorSourceSink", + "RemoteConnectorWorker", +] diff --git a/src/powercontext/errors.py b/src/powercontext/errors.py index 81214d4e9..def1e0515 100644 --- a/src/powercontext/errors.py +++ b/src/powercontext/errors.py @@ -99,6 +99,80 @@ def __init__( ) +class InvalidSourceDefinitionError(SourceError, TypeError): + """Raised when a Source Definition violates its registration contract.""" + + def __init__(self, definition_type: type[object], field: str, detail: str) -> None: + self.definition_type = definition_type + self.field = field + self.detail = detail + super().__init__(f"invalid Source Definition {_type_name(definition_type)} {field}: {detail}") + + +class SourceDefinitionNotFoundError(SourceError, LookupError): + """Raised when the active registry does not contain a Source Definition.""" + + def __init__(self, name: str, version: str | None = None) -> None: + self.name = name + self.version = version + suffix = "" if version is None else f" version {version!r}" + super().__init__(f"Source Definition {name!r}{suffix} is not registered") + + +class SourceProjectionNotFoundError(SourceError, LookupError): + """Raised when a Source Definition does not provide a requested projection.""" + + def __init__(self, source_type: str, projection_name: str, projection_version: str) -> None: + self.source_type = source_type + self.projection_name = projection_name + self.projection_version = projection_version + super().__init__( + f"Source Definition {source_type!r} does not provide projection " + f"{projection_name!r} version {projection_version!r}" + ) + + +class InvalidSourceProjectionError(SourceError, TypeError): + """Raised when a named Source projection violates its declared contract.""" + + def __init__(self, projection_name: str, field: str, detail: str) -> None: + self.projection_name = projection_name + self.field = field + self.detail = detail + super().__init__(f"invalid Source projection {projection_name!r} {field}: {detail}") + + +class InvalidSourceObservationError(SourceError, ValueError): + """Raised when a worker-projected observation violates its registered manifest.""" + + def __init__(self, issue: str, detail: str) -> None: + self.issue = issue + self.detail = detail + super().__init__(f"invalid Source observation {issue}: {detail}") + + +class ConnectorError(PowerContextError): + """Base exception for Connector contracts and run lifecycle failures.""" + + +class InvalidConnectorError(ConnectorError, TypeError): + """Raised when a Connector or binding violates its declared contract.""" + + def __init__(self, field: str, detail: str) -> None: + self.field = field + self.detail = detail + super().__init__(f"invalid Connector {field}: {detail}") + + +class InvalidConnectorRunError(ConnectorError, RuntimeError): + """Raised when a Connector run would violate replay or checkpoint safety.""" + + def __init__(self, issue: str, detail: str) -> None: + self.issue = issue + self.detail = detail + super().__init__(f"invalid Connector run {issue}: {detail}") + + class ArtifactError(PowerContextError): """Base exception for Artifact lookup and lifecycle failures.""" diff --git a/src/powercontext/http/__init__.py b/src/powercontext/http/__init__.py index 56c05bf89..8bce5c622 100644 --- a/src/powercontext/http/__init__.py +++ b/src/powercontext/http/__init__.py @@ -31,8 +31,11 @@ CaptureContentSourceRequest, CaptureContentSourceResponse, CaptureStatus, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorBinding, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -59,6 +62,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -136,6 +140,7 @@ PreparedWorkHandoff, PrepareHandoffRequest, ProjectDescriptor, + ProjectedSource, ProjectPage, ProposeExperienceRequest, ProposeSkillRequest, @@ -149,6 +154,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ReportActivitySource, @@ -171,10 +177,16 @@ SkillGenerationOrigin, SkillProposal, SkillValidationItem, + SourceDefinitionManifest, SourceInventoryStatistics, + SourceObservationReceipt, + SourceProjectionKey, + SourceProjectionManifest, + SourceProjectionValue, SourceReference, StatsPeriod, StoredHandoffReportActivity, + SubmitSourceObservationRequest, TaskCheck, TaskCheckStatus, TaskOutcome, @@ -210,8 +222,11 @@ "CaptureContentSourceRequest", "CaptureContentSourceResponse", "CaptureStatus", + "CommitConnectorCheckpointRequest", "CommitHandoffRequest", "CommittedHandoff", + "ConnectorBinding", + "ConnectorCheckpointState", "ContinueHandoffRequest", "CreateHandoffReportProjectRequest", "CreateWorkContractRequest", @@ -238,6 +253,7 @@ "GeneratedCandidateResponse", "GeneratedCandidateStatus", "GetArtifactCandidateRequest", + "GetConnectorCheckpointRequest", "GetExperienceRequest", "GetHandoffReportProjectRequest", "GetHandoffReportRequest", @@ -316,6 +332,7 @@ "PreparedWorkHandoff", "ProjectDescriptor", "ProjectPage", + "ProjectedSource", "ProposeExperienceRequest", "ProposeSkillRequest", "PurgeHandoffReportActivitiesRequest", @@ -328,6 +345,7 @@ "RecordHandoffReportActivityRequest", "RecordTaskOutcomeRequest", "RegisterHandoffReportWorkstreamRequest", + "RegisterSourceDefinitionRequest", "RejectArtifactCandidateRequest", "RememberMemoryRequest", "ReportActivitySource", @@ -350,10 +368,16 @@ "SkillGenerationOrigin", "SkillProposal", "SkillValidationItem", + "SourceDefinitionManifest", "SourceInventoryStatistics", + "SourceObservationReceipt", + "SourceProjectionKey", + "SourceProjectionManifest", + "SourceProjectionValue", "SourceReference", "StatsPeriod", "StoredHandoffReportActivity", + "SubmitSourceObservationRequest", "TaskCheck", "TaskCheckStatus", "TaskOutcome", diff --git a/src/powercontext/http/_generated/models.py b/src/powercontext/http/_generated/models.py index ad2a598c1..f7a42405d 100644 --- a/src/powercontext/http/_generated/models.py +++ b/src/powercontext/http/_generated/models.py @@ -285,6 +285,109 @@ class CaptureContentSourceRequest(BaseModel): metadata: dict[str, Any] | None = None +class SourceProjectionKey(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + name: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + version: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + + +class SourceProjectionManifest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + key: SourceProjectionKey + schema_: Annotated[dict[str, Any], Field(alias="schema")] + + +class SourceDefinitionManifest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + name: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + version: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + fingerprint: Annotated[StrictStr, Field(pattern="^sha256:[0-9a-f]{64}$")] + source_schema: dict[str, Any] + projections: Annotated[list[SourceProjectionManifest], Field(max_length=16)] + + +class RegisterSourceDefinitionRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + manifest: SourceDefinitionManifest + + +class ConnectorBinding(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + scope_id: Annotated[StrictStr, Field(max_length=256, min_length=1, pattern=".*\\S.*")] + binding_id: Annotated[StrictStr, Field(max_length=256, min_length=1, pattern=".*\\S.*")] + connector_name: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + connector_version: Annotated[StrictStr, Field(max_length=128, min_length=1, pattern=".*\\S.*")] + + +class GetConnectorCheckpointRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + + +class ConnectorCheckpointState(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + checkpoint: Annotated[Any | None, Field(...)] + + +class SourceProjectionValue(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + key: SourceProjectionKey + value: Any + + +class Materialization(StrEnum): + CAPTURED = "captured" + REFERENCED = "referenced" + + +class ProjectedSource(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + name: Annotated[StrictStr, Field(max_length=256, min_length=1)] + definition_version: Annotated[StrictStr, Field(max_length=128, min_length=1)] + materialization: Materialization + description: StrictStr | None = None + source_type: Annotated[StrictStr, Field(max_length=128, min_length=1)] + definition_fingerprint: Annotated[StrictStr, Field(pattern="^sha256:[0-9a-f]{64}$")] + payload: dict[str, Any] + projections: Annotated[list[SourceProjectionValue], Field(max_length=16)] + + +class SubmitSourceObservationRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + source: ProjectedSource + + +class CommitConnectorCheckpointRequest(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + binding: ConnectorBinding + expected: Annotated[Any | None, Field(...)] + checkpoint: Annotated[Any | None, Field(...)] + + class Kind(StrEnum): ARTIFACT = "artifact" @@ -1026,6 +1129,14 @@ class CaptureContentSourceResponse(BaseModel): position: Annotated[StrictInt, Field(ge=1)] +class SourceObservationReceipt(BaseModel): + model_config = ConfigDict( + extra="forbid", + ) + source: SourceReference + position: Annotated[StrictInt, Field(ge=1)] + + class HandoffMemoryCitation(BaseModel): model_config = ConfigDict( extra="forbid", diff --git a/src/powercontext/http/_generated/operations.py b/src/powercontext/http/_generated/operations.py index d344d87bd..e8671a1f0 100644 --- a/src/powercontext/http/_generated/operations.py +++ b/src/powercontext/http/_generated/operations.py @@ -16,8 +16,10 @@ Capabilities, CaptureContentSourceRequest, CaptureContentSourceResponse, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -31,6 +33,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -77,6 +80,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ResolveExternalSkillRequest, @@ -89,7 +93,10 @@ SearchMemoryRequest, SearchMemoryResponse, SkillArtifact, + SourceDefinitionManifest, + SourceObservationReceipt, StoredHandoffReportActivity, + SubmitSourceObservationRequest, UpdateHandoffReportProjectRequest, UpdateHandoffReportWorkstreamRequest, WorkSourceReceipt, @@ -201,6 +208,79 @@ class Operation(BaseModel, Generic[RequestT, ResponseT]): }, ) +REGISTER_SOURCE_DEFINITION = Operation[RegisterSourceDefinitionRequest, SourceDefinitionManifest]( + method="POST", + path="/v1/source-definitions/register", + operation_id="register_source_definition", + request_type=RegisterSourceDefinitionRequest, + request_location="body", + response_type=SourceDefinitionManifest, + success_status=200, + summary="Register a worker-owned Source Definition manifest", + tags=("source-ingestion",), + responses={ + 200: {"description": "The exact manifest is registered or was already registered identically."}, + 409: {"$ref": "#/components/responses/Conflict"}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + +GET_CONNECTOR_CHECKPOINT = Operation[GetConnectorCheckpointRequest, ConnectorCheckpointState]( + method="POST", + path="/v1/connector-checkpoints/get", + operation_id="get_connector_checkpoint", + request_type=GetConnectorCheckpointRequest, + request_location="body", + response_type=ConnectorCheckpointState, + success_status=200, + summary="Read a Connector binding checkpoint", + tags=("source-ingestion",), + responses={ + 200: {"description": "The current opaque checkpoint, including a normal null initial value."}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 409: {"$ref": "#/components/responses/Conflict"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + +SUBMIT_SOURCE_OBSERVATION = Operation[SubmitSourceObservationRequest, SourceObservationReceipt]( + method="POST", + path="/v1/source-observations", + operation_id="submit_source_observation", + request_type=SubmitSourceObservationRequest, + request_location="body", + response_type=SourceObservationReceipt, + success_status=202, + summary="Submit a worker-materialized Source observation", + tags=("source-ingestion",), + responses={ + 202: {"description": "The observation is durably accepted and can be referenced exactly."}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 409: {"$ref": "#/components/responses/Conflict"}, + 404: {"$ref": "#/components/responses/NotFound"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + +COMMIT_CONNECTOR_CHECKPOINT = Operation[CommitConnectorCheckpointRequest, ConnectorCheckpointState]( + method="POST", + path="/v1/connector-checkpoints/commit", + operation_id="commit_connector_checkpoint", + request_type=CommitConnectorCheckpointRequest, + request_location="body", + response_type=ConnectorCheckpointState, + success_status=200, + summary="Commit a Connector binding checkpoint", + tags=("source-ingestion",), + responses={ + 200: {"description": "The new opaque checkpoint is durable."}, + 401: {"$ref": "#/components/responses/Unauthorized"}, + 409: {"$ref": "#/components/responses/Conflict"}, + 422: {"$ref": "#/components/responses/InvalidRequest"}, + }, +) + PREPARE_CONTEXT = Operation[PrepareContextRequest, PreparedContext]( method="POST", path="/v1/context/prepare", diff --git a/src/powercontext/http/_generated/schema.py b/src/powercontext/http/_generated/schema.py index 6be425400..7caa0a2a7 100644 --- a/src/powercontext/http/_generated/schema.py +++ b/src/powercontext/http/_generated/schema.py @@ -90,6 +90,110 @@ }, } }, + "/v1/source-definitions/register": { + "post": { + "tags": ["source-ingestion"], + "summary": "Register a worker-owned Source Definition manifest", + "description": "Registers an immutable declarative manifest without loading worker plugin code.", + "operationId": "register_source_definition", + "requestBody": { + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/RegisterSourceDefinitionRequest"}} + }, + "required": True, + }, + "responses": { + "200": { + "description": "The exact manifest is registered or was already registered identically.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/SourceDefinitionManifest"}} + }, + }, + "409": {"$ref": "#/components/responses/Conflict"}, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, + "/v1/connector-checkpoints/get": { + "post": { + "tags": ["source-ingestion"], + "summary": "Read a Connector binding checkpoint", + "operationId": "get_connector_checkpoint", + "requestBody": { + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/GetConnectorCheckpointRequest"}} + }, + "required": True, + }, + "responses": { + "200": { + "description": "The current opaque checkpoint, including a normal null initial value.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/ConnectorCheckpointState"}} + }, + }, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "409": {"$ref": "#/components/responses/Conflict"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, + "/v1/source-observations": { + "post": { + "tags": ["source-ingestion"], + "summary": "Submit a worker-materialized Source observation", + "description": "Validates the observation against " + "its registered manifest and " + "durably appends it before receipt.", + "operationId": "submit_source_observation", + "requestBody": { + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/SubmitSourceObservationRequest"}} + }, + "required": True, + }, + "responses": { + "202": { + "description": "The observation is durably accepted and can be referenced exactly.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/SourceObservationReceipt"}} + }, + }, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "409": {"$ref": "#/components/responses/Conflict"}, + "404": {"$ref": "#/components/responses/NotFound"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, + "/v1/connector-checkpoints/commit": { + "post": { + "tags": ["source-ingestion"], + "summary": "Commit a Connector binding checkpoint", + "description": "Replaces the checkpoint only when its expected starting value still matches.", + "operationId": "commit_connector_checkpoint", + "requestBody": { + "content": { + "application/json": { + "schema": {"$ref": "#/components/schemas/CommitConnectorCheckpointRequest"} + } + }, + "required": True, + }, + "responses": { + "200": { + "description": "The new opaque checkpoint is durable.", + "content": { + "application/json": {"schema": {"$ref": "#/components/schemas/ConnectorCheckpointState"}} + }, + }, + "401": {"$ref": "#/components/responses/Unauthorized"}, + "409": {"$ref": "#/components/responses/Conflict"}, + "422": {"$ref": "#/components/responses/InvalidRequest"}, + }, + } + }, "/v1/context/prepare": { "post": { "tags": ["context"], @@ -2210,6 +2314,133 @@ "type": "object", "required": ["status", "source", "position"], }, + "SourceProjectionKey": { + "properties": { + "name": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "version": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["name", "version"], + }, + "SourceProjectionManifest": { + "properties": { + "key": {"$ref": "#/components/schemas/SourceProjectionKey"}, + "schema": {"additionalProperties": True, "type": "object"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["key", "schema"], + }, + "SourceDefinitionManifest": { + "properties": { + "name": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "version": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "fingerprint": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}, + "source_schema": {"additionalProperties": True, "type": "object"}, + "projections": { + "items": {"$ref": "#/components/schemas/SourceProjectionManifest"}, + "type": "array", + "maxItems": 16, + }, + }, + "additionalProperties": False, + "type": "object", + "required": ["name", "version", "fingerprint", "source_schema", "projections"], + }, + "RegisterSourceDefinitionRequest": { + "properties": {"manifest": {"$ref": "#/components/schemas/SourceDefinitionManifest"}}, + "additionalProperties": False, + "type": "object", + "required": ["manifest"], + }, + "ConnectorBinding": { + "properties": { + "scope_id": {"type": "string", "maxLength": 256, "minLength": 1, "pattern": ".*\\S.*"}, + "binding_id": {"type": "string", "maxLength": 256, "minLength": 1, "pattern": ".*\\S.*"}, + "connector_name": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + "connector_version": {"type": "string", "maxLength": 128, "minLength": 1, "pattern": ".*\\S.*"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["scope_id", "binding_id", "connector_name", "connector_version"], + }, + "GetConnectorCheckpointRequest": { + "properties": {"binding": {"$ref": "#/components/schemas/ConnectorBinding"}}, + "additionalProperties": False, + "type": "object", + "required": ["binding"], + }, + "ConnectorCheckpointState": { + "properties": { + "binding": {"$ref": "#/components/schemas/ConnectorBinding"}, + "checkpoint": {"nullable": True}, + }, + "additionalProperties": False, + "type": "object", + "required": ["binding", "checkpoint"], + }, + "SourceProjectionValue": { + "properties": {"key": {"$ref": "#/components/schemas/SourceProjectionKey"}, "value": {}}, + "additionalProperties": False, + "type": "object", + "required": ["key", "value"], + }, + "ProjectedSource": { + "properties": { + "name": {"type": "string", "maxLength": 256, "minLength": 1}, + "definition_version": {"type": "string", "maxLength": 128, "minLength": 1}, + "materialization": {"type": "string", "enum": ["captured", "referenced"]}, + "description": {"type": "string", "nullable": True}, + "source_type": {"type": "string", "maxLength": 128, "minLength": 1}, + "definition_fingerprint": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}, + "payload": {"additionalProperties": True, "type": "object"}, + "projections": { + "items": {"$ref": "#/components/schemas/SourceProjectionValue"}, + "type": "array", + "maxItems": 16, + }, + }, + "additionalProperties": False, + "type": "object", + "required": [ + "name", + "definition_version", + "materialization", + "source_type", + "definition_fingerprint", + "payload", + "projections", + ], + }, + "SubmitSourceObservationRequest": { + "properties": { + "binding": {"$ref": "#/components/schemas/ConnectorBinding"}, + "source": {"$ref": "#/components/schemas/ProjectedSource"}, + }, + "additionalProperties": False, + "type": "object", + "required": ["binding", "source"], + }, + "SourceObservationReceipt": { + "properties": { + "source": {"$ref": "#/components/schemas/SourceReference"}, + "position": {"type": "integer", "minimum": 1.0}, + }, + "additionalProperties": False, + "type": "object", + "required": ["source", "position"], + }, + "CommitConnectorCheckpointRequest": { + "properties": { + "binding": {"$ref": "#/components/schemas/ConnectorBinding"}, + "expected": {"nullable": True}, + "checkpoint": {"nullable": True}, + }, + "additionalProperties": False, + "type": "object", + "required": ["binding", "expected", "checkpoint"], + }, "CommitHandoffRequest": { "properties": { "scope_id": {"type": "string", "maxLength": 256, "minLength": 1, "pattern": ".*\\S.*"}, diff --git a/src/powercontext/server/app.py b/src/powercontext/server/app.py index 8cfd96edd..59990c673 100644 --- a/src/powercontext/server/app.py +++ b/src/powercontext/server/app.py @@ -134,6 +134,12 @@ from powercontext.builtin.runtime import ( ApproveArtifactCandidateRequest as RuntimeApproveArtifactCandidateRequest, ) +from powercontext.builtin.runtime import ( + CommitConnectorCheckpoint as RuntimeCommitConnectorCheckpoint, +) +from powercontext.builtin.runtime import ( + ConnectorCheckpointState as RuntimeConnectorCheckpointState, +) from powercontext.builtin.runtime import ( GenerateExperienceRequest as RuntimeGenerateExperienceRequest, ) @@ -192,6 +198,9 @@ from powercontext.builtin.runtime import ( StatisticsPeriod as RuntimeStatisticsPeriod, ) +from powercontext.builtin.runtime import ( + SubmitSourceObservation as RuntimeSubmitSourceObservation, +) from powercontext.builtin.work import ( AcknowledgeHandoff as RuntimeAcknowledgeHandoff, ) @@ -209,9 +218,13 @@ from powercontext.builtin.work import WorkSourceReceipt as RuntimeWorkSourceReceipt from powercontext.errors import ( ArtifactNotFoundError, + InvalidConnectorRunError, + InvalidSourceDefinitionError, + InvalidSourceObservationError, PowerContextError, RevisionConflictError, SourceConflictError, + SourceDefinitionNotFoundError, ) from powercontext.http import ( AcknowledgeHandoffRequest, @@ -223,8 +236,10 @@ Capabilities, CaptureContentSourceRequest, CaptureContentSourceResponse, + CommitConnectorCheckpointRequest, CommitHandoffRequest, CommittedHandoff, + ConnectorCheckpointState, ContinueHandoffRequest, CreateHandoffReportProjectRequest, CreateWorkContractRequest, @@ -240,6 +255,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetHandoffReportProjectRequest, GetHandoffReportRequest, @@ -286,6 +302,7 @@ RecordHandoffReportActivityRequest, RecordTaskOutcomeRequest, RegisterHandoffReportWorkstreamRequest, + RegisterSourceDefinitionRequest, RejectArtifactCandidateRequest, RememberMemoryRequest, ResolveExternalSkillRequest, @@ -298,7 +315,10 @@ SearchMemoryRequest, SearchMemoryResponse, SkillArtifact, + SourceDefinitionManifest, + SourceObservationReceipt, StoredHandoffReportActivity, + SubmitSourceObservationRequest, UpdateHandoffReportProjectRequest, UpdateHandoffReportWorkstreamRequest, WorkSourceReceipt, @@ -326,6 +346,7 @@ APPROVE_ARTIFACT_CANDIDATE, ATTACH_HANDOFF_REPORT_WORKSPACE, CAPTURE_CONTENT_SOURCE, + COMMIT_CONNECTOR_CHECKPOINT, COMMIT_HANDOFF, CONTINUE_HANDOFF, CREATE_HANDOFF_REPORT_PROJECT, @@ -337,6 +358,7 @@ GENERATE_SKILL, GET_ARTIFACT_CANDIDATE, GET_CAPABILITIES, + GET_CONNECTOR_CHECKPOINT, GET_EXPERIENCE, GET_HANDOFF_REPORT, GET_HANDOFF_REPORT_PROJECT, @@ -365,6 +387,7 @@ RECORD_HANDOFF_REPORT_ACTIVITY, RECORD_TASK_OUTCOME, REGISTER_HANDOFF_REPORT_WORKSTREAM, + REGISTER_SOURCE_DEFINITION, REJECT_ARTIFACT_CANDIDATE, REMEMBER_MEMORY, RESOLVE_EXTERNAL_SKILL, @@ -373,6 +396,7 @@ REVISE_MEMORY_ENTRY, SCAN_EXTERNAL_SKILLS, SEARCH_MEMORY, + SUBMIT_SOURCE_OBSERVATION, UPDATE_HANDOFF_REPORT_PROJECT, UPDATE_HANDOFF_REPORT_WORKSTREAM, Operation, @@ -385,6 +409,8 @@ reset_request_id, ) from powercontext.server.tracing import request_id_from_span +from powercontext.sources import ConnectorBinding as RuntimeConnectorBinding +from powercontext.sources import SourceDefinitionManifest as RuntimeSourceDefinitionManifest if TYPE_CHECKING: from powercontext.server.metrics import ServerMetrics @@ -410,6 +436,16 @@ class _SourceApplication(Protocol): def for_scope(self, scope_id: str, /) -> _ScopedSourceApplication: ... +class _RemoteIngestionApplication(Protocol): + async def register(self, manifest: RuntimeSourceDefinitionManifest, /) -> RuntimeSourceDefinitionManifest: ... + + async def checkpoint(self, binding: RuntimeConnectorBinding, /) -> RuntimeConnectorCheckpointState: ... + + async def submit(self, request: RuntimeSubmitSourceObservation, /) -> SourceReceipt: ... + + async def commit(self, request: RuntimeCommitConnectorCheckpoint, /) -> RuntimeConnectorCheckpointState: ... + + class _ScopedContextApplication(Protocol): async def prepare(self, request: RuntimePrepareContextRequest, /) -> RuntimePreparedContext: ... @@ -538,6 +574,7 @@ def for_scope(self, scope_id: str, /) -> _ScopedStatisticsApplication: ... class ServerApplication(Protocol): sources: _SourceApplication + ingestion: _RemoteIngestionApplication context: _ContextApplication experience: _ExperienceApplication external_skills: _ExternalSkillApplication @@ -653,6 +690,10 @@ async def unexpected_error(request: Request, error: Exception) -> JSONResponse: _add_route(app, DETACH_HANDOFF_REPORT_WORKSPACE, detach_handoff_report_workspace) _add_route(app, GET_HANDOFF_REPORT, get_handoff_report) _add_route(app, CAPTURE_CONTENT_SOURCE, capture_content_source) + _add_route(app, REGISTER_SOURCE_DEFINITION, register_source_definition) + _add_route(app, GET_CONNECTOR_CHECKPOINT, get_connector_checkpoint) + _add_route(app, SUBMIT_SOURCE_OBSERVATION, submit_source_observation) + _add_route(app, COMMIT_CONNECTOR_CHECKPOINT, commit_connector_checkpoint) _add_route(app, FLUSH_MEMORY, flush_memory) _add_route(app, REMEMBER_MEMORY, remember_memory) _add_route(app, SEARCH_MEMORY, search_memory) @@ -995,6 +1036,38 @@ async def capture_content_source( return mapping.capture_response(result) +async def register_source_definition( + request: RegisterSourceDefinitionRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> SourceDefinitionManifest: + result = await application.ingestion.register(mapping.runtime_source_definition_manifest(request.manifest)) + return mapping.source_definition_manifest_response(result) + + +async def get_connector_checkpoint( + request: GetConnectorCheckpointRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> ConnectorCheckpointState: + result = await application.ingestion.checkpoint(mapping.connector_checkpoint_request(request)) + return mapping.connector_checkpoint_response(result) + + +async def submit_source_observation( + request: SubmitSourceObservationRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> SourceObservationReceipt: + result = await application.ingestion.submit(mapping.submit_source_observation_request(request)) + return mapping.source_observation_receipt_response(result) + + +async def commit_connector_checkpoint( + request: CommitConnectorCheckpointRequest, + application: Annotated[ServerApplication, Depends(_require_application)], +) -> ConnectorCheckpointState: + result = await application.ingestion.commit(mapping.commit_connector_checkpoint_request(request)) + return mapping.connector_checkpoint_response(result) + + async def flush_memory( request: FlushMemoryRequest, application: Annotated[ServerApplication, Depends(_require_application)], @@ -1619,12 +1692,13 @@ def _map_report_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | def _map_domain_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | None]: + source_ingestion = _map_source_ingestion_error(error) + if source_ingestion is not None: + return source_ingestion if isinstance(error, ArtifactNotFoundError): return status.HTTP_404_NOT_FOUND, "artifact_not_found", "The requested Artifact was not found.", None if isinstance(error, MemoryEntryNotFoundError): return status.HTTP_404_NOT_FOUND, "memory_not_found", "The requested Memory value was not found.", None - if isinstance(error, SourceConflictError): - return status.HTTP_409_CONFLICT, "source_conflict", "The Source identity has different content.", None if isinstance(error, RevisionConflictError): return status.HTTP_409_CONFLICT, "revision_conflict", "The Memory Revision is stale.", None if isinstance(error, MemoryEntryInactiveError): @@ -1655,6 +1729,23 @@ def _map_domain_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | return status.HTTP_500_INTERNAL_SERVER_ERROR, "internal_error", "The Server failed.", None +def _map_source_ingestion_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | None] | None: + if isinstance(error, SourceConflictError): + return status.HTTP_409_CONFLICT, "source_conflict", "The Source identity has different content.", None + if isinstance(error, InvalidConnectorRunError): + return status.HTTP_409_CONFLICT, "connector_checkpoint_conflict", "The Connector checkpoint is stale.", None + if isinstance(error, SourceDefinitionNotFoundError): + return ( + status.HTTP_404_NOT_FOUND, + "source_definition_not_found", + "The Source Definition is not registered.", + None, + ) + if isinstance(error, (InvalidSourceDefinitionError, InvalidSourceObservationError)): + return status.HTTP_422_UNPROCESSABLE_CONTENT, "invalid_source_ingestion", "Source ingestion is invalid.", None + return None + + def _map_availability_error(error: Exception) -> tuple[int, str, str, dict[str, Any] | None] | None: if isinstance(error, _RuntimeNotReadyError): return status.HTTP_503_SERVICE_UNAVAILABLE, "runtime_not_ready", "The Runtime is not ready.", None diff --git a/src/powercontext/server/mapping.py b/src/powercontext/server/mapping.py index c26efc143..60ce5b1f4 100644 --- a/src/powercontext/server/mapping.py +++ b/src/powercontext/server/mapping.py @@ -75,6 +75,12 @@ from powercontext.builtin.runtime import ( ApproveArtifactCandidateRequest as RuntimeApproveArtifactCandidateRequest, ) +from powercontext.builtin.runtime import ( + CommitConnectorCheckpoint as RuntimeCommitConnectorCheckpoint, +) +from powercontext.builtin.runtime import ( + ConnectorCheckpointState as RuntimeConnectorCheckpointState, +) from powercontext.builtin.runtime import ( GenerateExperienceRequest as RuntimeGenerateExperienceRequest, ) @@ -127,6 +133,9 @@ from powercontext.builtin.runtime import ( Statistics as RuntimeStatistics, ) +from powercontext.builtin.runtime import ( + SubmitSourceObservation as RuntimeSubmitSourceObservation, +) from powercontext.builtin.sources import ExternalSkillImportMode as RuntimeExternalSkillImportMode from powercontext.builtin.work import ( AcknowledgeHandoff as RuntimeAcknowledgeHandoff, @@ -163,7 +172,9 @@ CaptureContentSourceRequest, CaptureContentSourceResponse, CaptureStatus, + CommitConnectorCheckpointRequest, CommittedHandoff, + ConnectorCheckpointState, CreateWorkContractRequest, EntryChange, EntryChangeOperation, @@ -179,6 +190,7 @@ GenerateExperienceRequest, GenerateSkillRequest, GetArtifactCandidateRequest, + GetConnectorCheckpointRequest, GetExperienceRequest, GetMemoryEntryRequest, GetSkillRequest, @@ -224,12 +236,16 @@ SkillArtifact, SkillProposal, SkillValidationItem, + SourceDefinitionManifest, + SourceObservationReceipt, SourceReference, + SubmitSourceObservationRequest, TaskCheck, WorkClaim, WorkSourceKind, WorkSourceReceipt, ) +from powercontext.http import ConnectorBinding as HttpConnectorBinding from powercontext.http import ( HandoffActivation as TransportHandoffActivation, ) @@ -278,6 +294,15 @@ from powercontext.http import ( RememberMemoryRequest as TransportRememberMemoryRequest, ) +from powercontext.sources import ( + ConnectorBinding as RuntimeConnectorBinding, +) +from powercontext.sources import ( + ProjectedSource as RuntimeProjectedSource, +) +from powercontext.sources import ( + SourceDefinitionManifest as RuntimeSourceDefinitionManifest, +) from powercontext.sources import SourceRef @@ -421,6 +446,62 @@ def capture_response(value: SourceReceipt) -> CaptureContentSourceResponse: ) +def runtime_source_definition_manifest(value: SourceDefinitionManifest) -> RuntimeSourceDefinitionManifest: + try: + return RuntimeSourceDefinitionManifest.model_validate(value.model_dump(mode="json", by_alias=True)) + except ValidationError as error: + raise InvalidRuntimeRequestError("source-definition-manifest") from error + + +def source_definition_manifest_response(value: RuntimeSourceDefinitionManifest) -> SourceDefinitionManifest: + return SourceDefinitionManifest.model_validate(value.model_dump(mode="json", by_alias=True)) + + +def runtime_connector_binding(value: HttpConnectorBinding) -> RuntimeConnectorBinding: + try: + return RuntimeConnectorBinding.model_validate(value.model_dump(mode="json")) + except ValidationError as error: + raise InvalidRuntimeRequestError("connector-binding") from error + + +def connector_checkpoint_request(value: GetConnectorCheckpointRequest) -> RuntimeConnectorBinding: + return runtime_connector_binding(value.binding) + + +def submit_source_observation_request(value: SubmitSourceObservationRequest) -> RuntimeSubmitSourceObservation: + try: + return RuntimeSubmitSourceObservation( + binding=runtime_connector_binding(value.binding), + source=RuntimeProjectedSource.model_validate(value.source.model_dump(mode="json")), + ) + except ValidationError as error: + raise InvalidRuntimeRequestError("source-observation") from error + + +def commit_connector_checkpoint_request( + value: CommitConnectorCheckpointRequest, +) -> RuntimeCommitConnectorCheckpoint: + try: + return RuntimeCommitConnectorCheckpoint( + binding=runtime_connector_binding(value.binding), + expected=value.expected, + checkpoint=value.checkpoint, + ) + except ValidationError as error: + raise InvalidRuntimeRequestError("connector-checkpoint") from error + + +def connector_checkpoint_response(value: RuntimeConnectorCheckpointState) -> ConnectorCheckpointState: + return ConnectorCheckpointState.model_validate(value.model_dump(mode="json")) + + +def source_observation_receipt_response(value: SourceReceipt) -> SourceObservationReceipt: + return SourceObservationReceipt( + source=source_reference(value.source_ref), + position=value.sequence, + ) + + def statistics_response(value: RuntimeStatistics) -> ScopedStats: return ScopedStats.model_validate(value.model_dump(mode="json")) diff --git a/src/powercontext/sources/__init__.py b/src/powercontext/sources/__init__.py index ead637bd8..fea55564a 100644 --- a/src/powercontext/sources/__init__.py +++ b/src/powercontext/sources/__init__.py @@ -14,15 +14,75 @@ from powercontext.sources.adapters import SourceAdapter from powercontext.sources.catalog import SourceCatalog -from powercontext.sources.models import Source, SourceMaterialization, SourceRef +from powercontext.sources.connectors import ( + CatalogConnectorSourceSink, + Connector, + ConnectorBinding, + ConnectorCapability, + ConnectorCheckpointStore, + ConnectorItemOutcome, + ConnectorLifecycle, + ConnectorRunCompletion, + ConnectorRunResult, + ConnectorRunSession, + ConnectorRunStatus, + ConnectorSourceSink, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, + validate_connector, +) +from powercontext.sources.definitions import ( + AdapterSourceDefinition, + SourceDefinition, + SourceDefinitionRegistry, + SourceProjection, +) +from powercontext.sources.models import Source, SourceMaterialization, SourceProjectionKey, SourceRef +from powercontext.sources.observations import ( + ProjectedSource, + SourceDefinitionManifest, + SourceProjectionManifest, + SourceProjectionValue, + manifest_for_definition, + project_source_for_transport, +) +from powercontext.sources.projections import TEXT_EVIDENCE_PROJECTION_KEY, TextEvidence from powercontext.sources.protocols import SourceCatalogBackend, SourceStore __all__ = [ + "TEXT_EVIDENCE_PROJECTION_KEY", + "AdapterSourceDefinition", + "CatalogConnectorSourceSink", + "Connector", + "ConnectorBinding", + "ConnectorCapability", + "ConnectorCheckpointStore", + "ConnectorItemOutcome", + "ConnectorLifecycle", + "ConnectorRunCompletion", + "ConnectorRunResult", + "ConnectorRunSession", + "ConnectorRunStatus", + "ConnectorSourceSink", + "ConnectorSubmissionResult", + "ConnectorSubmissionStatus", + "ProjectedSource", "Source", "SourceAdapter", "SourceCatalog", "SourceCatalogBackend", + "SourceDefinition", + "SourceDefinitionManifest", + "SourceDefinitionRegistry", "SourceMaterialization", + "SourceProjection", + "SourceProjectionKey", + "SourceProjectionManifest", + "SourceProjectionValue", "SourceRef", "SourceStore", + "TextEvidence", + "manifest_for_definition", + "project_source_for_transport", + "validate_connector", ] diff --git a/src/powercontext/sources/catalog.py b/src/powercontext/sources/catalog.py index 37a04238b..52b0717f6 100644 --- a/src/powercontext/sources/catalog.py +++ b/src/powercontext/sources/catalog.py @@ -14,19 +14,18 @@ from __future__ import annotations -from collections.abc import Iterable, Mapping -from typing import Any, cast +from collections.abc import Iterable +from typing import Any + +from pydantic import JsonValue from powercontext.errors import ( - InvalidSourceAdapterError, - InvalidSourceEntryError, - InvalidSourceResultError, - SourceAdapterNotFoundError, - SourceConflictError, SourceNotFoundError, ) from powercontext.sources.adapters import SourceAdapter -from powercontext.sources.models import Source, SourceRef +from powercontext.sources.definitions import SourceDefinitionRegistry +from powercontext.sources.models import Source, SourceProjectionKey, SourceRef +from powercontext.sources.observations import ProjectedSource from powercontext.sources.protocols import SourceCatalogBackend _AnySourceAdapter = SourceAdapter[Any, Any, Any] @@ -39,21 +38,14 @@ def __init__( self, *, backend: SourceCatalogBackend, - adapters: Iterable[_AnySourceAdapter], + adapters: Iterable[_AnySourceAdapter] = (), + registry: SourceDefinitionRegistry | None = None, ) -> None: - by_input: dict[type[object], _AnySourceAdapter] = {} - by_source: dict[type[Source], _AnySourceAdapter] = {} - for adapter in adapters: - input_class, source_class = _validate_adapter(adapter) - if input_class in by_input: - raise SourceConflictError("input_class", input_class) - if source_class in by_source: - raise SourceConflictError("source_class", source_class) - by_input[input_class] = adapter - by_source[source_class] = adapter + adapter_values = tuple(adapters) + if registry is not None and adapter_values: + raise TypeError("SourceCatalog accepts either registry or adapters, not both") # noqa: TRY003 self._backend = backend - self._by_input = by_input - self._by_source = by_source + self._registry = registry or SourceDefinitionRegistry.from_adapters(adapter_values) async def list(self) -> tuple[Source, ...]: sources = await self._backend.list() @@ -70,50 +62,29 @@ async def get(self, source: Source, /) -> Source: return stored def as_ref(self, source: Source, /) -> SourceRef: - adapter = _adapter_for_source(source, self._by_source) - return SourceRef(source_type=adapter.name, source_id=source.name) + if isinstance(source, ProjectedSource): + return SourceRef(source_type=source.source_type, source_id=source.name) + definition = self._registry.definition_for_source(source) + return SourceRef(source_type=definition.name, source_id=source.name) async def resolve(self, value: object, /) -> Source: - input_class = type(value) - try: - adapter = self._by_input[input_class] - except KeyError: - raise SourceAdapterNotFoundError("input", input_class) from None - source = await adapter.resolve(value) - if type(source) is not adapter.source_class: - raise InvalidSourceResultError(adapter.name, "resolve", adapter.source_class, type(source)) - self.as_ref(source) - return cast(Source, source) + return await self._registry.resolve(value) async def read(self, source: Source, /) -> object: - adapter = _adapter_for_source(source, self._by_source) - return await adapter.read(source) - - -def _validate_adapter(adapter: object) -> tuple[type[object], type[Source]]: - adapter_type = type(adapter) - input_class = getattr(adapter, "input_class", None) - if not isinstance(input_class, type): - raise InvalidSourceAdapterError(adapter_type, "input_class", "must be a type") - name = getattr(adapter, "name", None) - if not isinstance(name, str) or not name.strip(): - raise InvalidSourceAdapterError(adapter_type, "name", "must be a non-empty string") - source_class = getattr(adapter, "source_class", None) - if not isinstance(source_class, type) or not issubclass(source_class, Source): - raise InvalidSourceAdapterError(adapter_type, "source_class", "must be a Source subclass") - for method_name in ("resolve", "read"): - if not callable(getattr(adapter, method_name, None)): - raise InvalidSourceAdapterError(adapter_type, method_name, "must be callable") - return cast(type[object], input_class), cast(type[Source], source_class) - - -def _adapter_for_source( - source: object, - adapters: Mapping[type[Source], _AnySourceAdapter], -) -> _AnySourceAdapter: - if not isinstance(source, Source): - raise InvalidSourceEntryError(type(source)) - try: - return adapters[type(source)] - except KeyError: - raise SourceAdapterNotFoundError("source", type(source)) from None + if isinstance(source, ProjectedSource): + return source.payload + return await self._registry.read(source) + + def projection_keys(self, source: Source, /) -> tuple[SourceProjectionKey, ...]: + """Return the exact named projection capabilities advertised for ``source``.""" + + if isinstance(source, ProjectedSource): + return tuple(projection.key for projection in source.projections) + return self._registry.projection_keys(source) + + def project(self, source: Source, key: SourceProjectionKey, /) -> JsonValue: + """Evaluate one named projection against an exact Source value.""" + + if isinstance(source, ProjectedSource): + return source.projection(key) + return self._registry.project(source, key) diff --git a/src/powercontext/sources/connectors.py b/src/powercontext/sources/connectors.py new file mode 100644 index 000000000..b5ca098fa --- /dev/null +++ b/src/powercontext/sources/connectors.py @@ -0,0 +1,385 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Provider-neutral Connector run and durable checkpoint contracts.""" + +from __future__ import annotations + +from enum import StrEnum +from typing import Protocol + +from pydantic import BaseModel, ConfigDict, Field, JsonValue, field_validator, model_validator + +from powercontext.errors import InvalidConnectorError, InvalidConnectorRunError +from powercontext.limits import MAX_SCOPE_ID_LENGTH, MAX_SOURCE_ID_LENGTH, MAX_SOURCE_TYPE_LENGTH +from powercontext.sources.catalog import SourceCatalog +from powercontext.sources.models import Source, SourceRef +from powercontext.sources.protocols import SourceStore + + +class ConnectorCapability(StrEnum): + """Acquisition guarantees a Connector can actually enforce.""" + + COMPLETE_SNAPSHOT = "complete_snapshot" + CHANGE_FEED = "change_feed" + CHECKPOINT_RESUME = "checkpoint_resume" + AUTHORITATIVE_DELETION = "authoritative_deletion" + + +class ConnectorRunStatus(StrEnum): + """Whether the Connector completed the provider work represented by a run.""" + + COMPLETE = "complete" + INCOMPLETE = "incomplete" + + +class ConnectorSubmissionStatus(StrEnum): + """Durable outcome for one definition-native item submission.""" + + ACCEPTED = "accepted" + REPLAYED = "replayed" + REJECTED = "rejected" + FAILED = "failed" + + +class ConnectorBinding(BaseModel): + """Activate one Connector identity for exactly one Scope.""" + + model_config = ConfigDict(frozen=True) + + scope_id: str = Field(max_length=MAX_SCOPE_ID_LENGTH) + binding_id: str = Field(max_length=MAX_SOURCE_ID_LENGTH) + connector_name: str = Field(max_length=MAX_SOURCE_TYPE_LENGTH) + connector_version: str = Field(max_length=MAX_SOURCE_TYPE_LENGTH) + + @field_validator("scope_id", "binding_id", "connector_name", "connector_version") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value or not value.strip(): + raise ValueError("Connector identity must be non-empty") # noqa: TRY003 + if value != value.strip(): + raise ValueError("Connector identity must be trimmed") # noqa: TRY003 + return value + + +class ConnectorSubmissionResult(BaseModel): + """Sink result after one item has reached a durable acceptance boundary.""" + + model_config = ConfigDict(frozen=True) + + status: ConnectorSubmissionStatus + source_ref: SourceRef | None = None + detail: str | None = None + + @model_validator(mode="after") + def validate_source_ref(self) -> ConnectorSubmissionResult: + accepted = self.status in {ConnectorSubmissionStatus.ACCEPTED, ConnectorSubmissionStatus.REPLAYED} + if accepted != (self.source_ref is not None): + raise ValueError("accepted and replayed submissions require exactly one SourceRef") # noqa: TRY003 + return self + + +class ConnectorItemOutcome(BaseModel): + """Visible result for every item a Connector submitted during one run.""" + + model_config = ConfigDict(frozen=True) + + item_id: str + definition_name: str + status: ConnectorSubmissionStatus + source_ref: SourceRef | None = None + detail: str | None = None + + +class ConnectorRunCompletion(BaseModel): + """Connector-owned completion signal and next opaque checkpoint.""" + + model_config = ConfigDict(frozen=True) + + status: ConnectorRunStatus + checkpoint: JsonValue | None = None + + +class ConnectorRunResult(BaseModel): + """Observable lifecycle result after any safe checkpoint commit.""" + + model_config = ConfigDict(frozen=True) + + binding: ConnectorBinding + status: ConnectorRunStatus + previous_checkpoint: JsonValue | None + proposed_checkpoint: JsonValue | None + committed_checkpoint: JsonValue | None + items: tuple[ConnectorItemOutcome, ...] + + +class ConnectorSourceSink(Protocol): + """Accept definition-native input and return its durable local SourceRef.""" + + async def submit( + self, + binding: ConnectorBinding, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: ... + + +class ConnectorCheckpointStore(Protocol): + """Persist opaque binding checkpoints using optimistic comparison.""" + + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: ... + + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: ... + + +class Connector(Protocol): + """Acquire provider items through one lifecycle session.""" + + name: str + version: str + source_definitions: frozenset[str] + capabilities: frozenset[ConnectorCapability] + + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: ... + + +class ConnectorRunSession: + """Constrain one Connector run to declared Definitions and visible outcomes.""" + + def __init__( + self, + *, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + source_definitions: frozenset[str], + sink: ConnectorSourceSink, + ) -> None: + self.binding = binding + self.checkpoint = checkpoint + self._source_definitions = source_definitions + self._sink = sink + self._outcomes: list[ConnectorItemOutcome] = [] + self._item_ids: set[str] = set() + + @property + def outcomes(self) -> tuple[ConnectorItemOutcome, ...]: + return tuple(self._outcomes) + + async def submit( + self, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: + """Submit one item and record success, rejection, or sink failure exactly once.""" + + self._claim_item(item_id, definition_name) + try: + result = await self._sink.submit(self.binding, item_id, definition_name, value) + except Exception as error: + result = ConnectorSubmissionResult( + status=ConnectorSubmissionStatus.FAILED, + detail=type(error).__name__, + ) + if result.source_ref is not None and result.source_ref.source_type != definition_name: + raise InvalidConnectorRunError( + "definition-mismatch", + f"sink returned {result.source_ref.source_type!r} for {definition_name!r}", + ) + self._outcomes.append( + ConnectorItemOutcome( + item_id=item_id, + definition_name=definition_name, + status=result.status, + source_ref=result.source_ref, + detail=result.detail, + ) + ) + return result + + def reject(self, item_id: str, definition_name: str, detail: str, /) -> ConnectorSubmissionResult: + """Record one provider item that cannot satisfy its Source Definition.""" + + return self._record_provider_outcome( + item_id, + definition_name, + ConnectorSubmissionStatus.REJECTED, + detail, + ) + + def fail(self, item_id: str, definition_name: str, detail: str, /) -> ConnectorSubmissionResult: + """Record one provider item that could not be acquired safely.""" + + return self._record_provider_outcome( + item_id, + definition_name, + ConnectorSubmissionStatus.FAILED, + detail, + ) + + def _record_provider_outcome( + self, + item_id: str, + definition_name: str, + status: ConnectorSubmissionStatus, + detail: str, + ) -> ConnectorSubmissionResult: + _require_trimmed("detail", detail) + if status not in {ConnectorSubmissionStatus.REJECTED, ConnectorSubmissionStatus.FAILED}: + raise InvalidConnectorRunError("provider-outcome", "must be rejected or failed") + self._claim_item(item_id, definition_name) + result = ConnectorSubmissionResult(status=status, detail=detail) + self._outcomes.append( + ConnectorItemOutcome( + item_id=item_id, + definition_name=definition_name, + status=status, + detail=detail, + ) + ) + return result + + def _claim_item(self, item_id: str, definition_name: str) -> None: + _require_trimmed("item_id", item_id) + if item_id in self._item_ids: + raise InvalidConnectorRunError("duplicate-item", f"item {item_id!r} was submitted more than once") + if definition_name not in self._source_definitions: + raise InvalidConnectorRunError( + "undeclared-definition", + f"Connector did not declare Source Definition {definition_name!r}", + ) + self._item_ids.add(item_id) + + +class ConnectorLifecycle: + """Run Connectors while enforcing durable checkpoint ordering.""" + + def __init__(self, *, sink: ConnectorSourceSink, checkpoints: ConnectorCheckpointStore) -> None: + self._sink = sink + self._checkpoints = checkpoints + + async def run(self, connector: Connector, binding: ConnectorBinding, /) -> ConnectorRunResult: + source_definitions, capabilities = validate_connector(connector, binding) + previous = await self._checkpoints.load(binding) + if previous is not None and ConnectorCapability.CHECKPOINT_RESUME not in capabilities: + raise InvalidConnectorRunError( + "unsupported-resume", + "binding has a checkpoint but Connector does not advertise checkpoint resume", + ) + session = ConnectorRunSession( + binding=binding, + checkpoint=previous, + source_definitions=source_definitions, + sink=self._sink, + ) + completion = await connector.run(session) + if not isinstance(completion, ConnectorRunCompletion): + raise InvalidConnectorRunError("completion", "Connector must return ConnectorRunCompletion") + + unsafe = tuple( + outcome + for outcome in session.outcomes + if outcome.status in {ConnectorSubmissionStatus.REJECTED, ConnectorSubmissionStatus.FAILED} + ) + committed = previous + if completion.status is ConnectorRunStatus.COMPLETE and completion.checkpoint != previous and not unsafe: + await self._checkpoints.save(binding, completion.checkpoint, expected=previous) + committed = completion.checkpoint + return ConnectorRunResult( + binding=binding, + status=ConnectorRunStatus.INCOMPLETE if unsafe else completion.status, + previous_checkpoint=previous, + proposed_checkpoint=completion.checkpoint, + committed_checkpoint=committed, + items=session.outcomes, + ) + + +class CatalogConnectorSourceSink: + """Bridge lifecycle submissions to one scope-bound catalog and Source store.""" + + def __init__(self, *, scope_id: str, catalog: SourceCatalog, store: SourceStore[Source]) -> None: + _require_trimmed("scope_id", scope_id) + self._scope_id = scope_id + self._catalog = catalog + self._store = store + + async def submit( + self, + binding: ConnectorBinding, + item_id: str, + definition_name: str, + value: object, + /, + ) -> ConnectorSubmissionResult: + del item_id + if binding.scope_id != self._scope_id: + raise InvalidConnectorRunError( + "scope-mismatch", + f"sink is bound to {self._scope_id!r}, got {binding.scope_id!r}", + ) + source = await self._catalog.resolve(value) + source_ref = self._catalog.as_ref(source) + if source_ref.source_type != definition_name: + raise InvalidConnectorRunError( + "definition-mismatch", + f"input resolved as {source_ref.source_type!r}, expected {definition_name!r}", + ) + stored = await self._store.add(source) + stored_ref = self._catalog.as_ref(stored) + if stored_ref != source_ref: + raise InvalidConnectorRunError("identity-mismatch", "Source store changed the accepted identity") + return ConnectorSubmissionResult(status=ConnectorSubmissionStatus.ACCEPTED, source_ref=stored_ref) + + +def validate_connector( + connector: Connector, + binding: ConnectorBinding, +) -> tuple[frozenset[str], frozenset[ConnectorCapability]]: + """Validate one Connector declaration against the binding it will execute.""" + + name = getattr(connector, "name", None) + version = getattr(connector, "version", None) + if name != binding.connector_name: + raise InvalidConnectorError("name", f"binding expects {binding.connector_name!r}, got {name!r}") + if version != binding.connector_version: + raise InvalidConnectorError("version", f"binding expects {binding.connector_version!r}, got {version!r}") + source_definitions = getattr(connector, "source_definitions", None) + if not isinstance(source_definitions, frozenset) or not source_definitions: + raise InvalidConnectorError("source_definitions", "must be a non-empty frozenset") + if not all(isinstance(value, str) and value.strip() == value and value for value in source_definitions): + raise InvalidConnectorError("source_definitions", "must contain non-empty trimmed names") + capabilities = getattr(connector, "capabilities", None) + if not isinstance(capabilities, frozenset) or not all( + isinstance(value, ConnectorCapability) for value in capabilities + ): + raise InvalidConnectorError("capabilities", "must be a frozenset of ConnectorCapability values") + if not callable(getattr(connector, "run", None)): + raise InvalidConnectorError("run", "must be callable") + return source_definitions, capabilities + + +def _require_trimmed(field: str, value: object) -> None: + if not isinstance(value, str) or not value or value.strip() != value: + raise InvalidConnectorRunError(field, "must be a non-empty trimmed string") diff --git a/src/powercontext/sources/definitions.py b/src/powercontext/sources/definitions.py new file mode 100644 index 000000000..a67fcd2a9 --- /dev/null +++ b/src/powercontext/sources/definitions.py @@ -0,0 +1,246 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Explicit Source Definition registration and named projection routing.""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from typing import Any, Generic, Protocol, TypeVar, cast + +from pydantic import BaseModel, JsonValue, TypeAdapter + +from powercontext.errors import ( + InvalidSourceAdapterError, + InvalidSourceDefinitionError, + InvalidSourceEntryError, + InvalidSourceProjectionError, + InvalidSourceResultError, + SourceAdapterNotFoundError, + SourceConflictError, + SourceDefinitionNotFoundError, + SourceProjectionNotFoundError, +) +from powercontext.sources.adapters import SourceAdapter +from powercontext.sources.models import Source, SourceProjectionKey + +InputT = TypeVar("InputT") +SourceT = TypeVar("SourceT", bound=Source) +ValueT_co = TypeVar("ValueT_co", covariant=True) + +_JSON_VALUE = TypeAdapter(JsonValue) +_AnySourceAdapter = SourceAdapter[Any, Any, Any] + + +class SourceProjection(Protocol[SourceT]): + """Project one exact Source value through a named, versioned capability.""" + + name: str + version: str + source_class: type[SourceT] + output_class: type[BaseModel] + + def project(self, source: SourceT, /) -> object: ... + + +class SourceDefinition(SourceAdapter[InputT, SourceT, ValueT_co], Protocol[InputT, SourceT, ValueT_co]): + """Bind one adapter contract to a durable version and optional projections.""" + + version: str + projections: tuple[SourceProjection[SourceT], ...] + + +@dataclass(frozen=True, slots=True) +class AdapterSourceDefinition(Generic[InputT, SourceT, ValueT_co]): + """Promote an existing typed Source adapter into an explicit Definition.""" + + adapter: SourceAdapter[InputT, SourceT, ValueT_co] + version: str = "1" + projections: tuple[SourceProjection[SourceT], ...] = () + + @property + def input_class(self) -> type[InputT]: + return self.adapter.input_class + + @property + def name(self) -> str: + return self.adapter.name + + @property + def source_class(self) -> type[SourceT]: + return self.adapter.source_class + + async def resolve(self, value: InputT, /) -> SourceT: + return await self.adapter.resolve(value) + + async def read(self, source: SourceT, /) -> ValueT_co: + return await self.adapter.read(source) + + +_AnySourceDefinition = SourceDefinition[Any, Any, Any] + + +class SourceDefinitionRegistry: + """Provide one immutable routing view for Source persistence and consumers.""" + + def __init__(self, definitions: Iterable[_AnySourceDefinition], /) -> None: + by_input: dict[type[object], _AnySourceDefinition] = {} + by_source: dict[type[Source], _AnySourceDefinition] = {} + by_name: dict[str, _AnySourceDefinition] = {} + projections: dict[type[Source], Mapping[SourceProjectionKey, SourceProjection[Any]]] = {} + registered: list[_AnySourceDefinition] = [] + + for definition in definitions: + input_class, source_class = _validate_definition(definition) + if input_class in by_input: + raise SourceConflictError("input_class", input_class) + if source_class in by_source: + raise SourceConflictError("source_class", source_class) + if definition.name in by_name: + raise SourceConflictError("name", definition.name) + + projection_routes: dict[SourceProjectionKey, SourceProjection[Any]] = {} + for projection in definition.projections: + key = _validate_projection(definition, projection) + if key in projection_routes: + raise SourceConflictError("projection", (definition.name, key)) + projection_routes[key] = projection + + by_input[input_class] = definition + by_source[source_class] = definition + by_name[definition.name] = definition + projections[source_class] = projection_routes + registered.append(definition) + + self._definitions = tuple(registered) + self._by_input = by_input + self._by_source = by_source + self._by_name = by_name + self._projections = projections + + @classmethod + def from_adapters(cls, adapters: Iterable[_AnySourceAdapter], /) -> SourceDefinitionRegistry: + """Wrap legacy adapters as version ``1`` Definitions without projections.""" + + return cls(AdapterSourceDefinition(adapter) for adapter in adapters) + + @property + def definitions(self) -> tuple[_AnySourceDefinition, ...]: + return self._definitions + + def definition_for_name(self, name: str, /) -> _AnySourceDefinition: + try: + return self._by_name[name] + except KeyError: + raise SourceDefinitionNotFoundError(name) from None + + def definition_for_source(self, source: object, /) -> _AnySourceDefinition: + if not isinstance(source, Source): + raise InvalidSourceEntryError(type(source)) + try: + definition = self._by_source[type(source)] + except KeyError: + raise SourceAdapterNotFoundError("source", type(source)) from None + if source.definition_version != definition.version: + raise InvalidSourceDefinitionError( + type(definition), + "version", + f"Source declares {source.definition_version!r}, expected {definition.version!r}", + ) + return definition + + async def resolve(self, value: object, /) -> Source: + input_class = type(value) + try: + definition = self._by_input[input_class] + except KeyError: + raise SourceAdapterNotFoundError("input", input_class) from None + source = await definition.resolve(value) + if type(source) is not definition.source_class: + raise InvalidSourceResultError(definition.name, "resolve", definition.source_class, type(source)) + self.definition_for_source(source) + return cast(Source, source) + + async def read(self, source: Source, /) -> object: + definition = self.definition_for_source(source) + return await definition.read(source) + + def projection_keys(self, source: Source, /) -> tuple[SourceProjectionKey, ...]: + self.definition_for_source(source) + return tuple(self._projections[type(source)]) + + def project(self, source: Source, key: SourceProjectionKey, /) -> JsonValue: + definition = self.definition_for_source(source) + try: + projection = self._projections[type(source)][key] + except KeyError: + raise SourceProjectionNotFoundError(definition.name, key.name, key.version) from None + value = projection.project(source) + try: + validated = projection.output_class.model_validate(value) + return _JSON_VALUE.validate_python(validated.model_dump(mode="json")) + except (TypeError, ValueError) as error: + raise InvalidSourceProjectionError(key.name, "result", "must match the declared output schema") from error + + +def _validate_definition(definition: object) -> tuple[type[object], type[Source]]: + definition_type = type(definition) + input_class = getattr(definition, "input_class", None) + if not isinstance(input_class, type): + raise InvalidSourceAdapterError(definition_type, "input_class", "must be a type") + name = getattr(definition, "name", None) + if not isinstance(name, str) or not name.strip() or name != name.strip(): + raise InvalidSourceDefinitionError(definition_type, "name", "must be a non-empty trimmed string") + version = getattr(definition, "version", None) + if not isinstance(version, str) or not version.strip() or version != version.strip(): + raise InvalidSourceDefinitionError(definition_type, "version", "must be a non-empty trimmed string") + source_class = getattr(definition, "source_class", None) + if not isinstance(source_class, type) or not issubclass(source_class, Source): + raise InvalidSourceAdapterError(definition_type, "source_class", "must be a Source subclass") + projections = getattr(definition, "projections", None) + if not isinstance(projections, tuple): + raise InvalidSourceDefinitionError(definition_type, "projections", "must be a tuple") + for method_name in ("resolve", "read"): + if not callable(getattr(definition, method_name, None)): + raise InvalidSourceAdapterError(definition_type, method_name, "must be callable") + return cast(type[object], input_class), cast(type[Source], source_class) + + +def _validate_projection( + definition: _AnySourceDefinition, + projection: object, +) -> SourceProjectionKey: + projection_type = type(projection) + name = getattr(projection, "name", None) + version = getattr(projection, "version", None) + if not isinstance(name, str) or not isinstance(version, str): + raise InvalidSourceProjectionError(str(name), "key", "must contain string name and version") + try: + key = SourceProjectionKey(name=name, version=version) + except (TypeError, ValueError) as error: + raise InvalidSourceProjectionError(str(name), "key", "must contain valid name and version") from error + source_class = getattr(projection, "source_class", None) + if source_class is not definition.source_class: + raise InvalidSourceProjectionError( + key.name, + "source_class", + f"must be {definition.source_class.__module__}.{definition.source_class.__qualname__}", + ) + output_class = getattr(projection, "output_class", None) + if not isinstance(output_class, type) or not issubclass(output_class, BaseModel): + raise InvalidSourceProjectionError(key.name, "output_class", "must be a BaseModel subclass") + if not callable(getattr(projection, "project", None)): + raise InvalidSourceProjectionError(key.name, "project", f"must be callable on {projection_type.__name__}") + return key diff --git a/src/powercontext/sources/models.py b/src/powercontext/sources/models.py index 9987e0620..6e2cd48df 100644 --- a/src/powercontext/sources/models.py +++ b/src/powercontext/sources/models.py @@ -16,7 +16,7 @@ from enum import StrEnum -from pydantic import BaseModel, field_validator +from pydantic import BaseModel, ConfigDict, field_validator from powercontext.errors import InvalidSourceReferenceError from powercontext.limits import MAX_SOURCE_ID_LENGTH, MAX_SOURCE_TYPE_LENGTH @@ -42,13 +42,35 @@ def validate_reference_part(cls, value: str, info) -> str: return value +class SourceProjectionKey(BaseModel): + """Select one independently versioned named projection capability.""" + + model_config = ConfigDict(frozen=True) + + name: str + version: str + + @field_validator("name", "version") + @classmethod + def validate_key_part(cls, value: str, info) -> str: + _validate_reference_part(info.field_name, value) + return value + + class Source(BaseModel): """Base value for an adapter-owned Source description.""" name: str + definition_version: str = "1" materialization: SourceMaterialization description: str | None = None + @field_validator("definition_version") + @classmethod + def validate_definition_version(cls, value: str) -> str: + _validate_reference_part("definition_version", value) + return value + def _validate_reference_part(field: str, value: object) -> None: if not isinstance(value, str) or not value.strip(): diff --git a/src/powercontext/sources/observations.py b/src/powercontext/sources/observations.py new file mode 100644 index 000000000..60a54e427 --- /dev/null +++ b/src/powercontext/sources/observations.py @@ -0,0 +1,228 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Worker-owned Source Definition manifests and projected observations.""" + +from __future__ import annotations + +import hashlib +from typing import Any + +import rfc8785 +from pydantic import BaseModel, ConfigDict, Field, JsonValue, TypeAdapter, field_validator, model_validator + +from powercontext.errors import InvalidSourceDefinitionError, InvalidSourceProjectionError +from powercontext.limits import MAX_SOURCE_TYPE_LENGTH +from powercontext.sources.definitions import SourceDefinition, SourceDefinitionRegistry +from powercontext.sources.models import Source, SourceProjectionKey + +_JSON_VALUE = TypeAdapter(JsonValue) + + +class SourceProjectionManifest(BaseModel): + """Declarative schema for one worker-computed named projection.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + key: SourceProjectionKey + schema_: dict[str, JsonValue] = Field(alias="schema") + + +class SourceDefinitionManifest(BaseModel): + """Immutable declarative identity registered by a remote worker.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + name: str + version: str + fingerprint: str + source_schema: dict[str, JsonValue] + projections: tuple[SourceProjectionManifest, ...] = () + + @field_validator("name", "version") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value or value.strip() != value or len(value) > MAX_SOURCE_TYPE_LENGTH: + raise ValueError("manifest identity must be a bounded non-empty trimmed string") # noqa: TRY003 + return value + + @field_validator("projections") + @classmethod + def validate_projection_limit( + cls, + value: tuple[SourceProjectionManifest, ...], + ) -> tuple[SourceProjectionManifest, ...]: + if len(value) > 16: + raise ValueError("manifest must not declare more than 16 projections") # noqa: TRY003 + return value + + @field_validator("fingerprint") + @classmethod + def validate_fingerprint_shape(cls, value: str) -> str: + if not value.startswith("sha256:") or len(value) != 71: + raise ValueError("manifest fingerprint must use sha256:") # noqa: TRY003 + try: + int(value.removeprefix("sha256:"), 16) + except ValueError as error: + raise ValueError("manifest fingerprint must contain lowercase hexadecimal") from error # noqa: TRY003 + if value != value.lower(): + raise ValueError("manifest fingerprint must contain lowercase hexadecimal") # noqa: TRY003 + return value + + @model_validator(mode="after") + def validate_manifest(self) -> SourceDefinitionManifest: + keys = tuple(projection.key for projection in self.projections) + if len(set(keys)) != len(keys): + raise ValueError("manifest projection keys must be unique") # noqa: TRY003 + expected = source_definition_fingerprint( + name=self.name, + version=self.version, + source_schema=self.source_schema, + projections=self.projections, + ) + if self.fingerprint != expected: + raise ValueError("manifest fingerprint does not match its declaration") # noqa: TRY003 + return self + + +class SourceProjectionValue(BaseModel): + """One named projection computed by the worker that owns the Definition.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + key: SourceProjectionKey + value: JsonValue + + +class ProjectedSource(Source): + """Canonical worker-materialized Source stored without loading plugin code.""" + + source_type: str + definition_fingerprint: str + payload: dict[str, JsonValue] + projections: tuple[SourceProjectionValue, ...] = () + + @field_validator("source_type") + @classmethod + def validate_source_type(cls, value: str) -> str: + if not value or value.strip() != value or len(value) > MAX_SOURCE_TYPE_LENGTH: + raise ValueError("source_type must be a bounded non-empty trimmed string") # noqa: TRY003 + return value + + @model_validator(mode="after") + def validate_envelope_identity(self) -> ProjectedSource: + expected = { + "name": self.name, + "definition_version": self.definition_version, + "materialization": self.materialization.value, + "description": self.description, + } + for field, value in expected.items(): + if self.payload.get(field) != value: + raise ValueError(f"projected Source payload {field} does not match its envelope") # noqa: TRY003 + keys = tuple(projection.key for projection in self.projections) + if len(set(keys)) != len(keys): + raise ValueError("projected Source projection keys must be unique") # noqa: TRY003 + return self + + def projection(self, key: SourceProjectionKey, /) -> JsonValue: + for projection in self.projections: + if projection.key == key: + return projection.value + raise InvalidSourceProjectionError(key.name, "key", "was not supplied by the worker") + + +def manifest_for_definition(definition: SourceDefinition[Any, Any, Any], /) -> SourceDefinitionManifest: + """Build the immutable declaration transported by a remote worker.""" + + source_schema = _json_object(definition.source_class.model_json_schema()) + projections = tuple( + SourceProjectionManifest( + key=SourceProjectionKey(name=projection.name, version=projection.version), + schema=projection.output_class.model_json_schema(), + ) + for projection in definition.projections + ) + return SourceDefinitionManifest( + name=definition.name, + version=definition.version, + fingerprint=source_definition_fingerprint( + name=definition.name, + version=definition.version, + source_schema=source_schema, + projections=projections, + ), + source_schema=source_schema, + projections=projections, + ) + + +def project_source_for_transport( + registry: SourceDefinitionRegistry, + source: Source, + /, +) -> ProjectedSource: + """Execute one worker-owned Definition and serialize its durable result.""" + + definition = registry.definition_for_source(source) + manifest = manifest_for_definition(definition) + payload = _json_object(source.model_dump(mode="json")) + projections = tuple( + SourceProjectionValue(key=key, value=registry.project(source, key)) for key in registry.projection_keys(source) + ) + return ProjectedSource( + name=source.name, + definition_version=source.definition_version, + materialization=source.materialization, + description=source.description, + source_type=definition.name, + definition_fingerprint=manifest.fingerprint, + payload=payload, + projections=projections, + ) + + +def source_definition_fingerprint( + *, + name: str, + version: str, + source_schema: dict[str, JsonValue], + projections: tuple[SourceProjectionManifest, ...], +) -> str: + declaration = { + "name": name, + "version": version, + "source_schema": source_schema, + "projections": [projection.model_dump(mode="json", by_alias=True) for projection in projections], + } + encoded = rfc8785.dumps(_JSON_VALUE.validate_python(declaration)) + return f"sha256:{hashlib.sha256(encoded).hexdigest()}" + + +def _json_object(value: object) -> dict[str, JsonValue]: + validated = _JSON_VALUE.validate_python(value) + if not isinstance(validated, dict): + raise InvalidSourceDefinitionError(type(value), "schema", "must be a JSON object") + return validated + + +__all__ = [ + "ProjectedSource", + "SourceDefinitionManifest", + "SourceProjectionManifest", + "SourceProjectionValue", + "manifest_for_definition", + "project_source_for_transport", + "source_definition_fingerprint", +] diff --git a/src/powercontext/sources/projections.py b/src/powercontext/sources/projections.py new file mode 100644 index 000000000..3864a7534 --- /dev/null +++ b/src/powercontext/sources/projections.py @@ -0,0 +1,33 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Standard named projection schemas understood without Source plugin code.""" + +from pydantic import BaseModel, Field, JsonValue + +from powercontext.sources.models import SourceProjectionKey + +TEXT_EVIDENCE_PROJECTION_KEY = SourceProjectionKey(name="powercontext.text-evidence", version="1") + + +class TextEvidence(BaseModel): + """Canonical JSON shape consumed as textual Artifact evidence.""" + + source_type: str + source_id: str + content: str + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + +__all__ = ["TEXT_EVIDENCE_PROJECTION_KEY", "TextEvidence"] diff --git a/tests/builtin/persistence/test_provider.py b/tests/builtin/persistence/test_provider.py index 46a146c79..01de310d6 100644 --- a/tests/builtin/persistence/test_provider.py +++ b/tests/builtin/persistence/test_provider.py @@ -17,12 +17,45 @@ import asyncio import pytest - -from powercontext import ArtifactNotFoundError, SourceConflictError +from pydantic import BaseModel + +from powercontext import ( + AdapterSourceDefinition, + ArtifactNotFoundError, + Source, + SourceConflictError, + SourceDefinitionRegistry, + SourceMaterialization, +) from powercontext.builtin.artifacts.memory import MemoryCandidateRequest, MemoryEntryInput from powercontext.builtin.persistence.sqlite import SQLiteConfig from powercontext.builtin.runtime import BuiltinConfig, open_builtin_contexts -from powercontext.builtin.sources import ContentCapture, ContentSource, SourceCursor +from powercontext.builtin.sources import BUILTIN_SOURCE_REGISTRY, ContentCapture, ContentSource, SourceCursor + + +class CustomCapture(BaseModel): + source_id: str + value: str + + +class CustomSource(Source): + value: str + + +class CustomSourceAdapter: + input_class = CustomCapture + name = "test-custom" + source_class = CustomSource + + async def resolve(self, value: CustomCapture, /) -> CustomSource: + return CustomSource( + name=value.source_id, + materialization=SourceMaterialization.CAPTURED, + value=value.value, + ) + + async def read(self, source: CustomSource, /) -> str: + return source.value class EchoCandidatePipeline: @@ -54,6 +87,27 @@ class StateSaveFailure(RuntimeError): pass +def test_provider_uses_one_injected_source_registry_for_routing_and_persistence() -> None: + async def scenario() -> None: + registry = SourceDefinitionRegistry(( + *BUILTIN_SOURCE_REGISTRY.definitions, + AdapterSourceDefinition(CustomSourceAdapter()), + )) + async with open_builtin_contexts( + BuiltinConfig(database=SQLiteConfig()), + source_registry=registry, + ) as contexts: + context = await contexts.get("project") + source = await context.sources.resolve(CustomCapture(source_id="custom-1", value="typed value")) + stored = await context.sources.add(source) + + assert isinstance(stored, CustomSource) + assert await context.sources.read(stored) == "typed value" + assert await context.sources.list() == (stored,) + + asyncio.run(scenario()) + + def test_provider_translates_repository_source_identity_conflicts() -> None: async def scenario() -> None: async with open_builtin_contexts(BuiltinConfig(database=SQLiteConfig())) as contexts: diff --git a/tests/integrations/test_opendal_connector.py b/tests/integrations/test_opendal_connector.py new file mode 100644 index 000000000..57105ae8e --- /dev/null +++ b/tests/integrations/test_opendal_connector.py @@ -0,0 +1,296 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +from pathlib import Path + +import httpx +import pytest +from fastapi import FastAPI +from powercontext_connector_opendal import ( + OPENDAL_TEXT_FILE_CONNECTOR_NAME, + TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION, + OpenDALTextFileConnector, + TextFileSnapshotCapture, +) + +from powercontext.builtin.artifacts.memory import MemoryCandidateRequest, MemoryEntryInput +from powercontext.builtin.persistence.sqlite import SQLiteConfig +from powercontext.client import PowerContextClient, RemoteConnectorWorker, ServerResponseError +from powercontext.http import ( + CommitConnectorCheckpointRequest, + FlushMemoryRequest, + ListMemoryEntriesRequest, + ListMemoryEntriesResponse, + RegisterSourceDefinitionRequest, + SubmitSourceObservationRequest, +) +from powercontext.http import ( + ConnectorBinding as HttpConnectorBinding, +) +from powercontext.http import ( + ProjectedSource as HttpProjectedSource, +) +from powercontext.http import ( + SourceDefinitionManifest as HttpSourceDefinitionManifest, +) +from powercontext.server.factory import create_server_app +from powercontext.server.settings import McpConfig, ServerSettings +from powercontext.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + ConnectorBinding, + ConnectorCapability, + ConnectorRunResult, + ConnectorRunStatus, + ConnectorSubmissionStatus, + ProjectedSource, + SourceDefinitionRegistry, + TextEvidence, + manifest_for_definition, + project_source_for_transport, +) + + +class MemoryFileSystem: + def __init__(self) -> None: + self.files: dict[str, bytes] = {} + + def pipe_file(self, path: str, content: bytes) -> None: + self.files[path] = content + + def find(self, path: str, *, detail: bool) -> dict[str, dict[str, object]]: + assert detail + prefix = f"{path.rstrip('/')}/" if path else "" + return { + name: {"name": name, "size": len(content), "type": "file"} + for name, content in self.files.items() + if not prefix or name.startswith(prefix) + } + + def cat_file(self, path: str) -> bytes: + return self.files[path] + + +class TextEvidenceCandidatePipeline: + async def extract(self, request: MemoryCandidateRequest, /) -> tuple[MemoryEntryInput, ...]: + entries: list[MemoryEntryInput] = [] + for source in request.sources: + if not isinstance(source, ProjectedSource): + continue + evidence = TextEvidence.model_validate(source.projection(TEXT_EVIDENCE_PROJECTION_KEY)) + entries.append(MemoryEntryInput(kind="document", text=evidence.content, sources=(source,))) + return tuple(entries) + + +def _binding() -> ConnectorBinding: + return ConnectorBinding( + scope_id="project-a", + binding_id="documents-a", + connector_name=OPENDAL_TEXT_FILE_CONNECTOR_NAME, + connector_version="1", + ) + + +def _app(database: Path, *, memory: bool = False) -> FastAPI: + return create_server_app( + settings=ServerSettings( + database=SQLiteConfig(url=f"sqlite+aiosqlite:///{database}"), + mcp=McpConfig(enabled=False), + ), + candidate_pipeline=TextEvidenceCandidatePipeline() if memory else None, + ) + + +async def _run( + app: FastAPI, + connector: OpenDALTextFileConnector, + *, + flush_memory: bool = False, +) -> tuple[ConnectorRunResult, ListMemoryEntriesResponse | None]: + async with ( + app.router.lifespan_context(app), + httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), + base_url="http://testserver", + ) as transport, + ): + client = PowerContextClient("http://testserver", http_client=transport, trust_transport_security=True) + worker = RemoteConnectorWorker( + client=client, + registry=SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)), + ) + result = await worker.run(connector, _binding()) + memory = None + if flush_memory: + await client.flush_memory(FlushMemoryRequest(scope_id="project-a")) + memory = await client.list_memory_entries(ListMemoryEntriesRequest(scope_id="project-a")) + return result, memory + + +def test_remote_opendal_worker_persists_incremental_checkpoint_across_server_restart(tmp_path: Path) -> None: + async def scenario() -> None: + filesystem = MemoryFileSystem() + filesystem.pipe_file("docs/readme.md", b"First value") + filesystem.pipe_file("docs/nested/note.txt", b"Nested value") + filesystem.pipe_file("docs/image.bin", b"\x00\x01") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a", root="docs") + database = tmp_path / "powercontext.db" + + first, _ = await _run(_app(database), connector) + unchanged, _ = await _run(_app(database), connector) + filesystem.pipe_file("docs/readme.md", b"Second value") + changed, _ = await _run(_app(database), connector) + + assert first.status is ConnectorRunStatus.COMPLETE + assert [item.item_id for item in first.items] == ["nested/note.txt", "readme.md"] + assert all(item.status is ConnectorSubmissionStatus.ACCEPTED for item in first.items) + assert unchanged.previous_checkpoint == first.committed_checkpoint + assert unchanged.items == () + assert [item.item_id for item in changed.items] == ["readme.md"] + assert changed.previous_checkpoint == first.committed_checkpoint + assert changed.committed_checkpoint != first.committed_checkpoint + + asyncio.run(scenario()) + + +def test_remote_opendal_worker_keeps_checkpoint_before_a_rejected_item(tmp_path: Path) -> None: + async def scenario() -> None: + filesystem = MemoryFileSystem() + filesystem.pipe_file("good.md", b"Good value") + filesystem.pipe_file("invalid.txt", b"\xff") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") + database = tmp_path / "powercontext.db" + + rejected, _ = await _run(_app(database), connector) + filesystem.pipe_file("invalid.txt", b"Recovered value") + recovered, _ = await _run(_app(database), connector) + + assert rejected.status is ConnectorRunStatus.INCOMPLETE + assert rejected.committed_checkpoint is None + assert [(item.item_id, item.status) for item in rejected.items] == [ + ("good.md", ConnectorSubmissionStatus.ACCEPTED), + ("invalid.txt", ConnectorSubmissionStatus.REJECTED), + ] + assert recovered.status is ConnectorRunStatus.COMPLETE + assert recovered.previous_checkpoint is None + assert recovered.committed_checkpoint is not None + + asyncio.run(scenario()) + + +def test_remote_opendal_worker_completes_the_source_to_memory_loop(tmp_path: Path) -> None: + async def scenario() -> None: + filesystem = MemoryFileSystem() + filesystem.pipe_file("decision.md", b"Use exact snapshot references.") + connector = OpenDALTextFileConnector(filesystem, source_namespace="workspace-a") + + result, memory = await _run(_app(tmp_path / "powercontext.db", memory=True), connector, flush_memory=True) + + assert result.status is ConnectorRunStatus.COMPLETE + assert memory is not None + assert [entry.text for entry in memory.entries] == ["Use exact snapshot references."] + assert memory.entries[0].source_refs[0].name == "text-file-snapshot" + + asyncio.run(scenario()) + + +def test_opendal_connector_declares_only_enforced_capabilities() -> None: + connector = OpenDALTextFileConnector(MemoryFileSystem(), source_namespace="workspace-a") + + assert ConnectorCapability.CHECKPOINT_RESUME in connector.capabilities + assert ConnectorCapability.AUTHORITATIVE_DELETION not in connector.capabilities + assert ConnectorCapability.CHANGE_FEED not in connector.capabilities + + +def test_opendal_connector_reads_the_real_opendalfs_memory_backend(tmp_path: Path) -> None: + opendalfs = pytest.importorskip("opendalfs") + + async def scenario() -> None: + filesystem = opendalfs.OpendalFileSystem( + scheme="memory", + asynchronous=False, + skip_instance_cache=True, + ) + filesystem.pipe_file("docs/readme.md", b"OpenDAL value") + connector = OpenDALTextFileConnector( + filesystem, + source_namespace="opendal-memory", + root="docs", + ) + + result, _ = await _run(_app(tmp_path / "powercontext.db"), connector) + + assert result.status is ConnectorRunStatus.COMPLETE + assert result.items[0].item_id == "readme.md" + assert result.items[0].status is ConnectorSubmissionStatus.ACCEPTED + + asyncio.run(scenario()) + + +def test_remote_ingestion_rejects_invalid_projection_and_stale_checkpoint(tmp_path: Path) -> None: + async def scenario() -> None: + app = _app(tmp_path / "powercontext.db") + registry = SourceDefinitionRegistry((TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION,)) + manifest = manifest_for_definition(TEXT_FILE_SNAPSHOT_SOURCE_DEFINITION) + source = await registry.resolve( + TextFileSnapshotCapture(namespace="workspace-a", path="decision.md", content="Keep worker authority.") + ) + projected = project_source_for_transport(registry, source) + malformed = projected.model_copy(update={"projections": ()}) + binding = HttpConnectorBinding.model_validate(_binding().model_dump(mode="json")) + + async with ( + app.router.lifespan_context(app), + httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), + base_url="http://testserver", + ) as transport, + ): + client = PowerContextClient("http://testserver", http_client=transport, trust_transport_security=True) + with pytest.raises(ServerResponseError) as missing: + await client.submit_source_observation( + SubmitSourceObservationRequest( + binding=binding, + source=HttpProjectedSource.model_validate(projected.model_dump(mode="json")), + ) + ) + await client.register_source_definition( + RegisterSourceDefinitionRequest( + manifest=HttpSourceDefinitionManifest.model_validate( + manifest.model_dump(mode="json", by_alias=True) + ) + ) + ) + with pytest.raises(ServerResponseError) as invalid: + await client.submit_source_observation( + SubmitSourceObservationRequest( + binding=binding, + source=HttpProjectedSource.model_validate(malformed.model_dump(mode="json")), + ) + ) + await client.commit_connector_checkpoint( + CommitConnectorCheckpointRequest(binding=binding, expected=None, checkpoint={"cursor": 1}) + ) + with pytest.raises(ServerResponseError) as stale: + await client.commit_connector_checkpoint( + CommitConnectorCheckpointRequest(binding=binding, expected=None, checkpoint={"cursor": 2}) + ) + + assert (missing.value.status_code, missing.value.code) == (404, "source_definition_not_found") + assert (invalid.value.status_code, invalid.value.code) == (422, "invalid_source_ingestion") + assert (stale.value.status_code, stale.value.code) == (409, "connector_checkpoint_conflict") + + asyncio.run(scenario()) diff --git a/tests/test_connectors.py b/tests/test_connectors.py new file mode 100644 index 000000000..b1708e908 --- /dev/null +++ b/tests/test_connectors.py @@ -0,0 +1,272 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +import builtins +from copy import deepcopy + +import pytest +from pydantic import JsonValue + +from powercontext import ( + CatalogConnectorSourceSink, + ConnectorBinding, + ConnectorCapability, + ConnectorLifecycle, + ConnectorRunCompletion, + ConnectorRunSession, + ConnectorRunStatus, + ConnectorSubmissionResult, + ConnectorSubmissionStatus, + InvalidConnectorError, + InvalidConnectorRunError, + Source, + SourceCatalog, + SourceConflictError, + SourceRef, +) +from powercontext.builtin.sources import ( + BUILTIN_SOURCE_REGISTRY, + CONTENT_SOURCE_NAME, + TEXT_EVIDENCE_PROJECTION_KEY, + ContentCapture, +) + + +class IdempotentSourceStore: + def __init__(self, events: builtins.list[str]) -> None: + self.events = events + self.sources: dict[tuple[str, str], Source] = {} + + async def add(self, source: Source, /) -> Source: + definition = BUILTIN_SOURCE_REGISTRY.definition_for_source(source) + ref = SourceRef(source_type=definition.name, source_id=source.name) + key = (ref.source_type, ref.source_id) + existing = self.sources.get(key) + if existing is not None and existing != source: + raise SourceConflictError("identity", ref) + self.events.append(f"source:{ref.source_id}") + self.sources.setdefault(key, deepcopy(source)) + return self.sources[key] + + async def get(self, source: Source, /) -> Source: + definition = BUILTIN_SOURCE_REGISTRY.definition_for_source(source) + return self.sources[(definition.name, source.name)] + + async def list(self) -> tuple[Source, ...]: + return tuple(self.sources.values()) + + +class MemoryCheckpointStore: + def __init__(self, events: list[str]) -> None: + self.events = events + self.values: dict[str, JsonValue | None] = {} + + async def load(self, binding: ConnectorBinding, /) -> JsonValue | None: + return deepcopy(self.values.get(binding.binding_id)) + + async def save( + self, + binding: ConnectorBinding, + checkpoint: JsonValue | None, + /, + *, + expected: JsonValue | None, + ) -> None: + assert self.values.get(binding.binding_id) == expected + self.events.append("checkpoint") + self.values[binding.binding_id] = deepcopy(checkpoint) + + +class ContentConnector: + name = "test-content" + version = "1" + source_definitions = frozenset({CONTENT_SOURCE_NAME}) + capabilities = frozenset({ConnectorCapability.CHECKPOINT_RESUME}) + + def __init__(self, capture: ContentCapture) -> None: + self.capture = capture + + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + await session.submit(self.capture.source_id, CONTENT_SOURCE_NAME, self.capture) + return ConnectorRunCompletion(status=ConnectorRunStatus.COMPLETE, checkpoint={"cursor": 1}) + + +def _binding() -> ConnectorBinding: + return ConnectorBinding( + scope_id="scope-a", + binding_id="content-a", + connector_name="test-content", + connector_version="1", + ) + + +def test_connector_commits_checkpoint_after_durable_source_acceptance() -> None: + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + catalog = SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY) + checkpoints = MemoryCheckpointStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink(scope_id="scope-a", catalog=catalog, store=store), + checkpoints=checkpoints, + ) + connector = ContentConnector(ContentCapture(source_id="note-1", content="Remember this.")) + + result = await lifecycle.run(connector, _binding()) + + assert events == ["source:note-1", "checkpoint"] + assert result.status is ConnectorRunStatus.COMPLETE + assert result.previous_checkpoint is None + assert result.committed_checkpoint == {"cursor": 1} + assert result.items[0].status is ConnectorSubmissionStatus.ACCEPTED + assert result.items[0].source_ref == SourceRef(source_type=CONTENT_SOURCE_NAME, source_id="note-1") + assert len(store.sources) == 1 + stored = next(iter(store.sources.values())) + assert catalog.project(stored, TEXT_EVIDENCE_PROJECTION_KEY) == { + "source_type": CONTENT_SOURCE_NAME, + "source_id": "note-1", + "content": "Remember this.", + "metadata": {}, + } + assert catalog.project(stored, TEXT_EVIDENCE_PROJECTION_KEY) == catalog.project( + stored, + TEXT_EVIDENCE_PROJECTION_KEY, + ) + + replay = await lifecycle.run(connector, _binding()) + assert replay.previous_checkpoint == {"cursor": 1} + assert len(store.sources) == 1 + assert events == ["source:note-1", "checkpoint", "source:note-1"] + + asyncio.run(scenario()) + + +def test_connector_exposes_failed_items_and_does_not_advance_checkpoint() -> None: + class RejectingSink: + async def submit(self, binding, item_id, definition_name, value, /) -> ConnectorSubmissionResult: + return ConnectorSubmissionResult(status=ConnectorSubmissionStatus.REJECTED, detail="unsupported value") + + async def scenario() -> None: + events: list[str] = [] + checkpoints = MemoryCheckpointStore(events) + lifecycle = ConnectorLifecycle(sink=RejectingSink(), checkpoints=checkpoints) + + result = await lifecycle.run( + ContentConnector(ContentCapture(source_id="note-1", content="Remember this.")), + _binding(), + ) + + assert result.status is ConnectorRunStatus.INCOMPLETE + assert result.proposed_checkpoint == {"cursor": 1} + assert result.committed_checkpoint is None + assert result.items[0].status is ConnectorSubmissionStatus.REJECTED + assert events == [] + + asyncio.run(scenario()) + + +def test_connector_does_not_advance_an_incomplete_run_checkpoint() -> None: + class IncompleteConnector(ContentConnector): + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + await session.submit(self.capture.source_id, CONTENT_SOURCE_NAME, self.capture) + return ConnectorRunCompletion( + status=ConnectorRunStatus.INCOMPLETE, + checkpoint={"cursor": 1}, + ) + + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink( + scope_id="scope-a", + catalog=SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY), + store=store, + ), + checkpoints=MemoryCheckpointStore(events), + ) + + result = await lifecycle.run( + IncompleteConnector(ContentCapture(source_id="note-1", content="Remember this.")), + _binding(), + ) + + assert result.status is ConnectorRunStatus.INCOMPLETE + assert result.proposed_checkpoint == {"cursor": 1} + assert result.committed_checkpoint is None + assert events == ["source:note-1"] + + asyncio.run(scenario()) + + +def test_connector_rejects_duplicate_items_and_binding_mismatches() -> None: + class DuplicateConnector(ContentConnector): + async def run(self, session: ConnectorRunSession, /) -> ConnectorRunCompletion: + await session.submit("note-1", CONTENT_SOURCE_NAME, self.capture) + await session.submit("note-1", CONTENT_SOURCE_NAME, self.capture) + return ConnectorRunCompletion(status=ConnectorRunStatus.COMPLETE) + + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink( + scope_id="scope-a", + catalog=SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY), + store=store, + ), + checkpoints=MemoryCheckpointStore(events), + ) + capture = ContentCapture(source_id="note-1", content="Remember this.") + + with pytest.raises(InvalidConnectorRunError) as duplicate: + await lifecycle.run(DuplicateConnector(capture), _binding()) + assert duplicate.value.issue == "duplicate-item" + + mismatched = _binding().model_copy(update={"connector_version": "2"}) + with pytest.raises(InvalidConnectorError) as binding_error: + await lifecycle.run(ContentConnector(capture), mismatched) + assert binding_error.value.field == "version" + + asyncio.run(scenario()) + + +def test_catalog_connector_sink_rejects_a_different_scope_before_storage() -> None: + async def scenario() -> None: + events: list[str] = [] + store = IdempotentSourceStore(events) + lifecycle = ConnectorLifecycle( + sink=CatalogConnectorSourceSink( + scope_id="scope-b", + catalog=SourceCatalog(backend=store, registry=BUILTIN_SOURCE_REGISTRY), + store=store, + ), + checkpoints=MemoryCheckpointStore(events), + ) + + result = await lifecycle.run( + ContentConnector(ContentCapture(source_id="note-1", content="Remember this.")), + _binding(), + ) + + assert result.status is ConnectorRunStatus.INCOMPLETE + assert result.items[0].status is ConnectorSubmissionStatus.FAILED + assert result.items[0].detail == "InvalidConnectorRunError" + assert store.sources == {} + assert events == [] + + asyncio.run(scenario()) diff --git a/tests/test_source_observations.py b/tests/test_source_observations.py new file mode 100644 index 000000000..635bf58b7 --- /dev/null +++ b/tests/test_source_observations.py @@ -0,0 +1,71 @@ +# Copyright (c) 2026 OceanBase. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio + +import pytest +from pydantic import ValidationError + +from powercontext.builtin.sources import CONTENT_SOURCE_DEFINITION, ContentCapture +from powercontext.sources import ( + TEXT_EVIDENCE_PROJECTION_KEY, + ProjectedSource, + Source, + SourceCatalog, + SourceDefinitionManifest, + SourceDefinitionRegistry, + TextEvidence, + manifest_for_definition, + project_source_for_transport, +) + + +class EmptySourceBackend: + async def list(self) -> tuple[Source, ...]: + return () + + async def get(self, source: Source, /) -> Source: + raise AssertionError(source) + + +def test_definition_manifest_has_a_stable_content_addressed_identity() -> None: + first = manifest_for_definition(CONTENT_SOURCE_DEFINITION) + second = manifest_for_definition(CONTENT_SOURCE_DEFINITION) + + assert first == second + assert first.fingerprint.startswith("sha256:") + with pytest.raises(ValidationError, match="fingerprint does not match"): + SourceDefinitionManifest.model_validate(first.model_dump(mode="json", by_alias=True) | {"version": "2"}) + + +def test_projected_source_remains_usable_without_worker_definition_code() -> None: + async def scenario() -> None: + registry = SourceDefinitionRegistry((CONTENT_SOURCE_DEFINITION,)) + source = await registry.resolve( + ContentCapture(source_id="turn-1", content="Keep the remote contract declarative.") + ) + projected = project_source_for_transport(registry, source) + catalog = SourceCatalog(backend=EmptySourceBackend()) + payload = await catalog.read(projected) + projection = catalog.project(projected, TEXT_EVIDENCE_PROJECTION_KEY) + + assert isinstance(projected, ProjectedSource) + assert catalog.as_ref(projected).model_dump() == {"source_type": "content", "source_id": "turn-1"} + assert payload == projected.payload + assert catalog.projection_keys(projected) == (TEXT_EVIDENCE_PROJECTION_KEY,) + assert TextEvidence.model_validate(projection).content == "Keep the remote contract declarative." + + asyncio.run(scenario()) diff --git a/tests/test_sources.py b/tests/test_sources.py index b757b7b88..b9707f751 100644 --- a/tests/test_sources.py +++ b/tests/test_sources.py @@ -20,12 +20,17 @@ from typing import TypeVar import pytest +from pydantic import BaseModel from powercontext import ( + AdapterSourceDefinition, + InvalidSourceDefinitionError, InvalidSourceEntryError, + InvalidSourceProjectionError, InvalidSourceResultError, SourceAdapterNotFoundError, SourceNotFoundError, + SourceProjectionNotFoundError, ) from powercontext.context import Sources from powercontext.sources import ( @@ -33,7 +38,9 @@ SourceAdapter, SourceCatalog, SourceCatalogBackend, + SourceDefinitionRegistry, SourceMaterialization, + SourceProjectionKey, SourceStore, ) @@ -105,6 +112,24 @@ async def read(self, source: TranscriptExportSource) -> object: return source +class ConversationSummary(BaseModel): + session_id: str + message_count: int + + +class ConversationSummaryProjection: + name = "test.conversation-summary" + version = "1" + source_class = ConversationSource + output_class: type[BaseModel] = ConversationSummary + + def project(self, source: ConversationSource, /) -> ConversationSummary: + return ConversationSummary( + session_id=source.session_id, + message_count=0 if source.captured_value is None else len(source.captured_value.messages), + ) + + StoredSourceT = TypeVar("StoredSourceT", bound=Source) @@ -282,3 +307,54 @@ async def scenario() -> None: assert backend.sources == [] asyncio.run(scenario()) + + +def test_definition_registry_routes_named_projections_without_source_type_checks() -> None: + async def scenario() -> None: + adapter = ConversationAdapter({"session-42": Conversation(("one", "two"))}) + registry = SourceDefinitionRegistry(( + AdapterSourceDefinition( + adapter, + version="1", + projections=(ConversationSummaryProjection(),), + ), + )) + catalog = SourceCatalog(backend=InMemorySourceStore(), registry=registry) + source = await catalog.resolve(ConversationCapture("snapshot", "session-42", capture=True)) + key = SourceProjectionKey(name="test.conversation-summary", version="1") + + assert catalog.projection_keys(source) == (key,) + assert catalog.project(source, key) == {"session_id": "session-42", "message_count": 2} + with pytest.raises(SourceProjectionNotFoundError): + catalog.project(source, SourceProjectionKey(name=key.name, version="2")) + + asyncio.run(scenario()) + + +def test_definition_registry_rejects_version_and_projection_contract_violations() -> None: + class InvalidProjection: + name = "test.invalid-json" + version = "1" + source_class = ConversationSource + output_class: type[BaseModel] = ConversationSummary + + def project(self, source: ConversationSource, /) -> object: + return object() + + adapter = ConversationAdapter({"session-42": Conversation(("one",))}) + registry = SourceDefinitionRegistry(( + AdapterSourceDefinition(adapter, version="2", projections=(InvalidProjection(),)), + )) + source = ConversationSource( + name="snapshot", + materialization=SourceMaterialization.CAPTURED, + session_id="session-42", + captured_value=Conversation(("one",)), + ) + + with pytest.raises(InvalidSourceDefinitionError): + registry.definition_for_source(source) + + compatible = source.model_copy(update={"definition_version": "2"}) + with pytest.raises(InvalidSourceProjectionError): + registry.project(compatible, SourceProjectionKey(name="test.invalid-json", version="1")) diff --git a/uv.lock b/uv.lock index 8191a6a45..cb9d22315 100644 --- a/uv.lock +++ b/uv.lock @@ -2089,6 +2089,7 @@ dependencies = [ builtin = [ { name = "aiosqlite" }, { name = "apscheduler" }, + { name = "jsonschema" }, { name = "pydantic-ai-slim", extra = ["anthropic", "openai"] }, { name = "pydantic-settings" }, { name = "pyobvector" }, @@ -2111,6 +2112,7 @@ client = [ seekdb = [ { name = "aiosqlite" }, { name = "apscheduler" }, + { name = "jsonschema" }, { name = "pydantic-ai-slim", extra = ["anthropic", "openai"] }, { name = "pydantic-settings" }, { name = "pylibseekdb", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, @@ -2124,6 +2126,7 @@ server = [ { name = "fastapi" }, { name = "fastmcp" }, { name = "jinja2" }, + { name = "jsonschema" }, { name = "opentelemetry-api" }, { name = "opentelemetry-sdk" }, { name = "platformdirs" }, @@ -2174,6 +2177,9 @@ requires-dist = [ { name = "httpx", extras = ["socks"], marker = "extra == 'client'", specifier = ">=0.28,<1" }, { name = "inquirerpy", marker = "extra == 'cli'", specifier = ">=0.3,<1" }, { name = "jinja2", marker = "extra == 'server'", specifier = ">=3.1,<4" }, + { name = "jsonschema", marker = "extra == 'builtin'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'seekdb'", specifier = ">=4.23,<5" }, + { name = "jsonschema", marker = "extra == 'server'", specifier = ">=4.23,<5" }, { name = "opentelemetry-api", marker = "extra == 'cli'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'client'", specifier = ">=1.30,<2" }, { name = "opentelemetry-api", marker = "extra == 'server'", specifier = ">=1.30,<2" }, diff --git a/zensical.toml b/zensical.toml index 3365122da..31094001d 100644 --- a/zensical.toml +++ b/zensical.toml @@ -40,6 +40,7 @@ nav = [ { "Configure WorkBuddy" = "en/docs/how-to/configure-workbuddy.md" }, { "Configure OpenClaw" = "en/docs/how-to/configure-openclaw.md" }, { "Configure OpenCode" = "en/docs/how-to/configure-opencode.md" }, + { "Ingest text files with OpenDAL" = "en/docs/how-to/ingest-text-files-with-opendal.md" }, { "Trace with Phoenix" = "en/docs/how-to/trace-with-phoenix.md" }, ] }, { "Reference" = [ @@ -63,6 +64,7 @@ nav = [ { "RFCs & Meetings" = [ { "RFCs" = [ { "Overview" = "en/rfcs/README.md" }, + { "0000 Source Definition and Observation Model" = "en/rfcs/0000_source_definition_and_observation_model.md" }, { "1229 Unified Workloads and Long-Horizon Memory Evaluation" = "en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "1223 Human-Agent Work Continuity" = "en/rfcs/1223_human_agent_work_continuity.md" }, { "0082 Handoff Report" = "en/rfcs/0082_handoff_report.md" }, @@ -121,6 +123,7 @@ nav = [ { "配置 WorkBuddy" = "zh/docs/how-to/configure-workbuddy.md" }, { "配置 OpenClaw" = "zh/docs/how-to/configure-openclaw.md" }, { "配置 OpenCode" = "zh/docs/how-to/configure-opencode.md" }, + { "使用 OpenDAL 采集文本文件" = "zh/docs/how-to/ingest-text-files-with-opendal.md" }, { "用 Phoenix 查看 trace" = "zh/docs/how-to/trace-with-phoenix.md" }, ] }, { "参考" = [ @@ -144,6 +147,7 @@ nav = [ { "RFC 与会议纪要" = [ { "RFC" = [ { "概览" = "zh/rfcs/README.md" }, + { "0000 Source 定义与观察模型" = "zh/rfcs/0000_source_definition_and_observation_model.md" }, { "1229 统一工作负载与长程 Memory 评估" = "zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "1223 人与 Agent 工作连续性" = "zh/rfcs/1223_human_agent_work_continuity.md" }, { "0082 Handoff 报告" = "zh/rfcs/0082_handoff_report.md" },