📝 CodeRabbit Chat: Implement requested code changes - #136
📝 CodeRabbit Chat: Implement requested code changes#136coderabbitai[bot] wants to merge 14 commits into
Conversation
Add the execution plan for roadmap item 4.3.1, which delivers the first
half of the ADR-009 source-to-script vertical slice. The plan introduces
an object-store port, an idempotency-store port, a pre-generation
source-attachment entity, and the REST surface (POST /v1/uploads,
POST /v1/ingestion-jobs, POST /v1/ingestion-jobs/{job_id}/sources, and
their GET pairs), while deferring the two-step upload, magic-byte
sniffing, and S3 adapters to later roadmap items. The plan was revised
after a Logisphere community-of-experts review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first success criterion previously used PDF as the only example, which read as if the intake path were limited to PDF documents. Expand the criterion to enumerate the full allowlist (PDF, DOCX, plain text, Markdown, HTML) and call out the 415 response for unsupported types, so the contract scope matches the allowlist already pinned in `Constraints` and `Risks`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add ADR 015 for the upload object-store port and idempotency store contract that roadmap item `4.3.1` will implement. Update the system design, contents index, user guide, developer guide, and active ExecPlan with the source-intake schema sketch, reserved error codes, deferred resumable upload decision, validation evidence, and the CodeRabbit rate-limit blocker.
Remove the extra blank line before the source-intake maintainer section so markdownlint passes after the rebase.
Add focused tests for the refactored brief-loading and reference-binding facade paths so deleted module tests have replacement coverage for missing references and public re-export contracts. Define bounded intake metrics, storage-boundary trace spans, operational alerts, and log levels for upload and idempotency failures. Stabilize the local test gate by reusing one py-pglite process per pytest session, resetting schema per database-backed test, and avoiding xdist when only one worker is requested.
Replace identity-based bindings facade checks with async functional tests that call each public facade function through `bindings` and assert returned domain values. Record the follow-up review finding and decision in the execplan so the coverage rationale remains visible.
Expand the brief loader test module docstring to explain the helper functions, reference-document entities, brief-generation role, and in-memory repository stubs covered by the tests.
Replace HTTP-layer idempotency storage terms with operation-keyed domain records and opaque serialised outcomes. Update ADR 015, the system-design schema, and the source-intake execplan so adapters own response codecs while the domain port preserves the existing replay and conflict guarantees.
Add a shared async reset lock around the py-pglite public schema reset fixture so concurrent callers cannot drop `public` during another reset. Cover the path with a fixture test that verifies both lock use and the post-reset schema state. Document the xdist serialisation rule, and make the runtime wiring readiness test run lifespan shutdown so the database engine is disposed before the next py-pglite fixture starts.
Add upload, source attachment, idempotency, and object-store domain contracts for the source-intake workflow. Include a filesystem-backed object-store adapter so local intake paths can persist and stream opaque upload payloads behind the new port. Cover the ADR 015 multipart fingerprint, object-store path and size guards, and idempotency replay/conflict invariants with focused tests. Record the published ADR vector discrepancy in the execplan so the compatibility value remains explicit.
Apply formatter-driven line wrapping to existing documentation touched by the validation workflow. Keep the changes textual only so Markdown lint and diagram validation remain clean.
Add Alembic-backed upload, source-attachment, and idempotency storage plus source-intake application services and /v1 Falcon resources. Cover SQLAlchemy repository round trips, domain idempotency outcomes, the ADR 015 multipart worked vector, a BDD source-intake scenario, a response-envelope snapshot, and the end-to-end upload/job/source/poll workflow.
Document the origin/main rebase conflict resolution in the source-intake execplan and remove a Markdown blank-line lint issue exposed after the rebase.
|
Important Review skippedThis PR was authored by the user configured for CodeRabbit reviews. CodeRabbit does not review PRs authored by this user. It's recommended to use a dedicated user account to post CodeRabbit review feedback. ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Reviewer's GuideRefactors domain validation logic for ingestion sources, idempotency records/requests, and uploads into reusable private helper functions while preserving existing invariants and error messages. Flow diagram for shared upload validation helpersflowchart TD
Start["Upload validation entrypoints"]
Start --> UPost["Upload.__post_init__"]
Start --> IRequestPost["UploadInitRequest.__post_init__"]
UPost --> DeclaredSizeU["_require_non_negative_size(declared_size, declared_size)"]
UPost --> ActualSize["_require_non_negative_actual_size(actual_size)"]
UPost --> ContentTypeU["_require_non_empty_string(content_type, content_type)"]
UPost --> StorageKey["_require_non_empty_string(storage_key, storage_key)"]
IRequestPost --> DeclaredSizeR["_require_non_negative_size(declared_size, declared_size)"]
IRequestPost --> ContentTypeR["_require_non_empty_string(content_type, content_type)"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
ingestion_sources.py, the_require_exclusive_sourcedocstring doesn’t match the implementation logic (it currently says 'when exactly one source identifier is not populated' but the check is for both populated or both empty); consider rewording to describe the invariant more accurately (e.g., 'unless exactly one of upload_id or source_uri is populated'). - Both
idempotency.pyanduploads.pydefine a_require_non_empty_stringhelper with identical semantics; if this duplication is intentional for domain isolation that’s fine, but otherwise consider a shared internal utility to avoid divergence in future changes.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `ingestion_sources.py`, the `_require_exclusive_source` docstring doesn’t match the implementation logic (it currently says 'when exactly one source identifier is not populated' but the check is for both populated or both empty); consider rewording to describe the invariant more accurately (e.g., 'unless exactly one of upload_id or source_uri is populated').
- Both `idempotency.py` and `uploads.py` define a `_require_non_empty_string` helper with identical semantics; if this duplication is intentional for domain isolation that’s fine, but otherwise consider a shared internal utility to avoid divergence in future changes.
## Individual Comments
### Comment 1
<location path="episodic/canonical/ingestion_sources.py" line_range="21" />
<code_context>
SOURCE_URI = "source_uri"
+def _require_exclusive_source(has_upload: bool, has_uri: bool) -> None:
+ """Raise ValueError when exactly one source identifier is not populated."""
+ if has_upload == has_uri:
</code_context>
<issue_to_address>
**issue (complexity):** Consider inlining or consolidating these tiny validation helpers into one or two class-scoped methods so the dataclass’s control flow stays local and readable.
You’ve increased indirection without reducing complexity: five one-line module-level helpers for a single dataclass make control flow harder to follow and fragment the design.
You can keep all the validation and error messages exactly as-is while simplifying by:
1. Moving validation back into the dataclass (where the semantics clearly belong).
2. Grouping related checks into a single private method (or two at most), instead of five tiny helpers.
3. If you want to keep the logic reusable/testable, use a single `_validate` method or `@staticmethod` on the class.
For example:
```python
@dc.dataclass(frozen=True, slots=True)
class IngestionJobSource:
# ... fields ...
def __post_init__(self) -> None:
"""Validate attachment shape and weighting invariants."""
self._validate()
def _validate(self) -> None:
has_upload = self.upload_id is not None
has_uri = self.source_uri is not None
if has_upload == has_uri:
msg = "Exactly one of upload_id or source_uri must be populated."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.UPLOAD and not has_upload:
msg = "upload attachments must populate upload_id."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.SOURCE_URI and not has_uri:
msg = "source_uri attachments must populate source_uri."
raise ValueError(msg)
if not self.source_type.strip():
msg = "source_type must be a non-empty string."
raise ValueError(msg)
if not 0 <= self.weight <= 1:
msg = "weight must be between 0 and 1."
raise ValueError(msg)
```
If you still prefer some decomposition, keep it coarse-grained and class-scoped, not five separate module helpers, e.g.:
```python
def _validate_sources(self, has_upload: bool, has_uri: bool) -> None:
if has_upload == has_uri:
msg = "Exactly one of upload_id or source_uri must be populated."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.UPLOAD and not has_upload:
msg = "upload attachments must populate upload_id."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.SOURCE_URI and not has_uri:
msg = "source_uri attachments must populate source_uri."
raise ValueError(msg)
```
Then call this once from `_validate`. This keeps functionality identical but reduces surface area and improves readability.
</issue_to_address>
### Comment 2
<location path="episodic/canonical/idempotency.py" line_range="19" />
<code_context>
COMPLETED = "completed"
+def _require_non_empty_string(value: str, field_name: str) -> None:
+ """Raise ValueError when *value* is blank after stripping whitespace."""
+ if not value.strip():
</code_context>
<issue_to_address>
**issue (complexity):** Consider replacing the generic string validator and shared `_validate_idempotency_key_fields` helper with small field-specific helpers wired directly in each `__post_init__` to make the invariants clearer in place.
You can reduce the indirection while keeping the refactor’s DRY benefits by dropping the generic `_require_non_empty_string` and the “bag of fields” helper, and instead adding small, field-specific helpers.
This keeps messages and logic centralized, but makes each class’s invariants visible directly in `__post_init__` without stringly-typed field names.
```python
def _require_non_empty_operation(operation: str) -> None:
if not operation.strip():
raise ValueError("operation must be a non-empty string.")
def _require_non_empty_idempotency_key(idempotency_key: str) -> None:
if not idempotency_key.strip():
raise ValueError("idempotency_key must be a non-empty string.")
def _require_non_empty_body_hash(body_hash: str) -> None:
if not body_hash.strip():
raise ValueError("body_hash must be a non-empty string.")
```
Then wire them directly in each dataclass, keeping `_require_completed_has_outcome`:
```python
@dataclass(frozen=True, slots=True)
class IdempotencyRecord:
...
def __post_init__(self) -> None:
_require_non_empty_operation(self.operation)
_require_non_empty_idempotency_key(self.idempotency_key)
_require_non_empty_body_hash(self.body_hash)
_require_completed_has_outcome(self.state, self.serialised_outcome)
@dataclass(frozen=True, slots=True)
class IdempotencyAcquireRequest:
...
def __post_init__(self) -> None:
_require_non_empty_operation(self.operation)
_require_non_empty_idempotency_key(self.idempotency_key)
_require_non_empty_body_hash(self.body_hash)
```
This removes the extra layer `_validate_idempotency_key_fields` and avoids passing field names around as strings, making the invariants easier to read in place while preserving the shared validation logic and behavior.
</issue_to_address>
### Comment 3
<location path="episodic/canonical/uploads.py" line_range="23" />
<code_context>
EXPIRED = "expired"
+def _require_non_negative_size(value: int, field_name: str) -> None:
+ """Raise ValueError when an integer size field is negative."""
+ if value < 0:
</code_context>
<issue_to_address>
**issue (complexity):** Consider keeping validation logic per-class (with a single _validate method each) instead of multiple tiny module-level helper functions that add indirection without simplifying the code.
The new helpers add indirection without really simplifying the calling sites. Each function wraps a single if‑statement and message, so readers now have to jump between multiple tiny helpers to understand invariants that used to be local and domain-specific.
You can keep the validations cohesive and DRY without scattering them by grouping them into per-class helpers instead of multiple module-level utilities. For example:
```python
@dc.dataclass(frozen=True, slots=True)
class Upload:
...
def __post_init__(self) -> None:
"""Validate upload invariants at the domain boundary."""
self._validate()
def _validate(self) -> None:
if self.declared_size < 0:
msg = "declared_size must be non-negative."
raise ValueError(msg)
if self.actual_size is not None and self.actual_size < 0:
msg = "actual_size must be non-negative when provided."
raise ValueError(msg)
if not self.content_type.strip():
msg = "content_type must be a non-empty string."
raise ValueError(msg)
if not self.storage_key.strip():
msg = "storage_key must be a non-empty string."
raise ValueError(msg)
```
And for `UploadInitRequest`:
```python
@dc.dataclass(frozen=True, slots=True)
class UploadInitRequest:
...
def __post_init__(self) -> None:
"""Validate the client-declared upload metadata."""
self._validate()
def _validate(self) -> None:
if self.declared_size < 0:
msg = "declared_size must be non-negative."
raise ValueError(msg)
if not self.content_type.strip():
msg = "content_type must be a non-empty string."
raise ValueError(msg)
```
This keeps all checks close to their domain objects, avoids extra control-flow hops, and still gives you a single place per class to maintain the invariants. If you still want some shared logic, you can extract only the genuinely reusable pieces (e.g., a single `_require_non_negative_size`) and leave the rest inline to keep the code readable.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| SOURCE_URI = "source_uri" | ||
|
|
||
|
|
||
| def _require_exclusive_source(has_upload: bool, has_uri: bool) -> None: |
There was a problem hiding this comment.
issue (complexity): Consider inlining or consolidating these tiny validation helpers into one or two class-scoped methods so the dataclass’s control flow stays local and readable.
You’ve increased indirection without reducing complexity: five one-line module-level helpers for a single dataclass make control flow harder to follow and fragment the design.
You can keep all the validation and error messages exactly as-is while simplifying by:
- Moving validation back into the dataclass (where the semantics clearly belong).
- Grouping related checks into a single private method (or two at most), instead of five tiny helpers.
- If you want to keep the logic reusable/testable, use a single
_validatemethod or@staticmethodon the class.
For example:
@dc.dataclass(frozen=True, slots=True)
class IngestionJobSource:
# ... fields ...
def __post_init__(self) -> None:
"""Validate attachment shape and weighting invariants."""
self._validate()
def _validate(self) -> None:
has_upload = self.upload_id is not None
has_uri = self.source_uri is not None
if has_upload == has_uri:
msg = "Exactly one of upload_id or source_uri must be populated."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.UPLOAD and not has_upload:
msg = "upload attachments must populate upload_id."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.SOURCE_URI and not has_uri:
msg = "source_uri attachments must populate source_uri."
raise ValueError(msg)
if not self.source_type.strip():
msg = "source_type must be a non-empty string."
raise ValueError(msg)
if not 0 <= self.weight <= 1:
msg = "weight must be between 0 and 1."
raise ValueError(msg)If you still prefer some decomposition, keep it coarse-grained and class-scoped, not five separate module helpers, e.g.:
def _validate_sources(self, has_upload: bool, has_uri: bool) -> None:
if has_upload == has_uri:
msg = "Exactly one of upload_id or source_uri must be populated."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.UPLOAD and not has_upload:
msg = "upload attachments must populate upload_id."
raise ValueError(msg)
if self.attachment_kind is AttachmentKind.SOURCE_URI and not has_uri:
msg = "source_uri attachments must populate source_uri."
raise ValueError(msg)Then call this once from _validate. This keeps functionality identical but reduces surface area and improves readability.
| COMPLETED = "completed" | ||
|
|
||
|
|
||
| def _require_non_empty_string(value: str, field_name: str) -> None: |
There was a problem hiding this comment.
issue (complexity): Consider replacing the generic string validator and shared _validate_idempotency_key_fields helper with small field-specific helpers wired directly in each __post_init__ to make the invariants clearer in place.
You can reduce the indirection while keeping the refactor’s DRY benefits by dropping the generic _require_non_empty_string and the “bag of fields” helper, and instead adding small, field-specific helpers.
This keeps messages and logic centralized, but makes each class’s invariants visible directly in __post_init__ without stringly-typed field names.
def _require_non_empty_operation(operation: str) -> None:
if not operation.strip():
raise ValueError("operation must be a non-empty string.")
def _require_non_empty_idempotency_key(idempotency_key: str) -> None:
if not idempotency_key.strip():
raise ValueError("idempotency_key must be a non-empty string.")
def _require_non_empty_body_hash(body_hash: str) -> None:
if not body_hash.strip():
raise ValueError("body_hash must be a non-empty string.")Then wire them directly in each dataclass, keeping _require_completed_has_outcome:
@dataclass(frozen=True, slots=True)
class IdempotencyRecord:
...
def __post_init__(self) -> None:
_require_non_empty_operation(self.operation)
_require_non_empty_idempotency_key(self.idempotency_key)
_require_non_empty_body_hash(self.body_hash)
_require_completed_has_outcome(self.state, self.serialised_outcome)
@dataclass(frozen=True, slots=True)
class IdempotencyAcquireRequest:
...
def __post_init__(self) -> None:
_require_non_empty_operation(self.operation)
_require_non_empty_idempotency_key(self.idempotency_key)
_require_non_empty_body_hash(self.body_hash)This removes the extra layer _validate_idempotency_key_fields and avoids passing field names around as strings, making the invariants easier to read in place while preserving the shared validation logic and behavior.
| EXPIRED = "expired" | ||
|
|
||
|
|
||
| def _require_non_negative_size(value: int, field_name: str) -> None: |
There was a problem hiding this comment.
issue (complexity): Consider keeping validation logic per-class (with a single _validate method each) instead of multiple tiny module-level helper functions that add indirection without simplifying the code.
The new helpers add indirection without really simplifying the calling sites. Each function wraps a single if‑statement and message, so readers now have to jump between multiple tiny helpers to understand invariants that used to be local and domain-specific.
You can keep the validations cohesive and DRY without scattering them by grouping them into per-class helpers instead of multiple module-level utilities. For example:
@dc.dataclass(frozen=True, slots=True)
class Upload:
...
def __post_init__(self) -> None:
"""Validate upload invariants at the domain boundary."""
self._validate()
def _validate(self) -> None:
if self.declared_size < 0:
msg = "declared_size must be non-negative."
raise ValueError(msg)
if self.actual_size is not None and self.actual_size < 0:
msg = "actual_size must be non-negative when provided."
raise ValueError(msg)
if not self.content_type.strip():
msg = "content_type must be a non-empty string."
raise ValueError(msg)
if not self.storage_key.strip():
msg = "storage_key must be a non-empty string."
raise ValueError(msg)And for UploadInitRequest:
@dc.dataclass(frozen=True, slots=True)
class UploadInitRequest:
...
def __post_init__(self) -> None:
"""Validate the client-declared upload metadata."""
self._validate()
def _validate(self) -> None:
if self.declared_size < 0:
msg = "declared_size must be non-negative."
raise ValueError(msg)
if not self.content_type.strip():
msg = "content_type must be a non-empty string."
raise ValueError(msg)This keeps all checks close to their domain objects, avoids extra control-flow hops, and still gives you a single place per class to maintain the invariants. If you still want some shared logic, you can extract only the genuinely reusable pieces (e.g., a single _require_non_negative_size) and leave the rest inline to keep the code readable.
Code changes was requested by @leynos.
The following files were modified:
episodic/canonical/idempotency.pyepisodic/canonical/ingestion_sources.pyepisodic/canonical/uploads.pySummary by Sourcery
Extract shared validation logic into reusable helper functions across ingestion source, idempotency, and upload domain models.
Enhancements: