Release 1.1.0-rc.3#20
Merged
Merged
Conversation
Replace Meaningfy-specific attributions, emails, and example hosts with project-neutral wording, GitHub-native contact channels, and placeholder hosts. Drop stale copyright line from LICENSE. Update pyproject authors to the Publications Office of the European Union.
Closes TEDSWS-524 and TEDSWS-522. - Sort-by-cluster-size and cluster-size distribution stats via maintained cluster_sizes projection collection (ClusterSizeIndex port + Mongo adapter + integrator hooks in DecisionStoreService.store_decision + backfill / verify scripts). - previous_review_count counter on every decision, atomically incremented on each user-action save and preserved across ERE re-integration; decision_id filter on /curation/user-actions for the full curator timeline. - Optional ?reviewed=true|false filter on /curation/decisions via gated $lookup against user_actions on the curation path; bulk-sync path (query_decisions_paginated) unchanged. - cluster_size field on CanonicalEntityPreview, populated from ClusterSizeIndex.get_size. - RegistryStatistics gains cluster_size_average (renamed from average_cluster_size), cluster_size_median, cluster_size_p95, cluster_size_max, cluster_singletons_count. - Idempotency fix in user_action_service._check_not_already_curated: removes the updated_at-gated guard so re-actions on fresh decisions are rejected consistently. Indexes added: user_actions.about_entity_mention, cluster_sizes.size.
Task outcome files written by the implementer agents during the phased TEDSWS-524 + TEDSWS-522 implementation. Filed under TEDSWS-528 by the agents (branch-name aliasing); content covers phases 2 through 7.
Short-circuit storing an ERE outcome only when the full outcome (placement and candidates) is unchanged, instead of keying on cluster_id alone. A re-assessment that returns the same cluster with a changed confidence now writes through and advances updated_at, so the decision re-surfaces for curator review and shows the fresh confidence.
Add reviewed_since_placement (derived per request) alongside previous_review_count on DecisionSummary, and filter the decisions list by ever_reviewed and reviewed_since_placement. The UI composes the four curator states (not reviewed, up to date, needs revisit, reviewed more than once) from the two primitives; the old reviewed parameter is kept as a deprecated alias. Apply the review filter before sort/limit so review-filtered pages no longer under-fill and terminate pagination early.
Correct the user_actions join key to the about_entity_mention triad, describe the cursor and filter-before-limit pipeline behaviour, and add an implementation delta plan covering the material-outcome short-circuit, the four-state read surface, and the pagination fix.
Use $in:[0,None] for the ever_reviewed=false filter (engine-portable, matches a missing counter), dump the entity-mention triad once for both the query and the result key, and reference field-path constants in the lookup. Add tests for the never-reviewed counter branch, the combined ever_reviewed + reviewed_since_placement path, and the never-reviewed row.
Specify the post-review follow-ups, the effect of ERE-outcome integration on the decision/cluster-size/user-action stores, and TEDSWS-530 (curator reject/assign must reach ERE). Reject-all now excludes the current placement plus candidates; delivery stays best-effort with the outgoing payload logged for auditability.
…ines Cross-check against entity-resolution-docs: add normative-basis anchors, correct the ERE-integration write guards (separate stale-ordering guard from material-change idempotency), confirm the reject-all exclusion-set bug via the candidates[0]/[1:] split, and record the best-effort delivery deviation.
…ize delete-on-zero, review-state read-port) C (TEDSWS-530): "reject all" now excludes the current placement AND all candidates (deduplicated, order-preserving); the outgoing ERE payload summary is logged at INFO for auditability; best-effort delivery is retained. B: cluster_sizes shift deletes an entry once it reaches zero and never persists a negative count (no-upsert decrement + delete-on-<=0), so stats never observe a size:0 row. A: the reviewed_since_placement read moved to a curation-owned ReviewStateReader port/adapter so the decision-store no longer reads the user_actions collection (A1); has_current_action aligned to strict $gt (A5); list_decisions issues its independent reads concurrently (A4); CursorPage.count documented as an upper bound when the review filter is active (A3). Fixes two latent bugs surfaced by new FerretDB integration tests: - missing await on the review-filter and cluster-size aggregations, and - previous_review_count not stripped before Decision model validation.
- Consolidate derived-field stripping in MongoDecisionRepository._from_document (previous_review_count + cluster_size) so there is one strip funnel for all read paths; drop the inline cluster_size comprehension. - Scope the ReviewStateReader docstring honestly: it moves the flag-attachment read to curation; the review-filter $lookup intentionally remains in the decision-store aggregation for keyset pagination. - Replace stale action-vocabulary in accept/reject/assign docstrings with the contract field names. - Tests: assert the curation action still succeeds when the ERE publish is swallowed; pin the empty-candidates reject (exclusion = [placement]); add integration coverage for find_by_id on a counter-bearing decision and a B5 re-integration acceptance (counter preserved, updated_at advances, reviewed_since_placement flips false).
chore: remove Meaningfy attributions and contact references (TEDSWS-528)
…(TEDSWS-524-2)
Milestone 1 of TEDSWS-524-2. Replaces the derived `reviewed_since_placement`
boolean with a stored field on the decision row, joining the existing
`previous_review_count` counter. The cross-collection `$lookup user_actions`
and `ReviewStateReader` are removed from the curation list hot path; the
review predicate now lives in one place (the writer) instead of three.
Lifecycle:
- Integrator resets `reviewed_since_placement=false` on insert and on every
material placement advance, in the same atomic `$set` as `updated_at`.
- `record_review(decision_id, action_created_at)` replaces
`increment_review_count`. It uses an aggregation-update pipeline to
increment the counter unconditionally and conditionally set the flag to
true via `$cond` against the stored placement boundary, so delayed or
out-of-order curator actions never regress the flag.
Reads:
- `find_review_metadata` returns both primitives in one projection,
replacing the separate counter fetch and per-page `ReviewStateReader`
`$or`-of-N read.
- `find_with_filters` applies `reviewed_since_placement` as a plain
stored-field `$match`. `CursorPage.count` is exact for every filter
combination (resolves A3).
Resolves from `.claude/memory/epics/TEDSWS-524-1-specs.md`:
- A1 read side, A2, A3, A4, A5, B suite, S-1/S-2/S-3/S-5, F-1/F-2/F-3/F-4.
- C1 (cluster-size cursor placement) deferred to milestone 2.
Includes:
- new compound index `decisions_reviewed_since_placement_id`
- `seed_db.py` aligned to mirror the production lifecycle (no separate
backfill script)
- FerretDB aggregation-update pipeline smoke test
- lifecycle + paginated-across-pages integration tests (the regression
net previously missing)
- BDD scenarios for the four UI review states
- OpenAPI schema regenerated
…elds (TEDSWS-524-2)
Milestone 2 of TEDSWS-524-2 (C1 from .claude/memory/epics/TEDSWS-524-1-specs.md).
The keyset cursor predicate on the derived `cluster_size` field was
previously merged into the stage-1 `$match` of the aggregation pipeline,
where the field does not yet exist. Page 2+ matched nothing and pagination
terminated prematurely; the existing integration test only exercised page 1
so CI never caught it.
Fix: extract `_apply_cursor_condition` to decide cursor placement, and
forward the cursor predicate to `_fetch_with_cluster_size_sort` as a
separate kwarg. The aggregation now inserts the cursor `$match` between
`$project` (which removes `_cluster_meta`) and `$sort`, where
`cluster_size` is materialised. Stored-field filters (entity_type,
confidence, ever_reviewed, reviewed_since_placement, mention_identifiers)
remain in stage 1 — still indexable.
`cluster_size` stays a derived projection; no denormalisation onto the
decision row. Trade-off: cluster-size ordering still costs one `$lookup`
on PK per page, accepted at the 3k/day volume.
Resolves: C1 and (by extension) C2 from TEDSWS-524-1-specs.md.
Coverage:
- unit: cursor-not-in-stage-1 when cursor=None; cursor-after-\$addFields
when cursor present; stored-field filters stay in stage 1.
- integration: cluster_size ASC and DESC paginated-across-pages tests
(the regression net previously missing) + non-under-fill invariant.
The previous _check_not_already_curated guard issued a count_documents
read followed by a separate save, leaving a TOCTOU window in which two
concurrent submissions could both pass and both record. Empirically
reproduced on 2026-06-07: two ACCEPT_TOP rows 1 ms apart, counter 2,
duplicate ERE publishes.
Fix:
- record_review keeps its name but now returns bool and gates the
update_one filter on reviewed_since_placement != True. First concurrent
caller wins (modified_count == 1 -> True); others see modified_count == 0
-> False. Atomic at the single-document level.
- user_action_service runs save -> record_review (claim) -> compensate.
On False, the just-saved user_action is deleted via delete_by_id and
AlreadyCuratedError is raised (HTTP 409 unchanged).
- _check_not_already_curated removed; replaced by _record_or_compensate.
- has_current_action kept as a repository read-only utility with an
updated docstring (no longer used for write-path idempotency).
Boundary discipline:
- record_review signals race-lost via bool, not via a curation exception.
No new cross-module import from resolution_decision_store to curation.
- delete_by_id added on UserActionCurationRepository only; the generic
AsyncWriteRepository base stays save-only.
- make check-architecture: 10/10 contracts kept.
Known follow-up (deferred): if delete_by_id itself fails on the
compensation path, the original exception propagates as HTTP 500 instead
of AlreadyCuratedError. Rare-on-rare; will be addressed in a follow-up.
Coverage:
- unit: filter shape, True/False return on modified_count, save -> claim
order, compensation called when claim fails, no compensation when it
succeeds (across all three record_* methods).
- integration (FerretDB): 8-way concurrent record_review yields exactly
one winner; counter ends at 1, flag at True. Plus sequential second-
claim returns False, missing-decision returns False.
Add unit test suite for the three operational scripts in src/scripts/: backfill_cluster_sizes, backfill_previous_review_count, and verify_cluster_sizes. Includes one intentionally failing TDD-driver test (test_run_backfill_deletes_stale_entries) that will pass once Task 4 adds stale-entry deletion to backfill_cluster_sizes.
…LL.md and Makefile
…ir loop and upgrade callout
…y repair message
Add _delete_stale_entries to backfill_cluster_sizes.py, called at the end of
_run_backfill (both live and dry-run paths) after upserts complete. Uses
{"_id": {"$nin": sorted(live_cluster_ids)}} to delete cluster_sizes documents
whose cluster no longer exists in decisions. The existing early return for empty
decisions is preserved so no deletion occurs when there are no clusters at all.
Update verify_cluster_sizes.py repair hint to reflect that backfill now also
removes stale entries. Remove xfail marker from test_run_backfill_deletes_stale_entries
and correct its filter assertion to match MongoDB $nin semantics.
…-state feature/TEDSWS-524-2-review-state
…-fixes fix(scripts): operational script tests, docs, stale-entry repair, batch-size clarification
TEDSWS-524-1: reject exclusions (TEDSWS-530), cluster-size delete-on-zero, review-state read-port
…nt and cluster_sizes Add BDD feature and step definitions covering the full reviewed_since_placement lifecycle (fresh placement, curator action, ERE re-placement). Fix scenario isolation by including cluster_sizes in the MongoDB cleanup list. Align injected decision documents with the two new materialised fields. Correct the bulk-reevaluation decision fixture to exclude current_placement from candidates per domain invariant.
…ince-placement fix(scripts): backfill also sets reviewed_since_placement
Aligns reject_decision with the intended invariant: when a curator rejects a decision, ERE should re-evaluate excluding both the current placement and all candidate clusters. Updates the unit and e2e test assertions accordingly. Resolves merge conflict in test_user_reevaluation.py (all_refs fixture variable name), restoring the corrected _make_decision helper.
… reject-exclusion helper
test(e2e): extend e2e contract tests for curation PRs 120/121/122
… scenarios Extends resolution_and_curation.http with two new full-cycle scenarios: S6 covers the accept curation action with confirming ERE re-placement; S7 covers the reviewed_since_placement filter lifecycle (set on action, reset on re-placement). All new cases marked as passing in the testing report. Also rewords the notification subscriber worker debug log to accurately describe both the cross-instance case and unsolicited re-evaluation outcomes.
test(manual): add S6 accept and S7 reviewed_since_placement lifecycle scenarios
- Add --no-cache rebuild tips (make rebuild-clean / make infra-rebuild-clean) in the ERS Getting Started section and INSTALL.md steps for ERS and ERE - Document that entity schema changes require clearing the database volume (make down-volumes) to avoid schema mismatch errors on restart - Fix API_BACKEND_URL default: curation-api:8000 (no http:// prefix); Nginx prepends the protocol internally — the http:// prefix caused API errors - Add host-machine override note for REDIS_HOST in src/infra/.env.example: export REDIS_HOST=localhost before running black-box test targets - Update CHANGELOG.md version tag to 1.1.0-rc.2
docs: practical notes on setup and configuration
Release 1.1.0-rc.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
reviewed_since_placementmetadata, cluster-size ordering, and review counter in decision summary responsesreviewed_since_placementbackfill_cluster_sizes,verify_cluster_sizes,backfill_previous_review_count