[upstream-sync] fix(engine): concurrent preempt race + pending_process_token refactor (upstream #1994, #1996)

[github-actions]

Upstream Change Summary

Type: bug-fix
Difficulty: Hard
Recommendation: Adopt

Two closely related Rust engine fixes that must be ported together — they address a concurrent preempt race in target-state ownership transfer and a follow-up rollback gap:

PR #1994 closes the initial race: adds pending_process_token: Option<u128> to TargetStateInfoItem, written by pre_commit when it queues a sink action and cleared by commit_in_txn. A detection sub-pass at the top of pre_commit handles tokens: live token from same process → PendingRetry with exponential backoff (5ms → 200ms, max 8 retries); dead token from a crashed prior process → force prev_may_be_missing=true on reconcile.

PR #1996 supersedes part of #1994's design: moves pending_process_token from per-item (TargetStateInfoItem) to per-component (StablePathEntryTrackingInfo), closes a rollback gap where rollback_pending_tokens clearing per-item tokens lost the dead-token signal, and uses item.is_pending() (multi-state) directly instead of the separate detection sub-pass. Also removes dead_token_paths, deduplicates tracking_info deserialization (was twice; now once), and eliminates the detection sub-pass's own-tracking branch.

Upstream References

• PR #1994: fix(engine): close concurrent preempt race in target-state ownership transfer cocoindex-io/cocoindex#1994
• PR #1996: refactor(engine): move pending_process_token to component level + close rollback gap cocoindex-io/cocoindex#1996
• Upstream repo: https://github.com/cocoindex-io/cocoindex

Relevant Upstream Files / Areas

• rust/cocoindex/src/engine/ — pre_commit, commit_in_txn, Committer, target-state reconcile
• rust/cocoindex/src/state_store/ — TargetStateInfoItem, StablePathEntryTrackingInfo
• Spec: specs/target_state_ownership_transfer/concurrent_preempt_race_fix.md

Recoco Considerations

• Affected files: crates/recoco-core/src/engine/ (pre_commit, committer), crates/recoco-core/src/state_store/
• These changes are architectural correctness fixes — they close real race conditions in concurrent execution and crash recovery scenarios that recoco inherits from the same engine design
• No Python code to exclude; no new feature gates needed
• The rollback_pending_tokens logic and the retry backoff constants (5ms → 200ms, max 8 retries) should be adopted as-is
• Related prior work: recoco issues [upstream-sync] Target state ownership transfer between components (upstream PR #1787) #133 (target state ownership transfer, upstream #1787) and [upstream-sync] fix: reset version on preempted target state ownership transfer (upstream #1797) #147 (reset version on preempted transfer, upstream #1797) — review those for integration context

Integration Notes

Port #1994 first to understand the problem fully, then apply #1996's structural cleanup which supersedes some of #1994's intermediate design. The final state after both PRs is the target. The is_pending() predicate (multi-state item shape) is the key correctness invariant — ensure recoco's state representation supports it.