[upstream-sync] feat(rust-sdk): standalone Rust SDK with App/Ctx/proc-macro APIs (upstream #1919, #1973)

[github-actions]

Upstream Change Summary

Type: new-feature / api-change
Difficulty: Hard
Recommendation: Adopt

Upstream added a standalone Rust SDK (rust/sdk/) that exposes App and Ctx APIs for building incremental Rust pipelines without Python, along with proc-macro annotations (#[cocoindex::function], #[cocoindex::function(memo)], #[cocoindex::function(memo, batching)]). A follow-up PR (#1973) folded the SDK into the root Cargo workspace (resolver bumped 2 → 3), eliminated the second target/ dir, added a CI guard (dev/check_sdk_no_pyo3.sh) that enforces zero pyo3 in the SDK's transitive deps, and moved examples under examples/rust/. This is essentially upstream discovering the same design space recoco was built to occupy.

Upstream References

• PR #1919: feat(rust-sdk): add standalone SDK, examples, and benchmark cocoindex-io/cocoindex#1919 — add standalone Rust SDK, examples, and benchmark
• PR #1973: refactor(rust): unify workspace, isolate SDK from PyO3, restructure benchmark cocoindex-io/cocoindex#1973 — unify workspace, isolate SDK from PyO3, restructure benchmarks
• Upstream repo: https://github.com/cocoindex-io/cocoindex

Relevant Upstream Files / Areas

• rust/sdk/cocoindex/src/app.rs — App / AppBuilder public API
• rust/sdk/cocoindex/src/ctx.rs — Ctx processing context
• rust/sdk/cocoindex/src/memo.rs — memoization utilities
• rust/sdk/cocoindex/src/fs.rs — file walking with stable keys and fingerprints
• rust/sdk/cocoindex/src/profile.rs — profiling hooks
• rust/sdk/cocoindex-macros/src/lib.rs — proc-macro definitions
• dev/check_sdk_no_pyo3.sh — CI guard for PyO3 isolation
• examples/rust/files-transform/, examples/rust/multi-codebase-summarization/

Recoco Considerations

• Direct API overlap: Recoco exposes FlowBuilder, App equivalents, and a similar public crate API. The upstream SDK's App / Ctx / #[cocoindex::function] design should be compared carefully against recoco's current API surface to identify patterns worth adopting or areas where recoco is ahead.
• Proc-macro layer (#[cocoindex::function(memo)], #[cocoindex::function(memo, batching)]): Recoco doesn't currently expose these proc-macros. Adopting this ergonomic layer would significantly improve the user-facing API for custom operations. Worth evaluating cocoindex-macros as a model for a recoco-macros crate.
• fs.rs file walker: The stable-key + fingerprint file walker is a direct recoco concern (maps to source-local-file feature area). Review for blake3 hashing opportunities and algorithm improvements.
• CI guard (check_sdk_no_pyo3.sh): Recoco should adopt an equivalent check since it has the same hard constraint. This is an easy win.
• Workspace resolver bump 2 → 3 in #1973: Recoco should evaluate bumping its own workspace resolver for consistency.
• Examples: The files-transform and multi-codebase-summarization Rust examples are useful reference implementations to compare against recoco's own examples.
• Benchmark structure (benchmarks/file_summarization/): Recoco lacks a formal benchmark suite. The upstream benchmark structure is a good reference for adding one.

Integration Notes

This is high-priority: upstream is now converging on recoco's mission. The most actionable items are (1) evaluate the App/Ctx API for ergonomic improvements to recoco's public API, (2) prototype a recoco-macros crate modeled on cocoindex-macros, and (3) adopt the check_sdk_no_pyo3.sh CI guard immediately as a trivially-easy win. The full file walker and memo implementation should be reviewed for algorithm improvements before adopting.