Add incremental sparse RREF C-API (libsparse_rref_inc)

[thibautbar]

Summary

Adds a stateful, row-by-row sparse RREF API alongside the existing one-shot
sparse_mat_rref path, exposed as a separate translation unit
sprref_incremental.{cpp,h} that builds into a shared library
libsparse_rref_inc.{so,dylib}.

The motivation comes from project-numina/project-feynman:
SparseRREFBatchSolver there used to drive the sparserref CLI binary for
end-of-run RREF. That works great when is_solved is queried only once at
the end (check_at_end=True), but for flows that need per-step is_solved
(early termination, FIRE6-style dynamic seed expansion, intermediate logs,
pivot tracking — check_at_end=False) re-running the batch binary on every
step is O(N²). This branch lets project-feynman keep a persistent in-memory
basis that's updated row-by-row.

The legacy CLI (main.cpp) is unchanged. This is purely additive.

Public C API

sprref_inc_t* sprref_inc_init(uint64_t field_order, int n_threads);
void          sprref_inc_free(sprref_inc_t*);

void sprref_inc_resize_nvars(sprref_inc_t*, uint32_t);
void sprref_inc_set_masters(sprref_inc_t*, const uint32_t* cols, size_t n);
void sprref_inc_set_pivot_order(sprref_inc_t*, const uint32_t* cols, const uint64_t* keys, size_t n);
void sprref_inc_add_pivot_keys(sprref_inc_t*, const uint32_t* cols, const uint64_t* keys, size_t n);
void sprref_inc_add_pivot_keys2(sprref_inc_t*, const uint32_t* cols, const uint64_t* k0, const uint64_t* k1, size_t n);

/* Single-call insert (uses the C-side pivot key map). */
int  sprref_inc_insert(sprref_inc_t*, const uint32_t* cols, const uint64_t* vals,
                       size_t nnz, uint64_t rhs, uint32_t preferred_pivot);

/* Two-phase insert: forward elim in C, pivot choice in caller, then commit.
   Designed for callers whose natural pivot ordering is too rich to encode
   in fixed-width C-side keys (e.g. Python tuple Laporta keys). */
int  sprref_inc_insert_forward(sprref_inc_t*, const uint32_t* cols, const uint64_t* vals,
                               size_t nnz, uint64_t rhs,
                               uint32_t** out_candidate_cols, size_t* out_n);
void sprref_inc_commit_pivot(sprref_inc_t*, uint32_t pivot_col);
void sprref_inc_abort_pending(sprref_inc_t*);
void sprref_inc_buffer_free_u32(uint32_t* ptr);

/* Query: returns NOT_PIVOT / HAS_FREE / SOLVED + the stored row's expression. */
int  sprref_inc_is_solved(sprref_inc_t*, uint32_t var_idx,
                          const uint32_t* acceptable_free, size_t n_free,
                          uint32_t** out_cols, uint64_t** out_vals,
                          size_t* out_nnz, uint64_t* out_rhs);
void sprref_inc_buffer_free(uint32_t* cols, uint64_t* vals);

size_t   sprref_inc_rank(const sprref_inc_t*);
uint32_t sprref_inc_nvars(const sprref_inc_t*);
const char* sprref_inc_version(void);

State is held entirely in the opaque sprref_inc_t* handle (no globals), so
multiple handles can coexist for multiprocessing.Pool workers.

How it works

• Forward elimination loops over pivot columns present in the incoming
  row, saxpy-ing against each basis row until none remain. Modular arithmetic
  uses 128-bit intermediate to support primes up to 2^63; modular inverse via
  FLINT's n_invmod.
• Pivot choice:
  • The single-call insert uses the C-side pivot key map (smallest
    lex (key0, key1) wins, masters excluded, column index breaks ties).
  • The two-phase insert_forward returns the candidate columns and lets
    the caller (e.g. Python) pick the pivot using its native ordering.
• Eager backsub: after a new pivot is chosen and normalised, every
  existing basis row touching that column is saxpy-ed to zero out the entry.
  Result: the basis is in true RREF after every commit, making is_solved
  a near-O(1) lookup.
• Reverse index col_to_pivots: col → set<pivot_col> lets eager backsub
  find affected basis rows in O(degree) instead of scanning the full basis.

Build

build_incremental.sh produces the shared library:

bash build_incremental.sh
# Output: libsparse_rref_inc.{so,dylib} in the current directory.

Honors CXX, FLINT_PREFIX, GMP_PREFIX env vars. Defaults to:

• g++-15 on macOS (Apple Clang lacks <execution> parallel STL — same
  caveat as main.cpp)
• g++ on Linux
• FLINT_PREFIX=/opt/homebrew on macOS, auto-detect on Linux
  (/usr/include/flint → empty / system search paths; else /usr/local)

Tested on macOS arm64 (Homebrew) and intended for Ubuntu (apt
libflint-dev libgmp-dev libtbb-dev); the Linux side is exercised by
project-feynman's CI.

Performance (project-feynman play_tri2lx 7D, 5696 steps, mod 2017)

backend	wall time	step count	notes
sparse_rref_batch (CLI binary, batch)	13.75s	5696	reference; check_at_end=True
sparse_rref_batch + mode=incremental	15.55s	5696	this PR; check_at_end=False
spotlight_solver_sparse (pure Python)	20.73s	5696	per-step Python sparse RREF

→ Within ~13% of the highly-optimised batch path while supporting per-step
queries; ~25% faster than the pure-Python equivalent.

Pivot decisions match SpotlightSolverSparseGF bit-for-bit (verified by
project-feynman's BL2EM end-to-end test, which compares the resolved
target's coefficients on each master).

Commits in this branch

1. 84ec9a3 — Scaffolding (M1): API surface, stub implementation, build script
2. 5028015 — Forward elimination + insert + eager backsub (M2)
3. 119adcc — is_solved + expression extraction (M3)
4. fb100a7 — 3-way is_solved return (NOT_PIVOT / HAS_FREE / SOLVED)
5. 077f23c — add_pivot_keys for delta upserts (avoid full-replace traffic)
6. fb8160d — build_incremental.sh Linux/Ubuntu auto-detect
7. 5f501f2 — Two-phase insert_forward + commit_pivot (caller-side pivot choice)

Notes for future work

• Numerator-powers tuple in C-side ordering: the optional 2-uint64 key
  pair (add_pivot_keys2) packs scalars + denom_powers but drops
  numerator_powers. Workloads that need that field for tie-breaking should
  prefer the two-phase API. A byte-buffer key API could be added if needed.
• Multi-threading: n_threads is accepted but unused (M2 is
  single-threaded); FLINT's parallel kernels could be wired into saxpy /
  backsub later.
• Upstream PR: this is a self-PR on the fork to track the work for
  project-feynman; opening an upstream PR against munuxi/sparse_rref is
  a separate follow-up if you want this contributed back.