Add a validated Julia statistics layer with exact counts/rationals and higher precision

[hyperpolymath]

Suggested labels: enhancement, statistics, scientific-validation.
Companion: Symbolic-engine proposal, which is blocked by this work and a separate approval gate.

User direction and scope

Add an optional Julia-owned statistical-analysis layer to MetaManifold, surfaced through Stipple/Vue. Preserve every existing feature and scientific workflow; do not silently change existing pipeline algorithms or previously saved analyses.

The user is interested in the first three forms of “exact” capability discussed, but prioritises only the first two for initial implementation:

1. Exact integer counts and rational arithmetic where mathematically appropriate.
2. Higher-precision floating-point computation for explicitly supported operations.
3. Exact statistical tests: a later, separately validated extension, not part of the initial implementation acceptance gate.

Higher precision is not exact arithmetic. Neither removes measurement error, sampling uncertainty, model misspecification or biological bias. Symbolic mathematics is explicitly out of scope for this issue.

Proposed work

Numeric integrity

• Preserve integer counts; detect overflow and use checked/wider or arbitrary-size integers where justified.
• Support exact rational transformations for selected operations, with explicit handling of zero denominators and resource limits.
• Introduce an explicit numeric policy covering supported ordinary precision and arbitrary precision, precision settings, rounding/conversion rules and provenance.
• Audit CSV, JSON, database, browser, R and external-tool boundaries for precision loss, integer range and serialization compatibility. Never silently downgrade a requested numeric mode.
• Isolate precision configuration across concurrent analyses; avoid shared mutable precision settings leaking between users or jobs.
• Clearly distinguish exact stored/computed values, numerical approximations and rounded display values.

Statistical methods

• Define a bounded initial catalogue of methods before implementation, documenting supported response types and study designs.
• Support maximum-likelihood fitting for justified models, with explicit parameter constraints, convergence, identifiability and boundary-estimate checks.
• Support appropriate parametric and nonparametric analyses. Nonparametric is a class of methods, not a universal safe distribution or an assumption-free alternative.
• Treat Poisson/negative-binomial and binomial/beta-binomial models as candidates for review, not automatically approved choices for sequencing data.
• Require appropriate handling of overdispersion, zeros, sequencing depth, compositionality, covariates, pairing, blocking and repeated measurements where relevant.
• For permutation/bootstrap methods, validate exchangeability or resampling units, record seeds and resampling settings, and report uncertainty and Monte Carlo limitations.
• Specify uncertainty estimation, diagnostics, effect sizes, multiple-testing policy and warnings for every supported method.

Safe configuration and user experience

• Attach versioned configurations to immutable source runs; create linked derived analyses rather than overwrite existing results.
• Default to descriptive summaries when the information needed for valid inference is unavailable. Do not guess missing study design.
• Never silently switch statistical methods or select a method solely by a normality test or favourable p-value.
• Failed fits, unsupported precision, invalid input and resource exhaustion must produce explicit unsuccessful states, not plausible-looking results.
• Record inputs, transformations, model, settings, seeds, software/reference versions and diagnostics for reproducibility.
• Explain assumptions and numerical limitations in accessible user-facing language.
• Keep application logic Julia-owned; no new application TypeScript. Small necessary JavaScript adapters and Bun-managed tooling remain permitted.

Required deep validation and acceptance criteria

This is a scientific capability, not merely a form and an optimiser. Define method-specific acceptance tolerances and study-design coverage before claiming support.

• Approve the bounded method catalogue and publish supported/unsupported conditions.
• Publish numeric conversion and cross-language/storage contracts, including integer/rational round trips and explicit rejection of unsupported representations.
• Test known-answer cases and independent reference implementations; shared implementation assumptions alone are not independent validation.
• Test numerical stability across precision levels and adversarial cases: huge counts, sparse/all-zero tables, tiny probabilities, near-singular fits, boundary parameters, overflow/underflow and invalid/nonfinite inputs.
• Add property/generative tests for meaningful invariants and metamorphic relationships, plus fuzzing at parsing/configuration boundaries.
• Demonstrate negative controls and mutation/sensitivity tests: intentionally wrong inputs or implementations must be detected. Record commands and failing evidence.
• Use simulation studies to assess estimator bias, uncertainty-interval coverage and relevant error rates across supported conditions. Set justified tolerances in advance.
• Validate representative biological datasets and appropriate null/alternative cases; document where model assumptions fail.
• Verify reproducibility, serialization, concurrent precision isolation, cancellation, resource limits and failure recovery.
• Test UI-to-backend-to-result workflows, accessibility, provenance display and clear unsuccessful states.
• Verify existing scientific behaviour remains unchanged when the new analysis layer is not selected.
• Benchmark time/memory and bound expensive precision/resampling operations on supported platforms.
• Run the applicable repository/owner standards gates in CI, retaining results, coverage and benchmark artifacts; justify genuine N/A entries.
• Obtain independent statistical/scientific review and record limitations and release approval.

“Fully and deeply tested” means documented coverage of the declared scope, independent review and explicit residual-risk acceptance—not a claim that exhaustive testing proves all possible behaviour correct. Critical scientific-correctness, security or data-integrity defects must be resolved before release or before unblocking symbolic work.

Follow-up: exact statistical tests

Exact statistical tests remain desired, but are deferred beyond the initial integer/rational and higher-precision work. Each future test needs its own assumptions, supported designs, computational limits, known-answer cases and validation. Do not substitute an approximate method silently when an exact method is infeasible.

Dependency gate for symbolic mathematics

The companion symbolic-engine issue must stay blocked until the statistics layer completes the above validation, evidence is reviewed, blockers are resolved, and the owner explicitly approves moving to the symbolic stage. Shipping a UI, obtaining green unit tests, or closing this issue administratively is not sufficient to open that gate.

[hyperpolymath]

Started, foundationally, in #49 — the numeric-integrity half of the layer, which the
methods half has to build on.

Delivered: exact integer counts and Rational{BigInt} proportions, a BigFloat
precision scope, and an explicit NumericPolicySpec (immutable, fingerprinted for
provenance) covering the numeric policy, rounding/conversion rules and the exact/
approximate/rounded distinction. Nothing existing changes behaviour: :ordinary
remains Float64 throughout, so saved analyses are untouched.

The refusals matter more than the arithmetic, and each is a real hazard rather than
a hypothetical: Julia answers 1//0 for a non-zero numerator, so an all-zero sample
would otherwise yield a non-finite Rational that renders as a number — that case now
returns nothing, an explicit unsuccessful state. Counts arriving as Float64 beyond
2^53 are refused, since such a value cannot prove which integer it was and may already
have been rounded by R, a CSV reader or a JavaScript client. Sums raise
CountOverflowError instead of wrapping, rational growth is bounded by a stated budget
rather than an OOM, and NaN/Inf are refused at the JSON boundary. Precision isolation
across concurrent analyses is by serialisation in a lock, because Julia's BigFloat
precision is a process global; six concurrent tasks at six precisions are asserted to
each observe their own.

Published: docs/statistics/numeric-contracts.md (the cross-language and storage
contracts, boundary by boundary: JSON, CSV, R, DuckDB, the browser — criterion 2) and
docs/statistics/method-catalogue-v1-draft.md (a bounded catalogue with
supported/unsupported conditions, drafted for your approval — criterion 1, which
this issue requires before methods are implemented).

Still open, and not claimed by #49: catalogue approval, per-method known-answer and
simulation validation, adversarial and property coverage at method level, UI-to-backend
workflow tests, and independent statistical review. The negative and metamorphic
controls do exist at the numeric layer (144 tests: overflowing sums, all-zero tables,
denominator exhaustion, non-finite input, 50 random tables whose proportions must sum
to exactly 1, and controls asserting the lossy alternatives really are lossy).

The symbolic gate in #2 stays blocked.

[hyperpolymath]

Second numeric-integrity step, in #52: the boundary audit this issue asks for,
done by crossing each boundary with a real partner rather than by restating the
contract.

• Browser — in a real JavaScript engine, 9007199254740993 as a JSON number
  arrives as 9007199254740992 (one short, unmarked); as the encoded string it
  crosses byte for byte.
• R — R has no integer type wider than 32 bits (as.integer(2^31) is NA) and
  every number is a double; neither format() nor as.character() renders that count
  back to its digits. So a count that has been through R is unproven, and the
  contract's route (send and read it as a string) is asserted by round trip.
• Database — BIGINT carries the count exactly, DOUBLE silently changes it; past
  BIGINT a BIGINT column refuses rather than rounds; HUGEINT is exact to
  2^127; beyond that, the string form.
• CSV — text in, text out: "9.007199254740992e15" and "9007199254740993.0"
  are refused by name.

parse_exact_count / parse_exact_rational now close the round trip as an executable
inverse of to_storage, because a value that has passed through a float cannot be
inspected for the damage — the string form is the only representation allowed to claim
to be a count. 45 boundary assertions alongside the existing 144.

Still outstanding on #1, unchanged and not claimed: approval of the bounded method
catalogue (docs/statistics/method-catalogue-v1-draft.md — your call), then per-method
known-answer and simulation validation, adversarial and property coverage at method
level, UI-to-backend workflow tests, benchmarks for the expensive precision paths, and
independent statistical review. The symbolic gate (#2) stays blocked.

[hyperpolymath]

Verification note — the boundary tests are not dead code in CI. Worth stating
because the CI log does not name them: Julia's test summary collapses passing nested
testsets into the parent row, so test_numeric_boundaries.jl contributes to
MetabarcodingPipeline | 1993 without ever being printed. I checked that this is
collapsing rather than absence, arithmetically:

• run on 2f59384 (before this work): 1938 passing
• run on 1a71048 (after ci(tools): retry pinned downloads and name a TLS failure as one #51 + test(analysis): measure the numeric boundaries instead of assuming them #52): 1993 passing
• the delta is +55 = 45 boundary assertions (this file) + 10 new pin-guard
  assertions (ci(tools): retry pinned downloads and name a TLS failure as one #51). Accounted for exactly, to the assertion.

Both summary rows show Pass / Total with no Broken column, and 1993 == 1993.
That matters because the boundary file skips loudly by name when a partner is missing
(@test_skip), so a missing node, Rscript or DuckDB would have appeared as a Broken
count. It did not: in the CI runner all three partners were present and every boundary
was crossed for real.

The alternative check — push a deliberately failing assertion and watch CI go red —
turned out to be unnecessary once the counts reconciled, so no mutation branch was left
behind.

State: #49 e1856af, #51 0ef5eee, #52 387fff5, #53 1a71048. CI on the current
main tip is green (Julia 1993/1993, hygiene green, gate triage skipped by design).

[hyperpolymath]

The bounded method catalogue is approved — recorded in
docs/statistics/method-catalogue-v1.md (PR #55), approved without amendment by the
repository owner on 2026-09-22.

That satisfies this issue's requirement that the catalogue and its supported/unsupported
conditions be agreed before methods are implemented. What the approval covers, in the
document's own words:

• v1 scope: (1) descriptive summaries at exact precision, (2) parametric fits with
  explicit constraint and convergence reporting, (3) nonparametric permutation/bootstrap
  with stated exchangeability and seeds.
• The per-method publication list as the acceptance gate for each method.
• The reading that a descriptive summary is the correct default when the information
  needed for valid inference is missing.

What it does not cover: exact statistical tests (item 4, tracked in #3), any method
outside the list, or the symbolic work (#2), which stays blocked. Approval is of scope and
gate, not of method behaviour — each method still publishes its conditions, evidence and
negative controls before it is implemented, and nothing is implemented merely by
appearing in the list.

Next increment, now unblocked: item 1 — exact descriptive summaries (counts and
proportions as exact integers/rationals, no inference claimed), building on
numeric_policy.jl and the boundary audit already on main. It will arrive with that
method's published conditions, known-answer and independent-reference evidence, and its
negative controls.

[hyperpolymath]

Catalogue item 1 has shipped — exact descriptive summaries.

• PR feat(analysis): exact descriptive summaries (catalogue item 1) #57 merged as 5441bd99 (main tip); PR docs(statistics): approve the v1 method catalogue #55, the catalogue approval, as ab68a6fa.
• src/analysis/exact_summaries.jl + 97 assertions in test/unit/test_exact_summaries.jl,
  held to the conditions published in docs/statistics/method-conditions/exact-descriptive-summaries.md
  (merged in docs(statistics): publish the conditions for exact descriptive summaries #56, before the implementation existed).

Counts are exact integers, proportions are rationals, a zero-total sample carries
nothing rather than 0 (the distinction a single number cannot express — the case
Execution.jl writes 0.0 for today, deliberately left unchanged), floats above 2^53 are
refused rather than trusted, and group summaries aggregate counts rather than averaging
proportions.

Three things worth recording, because they are the expensive kind of progress:

1. The conditions were not amended to fit the code. Holding the implementation to
   text written beforehand exposed two defects in the numeric layer underneath: a rounding
   labelled "6dp" that printed ~80 digits (round on a BigFloat keeps its significand),
   and exact rationals rendered as Julia's 2//3 in text a person reads. Both fixed. The
   rounding now happens in integer arithmetic — correct past Float64, and dependent on
   nothing outside Base.
2. CI caught a bug the local evidence could not. import Printf passed every local
   harness, because a harness mounts source into an ad-hoc module and cannot enforce
   package dependency rules; precompilation failed with
   Package MetaManifold does not have Printf in its dependencies. Fixed by removing the
   dependency rather than declaring it.
3. The lint check written to catch that class of bug had two false negatives — a
   "stdlibs need no [deps] entry" exemption list that is factually wrong (only Base,
   Core and Main are bound without a declaration; verified by experiment), and it
   examined only the first name in a comma list, so using JSON3, YAML never checked
   YAML. Rewritten to read the AST, with a self-test that replays the exact input which
   defeated it and the inverses that must not be reported — mutation-checked, because a
   gate that cannot fail is decoration.

Next, under the approved catalogue order: item 2, parametric fits with explicit
constraint and convergence reporting. Per the catalogue's own rule, the conditions
document comes first and the implementation is held to it.