Robust Context merge: rename, scope, conflict policy

[crowlogic]

Problem

Context merging is currently fragile and has caused recurring bugs (at least 10 instances) when one Context's bindings need to be visible in another Context's compilation/instantiation. The latest manifestation is #1021: the OPS Wheeler recurrence had to thread the Riccati muntz.context all the way down through three constructor layers because Context.registerFunction("m", m) discards the source Expression on the mapping, and any inner self-instance allocated lazily by the compiled class has no way to resolve m from the OPS-local context.

Context.mergeFrom(Context) exists but only handles the easy case: same name → same instance is coalesced; same name → different instance throws CompilerException. There is no rename, no scope qualification, no conflict-resolution strategy. Callers are forced to either (a) thread one context through every constructor (current OPS workaround), (b) accept the failure mode, or (c) write ad-hoc copy loops at every site.

Root Cause: Context Has No Package Identity

The fundamental problem is that Context has no notion of identity as a mathematical object — it is a flat bag of name→binding mappings with no scope hierarchy and no way to express "these bindings form a coherent unit." Merge is fragile because there is no first-class concept of a context package that defines the boundary of what merges atomically. Until Context has that self-knowledge, mergeFrom is just string-matching on a flat map.

This must be solved before the merge semantics in the sections below, and independently of any disk-caching of compiled classes (that is a downstream consequence, not a prerequisite).

Prerequisite Fix: Context.packageName + Expression.internalName()

Step 1 — Add packageName to Context

public String packageName = null;

public Context(String packageName) {
  this();
  this.packageName = packageName;
}

Context callers that represent named mathematical objects (e.g. Jacobi polynomials, Riccati solvers) pass the package at construction:

context = new Context("arb.jacobi");

Step 2 — Add internalName() to Expression

Expression.className stays as the short name ("P", "a", etc.) — that is what the functions map, FunctionMapping.functionName, and all cross-expression symbolic references use. A single derived accessor computes the bytecode-level identity:

public String internalName() {
  return (context != null && context.packageName != null)
    ? context.packageName.replace('.', '/') + "/" + className
    : className;
}

Step 3 — All ASM emit sites use internalName() instead of className

Every ClassWriter.visit(...), visitTypeInsn(NEW, ...), visitFieldInsn(..., owner, ...), visitMethodInsn(..., owner, ...), and every field descriptor of the form "L" + className + ";" in Expression.generate() must use internalName() instead of bare className. Since Expression owns the ClassWriter, all these sites are local to it.

Step 4 — registerBytecodes is called with internalName()

Wherever Expression.generate() calls classLoader.registerBytecodes(className, bytecodes), change to classLoader.registerBytecodes(internalName(), bytecodes). Now pendingBytecodes is keyed by e.g. "arb/jacobi/P".

Step 5 — Normalize the key in ExpressionClassLoader.findClass

The JVM calls findClass("arb.jacobi.P") (dots). pendingBytecodes is keyed with slashes. One normalization line fixes it:

byte[] bytecodes = pendingBytecodes.remove(name.replace('.', '/'));

Result

className remains the short symbolic name used throughout expression logic. internalName() is the bytecode-level identity. Two contexts with different packageName values can never have a genuine class identity clash — arb.jacobi.P and arb.riccati.P are different classes. This gives the merge semantics below a coherent foundation: merging two contexts means merging two named mathematical objects, not colliding flat symbol tables.

---

What is needed (merge semantics, now unblocked)

A first-class Context merge feature that handles every realistic case:

1. Identical bindings — same name, same instance → silent coalesce (current behaviour, keep).
2. Compatible bindings — same name, different but equivalent instances (same class, same expression source) → coalesce with a configurable policy: prefer-this | prefer-other | error.
3. Name clashes with rename — same name, genuinely different bindings → automatically rename the incoming binding (e.g. m → m_2) and rewrite every dependent Expression / FunctionMapping that referenced the old name in the incoming context. The rename must propagate through:
   • Expression.referencedFunctions keys and node graphs
   • Expression.expressionString / source representation
   • Every downstream FunctionMapping instance whose compiled class field of that name was emitted
   • Already-instantiated functions: their reflective field for the renamed binding has to be re-bound, or a fresh instance allocated and wired in
4. Unrenameable clashes — when rename is impossible (e.g. the incoming binding's name appears in a hand-written Java field, or its rewrite would collide with an already-bound name in this context that is itself unrenameable) → throw a structured exception that names both sides and explains why rename failed.
5. Variable AND function bindings — both namespaces must be handled. A variable named p and a function named p can coexist (different field descriptors), so the merge must operate per-namespace.
6. Transitive dependencies — when an incoming FunctionMapping's expression references functions that are themselves in the incoming context, those must be merged together as a unit (topological merge), not one at a time, so partially-merged contexts never exist.

API sketch

public enum ConflictPolicy { PREFER_THIS, PREFER_OTHER, RENAME_INCOMING, ERROR }

public MergeReport mergeFrom(Context other, ConflictPolicy policy);

public record MergeReport(
    Set<String> coalesced,         // names that matched identically
    Map<String, String> renamed,   // old name → new name in `this`
    Set<String> imported,          // names new to `this`
    List<MergeFailure> failures    // populated only with policy=ERROR or unrenameable
) {}

Acceptance criteria

• All existing mergeFrom(Context) callers continue to work with the default policy.
• A new test class ContextMergeTest covers each of the five cases above.
• The OPS #1021 workaround (threading muntz.context through three constructor layers) can be replaced by opsContext.mergeFrom(muntz.context, RENAME_INCOMING) at the construction site, and the OPS test passes without the constructor threading.
• The variable-namespace and function-namespace fixtures both have rename coverage.
• Documentation: a Context.md describing the merge model, with a worked example covering each policy.

References

• arb.expressions.Context.mergeFrom(Context) — current implementation
• arb.expressions.Context.registerFunctionMapping(...) — see comment about replace=true semantics (Context.java:485-498) for the precedent that already partially handles overwrite
• arb.functions.polynomials.orthogonal.complex.OrthogonalPolynomialMomentFunctionalSequence — current #1021 workaround that this issue would obsolete
• arb.functions.complex.RiccatiMuntzPadeFunctional.partialDerivativeWithRespectToV — another caller that depends on careful context lifetime management
• arb.expressions.ExpressionClassLoader — pendingBytecodes and findClass are the load-side of the package identity fix

[crowlogic]

Mitigations independent of the full merge feature

Most "context drift" bugs (today's NoSuchFieldError cascade is the canonical example) come from three patterns that can be locked down cheaply:

1. Single-context ownership invariant

Every Function instance is constructed against exactly one Context. Re-registering an existing instance into a second Context (registerFunction("m", m) where m.getExpression().context != this) is the root cause of "the inner self-instance can't resolve m on lazy initialize" (which today required threading muntz.context through three constructor layers as a workaround).

Enforcement: Context.registerFunction asserts the incoming function's getExpression().context is either null or == this. Violations either fail fast or trigger the future merge feature explicitly.

2. Context freeze barrier

After the construction phase, every consumer expects the variable set to be stable. Today there is no marker for "construction done" — the OPS-on-muntz bug happened because m and a were compiled when context held {p, q, r, μ}; then OPS registered p0, p1 into the same context and later compiles got the larger view, but m's frozen declaredVariables was the smaller set, so sibling propagation faulted.

API: Context.freeze() makes the variable and function maps immutable. registerVariable/registerFunction past freeze throw a structured ContextFrozenException with the freeze callsite. Tests opt out via Context.unfreeze() when intentionally exercising late-registration paths.

3. Context-fingerprint in compiled-class metadata

Today, the only way to detect that a compiled class was built against an outdated context view is to discover that a field doesn't exist at runtime (NoSuchFieldError). The fingerprint approach: at codegen time, hash context.variables.keySet() (and similarly for functions) and emit the hash as a static final field on the generated class. At first execution, the class's initialize() asserts the live context's current hash matches; on mismatch, fail with both fingerprints visible so the caller sees exactly which variables drifted.

This is a debug-mode assert (zero-cost in release builds — same condition flag as arb4j.verifyBytecode).

4. Copy-on-write internal maps

Context.functions and Context.variables are mutable HashMaps. Generated bytecode that captured context.variableEntryStream().toList() at compile time can be invalidated by later mutation. Switching to PersistentMap semantics (or simply snapshotting the maps once at freeze()) eliminates the entire class of "I saw this set during codegen but it has since changed" bugs without changing the API.

Acceptance: any one of (1), (2), (3) shipped would have caught today's bug at construction time instead of at first-evaluate time, with a clear error message pointing at the offending registration site.

[crowlogic]

Merge feature partial landing in becd8ba.

Shipped

• ConflictPolicy { PREFER_THIS, PREFER_OTHER, RENAME_INCOMING, ERROR }
• Context.MergeReport(coalesced, renamed, imported, failures)
• New MergeReport mergeFrom(Context, ConflictPolicy) overload
• Existing mergeFrom(Context) now delegates to the new overload with ERROR (legacy semantics preserved)
• Both variable and function namespaces handled per-policy
• New ContextMergeTest covering null no-op, import, coalesce, ERROR/PREFER_THIS/PREFER_OTHER on variable clash, ERROR on function clash, RENAME_INCOMING surfacing, and legacy single-arg delegation

Full mvn test: 822/822 (was 813 + 9 new merge tests).

Deferred

• RENAME_INCOMING is wired into the API surface but throws UnsupportedOperationException with a message that points back here. Implementing it cleanly requires rewriting every Expression in the incoming Context that references the renamed name — through referencedFunctions keys, expressionString / source representation, and the AST node graph — and re-binding the field on any already-instantiated peer classes. That is the bulk of the original issue and a separate session's worth of work.
• Transitive / topological multi-binding merge (the MergeReport.renamed slot and MergeFailure slot are present in the record but unused by the current code paths).

Keeping #1024 open for the RENAME_INCOMING + propagation follow-up.

[crowlogic]

Followup 16b7f6e: dropped MergeReport and the Renamed / MergeFailure records.

Returned reports get ignored. Anything caller code needs to react to is now an exception — CompilerException for ERROR clashes, UnsupportedOperationException for RENAME_INCOMING. Successful merges return silently.

Also pinning the rename design: when RENAME_INCOMING is implemented, it MUST go through the AST via the existing Node/Expression rename infrastructure. No string manipulation of expressionString under any circumstances — every reference rewrite has to be parsed and rebuilt structurally.