Skip to content

Raise sublibrary SciMLBase compat floor to match parent Corleone (fix downgrade CI)#86

Open
ChrisRackauckas-Claude wants to merge 6 commits into
SciML:mainfrom
ChrisRackauckas-Claude:fix-sublib-scimlbase-downgrade-floor
Open

Raise sublibrary SciMLBase compat floor to match parent Corleone (fix downgrade CI)#86
ChrisRackauckas-Claude wants to merge 6 commits into
SciML:mainfrom
ChrisRackauckas-Claude:fix-sublib-scimlbase-downgrade-floor

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Problem

The downgrade-sublibraries CI on main is failing for both lib/CorleoneOED and lib/OptimalControlBenchmarks with:

Unsatisfiable requirements detected for package SciMLBase [0bca4576]:
 ├─restricted to versions 2 by <sublibrary>, leaving only versions: 2.0.0-3.21.0
 ├─restricted to versions 2.141.0-3 by Corleone, leaving only versions: 2.141.0-3.21.0
 └─restricted to versions 2.130.0 by an explicit requirement — no versions left

Root cause

Both sublibraries depend on the in-tree Corleone, which declares SciMLBase = "2.141.0, 3", but the sublibraries themselves declared the looser SciMLBase = "2, 3". The centralized sublibrary-downgrade.yml workflow pins every direct dependency to its declared floor, so SciMLBase got pinned to its lowest registered >= 2 version (2.130.0) while the developed in-tree Corleone still required >= 2.141.0 — an unsatisfiable resolve. The divergence has existed since commit fa6a2b9a2 (2026-06-07), where the root's SciMLBase floor was bumped to 2.141.0 but the sublibraries were left behind.

Fix

Raise both sublibraries' SciMLBase compat floor to 2.141.0, 3 to match the parent Corleone package. This makes the declared floor honest about what the dependency tree actually supports. No tests are skipped, no tolerances loosened.

Local verification

Reproduced the downgrade resolution on Julia 1.10.11 (the version CI uses), developing the in-tree Corleone and pinning SciMLBase to the sublibrary's downgrade floor:

  • Old floor (=2.130.0): RESOLVE_FAIL — reproduces the exact CI Unsatisfiable requirements detected for package SciMLBase error, for both sublibraries.
  • New floor (=2.141.0): RESOLVE_OK SciMLBase=>2.141.0 — resolves cleanly, for both sublibraries.

Please ignore until reviewed by @ChrisRackauckas.


Follow-up (Reexport + SymbolicIndexingInterface floors)

After the SciMLBase floor bump landed, the downgrade-sublibraries resolver surfaced the next two diverging floors. In-tree Corleone declares Reexport = "1.2.2" and SymbolicIndexingInterface = "0.3.43", but both sublibraries still declared the looser Reexport = "1.2" and SymbolicIndexingInterface = "0.3". The downgrade workflow pins each direct dep to its declared floor (Reexport=1.2.0, SymbolicIndexingInterface=0.3.39), which the developed in-tree Corleone rejects — the same unsatisfiable-resolve class as the SciMLBase issue. Raised both sublibrary floors to match the parent.

Local verification (Julia 1.10.11, develop in-tree Corleone, pin the two deps to the sublibrary downgrade floor):

  • Old floors (Reexport=1.2.0, SII=0.3.39): RESOLVE_FAIL — reproduces the CI Unsatisfiable requirements detected for package Reexport error, for both sublibraries.
  • New floors (Reexport=1.2.2, SII=0.3.43): RESOLVE_OK, for both sublibraries.

Note: the [QA] sublibrary checks are red on a separate, pre-existing set of Aqua failures (piracy on shooting_constraints, method ambiguity, unbound type params, undefined exports, stale StableRNGs dep, and a missing Pkg compat entry for the test extras). These reproduce on main, are unrelated to this PR's compat-floor change, and are out of scope here.


QA (Aqua) fixes (now in scope)

The lib/CorleoneOED [QA] Aqua failures that the original PR body called "out of scope" are now fixed at their real sources (commit 4daaea5). Aqua.test_all(CorleoneOED) failed 6 of 11 checks; each fix:

  • Type piracyshooting_constraints/shooting_constraints! for AbstractVector{<:Trajectory} dispatched only on Corleone-owned types (both the function and Trajectory belong to Corleone). Moved both vector methods into the parent Corleone package (src/multiple_shooting.jl), where they are legitimate extensions, not piracy. The Optimization extension already calls them via the Corleone.-qualified names, so call sites are unchanged.
  • Method ambiguityget_sampling_sums/get_sampling_sums! were ambiguous for OEDLayer{<:Any, false, <:Any, <:MultipleShootingLayer} with a NamedTuple state. Added the intersection methods returning the empty SAMPLED == false result.
  • Unbound type parameters — four update_fim methods declared where variables (DISCRETE in oed.jl, SPLIT in multiexperiments.jl) absent from the signature. Removed them.
  • Undefined exports — Symbolics (through v7) still exports Variable despite the binding being removed, so @reexport using Symbolics propagated an undefined export. Replaced it with using Symbolics + re-export of only the Symbolics exports that still resolve (this is an upstream Symbolics defect worked around locally; the usable API such as Num/@variables is still re-exported).
  • Stale dependency — moved StableRNGs from [deps] to [extras]/test target (test-only use).
  • Missing extras compat — added Pkg = "1".

Local verification (showed output): Aqua.test_all(CorleoneOED) passes 11/11 on Julia 1.10 and 1.12; CorleoneOED Core tests pass on 1.12 (1D Example 36/36, Lotka Volterra 125/125, Lotka Volterra SVD 17/17, including the @inferred get_sampling_sums checks and the multi-experiment Ipopt optimization that exercises the relocated shooting_constraints); parent Corleone Aqua still passes 11/11. No tests skipped, no tolerances loosened.


Follow-up (OptimalControlBenchmarks downgrade: ModelingToolkit ispublic on Julia 1.10)

After the SciMLBase / Reexport / SII floor bumps landed, the lib/CorleoneOED downgrade went green but lib/OptimalControlBenchmarks surfaced the next downgrade failure: ModelingToolkit fails to precompile on Julia 1.10 (the downgrade CI version) with UndefVarError: ispublic not defined (from MTK's @import_mtkbase macro), which also breaks the in-tree CorleoneModelingToolkitExtension.

Root cause — two distinct upstream-floor problems:

  1. ModelingToolkitBase was declared as a direct dependency of OptimalControlBenchmarks but is never imported or used in its source or tests (only the bare [deps] entry). As a direct dep with no compat entry, the downgrade workflow pinned it to its floor 1.0.0. MTK's @import_mtkbase references ModelingToolkitBase.ispublic/.mergedefaults, neither of which exists in MTKBase 1.0.0. Removed the spurious direct dep so MTKBase floats (still pulled transitively via MTK) and resolves to the latest compatible version (1.41).
  2. With MTKBase fixed, the downgrade still pinned ModelingToolkit to its lowest 11 version (11.0.0). MTK 11.0.0/11.1.0 call Base.ispublic unconditionally — it does not exist on Julia 1.10 (added in 1.11; later MTK versions guard it). MTK 11.2.0 hits a separate generate_ODENLStepData must be explicitly imported precompile error on 1.10. Raised the ModelingToolkit compat floor to 11.7.2, the earliest 11.x verified to precompile and load on Julia 1.10.

Local verification (Julia 1.10.11, develop in-tree Corleone, simulate the sublibrary downgrade resolve):

  • Before: UndefVarError: ispublic not defined on precompile (MTK 11.0.0 + MTKBase 1.0.0) — reproduces the CI failure.
  • After: RESOLVE_OK; resolved ModelingToolkit=11.7.2, ModelingToolkitBase=1.41.0; import OptimalControlBenchmarks precompiles and loads cleanly.

No tests skipped, no tolerances loosened.

Note: the non-downgrade sublibraries / ... matrix jobs on this PR are red on a separate centralized-CI infra failure — the Develop in-repo [sources] step fails with SystemError: opening file ".sciml-dotgithub/scripts/develop_sources.jl": No such file or directory. This is the shared SciML/.github monorepo-sublibrary helper, not anything in this repo; it was addressed upstream by SciML/.github#99 and a re-run should clear it. These jobs are skipped on main, so they are out of scope for the main reds this PR targets.

Status after this push (verified in CI on HEAD)

  • downgrade-sublibraries / test (lib/CorleoneOED) -> success
  • downgrade-sublibraries / test (lib/OptimalControlBenchmarks) -> success (the two checks that were red on main)
  • sublibraries / lib/{CorleoneOED,OptimalControlBenchmarks} / Core (Julia 1/lts/pre) -> success
  • sublibraries / lib/CorleoneOED [QA] -> success
  • All root tests/..., Downgrade, Runic, Spell Check -> success

Remaining red: sublibraries / lib/OptimalControlBenchmarks [QA] (Aqua). These are pre-existing and unrelated to the downgrade fix (they fail identically before this push and are skipped on main, since the sublibraries matrix is PR-only). Aqua flags, for OptimalControlBenchmarks:

  • Stale [deps]: OptimizationLBFGSB, Ipopt, UnoSolver, OrdinaryDiffEq, Runic (none imported by src/; Ipopt/UnoSolver are used only by the top-level main.jl demo script, OrdinaryDiffEqTsit5 rather than full OrdinaryDiffEq is what src/ uses).
  • Missing [compat] for BenchmarkTools, CairoMakie, IntervalSets, OptimizationLBFGSB, Runic, UnoSolver, and the Pkg extra.

These are left for a focused follow-up: cleanly resolving them means relocating the demo-only deps and choosing compat bounds, which is separable from the downgrade-CI fix this PR targets.


Follow-up (OptimalControlBenchmarks [QA] Aqua fixes — now in scope)

The sublibraries / lib/OptimalControlBenchmarks [QA] failures the section above called a "focused follow-up" are now fixed at their real sources. Aqua.test_all(OptimalControlBenchmarks) failed on Stale dependencies and (once that was fixed) on extras Compat bounds:

  • Stale dependenciesOptimizationLBFGSB, Ipopt, UnoSolver, OrdinaryDiffEq, Runic (and the unused StableRNGs) were declared in [deps] but never loaded by the package. OptimizationLBFGSB/OrdinaryDiffEq/Runic are referenced nowhere; Ipopt/UnoSolver are used only by the loose top-level main.jl demo script, which is not part of the package or its tests. StableRNGs stays available transitively via CairoMakie. Removed all of them from [deps]/[compat]. main.jl keeps its using Ipopt/using UnoSolver with a header comment noting these are demo-only solver back ends to Pkg.add before running it.
  • Compat bounds — added the missing [compat] entries: BenchmarkTools = "1", CairoMakie = "0.15", IntervalSets = "0.7" (direct deps that had none), and Pkg = "1" (Aqua's extras compat check does not exempt stdlibs).

The test target (Pkg, SafeTestsets, Test) is unchanged, so the downgrade-sublibraries Core resolve is not perturbed; the added compat floors only raise previously-uncapped lower bounds.

Local verification (Julia 1.10.11, develop in-tree Corleone + the sublibrary, showed output):

  • Aqua.test_all(OptimalControlBenchmarks) now passes every check (Method ambiguity, Unbound type parameters, Undefined exports, Project extras, Stale dependencies, Compat bounds 4/4, Piracy, Persistent tasks). Before this commit it failed on Stale dependencies, then on the extras Compat bounds.
  • Simulated the sublibrary downgrade resolve (pin every direct dep to its declared [compat] floor): RESOLVE_OK; import OptimalControlBenchmarks precompiles and loads cleanly (BenchmarkTools=1.8.0, CairoMakie=0.15.12, IntervalSets=0.7.14, ModelingToolkit=11.7.2, SciMLBase=2.141.0).

No tests skipped, no tolerances loosened.

Please ignore until reviewed by @ChrisRackauckas.

ChrisRackauckas and others added 5 commits June 15, 2026 10:11
The lib/* sublibraries depend on the in-tree Corleone, which declares
`SciMLBase = "2.141.0, 3"`, but both sublibraries declared the looser
`SciMLBase = "2, 3"`. The sublibrary-downgrade CI pins each direct dep to
its declared floor, so SciMLBase was pinned to 2.130.0 while the developed
in-tree Corleone still required >= 2.141.0, producing an unsatisfiable
resolve (`Unsatisfiable requirements detected for package SciMLBase`).

Bump both sublibraries' SciMLBase floor to 2.141.0 so their declared floor
is honest about what the dependency tree actually supports.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ch parent Corleone

After the SciMLBase floor bump, the downgrade-sublibraries resolver surfaced
the next two diverging floors: in-tree Corleone requires Reexport >= 1.2.2 and
SymbolicIndexingInterface >= 0.3.43, but both sublibraries still declared the
looser 1.2 and 0.3. The downgrade workflow pins each direct dep to its declared
floor (Reexport=1.2.0, SymbolicIndexingInterface=0.3.39), which the developed
in-tree Corleone rejects -> unsatisfiable resolve. Raise both floors to match
the parent so the declared floors are honest about what the tree supports.

Local verification on Julia 1.10.11 (develop in-tree Corleone, pin the two deps
to the sublibrary downgrade floor):
- Old floors: RESOLVE_FAIL, reproducing the CI Unsatisfiable Reexport error for
  both sublibraries.
- New floors: RESOLVE_OK for both sublibraries.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, exports, deps)

The lib/CorleoneOED [QA] group ran Aqua.test_all and failed 6 of 11 checks
(reproduced on Julia 1.10 and 1.12). Each is fixed at its real source:

- Type piracy: `shooting_constraints`/`shooting_constraints!` for
  `AbstractVector{<:Trajectory}` dispatched only on Corleone-owned types
  (the function and `Trajectory` both belong to Corleone). Moved both vector
  methods into the parent Corleone package (src/multiple_shooting.jl) next to
  the existing single-`Trajectory` methods, where they are no longer piracy.
  Call sites (CorleoneOED's Optimization extension) already use the
  `Corleone.`-qualified names, so they are unaffected.

- Method ambiguity: `get_sampling_sums`/`get_sampling_sums!` were ambiguous
  for an `OEDLayer{<:Any, false, <:Any, <:MultipleShootingLayer}` with a
  NamedTuple state (the `SAMPLED == false` method vs the MultipleShootingLayer
  NamedTuple method). Added the intersection methods; with no sampling the
  result is empty regardless of the shooting layer, matching the existing
  `SAMPLED == false` behaviour.

- Unbound type parameters: four `update_fim` methods declared `where` type
  variables that never appeared in the signature (`DISCRETE` in oed.jl, `SPLIT`
  in multiexperiments.jl). Dropped the unused parameters.

- Undefined exports: Symbolics (through v7) still lists `Variable` in its
  `export`s even though that binding was removed, so `@reexport using Symbolics`
  re-exported an undefined name. Reproduce Reexport's behaviour (`using` +
  re-export the package's exported names) while skipping names that no longer
  resolve in Symbolics. The genuinely-usable Symbolics API (`Num`, `@variables`,
  etc.) is still re-exported.

- Stale dependency: `StableRNGs` was a direct `[deps]` entry but is only used in
  tests. Moved it to `[extras]` and the test target.

- Missing extras compat: added a `Pkg = "1"` compat entry for the `Pkg` test
  extra.

Verified locally: Aqua.test_all(CorleoneOED) passes 11/11 on Julia 1.10 and
1.12; CorleoneOED Core tests pass on 1.12 (1D Example 36/36, Lotka Volterra
125/125, Lotka Volterra SVD 17/17, including the `@inferred get_sampling_sums`
assertions and the multi-experiment optimization that exercises the relocated
`shooting_constraints`); parent Corleone Aqua still passes 11/11.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e MTK floor

The lib/OptimalControlBenchmarks downgrade-sublibraries job failed at the
SciMLBase floor (fixed earlier in this PR); with that resolved the next
downgrade failure surfaced: ModelingToolkit precompilation fails on Julia
1.10 with `UndefVarError: ispublic not defined`.

Root cause (two distinct problems, both upstream-floor issues):

1. `ModelingToolkitBase` was a *direct* dependency of OptimalControlBenchmarks
   but is never imported or used anywhere in its source or tests (only the
   `[deps]` entry referenced it). Because it was a direct dep with no compat
   entry, the downgrade workflow pinned it to its floor `1.0.0`. MTK's
   `@import_mtkbase` macro references `ModelingToolkitBase.ispublic` /
   `.mergedefaults`, neither of which exists in MTKBase 1.0.0. Removing the
   spurious direct dep lets MTKBase float (it is still pulled transitively by
   ModelingToolkit), so the resolver picks the latest compatible MTKBase
   (1.41) instead of the broken 1.0.0.

2. With MTKBase fixed, the downgrade still pinned ModelingToolkit to its
   lowest version in `11` (= 11.0.0). MTK 11.0.0 and 11.1.0 call
   `Base.ispublic` unconditionally, which does not exist on Julia 1.10
   (it was added in Julia 1.11; later MTK versions guard the call). MTK
   11.2.0 hits a separate `generate_ODENLStepData must be explicitly
   imported` precompile error on 1.10. Raised the ModelingToolkit compat
   floor to `11.7.2`, the earliest 11.x verified to precompile and load on
   Julia 1.10.

Local verification (Julia 1.10.11, develop in-tree Corleone, simulate the
sublibrary downgrade resolve):
- Before: RESOLVE_OK but precompile fails with `UndefVarError: ispublic not
  defined` (MTK 11.0.0 + MTKBase 1.0.0), reproducing the CI failure.
- After: RESOLVE_OK; resolved ModelingToolkit=11.7.2, ModelingToolkitBase=1.41.0;
  `import OptimalControlBenchmarks` precompiles and loads cleanly.

No tests skipped, no tolerances loosened.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ompat)

The lib/OptimalControlBenchmarks [QA] group ran Aqua.test_all and failed:

- Stale dependencies: `OptimizationLBFGSB`, `Ipopt`, `UnoSolver`,
  `OrdinaryDiffEq`, and `Runic` were declared in `[deps]` but never loaded
  by the package. `OptimizationLBFGSB`, `OrdinaryDiffEq`, and `Runic` are
  not referenced anywhere; `Ipopt`/`UnoSolver` are used only by the loose
  top-level `main.jl` demo script (not part of the package or its tests).
  `StableRNGs` was likewise an unused direct dep (it stays available
  transitively via CairoMakie). Removed all of them from `[deps]`/`[compat]`
  so the declared deps are honest about what the package actually loads.
  `main.jl` keeps its `using Ipopt`/`using UnoSolver`; a header comment notes
  they are demo-only solver back ends to add to the env before running it.

- Compat bounds: `BenchmarkTools`, `CairoMakie`, and `IntervalSets` were
  direct deps without a `[compat]` entry, and the `Pkg` test extra lacked
  one too (Aqua's extras compat check does not exempt stdlibs). Added
  `BenchmarkTools = "1"`, `CairoMakie = "0.15"`, `IntervalSets = "0.7"`, and
  `Pkg = "1"`.

The test target (`Pkg`, `SafeTestsets`, `Test`) is unchanged, so the
downgrade-sublibraries Core resolve is not perturbed; the added compat
floors only raise previously-uncapped lower bounds.

Local verification (Julia 1.10.11, develop in-tree Corleone + the
sublibrary):
- `Aqua.test_all(OptimalControlBenchmarks)` passes all checks (Method
  ambiguity, Unbound type parameters, Undefined exports, Project extras,
  Stale dependencies, Compat bounds 4/4, Piracy, Persistent tasks). Before
  this change it failed on Stale dependencies, then on extras Compat bounds.
- Simulated the sublibrary downgrade resolve (pin every direct dep to its
  declared `[compat]` floor): RESOLVE_OK; `import OptimalControlBenchmarks`
  precompiles and loads cleanly.

No tests skipped, no tolerances loosened.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The manual Symbolics re-export loop works around an undefined `Variable`
export (and `@public Unknown`) in Symbolics. Both are fixed upstream in
JuliaSymbolics/Symbolics.jl#1906; the comment now records that the loop can be
replaced with `@reexport using Symbolics` plus a Symbolics compat-floor bump
once that lands.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor Author

The CorleoneOED.jl manual Symbolics re-export loop works around two stale entries in names(Symbolics) that don't resolve in the module (Variable, exported but its struct was removed; Unknown, @public but never imported from SymbolicUtils). That triggers Aqua's undefined-exports check on a blanket @reexport using Symbolics.

Fixed upstream: JuliaSymbolics/Symbolics.jl#1906 (verified on Julia 1.12 that Aqua.test_undefined_exports(Symbolics) fails before and passes after).

Once #1906 is released, this loop can be reverted to @reexport using Symbolics and the Symbolics compat floor bumped to the fixed version. Added a code comment to that effect in this PR.

Please ignore until reviewed by @ChrisRackauckas.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor Author

Upstream status re-check (the CorleoneOED.jl Symbolics re-export loop in this PR):

The loop works around two stale entries in names(Symbolics) that don't resolve in the module, which makes Aqua.test_undefined_exports fail on a blanket @reexport using Symbolics. Reproduced locally on Julia 1.12 against the latest released Symbolics v7.28.1:

SYMBOLICS_VERSION=7.28.1
UNDEFINED_EXPORTS=[:Unknown, :Variable]

Root-cause fix: JuliaSymbolics/Symbolics.jl#1906 (drops Variable from the export list; imports Unknown from SymbolicUtils so the @public Unknown resolves). As of today that PR is still open / not merged, and no Symbolics release contains it (latest tag is v7.28.1, which still fails as shown above).

Recommendation: keep the workaround as a stopgap to stay green. The in-code comment already documents the revert path (drop the loop, restore @reexport using Symbolics, and bump the Symbolics compat floor once #1906 ships). No change needed here until #1906 is merged and released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants