Skip to content

Make bevy_parity tests a superset of every runner↔JEOD Tier 3 scenario #389

@simnaut

Description

@simnaut

Background

For each Tier 3 scenario the workspace has up to three comparisons:

  1. runner ↔ JEOD CSV (physics correctness against an external oracle)
  2. runner ↔ bevy (the two consumers of astrodyn produce bit-identical state)
  3. bevy ↔ JEOD (the Bevy adapter is correct against the oracle)

(3) isn't tested directly today. It's implied by transitivity from (1) + (2): if runner matches JEOD within tolerance, and bevy matches runner bit-for-bit, then bevy matches JEOD within the same tolerance. For that argument to hold, every (1) needs a corresponding (2).

Architectural intent

Parity tests should be a superset of runner↔JEOD tests:

  • For every crates/astrodyn_verif_jeod/tests/tier3_*.rs test there is a sibling bevy_parity_* test that propagates the same scenario through Bevy and asserts bit-identical state with the runner.
  • Plus additional parity tests for Bevy-adapter mechanisms that don't have a JEOD analog (chained attach events, body-action lifecycle, frame switch, source mutation, …).

The first set carries transitivity to JEOD. The second set policies Bevy-adapter-specific mechanisms.

Current state

Mapping by topic name (best-effort match between tier3_* and bevy_parity_* filenames after stripping the family prefixes):

Set Count
Topics with both runner↔JEOD and runner↔bevy 6
Tier 3 topics missing a parity counterpart (the gap) 64
Parity topics with no Tier 3 counterpart (Bevy-mechanism stress, intentional) 23

The 6 already-covered topics are: attach_detach_trajectory, gj, kinematic_propagation, mass_attach_detach, relative, srp.

Gap inventory (64 Tier 3 scenarios with no parity counterpart)

apollo8_frame_switch
apollo_mass_tree
apollo_trajectory
attach_mass
battin
complex_attach_detach
contact
drag_6dof
drag_analytical
drag_flatplate_ver
drag_rot_verif
drag_ver
drag_verif
dyncomp_combinations
dyncomp_run10
dyncomp_run2
dyncomp_run3
dyncomp_run4
dyncomp_run5
dyncomp_run6
dyncomp_run7
dyncomp_run9
dyncomp_run_attach_to_ref_frame
earth_moon
euler
euler_edge
force_torque_response
integ_analytical
integ_comparison
integ_gj_orders
lsode
lvlh
lvlh_edge
lvlh_extended
lvlh_relative
lvlh_rot_init_propagation
mars_orbit
mercury
met
ned
ned_edge
orbelem
orbelem_comprehensive
orbinit_docker
orbinit_edge
orbinit_families
orbinit_roundtrip
planetary
polar_motion
ref_attach
relative_extended
shadow_2a
solar_beta
solar_beta_edge
solar_beta_extended
srp_1st_order
srp_basic
srp_rk4_thermal
tide_verif
time_docker
time_reversal
time_scales_comprehensive
timescale
torque_simple

Most are physics scenarios — Apollo lunar transfer, Mars/Mercury/Earth-Moon trajectories, the SIM_dyncomp run2–run10 family, every LVLH/NED/orbelem/orbinit derived-state edge case, the drag and integrator-comparison families, and so on.

Proposed implementation shape

Rather than hand-writing 64 parallel parity test files, the cost-effective approach is a single extension on VerificationCase that turns any Tier 3 scenario definition into a parity test:

// crates/astrodyn_verif_parity/src/lib.rs (new) — or in verif_jeod
pub trait VerificationCaseExt: Sized {
    /// Materialize the scenario into both an `astrodyn_runner::Simulation`
    /// and an `astrodyn_bevy::App`, step both for the case's duration,
    /// and assert every component is bit-identical (`f64::to_bits()`)
    /// at every reference-CSV time step.
    fn run_and_assert_parity(self);
}

impl VerificationCaseExt for VerificationCase {
    fn run_and_assert_parity(self) { /* ... */ }
}

Then each gap test becomes a one-liner:

// crates/astrodyn_verif_parity/tests/bevy_parity_dyncomp_run2.rs
#[test]
fn bevy_parity_dyncomp_run2_3dof() {
    astrodyn_verif_jeod::run_verification::sim_dyncomp::run2_3dof()
        .run_and_assert_parity();
}

This way the scenario definition stays in one place (verif_jeod::run_verification::sim_*), the parity assertion is one extension implementation, and the per-scenario test files are mechanical wrappers that are cheap to add and maintain.

Open design questions

  1. Where does run_and_assert_parity live? Options: (a) on VerificationCase itself in astrodyn_verif_jeod; (b) on a parity-side trait in astrodyn_verif_parity. (a) consolidates "everything verification" but pulls bevy/astrodyn_bevy deps into verif_jeod; (b) keeps Bevy out of the JEOD-verification crate but introduces a parity-side extension. Probably (b) — verif_jeod should stay Bevy-free.
  2. Does the parity assertion need to run at the same cadence as the JEOD comparison? Tier 3 tests assert at the reference-CSV cadence (typically 1–60 s); parity asserts every integration tick (ms-scale). Two cadences = different assertion shapes.
  3. 6-DOF vs 3-DOF: the Tier 3 VerificationCase has 3-DOF and 6-DOF flavors of dyncomp_run2 etc.; both should get a parity sibling, or the parity test should iterate over both.
  4. Exclusion list for scenarios where parity can't hold structurally (e.g., scenarios using runner-only APIs that have no Bevy equivalent yet): mark with #[ignore] plus a tracking comment, or carve out a known-gap list checked by CI.

Done criteria

  • VerificationCaseExt::run_and_assert_parity (or equivalent) implemented and tested on a pilot scenario (suggest sim_dyncomp::run2_3dof).
  • Wrapper tests added for every Tier 3 scenario in the gap inventory above (or covered by an iterator-style #[test] per family).
  • CI's parity job runs the new tests; the matrix tier3_topics ⊂ bevy_parity_topics is true after the change.
  • If exclusions are needed, they are explicit (#[ignore = "..."]) and counted in CI to prevent silent regression.

Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions