Skip to content

Tests - #1

Open
FabioRossiArcetri wants to merge 7 commits into
jw-lin:packagefrom
FabioRossiArcetri:tests
Open

Tests#1
FabioRossiArcetri wants to merge 7 commits into
jw-lin:packagefrom
FabioRossiArcetri:tests

Conversation

@FabioRossiArcetri

@FabioRossiArcetri FabioRossiArcetri commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown

@jw-lin
Hi there Jonathan, we met briefly at last SPIE after your talk, if you remember I am developing a jax backend for cbeam and have multi-wavelength simulations in the works.
Before proceeding with any further developments of cbeam, I propose to add this test suite to cbeam, which I guess will help to track any newly introduced bug or numerical accuracy related problems. I heavily relied on Claude to develop this, you are welcome to let me know what you think!
Cheers
Fabio Rossi

More details about the pull-request itself:

Add a test suite (+ two fixes it surfaced)

What this adds

tests/ — 134 tests, split in two layers:
tests/unit/ (124 fast tests) — isolated coverage of every class and helper in cbeam.waveguide (Prim2D/Circle/Rectangle/Prim2DUnion, Prim3D/Pipe/LinearPipe/Box/BoxPipe, the base Waveguide + all pre-defined subclasses), cbeam.propagator (Propagator/ChainPropagator helpers and the full z-invariant solve_at → characterize → propagate → make_field → save/load path), and cbeam.FEval (BVH tree build/query/evaluate/grid/resample/transverse_gradient).
tests/integration/ (10 slow tests) — one per documentation example (basicusage, fib, PL, dc, mmi, PL19), each rewritten as a test. They build a waveguide, characterize it, and propagate a field, asserting on physical invariants (power conservation, mode ordering, matrix (anti)symmetry) rather than mesh-dependent exact numbers. The heavier ones shorten the device from the docs — same numerical code path, fewer z-steps.
pyproject.toml — pytest config: pythonpath = ["src"] (the suite always tests the in-repo code), slow/integration markers, warning filters.
tests/conftest.py — forces the Agg matplotlib backend, instantiates the src/cbeam/FEval Julia project on first run, gives each Propagator a temp save_dir (nothing touches ./data).
docs_source/testing.rst — a documentation page describing the suite, how to run it, and a sample run. Linked into the main toctree. (The built HTML in docs/ is not regenerated here.)
Fixes surfaced while writing the tests
import cbeam was broken on current juliacall. FEval.py / FEvalsetup.py did from juliacall import Pkg, which juliacall ≥ 0.9.30 removed — so import cbeam raised ImportError on any fresh install. Now Julia's Pkg is obtained via Main (jl.seval("import Pkg"); jlPkg = jl.Pkg). Verified on juliacall 0.9.26 and 0.9.35.
Rectangle.nearest_boundary_point bug (waveguide.py): the i==2, x>xmax branch assigned outx twice instead of outx+outy, so points past the right edge got a "boundary" point that isn't on the rectangle. Fixed to match the sibling branches.

How to run

pytest -m "not slow" # unit + light integration, ~30 s
pytest # everything, ~15–20 min
pytest -m slow # only the full characterizations
Results
Run in a clean Conda env (Python 3.12, juliacall 0.9.35, cbeam installed editable):

suite result time
pytest -m "not slow" 124 passed 32 s
pytest -m slow 10 passed 15 min 29 s
total 134 passed, 0 failed

Notes for the reviewer

No behavior change beyond the two fixes above; everything else is additive (tests/, one docs page, pytest config).
docs/ (generated HTML) is intentionally left alone — it's regenerated by the normal Sphinx build.

FabioRossiArcetri and others added 5 commits September 1, 2026 11:18
@FabioRossiArcetri @claude
Add unit + integration test suite
0db398d 
- tests/unit: fast isolated tests for every class/function in
  cbeam.waveguide, cbeam.propagator, and cbeam.FEval
- tests/integration: end-to-end tests rewritten from each docs_source
  example (basicusage, fib, PL, dc, mmi, PL19); full characterizations
  marked `slow`
- pyproject.toml: pytest config (pythonpath=src, markers, filters)
- conftest.py: Agg backend, one-time FEval Julia project instantiation
- documents a bug found by the suite: Rectangle.nearest_boundary_point
  writes outx instead of outy in one branch (xfail)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@FabioRossiArcetri @claude
Fix findings surfaced by the test suite
7a5d7c5 
- FEval.py / FEvalsetup.py: `from juliacall import Pkg` was removed in
  juliacall >= 0.9.30, which broke `import cbeam` outright on current
  installs. Get Julia's Pkg through Main instead (works on old and new
  juliacall). Verified against juliacall 0.9.26 and 0.9.35.
- waveguide.py: Rectangle.nearest_boundary_point assigned `outx` twice in
  the `i==2, x>xmax` branch, so points past the right edge got a point
  that wasn't on the rectangle. Assign `outy` as the sibling branches do.
- tests: the Rectangle case is now a normal regression test (was xfail).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@FabioRossiArcetri @claude
docs: add a testing page
e383525 
Explains the layout of tests/ (unit + integration), how to run the
suite (pytest, the `slow` marker, the src pythonpath / FEval setup),
and shows a typical successful run (134 passed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@FabioRossiArcetri @claude
tests: add golden-file numerical regression layer to integration tests
8c87e5f 
The integration tests assert on physical invariants with loose tolerances so
they survive a mesh / eigenbasis change. This adds a second, stricter layer:
a `golden` fixture (tests/conftest.py) records the actual arrays a run produces
under tests/integration/_golden/<id>.npz and, on later runs, checks them with
np.testing.assert_allclose.

  CBEAM_GOLDEN=record   write the reference set
  CBEAM_GOLDEN=check     (default) compare; a missing file skips the assertion
  CBEAM_GOLDEN=off       no-op

Each slow test now also fingerprints its stable numeric outputs: neffs(z)
(resampled onto the current z-grid before comparing), sorted channel/mode
power spectra, and mode-amplitude magnitudes. The reference set itself is
generated from the `tests` branch library and committed separately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@FabioRossiArcetri @claude
tests: commit golden reference set recorded from the tests branch
b413202 
13 .npz files under tests/integration/_golden/, one per instrumented
integration test, recorded with CBEAM_GOLDEN=record against the `tests`
branch library (the pre-refactor code path). These are the agreed numerical
ground truth; runs on other branches are compared against them.

Also drop the per-mode `uf_power` golden check in
test_chain_propagator_end_to_end: those 19 modes span one ~19-fold
degenerate subspace, so the per-mode power split is gauge-dependent (moves
~5e-2 between eigensolver bases). The physical channel-power spectrum
(`channel_out_power`) is stable and stays pinned.

Verified: `multi` (working tree) reproduces every pinned quantity to
<= 3e-9 abs; the committed merge without the get_19port_positions revert
reproduces all but the 19-port compute_neffs scan (~3e-6, mesh-ordering).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jw-lin

jw-lin commented Sep 1, 2026 via email

Copy link
Copy Markdown
Owner

jw-lin
jw-lin reviewed Sep 8, 2026
View reviewed changes
Comment thread tests/integration/test_photonic_lantern.py
jw-lin
jw-lin reviewed Sep 8, 2026
View reviewed changes
Comment thread tests/integration/test_mmi.py Outdated
jw-lin
jw-lin reviewed Sep 8, 2026
View reviewed changes
Comment thread docs_source/testing.rst
@@ -0,0 +1,119 @@
testing

@jw-lin jw-lin Sep 8, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the doc will need to be recompiled for the new page to show up.

@FabioRossiArcetri FabioRossiArcetri Sep 9, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave this to you, since small difference in the toolchain used would result in a messy diff (not terrible once the final result is ok, but still not great to see).

@FabioRossiArcetri FabioRossiArcetri Sep 9, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also have the file workflows/tests.yaml which would trigger the test in CI (every time a pull request is issued), let me know what you think about this. In the next days I will start to deliver pull requests for the jax path and more! Cheers, Fabio

@FabioRossiArcetri
tests: address reviewer comments on symmetry and peak-count checks
987b788 
- test_photonic_lantern.py: the 5 outer lantern ports sit on an exact
  5-fold rotational symmetry, so a centered LP01 launch should split
  ~evenly between them. Add an explicit spread check (relative to the
  golden reference's ~0.3% observed spread, a 15% bound still leaves
  large margin) on top of the existing loose bounds, which could pass
  even with a lopsided/asymmetric split.
- test_mmi.py: tighten the 3-fold self-image peak count from a
  2-4 range to an exact 3, and switch from a hand-rolled local-maxima
  counter (which can double-count a plateau) to scipy.signal.find_peaks
  with a minimum sample separation.
@FabioRossiArcetri
tests: tighten lantern outer-port symmetry tolerance to 1.5%
ea65cad 
Verified against a live run (venv with juliacall/wavesolve/cbeam built
from this checkout): the outer-port relative spread comes in well under
1.5%, so this is still a comfortable margin, just a stricter one than
the initial 15% bound.
jw-lin
jw-lin reviewed Sep 9, 2026
View reviewed changes
Comment thread tests/integration/test_fib.py
assert modes.shape[0] == Nmax
# effective indices come out sorted high -> low
assert np.all(np.diff(effective_indices) <= 1e-9)
golden.check("neffs", effective_indices, rtol=1e-6, atol=1e-9)

@jw-lin jw-lin Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get a fail here on when running the "not slow" tests. neffs for modes 3,4,5 (LP21a/b and LP02) don't match golden file. This is the only fail out of all the tests, including the slow ones, on my desktop install. Below is the printout from my test run. I was also able to reproduce this behavior on my work laptop. Both are Windows machines.

Probably a result of differences in gmsh install + low resolution of the solve.

E       AssertionError:
E       Not equal to tolerance rtol=1e-06, atol=1e-09
E       neffs: max|delta|=5.052e-06
E       Mismatched elements: 3 / 10 (30%)
E       Mismatch at indices:
E        [3]: 1.441335940415306 (ACTUAL), 1.4413389326193096 (DESIRED)
E        [4]: 1.4413345184221822 (ACTUAL), 1.441337124027709 (DESIRED)
E        [5]: 1.4408963204385143 (ACTUAL), 1.4409013722926474 (DESIRED)
E       Max absolute difference among violations: 5.05185413e-06
E       Max relative difference among violations: 3.50603742e-06
E        ACTUAL: array([1.444165, 1.442914, 1.442913, 1.441336, 1.441335, 1.440896,
E              1.439905, 1.439904, 1.439898, 1.43976 ])
E        DESIRED: array([1.444165, 1.442914, 1.442913, 1.441339, 1.441337, 1.440901,
E              1.439904, 1.439904, 1.439898, 1.43976 ])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jw-lin jw-lin jw-lin left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@FabioRossiArcetri @jw-lin