Skip to content

VALIDATION.md

Latest commit

 

History

History
216 lines (163 loc) · 8.56 KB

VALIDATION.md

File metadata and controls

216 lines (163 loc) · 8.56 KB
Raw

Validation

This document describes how to validate SIDpulse Tracker before committing, tagging, or publishing a release.

It intentionally does not contain permanent test-count snapshots. Test counts change as the suite evolves. Release-specific results belong in release notes, CI logs, or the release checkpoint.

1. Automated test suite

From the repository root, using the project virtual environment:

./.venv/bin/python -m pytest -q

The release must not proceed if any test fails.

The SID/PRG/channel CPU tests use tests/py65_nmos.py, an explicit test-only adapter for py65 1.2.0's DEC absolute ($CE) timing-table typo (3 versus 6 cycles). The adapter copies the table per instance and leaves instruction execution and installed py65 untouched. tests/test_py65_nmos.py checks isolation, instruction semantics and the documented cycle count; tests/test_replay_cycles.py checks the production verifier independently against fixed hardware-manual expectations. Do not remove cycle assertions or alter production timing to match the typo.

2. Headless smoke tests

Verify that the application initializes and exercises its basic playback paths:

./.venv/bin/python -m sidpulse --headless-smoke --example
./.venv/bin/python -m sidpulse --headless-smoke --play-welcome-song

Both commands must exit successfully.

3. Generated references

Regenerate the tracked command/effect references and inspect any resulting diff:

./.venv/bin/python scripts/build_command_reference.py
./.venv/bin/python scripts/build_effect_reference.py
git diff --check

Unexpected generated changes must be reviewed before release.

4. Source-tree checks

Before committing:

git diff --check
git diff --cached --check
git status --short --branch

Before tagging, the release commit should be pushed and the working tree should be clean.

5. Version consistency

The release version must agree in all declarations:

cat VERSION
grep -nE 'version|__version__' pyproject.toml sidpulse/__init__.py

For v0.2.23, each application-version declaration must resolve to 0.2.23.

6. Interactive tracker checks

Automated tests do not replace a short desktop run. Before release, verify at minimum:

  • application starts normally;
  • startup New song opens a blank Untitled project and does not leave the demo loaded;
  • startup Play demo song keeps and plays the editable welcome project;
  • example/welcome project loads and plays;
  • pattern editor accepts normal note input;
  • typing 0..7 on the octave digit edits only the octave;
  • instrument editing and audition work;
  • instrument activity dots react to actual playback/audition rather than mere selection;
  • ADSR display shows attack 00 at the left edge as intended;
  • F6 pattern looping remains independent;
  • F11/F12 song-end looping can be enabled and disabled;
  • F9 Load browser works;
  • F10 Save browser works;
  • Save As offers the current project filename and permits non-destructive caret editing;
  • filename edits survive directory navigation;
  • overwrite Cancel returns to the same editable draft;
  • SID/PRG export uses the shared browser without renaming the native project;
  • .sidpulse save/load round-trip works;
  • audio playback has no obvious hangs, underruns or stuck notes;
  • Alt+F12 lists real outputs; selecting one and OK persists it across restart;
  • Test arpeggio is audible on the selected draft device, then song playback resumes with its previous pause state and unchanged song;
  • Cancel and Reset defaults followed by Cancel leave the saved device unchanged;
  • Reset defaults followed by OK saves System default / 2048 / detection ON;
  • Refresh outputs finds a newly connected USB/HDMI device; an unavailable saved device falls back to default with a visible message in Audio Settings;
  • test the above on native Linux and Windows hardware, including switching between two physical outputs. SDL dummy output cannot prove audible routing.

7. Export validation

For SID and PRG export:

  • verify a small known project exports successfully;
  • verify exported files are non-empty;
  • verify an oversized project is rejected cleanly;
  • verify the error reports required and available memory;
  • verify the error tells the user to shorten or simplify the project;
  • verify a failed export does not modify the editable .sidpulse project.

The exporter must never silently discard musical material to make a project fit.

8. Platform validation

SIDpulse targets Linux and Windows. The GitHub Actions matrix should pass on the exact release commit before tagging/publishing.

Current CI covers Ubuntu and Windows on Python 3.10 and 3.12. Local validation on one platform does not substitute for the remote matrix.

If an expected platform or interactive behavior has not been tested for a release, state that explicitly rather than implying otherwise.

9. Release gate

A typical final local gate is:

set -euo pipefail
./.venv/bin/python -m pytest -q
./.venv/bin/python -m sidpulse --headless-smoke --example
./.venv/bin/python -m sidpulse --headless-smoke --play-welcome-song
./.venv/bin/python scripts/build_command_reference.py
./.venv/bin/python scripts/build_effect_reference.py
git diff --check
git diff --cached --check
git status --short --branch

Then confirm:

  • all automated tests pass;
  • both smoke tests pass;
  • intended interactive checks pass;
  • version declarations match;
  • documentation matches the current behavior;
  • no private/local files are staged;
  • the pushed release commit passes CI;
  • the annotated release tag points at that exact verified commit.

10. Release-specific evidence

Exact test counts, CI URLs, operating systems manually tested, known limitations and intentionally skipped checks belong in the release record or checkpoint.

Do not rewrite this document merely because the number of tests changed.

v0.2.19 export squeezer gate

Consult the candidate-specific validation record. Run the full suite, including both new dependency-backed test modules; a passing pure subset does not replace it. With 64tass installed, run python scripts/build_squeeze_players.py --check. Inspect the export dialog's checked defaults and independent opt-outs, low/high zoom scrolling, Cancel and S/E paths. Compare SID/PRG exports with squeezing on/off in matching PAL/NTSC VICE/native audio and real hardware where available. Check hard-restart attacks and filter/sync/ring timing, not merely final register snapshots.

Export-analysis responsiveness (v0.2.21)

The worker lifecycle and compiler-progress tests are dependency-light. Actual animation, keyboard/mouse, resize and save continuation tests require pygame-ce:

./.venv/bin/python -m pytest -q -ra tests/test_export_analysis_job.py tests/test_export_progress_gui.py tests/test_squeeze_gui.py tests/test_file_browser_gui.py tests/test_programs.py tests/test_prg.py

Do not replace the spawned compiler with a synchronous fallback in production. The native GUI fixtures use frame polling to await completion; they must not restore the old assumption that opening Export already returns a compiled result.

Interactively open SID and PRG export on a larger song. Confirm that the first progress frame appears promptly, the coloured band keeps moving through long phases, and elapsed time advances. Resize the window, move it off/on screen, cancel during squeezing and verification, reopen, and try closing the application while analyzing. No export/preferences should be written by cancellation and no compiler process should survive application shutdown.

After analysis, change squeeze options and exercise Analyze, Save + export and Export only, including native-save cancellation and export overwrite cancellation. Check animation/Cancel at minimum window size, 300% zoom and each theme. Confirm existing audio continues without introducing new dropouts on the supported hosts. A passing worker unit test alone does not establish desktop or audio behavior.

Audio isolation and underrun detection (v0.2.22)

  • Run the full test suite with native pygame/reSIDfp and py65 dependencies.
  • See VALIDATION-v0.2.22.md for the unchanged 0.2.21 comparison and rerun commands.
  • Verify Alt+F12, checkbox OK/Cancel and config true/false; warnings belong in the lower-left footer, never a modal. Distinguish missing PCM from late callbacks.
  • Exercise playback, audition, pause/resume, buffer changes, mute, live edits, repeated restarts, Info scopes, resizing, export, and clean shutdown.
  • Repeat the real-device checks on Linux and Windows before claiming either platform's hardware path is validated. Dummy-driver zeroes are not that proof.