Skip to content

Preserve baselines for skipped tests#34

Merged
mavam merged 2 commits intomainfrom
topic/improve-skipped-test-output
Mar 11, 2026
Merged

Preserve baselines for skipped tests#34
mavam merged 2 commits intomainfrom
topic/improve-skipped-test-output

Conversation

@mavam
Copy link
Copy Markdown
Member

@mavam mavam commented Mar 10, 2026

Summary

  • stop treating existing baseline files as an error for skipped tests
  • preserve skipped-test baselines during --update
  • add regression tests for verbose, quiet, and update-mode skip handling

Testing

  • uv run pytest tests/test_run.py -k 'handle_skip or run_skipped_reason_matches_static_skip_and_executes or suite_static_skip_short_circuits_fixture_activation or suite_static_skip_respects_run_skipped_selector'
  • uv run ruff check src/tenzir_test/run.py tests/test_run.py
  • uv run ruff format --check src/tenzir_test/run.py tests/test_run.py

Linear: https://linear.app/tenzir/issue/TNZ-438/dont-show-an-error-like-output-for-skipped-tests

resolves TNZ-438

mavam added 2 commits March 10, 2026 17:19
Skipped tests should stay skipped even when a baseline file already\nexists. Preserve existing baselines instead of treating them as\nfailures or clearing them during --update.\n\nAdd regression coverage for verbose and non-verbose skip output as\nwell as update mode preservation.\n\nRef: TNZ-438
@greptile-apps
Copy link
Copy Markdown

greptile-apps bot commented Mar 10, 2026

Confidence Score: 4/5

  • This PR is safe to merge; it removes a clearly incorrect baseline-file side-effect from skip handling with good test coverage.
  • The logic change is small and well-targeted, the new and updated tests directly verify the intended behaviour, and the changelog entry is accurate. The only concerns are cosmetic/maintenance: two now-unused parameters on handle_skip and a stale bool | str return-type annotation.
  • The update and output_ext parameters in handle_skip (src/tenzir_test/run.py line 4289) are now dead and worth cleaning up.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Worker._record_static_skip called] --> B[handle_skip invoked]
    B --> C{is_verbose_output?}
    C -- Yes --> D[Print SKIP message]
    C -- No --> E[No output]
    D --> F[return 'skipped']
    E --> F

    subgraph OLD ["Old behavior (removed)"]
        G[ref_path = test.with_suffix]
        H{update=True?}
        G --> H
        H -- Yes --> I[Write empty file to ref_path]
        H -- No --> J{ref_path exists and non-empty?}
        J -- Yes --> K[report_failure + return False]
        J -- No --> F
    end

    style OLD fill:#ffcccc,stroke:#cc0000,color:#000
Loading

Last reviewed commit: 5ba2cfc

@mavam mavam requested a review from IyeOnline March 10, 2026 19:08
@mavam mavam merged commit 33c245f into main Mar 11, 2026
7 checks passed
@mavam mavam deleted the topic/improve-skipped-test-output branch March 11, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants