Skip to content

test(scripts): add 56 tests for validate_sc_notebooks.py (cycle 71)#1882

Merged
jsboige merged 1 commit into
mainfrom
test/validate-sc-notebooks
May 30, 2026
Merged

test(scripts): add 56 tests for validate_sc_notebooks.py (cycle 71)#1882
jsboige merged 1 commit into
mainfrom
test/validate-sc-notebooks

Conversation

@jsboige
Copy link
Copy Markdown
Owner

@jsboige jsboige commented May 30, 2026

Summary

Add 56 tests for scripts/smartcontracts/validate_sc_notebooks.py — SC notebook structural validator.

Functions tested

Zone Tests Key invariants
source_text 7 String/list extraction, missing source
is_nav_cell 7 Back/forward link detection regex
has_exposed_solution 8 TODO/NotImplementedError exclusion, triple-quote Solidity, test function with assertEq
make_source_list 5 STRING→LIST conversion, trailing newline behavior
get_executable_cells 12 TODO/NotImplementedError/pip skip, anvil profile gating, web3/compile_and_deploy skip
validate_notebook 14 Title/nav/objectives/duration, STRING format detection, consecutive code, anvil web3 requirement, missing notebook, invalid JSON
Dataclasses 2 NotebookValidation defaults, CellInfo creation
Colors 1 disable() clears all ANSI codes

Test count

  • 56 new tests, 506 lines
  • Suite: 1008 passed (0 regression)
  • Total scripts tests across all cycles: 390→1008

Scope

  • Single file: scripts/tests/test_validate_sc_notebooks.py
  • Lane: smartcontracts (po-2024:CoursIA-2, non-validation-notebook)
  • No notebooks modified, no source scripts modified

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

Test pure functions: source_text (7), is_nav_cell (7), has_exposed_solution (8),
make_source_list (5), get_executable_cells (12), validate_notebook (14),
dataclasses (2), Colors (1).

Key invariants: nav link detection, solution exposure heuristics, executable
cell filtering (TODO/NotImplementedError/pip skip, anvil profile gating),
structural validation (title/nav/objectives/duration, STRING format, consecutive
code, anvil web3 requirement).

Suite: 1008 passed (0 regression).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copy link
Copy Markdown
Collaborator

@clusterManager-Myia clusterManager-Myia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NanoClaw]

Solid test file overall — good coverage of the pure functions in validate_sc_notebooks.py with well-organized class-based grouping and proper use of tmp_path fixtures. A few issues worth addressing:

1. sys.path.insert hack (line 14)

sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "smartcontracts"))

This is fragile and won't work if tests are invoked from a different working directory or by a test runner that copies files. Prefer:

  • A conftest.py at scripts/tests/conftest.py that adds the import path once
  • Or add scripts/smartcontracts to testpaths / pythonpath in pyproject.toml under [tool.pytest]

2. TestColors.test_disable — fragile global state (lines 498-506)

This test mutates class-level attributes (Colors.GREEN, Colors.RED, etc.) and tries to restore them afterward, but:

  • Only Colors.GREEN is saved/restored from the original value; the rest (RED, YELLOW, BLUE, CYAN, BOLD, END) are hard-coded back to their expected ANSI codes
  • If any other test runs in parallel and reads Colors during the restore window, it sees inconsistent state
  • Tests for ANSI color helpers are generally low-value — they test that string constants exist, not behavior

Suggestion: Either remove this test class entirely (it tests that class attributes can be set to ""), or refactor Colors to be instance-based so tests can isolate state without global mutation.

3. Dataclass tests are tautological (lines 485-496)

test_notebook_validation_defaults and test_cell_info_creation just verify that Python's dataclass __init__ assigns fields correctly. These tests will never break unless Python itself is broken. Consider either:

  • Removing them (no real value)
  • Or testing meaningful behavior (e.g., serialization, equality, or how defaults interact with validation logic)

4. Minor: test data naming mismatch (line 82-84)

def test_list_with_trailing_newline(self):
    cell = {"source": ["line1\n", "line2\n", ""]}
    assert source_text(cell) == "line1\nline2\n"

Comments/docstrings reference "hello/world" in surrounding tests but this one uses "line1/line2" — cosmetic but slightly confusing when reading the class as a whole.

5. Count discrepancy

Title says 56 tests. I count 59 test methods across 8 classes (7+7+8+5+12+17+2+1). Minor — worth fixing the title to match.

6. Missing edge case coverage

  • make_source_list with \r\n (Windows line endings) — no test for this, and the split logic (splitting on \n) may produce \r artifacts
  • get_executable_cells with foundry profilestandalone and anvil are tested, but foundry is only tested implicitly in test_profile_stored. A foundry-specific filtering test would close the gap

Verdict

Good structural quality and meaningful assertions for the core validation logic. The main actionable items are the sys.path hack (#1, should fix before merge) and the fragile Colors test (#2, consider dropping). The rest are suggestions.

@jsboige jsboige merged commit 53a2923 into main May 30, 2026
2 checks passed
@jsboige jsboige deleted the test/validate-sc-notebooks branch May 30, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants