Skip to content

feat: add python line and branch coverage governance to ci #317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
0xKT merged 2 commits into from
Aug 17, 2026
Merged

feat: add python line and branch coverage governance to ci #317

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
af70757
feat: add python line and branch coverage governance to ci
forrestjgq1982 Aug 14, 2026
e6e4b31
fix: reproduce coverage baseline on ci and restore locked install
forrestjgq1982 Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3,576 changes: 3,576 additions & 0 deletions .github/coverage-baseline.json
View file
Open in desktop

Large diffs are not rendered by default.

51 changes: 47 additions & 4 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,17 @@ jobs:
name: unit (py${{ matrix.python }} / ${{ matrix.os }})
runs-on: ${{ matrix.os }}
timeout-minutes: 20
env:
PYTHON_VERSION: ${{ matrix.python }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python: ["3.12"]
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0

- name: Install uv
uses: astral-sh/setup-uv@v8.2.0
Expand All @@ -91,15 +95,54 @@ jobs:
run: uv python install ${{ matrix.python }}

- name: Install dependencies
run: uv sync --locked --all-extras --dev
run: uv sync --locked --all-extras --dev --python ${{ matrix.python }}

- name: Unit tests (integration + e2e deselected by norecursedirs + addopts)
- name: Unit tests with line and branch coverage
# TERM=dumb: typer's rich_utils force-enables colour when GITHUB_ACTIONS is
# set, and the highlighter then splits option names with escape codes, so
# `"--flag" in result.stdout` assertions fail only on CI.
run: make coverage

- name: Coverage summary and lowest-covered files
if: ${{ !cancelled() && hashFiles('coverage.json') != '' }}
run: make coverage-summary

- name: Overall coverage ratchet
if: ${{ !cancelled() && hashFiles('coverage.json') != '' }}
run: make coverage-ratchet

- name: Coverage baseline cannot decrease
if: ${{ !cancelled() && github.event_name == 'pull_request' }}
env:
TERM: dumb
run: uv run --all-extras pytest -q
COVERAGE_BASE_REF: origin/${{ github.base_ref }}
run: make coverage-baseline-check

- name: PR diff coverage
if: ${{ !cancelled() && github.event_name == 'pull_request' && hashFiles('coverage.json') != '' }}
env:
COVERAGE_BASE_REF: origin/${{ github.base_ref }}
run: make coverage-diff

- name: Explain skipped diff coverage
if: ${{ !cancelled() && github.event_name != 'pull_request' }}
run: echo "Diff coverage is a PR-only gate; push builds still enforce the overall ratchet."

- name: Create auditable baseline candidate
if: ${{ !cancelled() && hashFiles('coverage.json') != '' }}
run: make coverage-baseline-candidate

- name: Upload coverage reports
if: always()
uses: actions/upload-artifact@v4
with:
name: python-coverage-py${{ matrix.python }}-${{ matrix.os }}
path: |
coverage.xml
coverage.json
htmlcov/
coverage-baseline-candidate.json
if-no-files-found: warn
retention-days: 30

windows-upgrade:
name: Windows self-upgrade
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ poetry.lock
.coverage
.coverage.*
coverage.xml
coverage.json
coverage-baseline-candidate.json
htmlcov/
botpy.log
nano.*.save
Expand Down
3 changes: 3 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ Hard constraints only (violations get reverted / rejected). Soft suggestions and

### §1.1 Top rule: don't add comments unless necessary

- Every new code file must document its purpose in English with an appropriate

@gloryfromca gloryfromca Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This is a new repo-wide hard constraint riding along in a CI-coverage PR. AGENTS.md is self-describing as "hard constraints only (violations get reverted / rejected)", so from merge onward every new file in the repo is subject to it, and reviewers looking at a coverage PR are not the audience that would weigh in on it. The bullet itself is fine and the follow-on sentence resolves the tension with the "don't comment unless necessary" top rule -- it is the bundling I would flag.

Also: the summary table at the top still gives section 1 as "Don't comment unless necessary; comments in English", which no longer describes a section that now mandates a module docstring in every new file. If the bullet stays, the gist row should mention it.

module docstring or equivalent file-level documentation. Inline comments
still follow the necessity rules below.
- Match the style of surrounding lines. If neighboring code has no comments, **don't** add one to your new line.
- Comment **only** when:
- the logic is non-obvious;
Expand Down
53 changes: 41 additions & 12 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,15 @@
.PHONY: help install install-deps lint lint-python lint-tui lint-bridge test test-python test-tui build build-tui build-bridge check-commits check-pr-title check-large-files ci clean
.PHONY: help install install-deps lint lint-python lint-tui lint-bridge test test-python test-tui coverage coverage-summary coverage-diff coverage-ratchet coverage-baseline-check coverage-baseline-candidate build build-tui build-bridge check-commits check-pr-title check-large-files ci clean

PYTHON ?= python3
PYTHON_LINT_TARGETS ?= scripts/check_commit_file.py scripts/check_commit_messages.py scripts/check_pr_title.py scripts/check_large_files.py scripts/commit_lint.py tests/test_commit_lint.py tests/test_large_file_check.py
PYTHON_VERSION ?= 3.12
PYTHON_LINT_TARGETS ?= scripts/check_commit_file.py scripts/check_commit_messages.py scripts/check_pr_title.py scripts/check_large_files.py scripts/commit_lint.py scripts/coverage_gate.py tests/test_commit_lint.py tests/test_coverage_gate.py tests/test_large_file_check.py
COMMIT_RANGE ?= origin/main..HEAD
COVERAGE_BASE_REF ?= origin/main
# Required coverage percentage for executable lines changed by a PR.
COVERAGE_DIFF_THRESHOLD ?= 90
# Allowed line or branch regression in percentage points to absorb rounding noise.
COVERAGE_RATCHET_TOLERANCE ?= 0.05
COVERAGE_REPORT_ARGS = --cov=raven --cov-branch --cov-report=term-missing:skip-covered --cov-report=xml --cov-report=json --cov-report=html

help:
@echo "Targets:"
Expand All @@ -13,27 +20,31 @@ help:
@echo " lint-tui TypeScript lint + RPC drift check"
@echo " lint-bridge Bridge package build check"
@echo " test Run focused Python checks and TUI tests"
@echo " coverage Run the default Python suite with line and branch coverage"
@echo " coverage-diff Check changed executable lines against COVERAGE_BASE_REF"
@echo " coverage-ratchet Check total line and branch coverage against the baseline"
@echo " coverage-baseline-check Ensure a proposed baseline never lowers the target branch"
@echo " check-commits Validate Conventional Commit subjects"
@echo " check-pr-title Validate the PR title in PR_TITLE"
@echo " check-large-files Validate PR files avoid blocked assets and size bloat"
@echo " ci Run the local CI gate"
@echo " clean Remove generated caches and build output"

install-deps:
uv sync --frozen --extra dev --dev
uv sync --frozen --python $(PYTHON_VERSION) --extra dev --dev

install: install-deps
uv run pre-commit install
uv run pre-commit install --hook-type commit-msg
uv run --frozen --python $(PYTHON_VERSION) pre-commit install
uv run --frozen --python $(PYTHON_VERSION) pre-commit install --hook-type commit-msg
npm ci
npm ci --prefix ui-tui
npm ci --prefix bridge

lint: lint-python lint-tui lint-bridge

lint-python:
uv run --extra dev ruff check $(PYTHON_LINT_TARGETS)
uv run --extra dev ruff format --check $(PYTHON_LINT_TARGETS)
uv run --frozen --python $(PYTHON_VERSION) --extra dev ruff check $(PYTHON_LINT_TARGETS)
uv run --frozen --python $(PYTHON_VERSION) --extra dev ruff format --check $(PYTHON_LINT_TARGETS)

lint-tui:
npm run lint --prefix ui-tui
Expand All @@ -46,7 +57,25 @@ lint-bridge:
test: test-python test-tui

test-python:
uv run --all-extras pytest -q
uv run --frozen --python $(PYTHON_VERSION) --all-extras pytest -q

coverage:
TERM=dumb uv run --frozen --python $(PYTHON_VERSION) --all-extras pytest -q $(COVERAGE_REPORT_ARGS)

coverage-summary:
uv run --frozen --python $(PYTHON_VERSION) python scripts/coverage_gate.py summary

coverage-diff:
uv run --frozen --python $(PYTHON_VERSION) python scripts/coverage_gate.py diff --base-ref $(COVERAGE_BASE_REF) --threshold $(COVERAGE_DIFF_THRESHOLD)

coverage-ratchet:
uv run --frozen --python $(PYTHON_VERSION) python scripts/coverage_gate.py ratchet --tolerance $(COVERAGE_RATCHET_TOLERANCE)

coverage-baseline-check:
uv run --frozen --python $(PYTHON_VERSION) python scripts/coverage_gate.py baseline-check --base-ref $(COVERAGE_BASE_REF)

coverage-baseline-candidate:
uv run --frozen --python $(PYTHON_VERSION) python scripts/coverage_gate.py baseline

test-tui:
npm test --prefix ui-tui
Expand All @@ -61,18 +90,18 @@ build-bridge:

check-commits:
npx commitlint --from origin/main --to HEAD --config commitlint.config.cjs
PYTHONPATH=. uv run --extra dev python scripts/check_commit_messages.py $(COMMIT_RANGE)
PYTHONPATH=. uv run --frozen --python $(PYTHON_VERSION) --extra dev python scripts/check_commit_messages.py $(COMMIT_RANGE)

check-pr-title:
PYTHONPATH=. uv run --extra dev python scripts/check_pr_title.py
PYTHONPATH=. uv run --frozen --python $(PYTHON_VERSION) --extra dev python scripts/check_pr_title.py

check-large-files:
PYTHONPATH=. uv run --extra dev python scripts/check_large_files.py $(COMMIT_RANGE)
PYTHONPATH=. uv run --frozen --python $(PYTHON_VERSION) --extra dev python scripts/check_large_files.py $(COMMIT_RANGE)

ci: lint test build

clean:
rm -rf .pytest_cache .ruff_cache .uv-cache .mypy_cache htmlcov dist build
rm -rf .pytest_cache .ruff_cache .uv-cache .mypy_cache htmlcov coverage.xml coverage.json coverage-baseline-candidate.json dist build
rm -rf ui-tui/dist ui-tui/coverage ui-tui/.vitest-cache ui-tui/packages/hermes-ink/dist
rm -rf bridge/dist
find . -type d -name __pycache__ -prune -exec rm -rf {} +
31 changes: 31 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,37 @@ markers = [
"llm_judge: uses an LLM as a grader for extraction quality (extra cost / non-determinism); opt-in",
]

[tool.coverage.run]
branch = true
source = ["raven"]
omit = [
"raven/__main__.py",
"raven/evolver/__main__.py",
"raven/utils/win_fcntl_shim.py",
]

[tool.coverage.report]
precision = 2
show_missing = true
omit = [
"raven/__main__.py",
"raven/evolver/__main__.py",
"raven/utils/win_fcntl_shim.py",
]
exclude_also = [
"if TYPE_CHECKING:",
"if __name__ == .__main__.:",
]

[tool.coverage.html]
directory = "htmlcov"

[tool.coverage.xml]
output = "coverage.xml"

[tool.coverage.json]
output = "coverage.json"

[dependency-groups]
dev = [
"json-repair>=0.59.5",
Expand Down
Loading
Loading