Skip to content

ci: fix coverage profile collection and move benchmarks to PR checks #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sentomk merged 1 commit into from
Jul 31, 2026
Merged

ci: fix coverage profile collection and move benchmarks to PR checks #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions .github/workflows/bench-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
name: Publish Benchmark Charts

# Runs on push to main only. Generates benchmark charts and commits
# them to docs/assets/bench/. This is separated from the main CI so
# that benchmark failures never cause the main branch CI to fail.
on:
push:
branches: [main]
paths-ignore:
- "README.md"
- "CONTRIBUTING.md"
- "docs/**"
- "LICENSE"
- ".github/ISSUE_TEMPLATE/**"

permissions:
contents: write

concurrency:
group: bench-publish-${{ github.ref }}
cancel-in-progress: true

jobs:
publish:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.BENCH_BOT_TOKEN || github.token }}

- name: Configure
run: |
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DPTN_BUILD_TESTS=OFF \
-DPTN_BUILD_BENCHMARKS=ON \
-DPTN_SKIP_COMPILER_CHECK=ON

- name: Build
run: cmake --build build --target ptn_bench ptn_bench_scale --parallel

- name: Run benchmarks
run: |
mkdir -p bench_results
build/bench/ptn_bench \
--benchmark_filter="VariantMixed|VariantAltHot|VariantMixedGuarded|VariantAltHotGuarded|ProtocolRouter|CommandParser|PacketMixed|PacketMixedHeavyBind" \
--benchmark_out=bench_results/ptn_bench.json \
--benchmark_out_format=json
build/bench/ptn_bench_scale \
--benchmark_filter="ScaleN" \
--benchmark_out=bench_results/ptn_bench_scale.json \
--benchmark_out_format=json

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.x"

- name: Install plotting dependency
run: python -m pip install matplotlib

- name: Generate benchmark charts
run: |
python scripts/bench_single_report.py \
--input bench_results/ptn_bench.json \
--include "VariantMixed|VariantAltHot|VariantMixedGuarded|VariantAltHotGuarded|ProtocolRouter|CommandParser|PacketMixed|PacketMixedHeavyBind" \
--outdir docs/assets/bench \
--prefix latest \
--title "Patternia vs Standard C++"
python scripts/bench_single_report.py \
--input bench_results/ptn_bench_scale.json \
--include "ScaleN" \
--outdir docs/assets/bench \
--prefix scale \
--title "Variant Dispatch Scalability (4-32 alternatives)"

- name: Commit and push chart update
run: |
if git diff --quiet -- docs/assets/bench/; then
echo "No chart changes."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add docs/assets/bench/
git commit -m "ci: update benchmark summary chart"
git push
39 changes: 39 additions & 0 deletions .github/workflows/ci-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,42 @@ jobs:
if: always()
run: |
echo "Compile-time benchmarks built successfully."

benchmark:
name: Benchmark (informational)
runs-on: ubuntu-latest
# Informational only — does not block PR merge.
# Benchmark failures surface as artifacts for review, not as red X.
continue-on-error: true

steps:
- uses: actions/checkout@v4

- name: Configure
run: |
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DPTN_BUILD_TESTS=OFF \
-DPTN_BUILD_BENCHMARKS=ON \
-DPTN_SKIP_COMPILER_CHECK=ON

- name: Build
run: cmake --build build --target ptn_bench ptn_bench_scale --parallel

- name: Run benchmarks
run: |
mkdir -p bench_results
build/bench/ptn_bench \
--benchmark_filter="VariantMixed|VariantAltHot|VariantMixedGuarded|VariantAltHotGuarded|ProtocolRouter|CommandParser|PacketMixed|PacketMixedHeavyBind" \
--benchmark_out=bench_results/ptn_bench.json \
--benchmark_out_format=json
build/bench/ptn_bench_scale \
--benchmark_filter="ScaleN" \
--benchmark_out=bench_results/ptn_bench_scale.json \
--benchmark_out_format=json

- name: Upload benchmark results
uses: actions/upload-artifact@v4
with:
name: benchmark-results-${{ github.event.pull_request.number || github.sha }}
path: bench_results/
89 changes: 11 additions & 78 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,12 +79,20 @@ jobs:
run: cmake --build build --parallel

- name: Run Tests
run: ctest --test-dir build --output-on-failure
env:
# Give each test process its own raw profile file.
# %p = PID, %m = binary module name → avoids overwrites across
# the 150+ gtest_discover_tests processes.
LLVM_PROFILE_FILE: ${{ github.workspace }}/build/profile/%p-%m.profraw
run: |
mkdir -p build/profile
ctest --test-dir build --output-on-failure

- name: Capture coverage
run: |
# Merge raw profiles from all test runs.
find build -name '*.profraw' > profraw_list.txt
# Merge raw profiles from all test processes.
find build/profile -name '*.profraw' > profraw_list.txt
echo "Found $(wc -l < profraw_list.txt) raw profiles"
llvm-profdata-18 merge \
-input-files profraw_list.txt \
-output coverage.profdata \
Expand All @@ -111,78 +119,3 @@ jobs:
token: ${{ secrets.CODECOV_TOKEN }}
files: coverage.info
fail_ci_if_error: false

benchmark:
needs: build-and-test
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.BENCH_BOT_TOKEN || github.token }}

- name: Configure
run: |
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DPTN_BUILD_TESTS=OFF \
-DPTN_BUILD_BENCHMARKS=ON \
-DPTN_SKIP_COMPILER_CHECK=ON

- name: Build
run: cmake --build build --target ptn_bench ptn_bench_scale --parallel

- name: Run benchmarks
run: |
mkdir -p bench_results
build/bench/ptn_bench \
--benchmark_filter="VariantMixed|VariantAltHot|VariantMixedGuarded|VariantAltHotGuarded|ProtocolRouter|CommandParser|PacketMixed|PacketMixedHeavyBind" \
--benchmark_out=bench_results/ptn_bench.json \
--benchmark_out_format=json
build/bench/ptn_bench_scale \
--benchmark_filter="ScaleN" \
--benchmark_out=bench_results/ptn_bench_scale.json \
--benchmark_out_format=json

- name: Upload benchmark results
uses: actions/upload-artifact@v4
with:
name: benchmark-results-${{ github.sha }}
path: bench_results/

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.x"

- name: Install plotting dependency
run: python -m pip install matplotlib

- name: Generate benchmark chart
run: |
python scripts/bench_single_report.py \
--input bench_results/ptn_bench.json \
--include "VariantMixed|VariantAltHot|VariantMixedGuarded|VariantAltHotGuarded|ProtocolRouter|CommandParser|PacketMixed|PacketMixedHeavyBind" \
--outdir docs/assets/bench \
--prefix latest \
--title "Patternia vs Standard C++"
python scripts/bench_single_report.py \
--input bench_results/ptn_bench_scale.json \
--include "ScaleN" \
--outdir docs/assets/bench \
--prefix scale \
--title "Variant Dispatch Scalability (4-32 alternatives)"

- name: Commit and push chart update
run: |
if git diff --quiet -- docs/assets/bench/; then
echo "No chart changes."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add docs/assets/bench/
git commit -m "ci: update benchmark summary chart"
git push
Loading