research: publish canonical pilot v1 dataset and analysis

[ErenAri]

Summary

Publish the first successful prospective BPFCompat pilot collection as an
immutable repository dataset snapshot, then generate descriptive RQ1–RQ4 tables
from that snapshot under CI.

Canonical collection

• GitHub Actions run: 35393833464
• source commit: 5de550e5cc8c659872a90fc262eb0250c229bf23
• Actions artifact: 10566859006
• artifact digest:
  sha256:7aae7b5c099739ef0bcad74fbb3731aaafb1177f1ba128020729b08803610910
• planned/observed executions: 70/70
• outcomes: 50 compatible / 13 incompatible / 7 inconclusive
• collection_complete: true
• fully_evaluable: false

The seven inconclusive executions are the frozen
oracle-linux-9-uek7-5.15 logical profile. The vendor image actually booted
6.12 UEK, so those executions retain the exact observed environment but are not
claimed as evidence about requested 5.15 compatibility.

Dataset publication

Adds research/data/v1/ with:

• execution provenance from the canonical run;
• verbatim normalized collection/environment outputs;
• lossless per-case JSONL execution shards;
• SHA-256 for every raw BPFCompat report;
• SHA-256 for every committed processed dataset file;
• a dataset manifest binding the repository snapshot to the successful Actions
  artifact and canonical executions.jsonl digest.

Raw reports are not duplicated into Git history. Their exact hashes and paths
are committed; the content-addressed full Actions artifact remains staging
evidence to copy into the later DOI archive.

Deterministic analysis

Adds scripts/research/analyze-study-v1.py and generated RQ1–RQ4 outputs under
research/analysis/v1/. test-analysis-v1.sh regenerates the outputs,
compares them byte-for-byte, checks dataset SHA-256s, and asserts key collection
invariants.

The main research preflight now runs that reproduction gate whenever the
dataset, analysis, or analysis scripts change.

Descriptive pilot findings

RQ1: For the controlled BPF_MAP_TYPE_RINGBUF probe, the simple upstream
Linux 5.8 version threshold agrees with 8/9 conclusive pilot observations
(88.889%). AlmaLinux 8's observed 4.18 kernel is the below-threshold compatible
exception, consistent with a vendor backport. The 5.8 threshold is bound to
Linux commit 457f44363a8894135c85b7a9afd2bd8196db24ab.

RQ2: Outside the calibration case there are 4 incompatible observations
among 54 evaluable executions: 2 UNSUPPORTED_MAP_TYPE and 2
COMMAND_VALIDATION_FAILURE. Calibration remains separately reported and is
never mixed into field-failure prevalence.

RQ3: The paired Cilium-derived paths have 0 verdict disagreements across 9
conclusive exact environments. This is intentionally not interpreted as a
pure loader-effect estimate because libbpf exercises load+attach while the
cilium/ebpf path is load-only.

RQ4: The pilot records 10 exact environments and two cross-vendor
version inversions where an older-numbered kernel passes while a newer-numbered
kernel fails. Patch-level longitudinal change is not estimable in v1 because
there is only one exact environment per logical profile.

These are selected, stratified x86_64 pilot results, not population estimates
for Linux deployments.

Still pending

• representative repeat-run stability sample;
• license/redistribution review for archival contents;
• final manuscript figures/tables;
• research-tagged release and DOI archive.

Summary by CodeRabbit

• New Features

  • Published the BPF compatibility pilot v1 dataset containing 70 execution records across seven cases and ten Linux environment profiles.
  • Added deterministic analysis outputs for descriptive RQ1–RQ4 results, including case outcomes and failure classifications.
  • Added integrity metadata, provenance records, report checksums, and reproducibility safeguards for the dataset and analysis.

• Documentation

  • Updated research documentation with collection status, dataset limitations, analysis methods, and roadmap progress.
  • Documented inconclusive Oracle Linux observations caused by a kernel-profile mismatch.

• Tests

  • Added automated checks that validate dataset integrity and reproduce analysis results exactly.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 33ee69c1-a2ab-4156-ace7-26747d08c7e8

📥 Commits

Reviewing files that changed from the base of the PR and between 5de550e and 04e54c4.

⛔ Files ignored due to path filters (8)

• research/analysis/v1/generated/RESULTS.md is excluded by !**/generated/**
• research/analysis/v1/generated/analysis-summary.json is excluded by !**/generated/**
• research/analysis/v1/generated/case-outcomes.csv is excluded by !**/*.csv, !**/generated/**
• research/analysis/v1/generated/failure-taxonomy.csv is excluded by !**/*.csv, !**/generated/**
• research/analysis/v1/generated/rq1-ringbuf-version.csv is excluded by !**/*.csv, !**/generated/**
• research/analysis/v1/generated/rq3-loader-pairs.csv is excluded by !**/*.csv, !**/generated/**
• research/analysis/v1/generated/rq4-environments.csv is excluded by !**/*.csv, !**/generated/**
• research/analysis/v1/generated/rq4-version-inversions.csv is excluded by !**/*.csv, !**/generated/**

📒 Files selected for processing (23)

• .github/workflows/research-pilot-v1.yml
• CHANGELOG.md
• research/README.md
• research/ROADMAP.md
• research/analysis/v1/README.md
• research/analysis/v1/analysis-plan.json
• research/corpus/v1/README.md
• research/data/v1/README.md
• research/data/v1/SHA256SUMS
• research/data/v1/dataset-manifest.json
• research/data/v1/execution-provenance.json
• research/data/v1/processed/collection-summary.json
• research/data/v1/processed/exact-environments.json
• research/data/v1/processed/executions/cilium-tracepoint-ebpf-go.jsonl
• research/data/v1/processed/executions/cilium-tracepoint-libbpf.jsonl
• research/data/v1/processed/executions/core-relocation-fail-libbpf.jsonl
• research/data/v1/processed/executions/falco-modern-bpf-scap-open.jsonl
• research/data/v1/processed/executions/perfbuf-fallback-libbpf.jsonl
• research/data/v1/processed/executions/ringbuf-modern-libbpf.jsonl
• research/data/v1/processed/executions/simple-pass-libbpf.jsonl
• research/data/v1/raw-report-checksums.json
• scripts/research/analyze-study-v1.py
• scripts/research/test-analysis-v1.sh

 _____________________________________________________________
< Tabs vs spaces? You somehow chose violence *and* confusion. >
 -------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.