Skip to content

feat(readiness): add reproducible validation and security evidence - #323

Draft
Bakar404 wants to merge 2 commits into
mainfrom
codex/ai-readiness-improvements
Draft

Bakar404 wants to merge 2 commits into
mainfrom
codex/ai-readiness-improvements

Conversation

@Bakar404

@Bakar404 Bakar404 commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Description

Refs #317. Partial, non-closing repository-readiness contribution. Draft for hosted validation and independent review.

  • Enroll existing Node regression suites alongside the retained migration, Rust and web gates; add mutation-free previews, fail-fast JSON receipts, observed test counts and source-change invalidation.
  • Pin Node/Rust toolchains and CI Actions; add optional local hooks and module guidance.
  • Check and narrowly regenerate the documented command/toolchain contract; test Windows EOL handling and the actual CLI failure/preview behavior.
  • Add checksum-pinned, redacted Gitleaks scanning plus a synthetic rejection control; remove the committed example master-key value without touching deployed secrets.
  • Add native Dependabot configuration and CODEOWNERS routing. These are configuration changes, not claims of hosted operation or required native review enforcement.
  • Preserve existing GitHub CodeQL default setup (Actions, JavaScript/TypeScript, Python and Rust). Publication review removed a conflicting JavaScript-only advanced job; no native CodeQL setting was disabled.
  • Preserve benchmark history, validation receipts and a readiness-focused review of the open PRs.

Source base: 0b1ad59da398e3dbd6a696d0264bcb6ebd620219 (main).
Branch: codex/ai-readiness-improvements -> main.
Scope: validation/CI/security configuration, contributor guidance and evidence. No application behavior, schema migration, provider SDK/runtime or ECorp execution authority is changed.

Harness first: use native Node test discovery, Cargo, pnpm, GitHub Actions/Dependabot/default CodeQL, and existing repository commands. The small driver adds source-bound receipts and a consistent entry point; it is not an agent scheduler, approval system or autonomous repair engine.

Coordination and remaining #317 scope

Type of change

  • Test/tooling and non-breaking validation improvements
  • Security/CI configuration
  • Documentation and evidence
  • Breaking application change

How Has This Been Tested?

The retained full local gate before publication follow-up passed 1035 Node tests, 554 Rust tests, zero failures; 343 Rust tests were explicitly ignored, not counted as passes. Migration, docs, formatting, Clippy, web build and web lint passed with no source change during that invocation.

Publication follow-up adds a review-skill discovery regression and removes the duplicate CodeQL job:

  • Focused helper/black-box CLI tests: 15 passed.
  • Four YAML configurations parsed successfully.
  • Gitleaks source scan: no findings after reviewed remediation/exceptions; synthetic credential-shaped control is rejected and redacted, including inside an otherwise allowlisted artifact.
  • Fresh broad Node run: 1035 passed, 0 failed, 1 cancelled. The existing Teams SDK authentication/startup test exceeded its 15-second limit. Previous complete passes and focused passes are retained, but do not erase this intermittent failure. No test was removed or timeout relaxed.
  • The six required repository gates were rerun separately after publication corrections: 41 migration checks, formatting, Clippy, 554 Rust tests (343 ignored), web build and web lint passed. This does not erase the separate broad Node cancellation above.

Reproduce from a trusted isolated worktree:

pnpm install --frozen-lockfile
npm ci --prefix scenarios/repo-steward --ignore-scripts --no-audit --no-fund
pnpm check:preview
pnpm check

The host's npmjs.org TLS/registry-identity issue remains explicit in docs/AI_READINESS.md; installed-dependency checks are not clean-bootstrap proof. TLS and supply-chain policies were not disabled. All existing dependency versions are retained; the previously undeclared ws@8.18.3 test dependency is the only root dependency addition.

Test Configuration:

  • Firmware: N/A, contributor tooling only.
  • Hardware/OS: Windows ARM64; no retained server, runner or database was restarted.
  • Toolchain: Node 24.19.0, pnpm 11.19.0, Rust/Cargo 1.98.1, Gitleaks 8.30.1.
  • Product Copilot SDK/runtime: unchanged.
  • Benchmark: hash-pinned CodeBlend distribution, approved Windows ARM64/x64 exception, isolated Copilot CLI 1.0.80-0, three valid judges in the completed runs.

Benchmark interpretation

The local development sequence was 54.2 -> 60.0 -> 67.1 -> 67.1, with final pre-publication foundations 87.0 and operations 51.75. These are historical local dirty-tree assessments with preserved run identities, not a new benchmark of this published commit. Both axes must reach 80 for AI-Ready; that verdict remains no. The CodeQL publication correction and any future native settings changes require fresh measurement before a new score claim.

Details: docs/AI_READINESS.md, docs/VALIDATION.md, docs/reports/ai-readiness-validation-iteration-*.json, and docs/reports/github-readiness-review-2026-09-18.md.

Checklist

  • Source base, scope and non-closing issue relationship documented
  • Author self-review; native CodeQL conflict corrected before publication
  • Regression tests and explicit failure evidence included
  • Ignored, cancelled, unexecuted and historical evidence distinguished
  • Existing source checkout, runner identity, databases and other worktrees preserved
  • Consistently green fresh broad Node gate, including the intermittent Teams SDK case
  • Hosted checks on the published head verified
  • Required independent/human/team approval
  • Template screenshot evidence completed where required; no browser/application screenshots are fabricated for this tooling-only change

Repository settings were applied only after the separate exact-preview approval; the verified follow-up is recorded below. No merge, auto-merge, deployment, live credential rotation or policy bypass is authorized by this PR.

Hosted publication checkpoint

At published head 6119296d475e94a45b643534a3a0bbd581b130f2, the new hosted secret scan passed. Native CodeQL Actions/JavaScript/Python checks, all three runner-platform jobs, and the Windows desktop job passed at inspection; Rust analysis and integration were still running.

The hosted quality job failed after the newly enrolled Node suite: 939 passed, 62 failed, 5 skipped, 0 cancelled. All 62 failures point to tools/e2e_checkpoint_verification.test.mjs with unsafe_posix_path: its Windows-shaped offline fixtures reach expectedWorkspace using the Linux host's default path implementation. This is different from the intermittent local Teams timeout. Exact failing job.

This original-head failure prompted the approved fixture/driver path-semantics repair recorded below. No suite was skipped, unsafe path accepted or gate waived. The generated failed validation receipt remains retained. The PR remains draft pending exact-head hosted validation and independent review.

Approved follow-up at 944dcbe

Published head: 944dcbe1c95c7fece0cc57fb29bc3f90c371f31d.

  • Make checkpoint assessment and offline suite path semantics explicit using native Node path.win32/path.posix. Native execution retains its host default; containment helpers, authority checks and live runtime configuration are unchanged.
  • Add Windows/POSIX safety regressions. Focused checkpoint/stopped-source suites: 266 passed, zero failed/cancelled/skipped. Four new negative-control regressions fail against the unfixed 6119296 driver and pass with the repair.
  • Final serialized pnpm check: all eight gates passed, 1054 Node passed, 554 Rust passed, 343 Rust ignored, no source change during validation and no unexecuted gates. Migration count: 41. Formatting, Clippy, docs, web build and web lint passed.
  • Preserve the earlier broad run with 1053 Node passed and one Teams SDK cancellation. The unchanged Teams suite passed 18/18 in isolation. A later green run does not erase this intermittent limitation.
  • Enable and read back the approved native secret-scanning/push-protection/Dependabot settings and main-protection checks/review requirements. Other rulesets, native CodeQL and its thresholds remain unchanged; auto-merge remains disabled. The new secrets job is not yet a main-wide required check.

Follow-up evidence and unaltered full validation receipt retain exact source fingerprints. The final summary paragraph/receipt were reporting-only additions after validation; the validated code/test files were unchanged.

New exact-head hosted CI is running; it is not yet claimed green. Independent review is still required. No benchmark was rerun, so no new readiness score is claimed.

Enabling dependency alerts surfaced Dependabot alert #1, a moderate glib advisory in the desktop Cargo lockfile. Dependency compatibility/remediation is a separate follow-up; this scoped repair does not silently upgrade desktop dependencies.

Copilot AI balanced review requested due to automatic review settings September 18, 2026 05:32

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings September 18, 2026 05:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce repository regressions and documentation contracts in reviewed maintenance

2 participants