Skip to content

refactor(benchmark): expose existing offline metric aggregation - #183

Draft
kaveh8866 wants to merge 3 commits into
mainfrom
refactor/semantic-core-metric-aggregation
Draft

kaveh8866 wants to merge 3 commits into
mainfrom
refactor/semantic-core-metric-aggregation

Conversation

@kaveh8866

@kaveh8866 kaveh8866 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

1. Contribution Path & Summary

  • Fast contribution path
  • Core change path — CORE_CHANGE_PATH

Expose the existing offline Semantic Core aggregation as semanticCoreMetricsFor and isolate returned metric identities and benchmark bindings, including nested benchmark identities, from shared canonical objects. Previously, mutation through a returned result could affect later results or canonical definitions. Each result now receives independent identity objects.

Type: fix / refactor. Domain: Benchmark / Security / Core API. Branch: refactor/semantic-core-metric-aggregation.

Core classification applies because the change repairs a canonical-identity trust boundary and intentionally changes object-reference behavior. Serialized metrics, formulas, eligibility, missingness, numerators, denominators, metric versions, evaluator identity and provenance-reference format remain unchanged.

Related historical scope: Issue #165 and Draft PR #166. API contract: Semantic Core pilot guide. These references do not establish human approval of this repair. Maintainers must confirm the applicable Core/RFC process; no accepted repair-specific RFC is asserted.

2. Tests & Verification

Current synchronized head: cf920d66873cff992daab0314881f9a4795eb664. API repair head before synchronization: 0be6214ff0fdb54c757392f13f5ff1b18978d1dd.

Current exact-head GitHub CI: PENDING — 8 required contexts passed, 0 failed, 1 pending; observed 2026-10-08T22:12:43.068Z.

Documentation Site: completed / success.
Security: completed / success.
CI: in_progress / pending.

Earlier checks on the pre-synchronization repair head were all SUCCESS. They are historical evidence and are not reused as CI for the new head.

Post-security isolated integration reviewed against main 24c885c0cfc46bd3eb35fc38189397683b47ff59. Conflict-free combined Git tree: 0fb29a78788dd2ec4473669b22365f30152d087e; the synchronized commit has exactly this tree. Current protected main was integrated with a normal non-force merge, normal repository hooks, and one branch push; no merge into main occurred.

Combined-source local results:

  • pnpm test: 1,526 passed, 36 skipped, 0 failed; 238 files passed, 10 skipped. Skips remain skips; this is not a claim that every test executed.
  • Full Python pytest: 40 passed, 0 failed, 0 skipped each on Python 3.10.11, 3.11.9 and 3.12.13. Wheel and sdist builds passed on all three.
  • Changeset status, format, lint, typecheck, package boundaries, IP policy and workspace build passed. Lint: zero errors, 67 warnings.
  • SDK tests: 6 passed; reference workflow: 3; shared contracts: 3; product contracts: 8; security suite: 16; security/credential boundaries: 7; architecture boundaries: 3; documentation tests: 4. These subsets overlap the full suite and must not be added to its total.
  • Python 3.12 JSON Schema parity: 54 assertions, four schemas, 18 cases, three validators. Canonical JSON: 27 vectors, 40 byte checks, 60 digest checks, zero divergences.
  • Twelve independent deterministic API scenarios reproduced pre-repair serialized output exactly; canonical, sibling and cross-call shared object references were zero. Mutation and no-input-mutation checks passed; harness fetch calls: zero.
  • Doctor, preflight, connector registry, deterministic local smoke, documentation build and validation passed.

Local environment: Windows, Node 24.20.0, pnpm 11.11.0. CI uses Linux/Node 22. The local installation used the frozen lockfile, offline cache and --ignore-scripts; it is not an identical CI environment. Python build retries followed installation of the declared Hatchling backend in the isolated environments; initial missing-backend failures are retained in the operational report. No source fix was needed.

3. Compatibility & Cross-Language Parity

Existing five S03 metrics and serialized results remain compatible. No new schema, metric, benchmark version, Python API, evaluator or transport was added. The benchmark package remains private and the helper remains an internal experimental API.

Trusted-input contract: callers supply already evaluated case records and attest packDigest. Aggregation does not independently validate schemas, identities, uniqueness, digest correctness or provenance. It does not reject untrusted data as a newly validated boundary. Callers requiring verification must validate before invoking this helper. Supported input is ordinary scalar case data; arbitrary object graphs/getters are outside this contract.

4. Security & Privacy Impact

Returned identity/binding objects are detached by explicit structural copies; shared canonical definitions are not frozen. This addresses mutation through results, not direct mutation of separately exported canonical constants. Future nested identity fields require renewed isolation review.

No provider calls, credentials, live experiments, transport changes or empirical evidence imports. Security baseline files from merged PR #182 are preserved exactly, including pnpm 11.11.0. Path sanitization is not a new aggregation guarantee; this helper accepts trusted evaluated data and performs no path-based verification.

5. Scientific Methodology & Behavioral Evidence Impact

qualificationOutcome = NOT_QUALIFIED; public maturity BM2 / UNVALIDATED_PROXY / NOT_PROMOTED; scientificAuthority = NONE.

Frozen historical evidence remains unchanged, including the four HTTP 429 infrastructure observations. No qualification recovery, empirical rerun, scientific validation, BM3/BM4/BM5 promotion, leaderboard, Cyber or release. Local regression demonstrates bounded engineering compatibility, not independent scientific replication.

6. Documentation & Guides

The existing pilot guide, function documentation and patch changeset disclose output isolation and the trusted-input contract. Operational integration reports remain outside Git.

7. Migration Impact & Breaking Changes

Intentional API behavior change: reference equality/aliasing with canonical definitions and between results is no longer preserved. Callers must compare identity field values rather than rely on shared object identity or mutate outputs to modify canonical definitions. Outputs remain mutable; immutability is not promised.

No serialized-contract or metric-mathematics breaking change. No new stable public API guarantee is introduced. The changeset accurately records the reference-behavior change.

8. Review Readiness

Draft; independent @Logorythmus-org/semantiq-maintainers review remains pending. The protected ruleset requires one approval including CODEOWNER review, stale-review dismissal and resolved review threads. Current required CI is PENDING. Independent human review remains pending.

Technical integration is conflict-free and locally validated. CONTRIBUTING.md's up-to-date-branch requirement was addressed by the authorized synchronization with protected main 24c885c0cfc46bd3eb35fc38189397683b47ff59. Fresh CI is being evaluated on cf920d66873cff992daab0314881f9a4795eb664; previous approvals must not be carried across a changed head contrary to stale-review protection.

This description update and the bounded branch synchronization were expressly authorized. No reviewer request, Ready transition, approval, merge into main or release is authorized. PR #166 remains Draft and unmerged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant