Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/AUTOGENERATION-PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ areas remain unfinished; they cannot disappear from the denominator.
| Shared binary strings | Merged in PR #747 at `8f0fdaf4`. It preserves raw bytes in saved state, distinguishes a one-byte default from a remainder string, and carries the bounded CameraInfo repair. | This is a shared capability, not a Canon migration. CameraInfo remains a legacy text-domain adapter, and no Canon manual reader has been removed. The prior full-pair supervisor-status limitation remains recorded. |
| Keyed-directory schema and compiler | Merged in PR #748 at `ebbe1ece`; all final hosted checks passed at `a422e8de`. | Native parent facts and expression declarations are checked. Shared reporting policy merged in #750; the inactive reader merged in #752 at `634e5616`. No production route is active. |
| Shared word-directory processor | Merged in PR #754 at `1138a880`; nine tables, 132 rows, 698 Python tests and all five hosted jobs pass. | Four of five unsupported child processors now have generated descriptors. Canon production routing and manual-reader retirement remain unfinished. |
| Shared serial processor | Native probe and inventory merged in #755/#756. The next combined checkpoint compiles eight generated tables, accounts for 106 emitted alternatives and 26 omissions, and passes Real AudioV3/V4 native/Rust replay. | Final hosted acceptance and merge remain. A separate V4 draft replaces its manual 31-slot sequence; carrier/corpus acceptance is still required before counting retirement. |
| Shared serial processor | Merged in #757 at `58849bc7`, after #755/#756. Eight tables account for 106 emitted alternatives and 26 omissions; all five hosted checks pass. | The V4 migration replaces its manual 31-slot sequence and passes bounded native output plus source-change execution proof. Full corpus and hosted acceptance remain before counting retirement. |
| Recorded source inventory | Merged in PR #749 at `18a8ef17`; all final hosted checks passed at `72e8e664`. The report accounts for 1,512 table identities and retains 119 tables with no named rows. | This establishes the captured source population. Classifying which rules are generated, manual, unsupported or unclassified remains open; source shape is not automation. |
| Sony plain generator recovery | PR #745 merged; six tables and 193 rows reproduced | These tables can be rebuilt. This alone does not prove that their behavior is fully automatic. |
| Sony enciphered recovery | Producer and independent verifier preserved; M4 review found five blockers; not landed | The draft still has a Sony-specific translation layer. Its review remains useful, but it is not the architecture target. |
Expand Down Expand Up @@ -151,8 +151,9 @@ Canonical artifacts and ledgers were regenerated successfully with local Perl
the native serial processor. The [native probe](reference/serial-processor-checkpoint.md)
is merged in #755. The reviewed JSON inventory captures all eight selected
tables and preserves every refusal. The combined Rust emitter and reader
compile and pass native Real AudioV3/V4 replay. Complete their final hosted
gate, then validate the V4 carrier and remove its manual sequence.
merged in #757 after native Real AudioV3/V4 replay and all final hosted
checks. The V4 carrier passes bounded/native and supported source-change
execution proof; complete its corpus and hosted gate, then land retirement.
Account for every condition and conversion before reducing the final
unsupported-child count from one to zero.
2. Resolve the four omitted parent rows using shared byte handling. An explicit
Expand Down
38 changes: 28 additions & 10 deletions docs/AUTOGENERATION-PROGRESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,11 @@ chronological callback trace. The combined checkpoint at `66c430c6` passes **37
failures and zero skips**, on both threaded and non-threaded Perl 5.38.2
against pinned ExifTool 13.59. The full hosted checks then passed for #756.

The shared serial reader and Rust emitter are now combined on
`codex/shared-serial-integration-20260913`, with pipeline checkpoint `9693ef9e`.
The shared serial reader and Rust emitter merged in PR #757 as `58849bc7`
at 12:47 UTC. All five hosted checks passed at `72fbebb1`, including 757
canonical Python tests with zero failures/skips in 641.913 seconds, 5,756
nextest tests passing (59 intentionally skipped), the complete Cargo test/doc
invocation, and the explicitly invoked native serial replay.
The [runtime checkpoint](reference/serial-runtime-checkpoint.md) records the
proof and its limits. All eight tables remain accounted for: **106 emitted
alternatives and 26 explicit omissions**, with zero independent verifier
Expand All @@ -86,14 +89,29 @@ local absolute invocation paths. Re-running its producer with the portable
invocation reproduces the committed ledger exactly. All 607 expressions agree
on 16,789 applicable comparisons; 14 inputs are inapplicable.

This checkpoint still needs its final hosted gate and merge. No production
route is enabled; **one unsupported Canon child processor, four omitted parent
rows and zero Canon manual readers retired** remain the production status.
The Real AudioV4 retirement draft is separately committed and published at
`118afac7`. Source review accepts its narrow behavior, but it still needs a
fresh Rust build, native output comparison, full-corpus pair and upgrade-flow
proof. Its metadata occurrence API retains the existing group-0/group-2
limitations; native warning output and V3/V5 activation also remain unfinished.
That merged checkpoint enabled no production caller. **One unsupported Canon
child processor, four omitted parent rows and zero Canon manual readers
retired** remain the Canon status.

The next [Real AudioV4 migration](reference/real-audio-v4-retirement.md) is
implemented on `codex/real-v4-retirement-integration-20260913`, runtime
`4f01db97` with additional regression assertions at `d3f51325`. Its generated
reader replaces the manual 31-slot sequence. All six carrier tests, full
Clippy and formatting pass. A fresh 19-file native/control/candidate comparison
has 15 scored cases matching native output and four explicitly retained scope
or diagnostic cases. Ten files correct existing legacy behavior: UTF-8 repair,
NUL truncation, or later fields invented after an incomplete string.

A supported source mutation also reaches actual output: changing AudioV4's
`Title` name to `UpgradeTitle` in a copied pinned source, regenerating and
compiling with the unchanged carrier, changes exactly that output key. No
tag-specific Python/Rust rule is edited. This proves that supported change;
it does not substitute for a real release upgrade or broader semantics.

The full 4,238-file paired comparison and hosted acceptance are required before
landing and counting the 31-slot retirement. The current occurrence API retains
its existing native group-0/group-2 limitations; native warning output and
V3/V5 activation remain unfinished.

The first additional retirement candidate is Real AudioV4's manually specified
31-entry sequence. It uses the same native serial processor and can prove reuse
Expand Down
2 changes: 1 addition & 1 deletion docs/UPGRADE-NEXT-STEPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ populations. They are implementation history, not work to restart.

| Priority | Remaining work | State | Completion evidence |
| --- | --- | --- | --- |
| 1 | Extend the source inventory and complete the next shared directory capability | Sony Tag202a migration/retirement merged in #746; source inventory and artifact join merged in #749/#751. Directory validation and nine word tables merged in #753/#754; canonical regeneration and all final gates pass. One child processor and four parent rows remain unsupported. Serial probe/inventory merged in #755/#756. Combined Rust emission/reading passes native V3/V4 replay; eight tables account for 106 emitted alternatives and 26 omissions. Final hosted gate is pending. Real AudioV4 retirement is published separately and still needs carrier/corpus/upgrade-flow acceptance. | Classify runtime/manual rules against the recorded inventory; resolve named processor blockers, verify both carriers, then retire the duplicate readers. Details in the [scoreboard](./AUTOGENERATION-PROGRESS.md). |
| 1 | Extend the source inventory and complete the next shared directory capability | Sony Tag202a migration/retirement merged in #746; source inventory and artifact join merged in #749/#751. Directory validation and nine word tables merged in #753/#754; canonical regeneration and all final gates pass. One child processor and four parent rows remain unsupported. Serial probe/inventory/runtime merged in #755–#757; eight tables account for 106 emitted alternatives and 26 omissions. Real AudioV4 migration passes six carrier tests, bounded native output and supported source-change execution proof. Complete its paired corpus and hosted acceptance before landing the 31-slot retirement. | Classify runtime/manual rules against the recorded inventory; resolve named processor blockers, verify both carriers, then retire the duplicate readers. Details in the [scoreboard](./AUTOGENERATION-PROGRESS.md). |
| 2 | Continue generated EXIF directory migration | IFD1, InteropIFD and ExifIFD E-2 landed; Claude's handoff reserves E-3, fallback retirement, IFD0 and numeric-coercion work | Refresh Claude's ownership before taking work; preserve occurrence behavior and measure each activation against its own control |
| 3 | Join producer and upgrade accounting | Source-to-artifact join merged; runtime and manually maintained rule classification remain open | Generated facts, manual rules, unsupported rules and actual execution remain distinct; ordinary source changes need no new tag rules |
| 4 | Preserve Sony/Nikon producer recovery evidence | Sony plain landed; enciphered/encrypted recovery preserved and unlanded | Recovery is labeled maintenance; no numbering-only reconstruction or duplicate per-vendor interpreter is promoted as the target |
Expand Down
73 changes: 73 additions & 0 deletions docs/reference/real-audio-v4-retirement.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Real AudioV4 manual-sequence retirement

The shared serial capability merged in #757 at `58849bc7`. This next change
uses it in the Real AudioV4 carrier and removes the manually copied 31-slot
field sequence. It adds no new tag-specific generator or interpreter.

## What changes

`real_audio.rs` retains signature/version selection and the native 512-byte
maximum body read. Generated `Real::AudioV4` supplies names, formats, field
order, dynamic lengths, visibility and groups to the common serial reader.
The carrier projects emitted rows through the existing metadata insertion API.

Fresh native comparisons also identify corrections to the old parser:

- Invalid UTF-8 uses native question-mark repair. This fixes Copyright on the
real corpus sample, not only synthetic input.
- Strings stop at NUL while consuming their full declared byte span.
- An incomplete Artist span stops the serial reader. The old cursor stayed in
place after the failed read and could invent Copyright and Comment from the
same bytes. Cases with 511, 512 and 600 body bytes prove the correction.

AudioV3/V5 activation, native warnings and complete occurrence groups remain
outside this migration. The current API derives stored group 0 from the visible
Real-RA4 key and cannot store group 2; this pre-existing limitation remains
explicit. The descriptor retains the native group facts. No new project
percentage or Canon retirement is claimed.

## Evidence before the full acceptance gate

Control is `58849bc7`; candidate runtime is `4f01db97`. `d3f51325` adds only
regression assertions. The six carrier unit tests, full all-feature Clippy and
formatting pass. The source builds reuse one coordinator-owned local cache:
control 1.486 seconds, candidate 7.443 seconds; the final six-test build/run is
61.502 seconds. Workers perform source/evidence work without Cargo.

The native carrier contract covers 19 fixtures against pinned ExifTool 13.59
and Perl 5.38.2. All 15 scored candidate projections match native Real-* output.
Ten fixtures improve legacy output; five are unchanged. Four pre-existing
scope/diagnostic cases remain recorded. Control/candidate exit codes agree.
Original failed assumptions that five control projections were already correct
are preserved alongside raw runs and the independently corrected contract.
These counts are bounded fixture evidence, not a full-corpus result.

A separate copied-source proof changes exactly one supported native property:
`Real::AudioV4[24].Name` from `Title` to `UpgradeTitle`. Fresh dump, generation,
independent verification and formatting change one generated Rust name. A
source archive compiled with that artifact and the unchanged carrier changes
exactly `Real-RA4:Title` to `Real-RA4:UpgradeTitle` on the real sample, retaining
its value and matching the modified native source. Compilation takes 15.461
seconds. Tag-specific Python/Rust edits: zero. This proves one supported source
change, not compatibility with an entire new release.

## Acceptance and retirement accounting

Before landing, require the complete paired 4,238-file census and final hosted
checks. The paired runner reuses `conformance.py` scoring, authenticates both
binary/build manifests and unchanged scoring/oracle helpers, probes DOCX oracle
capability, and preserves file hashes, raw outputs, exits and per-file results.
It caps concurrency at two and persists progress; interrupted or vacuous runs
cannot be reported as a pass. Record any pre-existing diagnostic exits apart
from new failures. Exact per-file changes, rather than equal totals alone,
determine acceptance.

After acceptance and squash merge, count one manual serial reader and its
31-slot sequence as retired. This is not 31 newly emitted tags: hidden fields
still advance the cursor, and visible output is separately measured. The
remaining Canon child processor and four parent omissions are separate work.

Evidence is under `$OXIDEX_WORK_EVIDENCE/shared-pilot/real-v4-integration-20260913/`:
`native-carrier-comparison.json`, `native-carrier-accepted.json`, build/test
records, `source-upgrade-proof/runtime-replay.json` and `full-pair-20260913/`.
Native fixture generation/contracts are in sibling `real-audio-retirement-20260913/`.
14 changes: 10 additions & 4 deletions docs/reference/serial-runtime-checkpoint.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ The base is merged PR #756 (`eb700430`), pinned to ExifTool 13.59. Runtime,
emitter and independent verifier are integrated at `9693ef9e` on
`codex/shared-serial-integration-20260913`. The earlier compiled checkpoint
`ff383b8e` is published. Its Rust source is unchanged by the later verifier and
pipeline additions. The final checkpoint still needs hosted acceptance.
pipeline additions. Final head `72fbebb1` passed all five hosted checks and
merged as `58849bc7` in PR #757 at 12:47 UTC on September 13. Canonical Python
passed 757 tests with zero failures/skips in 641.913 seconds; nextest passed
5,756 tests with 59 intentionally skipped. Full Cargo test/doc and explicit
serial native replay also passed.

Native inventory selects all tables whose processor is `ProcessSerialData`
before examining generated output. Eight tables contain 130 entries and 132
Expand Down Expand Up @@ -95,10 +99,12 @@ No i7 job or daemon change is part of this checkpoint.

## What remains

- Complete final hosted checks and merge this inactive runtime checkpoint.
- The inactive runtime checkpoint is merged. Complete acceptance of its first
production carrier, described in the [V4 retirement record](real-audio-v4-retirement.md).
- Validate the separately published AudioV4 carrier draft (`118afac7`) with a
fresh candidate/control, real and constructed files, source-change flow and
the full corpus, then remove its manual 31-slot sequence.
full corpus and hosted checks, then land its manual 31-slot retirement.
Fresh bounded candidate/control/native output and supported source-change
execution proof now pass.
- Preserve the carrier's known output-model gaps: native group 0/group 2 are
not stored by the current occurrence API; header-only native warning output
is absent; AudioV3/V5 are not activated. Do not call these complete parity.
Expand Down
Loading
Loading