data: rebake osint-v3 — store GUID tiers in declared field order (hi/lo fix)#86
Conversation
…lo fix) The bake stored each 6x(8:8) tier as a little-endian u16, landing the two bytes of every hi:lo pair as (lo, hi). The field cards in crates/cockpit-server/src/osint_classview.rs (OSINT_SYSTEM_FIELDS 12, OSINT_PERSON_FIELDS 5) and the contract position law (field position i = facet byte i, hi:lo reading) read position i as facet byte i. The two disagreed by a byte swap within each tier pair. Decoding the pre-fix bake against the codebook's declared order landed only 16/611 GUID1 and 52/133 GUID2 values in-vocabulary; swapping each tier pair's two bytes raises that to 609/611 GUID1 and 133/133 GUID2. (The 52/133 GUID2 "before" counts the 5 McClelland fields 1-based ignoring the TWIG "_" padding byte; the stricter count that also requires the padding byte empty is 0/133 -> 133/133, because pre-fix the motive value sat in the padding slot, not its own position.) Per operator ruling the DATA is fixed, not the field declarations. Transform: scripts/osint_v3_rebake_hilo.py (checked in) — no in-repo baker exists (baked from the external aiwar-neo4j-harvest, not regenerable offline), so a deliberate one-shot swaps tier bytes 4..16 pairwise per GUID while leaving the classid u32 (0x07011000), row_id, header, and every non-GUID node field byte-identical (unswap(new) == old for all 611 rows). The .soa and osint_v3_nodes.json are rewritten consistently and cross-verified byte-for-byte; row count 611 and persons 133 unchanged. The script refuses to re-run on already-corrected data (a pairwise swap is its own inverse). The residual 2/611 GUID1 out-of-vocab after the fix are genuine codebook-vs-harvest gaps, not swap artifacts: DreamSecurity MLType=30 (vocab max 29) and Shoebox civicUse=17 (vocab max 16). McClelland indexing (GUID2 persons): the corrected tiers are 1-indexed against mcclelland_vocab (value v -> vocab[v-1]). Every field's values fall in 1..=len with observed max equal to the vocab length and no meaningful zeros; under 0-indexing 26/133 would overflow their arrays. The TWIG "_" padding byte is 0 for all 133 persons after the fix (pre-fix it carried the motive value). No code or tests changed: no Rust source reads these tier bytes (osint_gotham.rs bakes its own separate osint_scene.soa) and the osint_classview.rs tests assert only field labels/counts, with no hardcoded GUID bytes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jEwwaT5JZ5x8qWvcnaMYC
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 2 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f637227. Configure here.
| print(f" GUID2 in-vocab: as-stored {g2}/{npers} if-swapped {g2s}/{npers}") | ||
|
|
||
| if check_only: | ||
| return |
There was a problem hiding this comment.
Check mode skips SoA JSON sync
Medium Severity
With --check, the script validates the .soa header and row count, then reports in-vocabulary decode rates using only osint_v3_nodes.json. It never compares each record’s GUID bytes in osint_v3.soa to the matching guid1/guid2 hex fields, so a successful --check exit can happen while the binary and JSON disagree.
Reviewed by Cursor Bugbot for commit f637227. Configure here.
….json Cursor Bugbot (PR #86, Medium) found that --check validated the .soa header and row count and reported in-vocabulary decode rates from osint_v3_nodes.json only, never comparing each row's stored GUID bytes to the matching guid1/guid2 hex fields — so a --check could exit 0 while the binary and the JSON disagreed. The byte-for-byte cross-verify that already lived inline in the transform path is now factored into a shared cross_verify_soa_vs_nodes() helper (row_id agreement, GUID1/GUID2 full-hex agreement, null-guid2 all-zero, canonical classid prefix), which both --check and the transform path call so they cannot drift. In check mode it iterates every stride-36 row, and on any mismatch prints the first few offending rows (row id + field + soa/json hex) and exits non-zero; the existing decode-rate reporting is unchanged. Verified against the current rebaked data: --check reports "CROSS-VERIFY OK: 611 rows" and exits 0. Detection proven on an isolated scratchpad copy (real files untouched) by flipping one GUID1 byte at row 42 — check printed "CROSS-VERIFY FAILED: 1 mismatch ... row_index=42 row_id=42 field=guid1" and exited 1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jEwwaT5JZ5x8qWvcnaMYC
|
Bugbot finding addressed in Proof both ways:
Generated by Claude Code |
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jEwwaT5JZ5x8qWvcnaMYC
2 participants
Resolves the byte-order mismatch flagged in #84 (operator-approved rebake): the V3 bake stored u16 GUID tiers little-endian
(lo,hi)while the field declarations (OSINT_SYSTEM_FIELDS/OSINT_PERSON_FIELDSand the contract law field position i = facet byte i) read(hi,lo). The data is now stored in declared field order — no code changes needed or made.Decode rates (measured, no swap, against the codebook)
The finding from #84 reproduced exactly before transforming. A second independent smoking gun surfaced: pre-fix, the person motive value sat in the TWIG
_padding byte; post-fix the padding byte is 0 for all 133 persons.Method
No in-repo baker exists (
data/osint-v3/comes from the externalaiwar-neo4j-harvest; the in-repoosint-bakecrate produces a different artifact). One-shot transform checked in atscripts/osint_v3_rebake_hilo.py: swaps tier bytes 4..16 pairwise per GUID, leaves classid (0x07011000)/row_id/header byte-identical, rewrites.soa+nodes.jsonconsistently, cross-verifies byte-for-byte, and refuses to re-run on already-corrected data (pairwise swap is self-inverse).Verification beyond the script's own asserts: header/stride/len intact (3/36/22012); 0
.soa↔nodes.jsonmismatches;unswap(new)==oldfor all 611 rows; non-GUID fields identical; 611 rows / 133 persons unchanged. Independent spot-probe (orchestrator):LiorDivguid2[4,1,2,3,2,0,…]→ stage=IV, need=nPow, receptor=STATUS, rubicon=ACTIONAL, motive=STATUS, padding=0 — all in-vocab.Findings recorded in the bake README (dated note)
osint_classview.rswithvocab[v-1].DreamSecurityMLType=30 (vocab max 29),ShoeboxcivicUse=17 (max 16).With this merged, the mechanical
OSINT_DECODEin the cockpit card starts hitting ~609/611 automatically — analyst-readable values in the node detail popup instead of raw bytes.🤖 Generated with Claude Code
Generated by Claude Code