Repository navigation
Conversation
Reuse sorted MAAD entries for HHI, top address shares, and fixed-prefix entropy. Recompute rollups from merged address weights and preserve MAAD scope skipping and minimum counts. Version the product contract and include concentration rows in SQLite/D1 schemas, verification, comparison, shard merging, and window exports. Cover both address families, zero weights, rollups, and contract rejection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
🤖 Claude Opus 5.5 on behalf of Oliver. The code was written by an agent and has not had human review. It changes the result contract, so products built with it need fresh database paths.
Problem
netflow-predictor found that packet- and byte-weighted MAAD measures of external address sets carry forecasting information about near-term campus traffic. Reviewers will ask whether ordinary heavy-hitter concentration explains that. The pipeline stores no single-scale concentration measure to test against.
Change
address_concentration_statsstores conventional concentration controls for every address set and measure that MAAD computes. It uses the same keys asaddress_maad_stats.hhi: sum of squared weight shares.top1_share,top10_share,top100_share: share of weight in the heaviest addresses.entropy_p8/p16/p24/p32for IPv4 andentropy_p32/p48/p64/p128for IPv6: Shannon entropy in bits after summing weights by fixed-length prefix.weight_totalandentry_count, for auditing.Concentration uses MAAD's per-address weights, zero-weight exclusion, internal-side skipping, and minimum count of two. Rollups recompute from merged per-address child maps and never average child statistics. The computation reuses MAAD's sorted entries, with no extra flow scan or full weight sort.
The schema and result configuration record the contract.
verify,compare,merge-shards, analysis exports, and the D1 schema handle the new table.docs/code/pipeline-contract.mdhas a new "Concentration statistics" section.Validation
bun run format,format:check,lint, andtypecheckpass.verifyproduce 216 concentration rows.Rollout
This branch will be built on pinot and run across the cluster for 2026-01-01 through 2026-06-30, for the full campus product and both
daily_active_sourcessubsets, before merging. The new products go to fresh paths and do not touch existing databases.Code by GPT-6.1-Sol (Codex) and description by Claude Opus 5.5 (Claude Code), coordinated in T3 Code.
🤖 Generated with Claude Code