Follow-ups from the 2026-09-02 session-replay pin work (charts#15090). Full pins close the cross-batch inference crashes for known fields, but three gaps remain in arrow_converter.py / main.py. Both adversarial review passes verified these empirically.
1. Uncaught OverflowError on integers ≥ 2^63 (poison-batch crash-loop)
A JSON integer in [2^63, ~1e21) raises OverflowError at pa.array() / from_pylist (arrow_converter.py:90, :354). Only ArrowInvalid/ArrowTypeError are caught; OverflowError escapes to the blanket handler in main.py:594 → fatal → restart → same offsets → crash-loop. JS emits integer notation below 1e21, so e.g. a corrupt timestamp inflating inter_action_gap_sum_of_squares_ms lands exactly in this window. typedColumns pins do not help — the raise happens before coercion.
Fix: catch OverflowError alongside the Arrow exceptions in the per-value fallback paths so the value NULLs and increments errors_total{type="column_coercion"}.
2. _floatify_integers_in_float_fields only fires on inferred-float batches
An all-int batch infers int64, and casting int64 > 2^53 to a double-pinned column raises ArrowInvalid → per-value NULL (silent data loss with metric) instead of nearest-double conversion. Mixed batches take the floatify path and convert correctly — same value, different outcome by batch composition.
Fix: floatify integers for any column pinned double via MILLPOND_TYPED_COLUMNS regardless of the inferred batch schema.
3. No structural defense against future unpinned numeric fields
The session tables are fixed-column ClickHouse schemas, so new fields arrive only via DDL migration in posthog/posthog — but when one does, it lands unpinned and can reintroduce the int64-vs-string concat crash (exactly how inter_action_gap_sum_ms bit us: absent from 32K sampled records, present in prod).
Fix: on pa.concat_tables type mismatch, unify the conflicting column to string (and count it) instead of dying; the pinned path stays authoritative for declared columns.
Also related
- Whole-float-for-bigint is order-dependent: float-first batch casts 2.0→2 exactly; int-first mixed batch stringifies to "2.0" which NULLs on bigint coercion.
- millpond pyducklake >=1.0.18 bump is still pending (Leak A parity).
Follow-ups from the 2026-09-02 session-replay pin work (charts#15090). Full pins close the cross-batch inference crashes for known fields, but three gaps remain in
arrow_converter.py/main.py. Both adversarial review passes verified these empirically.1. Uncaught OverflowError on integers ≥ 2^63 (poison-batch crash-loop)
A JSON integer in [2^63, ~1e21) raises
OverflowErroratpa.array()/from_pylist(arrow_converter.py:90, :354). OnlyArrowInvalid/ArrowTypeErrorare caught;OverflowErrorescapes to the blanket handler inmain.py:594→ fatal → restart → same offsets → crash-loop. JS emits integer notation below 1e21, so e.g. a corrupt timestamp inflatinginter_action_gap_sum_of_squares_mslands exactly in this window. typedColumns pins do not help — the raise happens before coercion.Fix: catch
OverflowErroralongside the Arrow exceptions in the per-value fallback paths so the value NULLs and incrementserrors_total{type="column_coercion"}.2. _floatify_integers_in_float_fields only fires on inferred-float batches
An all-int batch infers int64, and casting int64 > 2^53 to a double-pinned column raises
ArrowInvalid→ per-value NULL (silent data loss with metric) instead of nearest-double conversion. Mixed batches take the floatify path and convert correctly — same value, different outcome by batch composition.Fix: floatify integers for any column pinned
doublevia MILLPOND_TYPED_COLUMNS regardless of the inferred batch schema.3. No structural defense against future unpinned numeric fields
The session tables are fixed-column ClickHouse schemas, so new fields arrive only via DDL migration in posthog/posthog — but when one does, it lands unpinned and can reintroduce the int64-vs-string concat crash (exactly how
inter_action_gap_sum_msbit us: absent from 32K sampled records, present in prod).Fix: on
pa.concat_tablestype mismatch, unify the conflicting column to string (and count it) instead of dying; the pinned path stays authoritative for declared columns.Also related