Skip to content

Fix nullable type handling for column types#237

Merged
sert121 merged 5 commits into
mainfrom
yash/fix-nullable-types
Mar 26, 2026
Merged

Fix nullable type handling for column types#237
sert121 merged 5 commits into
mainfrom
yash/fix-nullable-types

Conversation

@sert121
Copy link
Copy Markdown
Contributor

@sert121 sert121 commented Mar 25, 2026

Summary

  • Previously only date/datetime types were wrapped in Nullable() when the source schema allowed nulls
  • Boolean, integer, float, and string columns with NULL values in the source DB were created as non-nullable in Clickhouse, silently converting NULLs to defaults (false, 0, empty string)
  • Now checks the JSON schema type array for "null" and wraps any non-primary-key column in Nullable() accordingly

Root cause

The to_sql_type() method in connectors.py had explicit Nullable() wrapping for DATE, TIMESTAMP, TIME, and DATETIME types, but no handling for BOOLEAN, INTEGER, FLOAT, VARCHAR, etc. The Singer SDK JSON schema represents nullable fields as {"type": ["boolean", "null"]} but this null indicator was ignored for non-date types.

Impact

Affects any source table with nullable boolean/integer/float/string columns. Confirmed on Outdoorsy's rentals table: 18 boolean columns all created as Bool instead of Nullable(Bool), with all source NULLs converted to false.

Test plan

  • Verify nullable bool column from Postgres creates Nullable(Bool) in Clickhouse
  • Verify non-nullable columns remain non-nullable
  • Verify primary key columns are not wrapped in Nullable

Ubuntu added 5 commits March 25, 2026 18:16
Previously only date/datetime types were wrapped in Nullable() when
the JSON schema included "null". This caused nullable boolean (and
other) columns from source databases to lose their NULL values —
NULLs were silently converted to default values (false for bools,
0 for ints, etc).

Now checks the JSON schema for "null" in the type array and wraps
any non-primary-key column in Nullable() accordingly.
@sert121 sert121 changed the title Fix nullable type handling for all column types Fix nullable type handling for column types Mar 25, 2026
@sert121 sert121 merged commit 876c77a into main Mar 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants