[backfill: ClickHouse/clickhouse-connect] Inserting empty bytes b'' into non-nullable FixedString(N) fails

[alex-clickhouse]

Target client repo

ClickHouse/clickhouse-connect

Severity

sev:1 - visible DataError is raised (not silent corruption), and there are several trivial workarounds: pass an empty Python str ("") instead of empty bytes (b""), pre-pad with b"\x00" * N, or use a Nullable(FixedString(N)) column. Per the rubric, easy client-side workaround puts this in sev:1.

Description

The binary writer for FixedString(N) handles "empty value" inconsistently across the three input paths:

• String input (any value whose first element is a str, or write_format == "string"): empty string "" is encoded to b"", then zero-padded with dest.extend(empty[: sz - len(b)]). Works.
• Nullable + bytes input: the loop uses if not b: which is truthy on both None and b"", so empty bytes are written as sz zero bytes. Works.
• Non-nullable + bytes input (the buggy path): the loop has only if len(b) != sz: raise ctx.data_error(...), with no special case for an empty value, so b"" triggers DataError: Failed to write column 'val': Fixed String binary value does not match column size N.

Same behavior reported in clickhouse-java (binary stream stream.writeBytes("".getBytes()) for FixedString(10)). In clickhouse-connect the bug surfaces only on the non-nullable bytes path, because the string path was already fixed in #244 and the nullable bytes path coincidentally handles it via the if not b: check.

The ClickHouse server itself pads short FixedString values with \x00, so the natural expectation is that b"" should round-trip as b"\x00" * N, just like "" already does on the string path.

ClickHouse server version

26.4.2.10 (verified against the running server at http://localhost:8123).

Reproduction

import clickhouse_connect

client = clickhouse_connect.get_client(host="localhost", port=8123)
client.command("CREATE TABLE test_fs (id UInt32, val FixedString(10)) ENGINE=Memory")
client.command("CREATE TABLE test_fs_n (id UInt32, val Nullable(FixedString(10))) ENGINE=Memory")

# String path - works (empty string accepted, server pads with \x00)
client.insert("test_fs", [[1, ""]], column_names=["id", "val"])

# Nullable + bytes path - works (handled by `if not b:`)
client.insert("test_fs_n", [[2, b""]], column_names=["id", "val"])

# Non-nullable + bytes path - FAILS
client.insert("test_fs", [[3, b""]], column_names=["id", "val"])
# clickhouse_connect.driver.exceptions.DataError:
# Failed to write column 'val': Fixed String binary value  does not match column size 10

Expected: row 3 inserts successfully with val = b"\x00" * 10, matching both the string path and the nullable path.
Actual: DataError raised before any bytes are sent to the server.

Suggested fix (optional)

clickhouse_connect/datatypes/string.py, in FixedString._write_column_binary, non-nullable bytes branch (the final else: near line 129):

else:
    for b in column:
        if not b:
            ext(empty)
        elif len(b) != sz:
            raise ctx.data_error(...)
        else:
            ext(b)

This mirrors the nullable branch directly above it (lines 121-128) and the string branch's zero-padding behavior, and is consistent with how the ClickHouse server itself pads short FixedString writes.

Dedup search performed

• gh issue list --repo ClickHouse/clickhouse-connect --search "FixedString empty" --state all --limit 20 -> 1 result (Inserting an empty string to a FixedString column fails #244, about empty STRING, fixed in 2023)
• gh issue list --repo ClickHouse/clickhouse-connect --search "FixedString bytes" --state all --limit 20 -> 3 unrelated results (Query with column_formats set on FixedString columns should convert to pandas.StringDtype  #356, FixedString values are returned as bytes #325, Writing bytes to String column fails (INTERNAL EXCEPTION WHILE SERIALIZING) #148)
• gh pr list --repo ClickHouse/clickhouse-connect --search "FixedString empty" --state all --limit 20 -> 0
• gh issue list --repo ClickHouse/integrations-ai-playground --label backfill --label "target:clickhouse-connect" --search "FixedString" --state all --limit 20 -> 0
• gh pr list --repo ClickHouse/integrations-ai-playground --search "FixedString" --state all --limit 20 -> 0

The 2023 fix for empty string ("") did not extend to the empty bytes (b"") non-nullable path.

Source bug

ClickHouse/clickhouse-java#662