[EPIC] Timezone handling bugs #6335

What / Why

I audited how Comet handles timezones, starting from #2730. The model is simple and mostly sound. Spark's TimestampType is a UTC instant, so nothing is converted at the JVM/native boundary. Comet passes the raw microseconds in both directions and labels them Timestamp(Microsecond, "UTC"). TimestampNTZType is Timestamp(Microsecond, None). The session timezone never becomes part of a value. Each timezone-aware expression carries it, and it's applied inside the native kernel, or the expression runs through the codegen dispatcher with Spark's own timeZoneId. #6337 adds a contributor guide page that describes the model in more detail.

The bugs cluster where that model breaks down:

• native expressions that emit a TimestampType value with some other label
• session timezone IDs that the native parser can't read
• timezone rules that come from a different database than the JVM's

Label drift is easy to miss. The scan and shuffle boundaries cast every column back to its declared type, so a test that only projects the result passes. It shows up when the result is compared, goes through a CASE, or feeds another native expression.

All of the new bugs below reproduce on main at 764936187, on Spark 3.5 and 4.1.

Bugs

• Native timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so downstream expressions ignore the session timezone #6328 timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so hour, casts and comparisons over it are wrong or fail in non-UTC sessions (critical). Fixed by fix: label native timestamp_seconds results as UTC timestamps #6341.
• Native date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions #6330 date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions, the default on Ubuntu and Debian images (high). Fixed by fix: keep the input's timezone label on native date_trunc results #6345.
• CASE and COALESCE over timestamps panic when the branches have different Arrow timezones #6327 CASE and COALESCE over timestamps panic when the branches have different Arrow timezones (high). Fixed by fix: relabel CASE and COALESCE timestamp branches instead of panicking #6347.
• Session timezone IDs such as GMT+8, Z and PST make native timestamp expressions fail #6329 session timezone IDs such as GMT+8, Z and PST make casts, hour and df.show() fail (high). Fixed by fix: normalize session timezone IDs before passing them to native code #6351.
• timestamp_trunc panics on DST-transition timestamps in a DST timezone #5633 timestamp_trunc panics on DST-transition timestamps in a DST timezone. Native only with allowIncompatible (high). Fixed by feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 for literal formats and fix: truncate per-row date_trunc formats with the literal-format kernel #6354 for formats that come from a column.
• Native timezone rules come from chrono-tz's bundled tzdata, so results differ from Spark when the JVM's tzdata differs #6331 native timezone rules come from chrono-tz's bundled tzdata (2025b), not the JVM's (medium). fix: warn when native and JVM timezone databases differ #6353 logs a warning when the two versions differ.
• Native CSV V2 scan parses timestamps as UTC and ignores the session timezone #6332 the native CSV V2 scan ignores the session timezone (low, testing-only config). Fixed by fix: fall back from the native CSV scan for timestamps outside UTC #6349.
• days transform is evaluated in the session timezone, while hours and Iceberg use UTC #6333 days is evaluated in the session timezone, while hours and Iceberg use UTC (low). Fixed by fix: count the days partition transform in UTC #6348.

Status (2026-10-10)

All eight bugs are fixed on main. #6351 also replaced every getOrElse("UTC") fallback with CometTimeZone.nativeId, and #6347 turned the asserts in array_with_timezone into errors, so the section on #2730 below describes the code before those changes. #2730 is closed.

For #5633, #5956 replaced the literal-format kernel. It truncates in local time the way Spark does, follows LocalDate.atStartOfDay for the date levels, and handles the whole timestamp range in UTC sessions. #6354 sends formats that come from a column through the same kernel. #6740 matched Spark 4.2, which raises long overflow for SECOND and MILLISECOND where earlier versions wrap.

#6686 brought the contributor guide page up to date, and #6340 points the PR review skills at it.

Follow-ups:

• Check the timezone label of every native expression that returns a timestamp #6849 checks the timezone label of every native expression that returns a timestamp, which is the test gap described below.
• Native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules #6816 native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules.
• Vectorize the lower-bound check in native date_trunc for fine units #6717 vectorizes the lower-bound check feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 added for the fine units.

The UTC fallbacks in #2730

I instrumented every timeZoneId.getOrElse("UTC") site. Then I ran the datetime, cast, SQL-file, JSON, CSV, fuzz and expression suites. About 11,700 serde calls happened across 1,020 tests, and about 900 of them arrived without a timezone. Every one of those was a cast that Spark doesn't consider timezone-sensitive: numeric casts, Comet's own nullability-widening casts, and the cast inside IntegralDivide. None was a timezone-aware expression, which fits Spark refusing to resolve one without a timezone. So the fallback isn't a correctness bug today. The helper proposed in #6329 would replace it. The fallback can't simply be removed, though, because array_with_timezone asserts a non-empty timezone even for casts that don't use one.

Related

• Date-to-timestamp casts can overflow or panic for wide dates #5456 date-to-timestamp casts overflow or panic for wide dates (fix: make wide date-to-timestamp casts safe #5457 is open)
• Datetime rebase: track the documented scan limitation, and spark.comet.exceptionOnDatetimeRebase is dead code #5010 datetime rebasing isn't supported (feat: add native Parquet datetime rebasing #5047 and fix: fall back for Parquet datetime rebasing #5048 are open). Spark's legacy rebase is itself timezone-dependent.
• Evaluate moving from chrono to Jiff #4754 moving from chrono to jiff
• DataFusion / DataFusion-Spark functions whose Arrow return type drifts from Spark catalyst's declared type #4515 return type drift, where Native timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so downstream expressions ignore the session timezone #6328 and Native date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions #6330 are the timestamp cases with consequences inside a stage
• Audit Spark SQL configs that affect query semantics and ensure Comet honors them (or falls back) #4180 has an item for honoring spark.sql.session.timeZone everywhere

Already documented: Python Arrow UDFs see timestamps labelled UTC rather than the session timezone, and chrono-tz's DST rules end around 2100. Not in the user guide yet: spark.sql.parquet.int96TimestampConversion=true disables Comet for the session.

Fixed earlier, same class: #2720 (SparkToColumnar labelled timestamps with the session timezone), #2649 via #4761 (the date_trunc schema mismatch, whose fix introduced the label in #6330), and #5556 (the Python runner accepts Etc/UTC for UTC).

Test gaps

The SQL-file tests use UTC, America/Los_Angeles, America/New_York, Asia/Kolkata and +05:30. None of them use Etc/UTC, or the offset and short-ID forms from #6329. Most expression tests only project their result. Adding GMT+8 to the datetime files' ConfigMatrix would have caught #6329. #6328, #6330 and #6327 need more than a timezone setting: a test that compares each native timestamp-returning expression with another timestamp, or uses it in a CASE, in both a non-UTC session and Etc/UTC.