feat(cudf): GPU timezone-aware date_trunc and date_add - #19
Draft
dan13bauer wants to merge 28 commits into
Draft
Conversation
GPU (cuDF) expression evaluation diverged from CPU on timezone-sensitive
operations. Two GPU-only suites (labeled cuda_driver) pin the target behavior as
an executable spec; both are red until the GPU path gains timezone support.
TimezoneExtractionTest runs year/month/day/hour/quarter/day_of_week/
day_of_year/week/year_of_week under a non-UTC session timezone and asserts the
GPU result equals CPU. The GPU path reads calendar fields off the raw UTC
instant, so for 2021-01-01 02:00:00 UTC under America/Los_Angeles it returns
hour 2 and year 2021 instead of the local 18 and 2020. UTC controls and the
sub-minute fields that no whole-minute offset can shift stay green and prove a
failure is timezone-driven.
TimezoneFunctionTest forces GPU evaluation of the TIMESTAMP WITH TIME ZONE
function family and compares against CPU. now() is non-deterministic, so it only
asserts the GPU returns a TIMESTAMP WITH TIME ZONE. It also pins the CPU/GPU
parity edge cases the happy-path tests miss: to_iso8601 of a zero-offset instant
("Z"), the Joda Z/ZZ/ZZZ/z zone tokens, from_unixtime overflow rejection, and
the ISO8601 shapes from_iso8601_timestamp must accept (date-only and shorter
forms, sub-second precision, hours-only and sub-hour-negative offsets).
GPU (cuDF) expression evaluation now honors the session timezone for datetime
extraction and implements the TIMESTAMP WITH TIME ZONE function family, matching
the CPU path. Previously extraction read calendar fields off the raw UTC instant
-- hour() of 2021-01-01 02:00:00 UTC returned 2 under any session timezone --
and the TIMESTAMP WITH TIME ZONE functions threw:
Unsupported expression for recursive evaluation: <name>
A CudfExpressionContext carrying the session timezone is threaded from
CudfFilterProject to every CudfFunction. TimezoneConversion converts a UTC
column to local wall clock with public libcudf APIs only
(make_timezone_transition_table plus a sorted upper_bound, gather, and a
duration add; no custom kernel). Extraction converts to local before reading the
field, leaving second and millisecond on the raw instant.
The TIMESTAMP WITH TIME ZONE functions operate on the packed millis/zone-key
int64 and derive per-row offsets from the zone transition table: to_unixtime,
at_timezone, from_unixtime, timezone_hour, timezone_minute, to_iso8601,
format_datetime, parse_datetime, from_iso8601_timestamp and now.
registerTimezoneFunctions registers the custom type itself, so a worker that
registers cuDF before the CPU prestosql functions starts cleanly. Offset and
render paths assume one time zone per column.
The family matches CPU on the edge cases too: to_iso8601 renders a zero-offset
instant as "Z"; format_datetime distinguishes the Joda zone tokens (single "Z"
without a colon, "ZZ" with a colon, "ZZZ" the zone id) and raises VELOX_NYI for
the DST-dependent zone-name token "z"; from_unixtime rejects an out-of-range
instant; and from_iso8601_timestamp accepts date-only and shorter forms,
sub-second precision, hours-only offsets, and sub-hour negative offsets such as
-00:30. No CPU-path or core changes.
Pin GPU/CPU parity for three timezone edge cases the existing happy-path tests miss. A NULL TIMESTAMP WITH TIME ZONE row must stay NULL through timezone_hour and timezone_minute, an all-NULL column must render all-NULL through to_iso8601, and an out-of-range offset must be rejected rather than silently corrupt the packed value:
from_iso8601_timestamp('2021-01-01T02:00:00+99:00')
Two reproducers are red against the current GPU code: timezone_hour/minute return 0 for a NULL row, and from_iso8601 packs a 6780 zone key that overflows the 12-bit zone field. The all-NULL to_iso8601 case passes today and stands as a contract test, since the underlying invalid-scalar read happens to yield GMT here and cannot be made to fail differentially.
Close GPU/CPU parity gaps in the TIMESTAMP WITH TIME ZONE functions. A NULL row now stays NULL through timezone_hour and timezone_minute instead of returning 0. An out-of-range offset is now rejected instead of silently corrupting the packed value. For example:
from_iso8601_timestamp('2021-01-01T02:00:00+99:00')
now throws an "Invalid timezone offset" user error rather than packing a 6780 zone key that overflows the 12-bit zone field.
The null gap lived in the shared offset primitive. utcOffsetSeconds builds its column with make_column_from_scalar or gather, and both return a fully-valid column regardless of the input's validity. A new withInputNullMask re-applies the input's mask there, so a null instant produces a null offset and flows through every caller: timezone_hour, timezone_minute, to_iso8601, and format_datetime.
The offset bound mirrors tz::getTimeZoneID. from_iso8601 reduces the parsed offset magnitude and rejects anything past 840 minutes (+/-14h) before packing it into the zone key. A companion guard in uniformZoneKey returns GMT for an all-NULL column, matching its existing empty-column path, so a column with no readable zone no longer reads an invalid reduce scalar off the device.
Pin three more GPU/CPU parity gaps the happy-path timezone tests miss. format_datetime must honor the Joda fractional-second run length -- a single 'S' renders 1 digit and 'SSSSSS' renders 6, where the GPU emits 3 regardless. from_unixtime must map NaN to the epoch and reject an infinite input, and its (double, hours, minutes) overload rounds a negative-fractional second 1 ms differently from the (double, varchar) overload. An offset-less from_iso8601_timestamp must be read in the session timezone rather than GMT:
from_iso8601_timestamp('2021-01-01T02:00:00')
under an Asia/Kolkata session is 02:00 local time, not 02:00 UTC.
To reach the session-timezone cases, CudfFunctionBaseTest now builds its evaluation context from the query config the way CudfFilterProject does, so a test can set a session zone like the CPU DateTimeFunctionsTest. The new reproducers are red against the current GPU code; the control cases (an explicit offset or "Z" under a session, and the 3-digit fraction) stay green.
Bring three GPU timezone behaviors into line with the CPU prestosql path. format_datetime now honors the Joda fractional-second run length, emitting "%<n>f" for 1-9 digits instead of a fixed 3. from_unixtime now maps NaN to pack(0), rejects an infinite input, and uses the rounding of the matching CPU overload -- the (double, hours, minutes) form floors the seconds and rounds the fraction on its own, which differs from the (double, varchar) form's llround by up to 1 ms on negative-fractional input. An offset-less from_iso8601_timestamp is now interpreted in the session timezone:
from_iso8601_timestamp('2021-01-01T02:00:00')
under an Asia/Kolkata session yields the UTC instant for 02:00 local time, packed with the Asia/Kolkata key, instead of treating the wall clock as GMT.
For the session case the parser now captures whether a zone suffix was present, so an explicit "Z" or numeric offset still wins and an absent suffix reuses the existing utcOffsetSeconds primitive to shift the wall clock. That local-to-UTC step is exact for fixed-offset zones and away from DST transitions; inside a transition window it can differ by the DST delta and does not reproduce CPU's throw on a nonexistent local time, as a comment notes.
Separately, the batch row count is threaded into CudfFunction::eval so now() sizes its constant output from it, replacing a hack that borrowed an arbitrary input column and a check that asserted an invariant enforced elsewhere.
Add a pre-1970 (negative-millis) TIMESTAMP WITH TIME ZONE instant through to_unixtime and to_iso8601. The unpack step uses an arithmetic right shift, which differs from a logical shift only for negative packed values, and every other timezone test uses a positive 2021 instant -- so this is the first to exercise it. Both match CPU; the unpack was already correct, the gap was coverage. Replace TimezoneExtractionTest's hand-rolled resultsEqual with facebook::velox::test::assertEqualVectors for a precise per-row diff on failure, as the sibling FilterProjectTest does. Also switch the zone-name constants to std::string_view and rename kJan2021_0200Utc.
Rename the anonymous-namespace helpers i64 -> int64Scalar (pairing with the existing int64Type) and binOp -> binaryOp, per the no-abbreviation rule. No behavior change. The remaining review nits, grouped: - digit separators on 3'600 / 60'000; - parse_datetime's non-UTC-session guard switched from VELOX_CHECK to VELOX_NYI, matching the other scoped GPU limitations; - a doc comment on CudfFunction::context_; - the timezone_hour/timezone_minute name added to their shared arity-check message; - a stream passed to the empty "no separator" string scalars in the concatenate calls.
from_iso8601_timestamp reads an offset-less string as local time in the session zone, so under a daylight-savings zone it must match CPU's Timestamp::toGMT, where shifting the wall clock by the offset read as if it were UTC diverges. The new cases assert GPU == CPU for a valid post-gap time (America/Los_Angeles 2021-03-14T03:30:00, which resolves to 2021-03-14T10:30:00 UTC, not 11:30), a nonexistent spring-forward gap time (2021-03-14T02:30:00, which CPU rejects), and an ambiguous fall-back overlap (Australia/Sydney 2021-04-04T02:30:00, which CPU resolves to the earliest reading, 2021-04-03T15:30:00 UTC). A western-hemisphere zone cannot exercise the overlap, so Sydney is used for that case.
from_iso8601_timestamp of an offset-less string now resolves the wall clock to a UTC instant exactly as CPU does, even across daylight-savings transitions. The GPU previously shifted by the offset it read with the wall clock keyed as if it were UTC, which picks the wrong side of a transition window and never fails on a nonexistent local time. Under America/Los_Angeles,
from_iso8601_timestamp('2021-03-14T03:30:00')
returned 2021-03-14T11:30:00 UTC instead of the correct 10:30, and a gap time like 2021-03-14T02:30:00 returned an instant where CPU throws.
Back the TimezoneConversion primitives with a per-zone offset table built once from Velox's own time zone database -- the same source the CPU path uses -- and cached for the process lifetime. It holds a forward (UTC-keyed) table behind toLocalTimestamp and utcOffsetSeconds and an inverse (local-keyed) table with a per-breakpoint gap flag behind toUtcTimestamp; a spring-forward gap raises a user error and a fall-back overlap resolves to the earliest instant. This replaces both the wall-clock-as-UTC shift in from_iso8601 and the libcudf make_timezone_transition_table that backed the forward path, so every GPU timezone conversion now reads the same database as the CPU.
`from_iso8601_timestamp` recompiled its field-extraction `regex_program` on every `eval()` call, though the pattern never changes. Build it once in the `FromIso8601Function` constructor and store it as a member, matching how the parse format string is already precomputed. `eval()` reuses the stored program.
cuDF evaluated timezone-sensitive functions like `hour()` on UTC inside hash-join filters, nested-loop-join conditions, and hive remaining filters, even when the query set a non-UTC session timezone. The CPU path reads the wall clock in the session zone, so the GPU and CPU diverged silently. For example, under `America/Los_Angeles` a join filter
hour(t_ts) = 18
kept the row whose timestamp is 18:00 UTC rather than the row that is 18:00 in Los Angeles, returning the wrong join output with no error.
The filter/project operator already threaded the session timezone into the cuDF expression it builds. This extends that threading to the remaining operators: each one now derives a `CudfExpressionContext` from the query config and passes it into the cuDF expression, including the AST precompute path that builds functions like `hour()`. A new `contextFromConfig` helper centralizes the derivation, and the filter/project operator and the function test base adopt it.
Test plan: the join and nested-loop tests use a GPU-versus-CPU differential oracle -- run the same plan with cuDF registered and unregistered under a non-UTC session timezone and assert the results match. The nested-loop test pins CPU fallback off for the GPU run so a missing GPU path fails loudly instead of silently matching the CPU result. The hive test cannot toggle the registry, since the fixture keeps the cuDF hive connector registered, so it compares against a hand-built expected row.
Signed-off-by: Daniel Bauer <dnb@zurich.ibm.com>
On GPU, now() / current_timestamp returned a TIMESTAMP WITH TIME ZONE even when the session had no usable timezone, diverging silently from CPU. When adjust_timestamp_to_session_timezone is off, or the session timezone is empty, CPU's CurrentTimestampFunction throws:
Timezone cannot be null
The GPU instead defaulted the zone key to GMT and produced a value. It now throws the same error in both cases.
The eval now requires adjust_timestamp_to_session_timezone and a non-empty session timezone before packing, and builds the result with pack() — which range-checks the UTC millis — instead of the manual shift-and-mask that skipped the check.
Test plan: now() is pinned to the deterministic pack(sessionStartTimeMs, sessionZone) contract rather than a live clock, since a live now() cannot be compared CPU-versus-GPU.
Signed-off-by: Daniel Bauer <dnb@zurich.ibm.com>
A filter predicate like:
hour(ts) = 5
compiles to an AST comparison whose hour(ts) operand is not itself an AST node, so it is precomputed into a column before the comparison runs. That precompute must evaluate hour() in the session timezone, or the GPU selects different rows than CPU — a silent wrong result rather than an error. Commit 850adba threaded the session-timezone context into that path; this adds the regression test that pins it.
The test runs the filter under Asia/Kolkata over a two-row input chosen so the UTC-hour-5 row and the local-hour-5 row differ, and compares the GPU and CPU outputs.
Signed-off-by: Daniel Bauer <dnb@zurich.ibm.com>
Resolve conflicts from main's cudf expression-subsystem refactor: - ExpressionEvaluator.cpp includes: keep both NullMask.h (main) and the timezone headers (ours). - SubstrFunction: drop our in-file copy; main relocated it to PrestoFunctions.cpp. - createCudfFunction: combine main's canEvaluate gate with our setContext(context) call so both features coexist. - Add the numRows param (our CudfFunction::eval signature change) to the eval overrides main added/moved: IsNullFunction, IsNotNullFunction, PrestoFunctions::SubstrFunction, and sparksql SubStringFunction.
The context carries only date/time session state; a narrower name matches its scope and leaves room for a broader expression context to contain it. Addresses review comment on facebookincubator#17899.
from_unixtime(double, bigint, bigint) computed hours*60 + minutes in int64 then truncated to int32 before validating the +/-840 bound, so INT64_MAX hours overflowed instead of erroring. Mirror CPU's checkedMultiply/checkedPlus. Addresses review comment on facebookincubator#17899.
parse_datetime with a Joda offset token (Z or ZZ) folded the offset into the UTC instant but packed the zone as GMT, so timezone_hour returned 0 and to_iso8601 printed a Z suffix instead of the parsed offset. Recover the trailing offset per row with a regex and pack the matching fixed-offset zone key. The offset-minutes and zone-key mapping already used by from_iso8601_timestamp is extracted into two shared free functions and reused. Addresses review comment on facebookincubator#17899.
A TIMESTAMP WITH TIME ZONE column whose rows carry different zone keys threw
cuDF timezone functions require a single time zone per column
on the GPU, while CPU handles each row's own zone. A column mixing
America/Los_Angeles and Asia/Kolkata at the same instant now returns
timezone_hour -8 and 5 as CPU does, instead of failing.
Replace the single-zone assumption with a per-row offset that loops over the
distinct zones present (small in practice), computes each zone's offset over
the whole column, and selects the rows carrying that zone. The format_datetime
zone-id token (ZZZ) renders each row's own zone name through the same
distinct-zone helper. This removes the one-zone-per-column limitation from the
offset and render paths and deletes uniformZoneKey.
Addresses review comment on facebookincubator#17899.
On the GPU, from_iso8601_timestamp diverged from CPU on every input CPU does
not parse cleanly. It accepted a space date/time separator that CPU rejects:
from_iso8601_timestamp('2021-01-02 11:38')
A year-only string like '2021' parsed as 2020-11-30 instead of 2021-01-01.
Malformed text returned NULL, and a nonexistent date like '2021-13-45' was
silently rolled into the next month. Every non-null row now ends as CPU would:
the correct value, or the same parse error.
Each non-null row is classified against the in-range ISO8601 pattern. A row
that fails the pattern, or names a month or day outside the calendar, throws
like CPU's fromTimestampWithTimezoneString. cudf's parser normalizes an
out-of-range date rather than rejecting it. The calendar check therefore parses
the date, reads the month and day back, and requires they equal the input. The
year-only underflow is a separate fix. cudf's field extraction leaves an absent
month or day as an empty string, which the parser read as 0; both now default
to "01".
A year that is signed or has five or more digits parses on CPU but exceeds what
the int16 %Y parser can hold, so it now raises
from_iso8601_timestamp does not support years outside [0000, 9999] on GPU
as a not-implemented error instead of a wrong instant.
Addresses review comment on facebookincubator#17899.
# Conflicts: # velox/experimental/cudf/expression/ExpressionEvaluator.h
Truncate date_trunc(timestamp) on the session-local wall clock instead of the raw UTC epoch, matching the CPU truncateTimestamp: convert to local, truncate, convert back (toLocalTimestamp/toUtcTimestamp), with a DST-safe UTC delta for the hour unit so fractional-offset zones (e.g. Asia/Kolkata +05:30) are exact. second/minute and DATE keep the raw UTC / zone-free path. Remove the adjust_timestamp_to_session_timezone CPU-fallback gate (DateTruncFunction::isTimezoneSensitive and the CudfFilterProject check) now that the GPU path is correct, so date_trunc(timestamp) is GPU-accelerated under a non-UTC session. Add non-UTC GPU-vs-CPU parity tests (America/Los_Angeles, Asia/Kolkata) and flip the ToCudfSelectionTest expectations to use cuDF.
Move the packed TIMESTAMP WITH TIME ZONE column primitives (per-row zone key, UTC instant, distinct zones, per-row offset, local wall clock) out of prestosql/TimezoneFunctions.cpp into a shared TimestampWithTimeZoneColumn unit, and add the per-row multi-zone local-to-UTC and repack primitives the date_trunc/date_add TSWTZ overloads will use. TimezoneFunctions.cpp now forwards to the shared helpers; no behavior change.
Add a TIMESTAMP WITH TIME ZONE path to DateTruncFunction that truncates on each row's embedded zone (per-row multi-zone, via the shared TimestampWithTimeZoneColumn helpers): sub-day units subtract the local-to-truncated delta from the UTC instant, day-and-above truncate the local wall clock then convert back per row's zone. Register the third Presto date_trunc signature. Value parity covered in FilterProjectTest (operator path) and routing in ToCudfSelectionTest.
Add DateAddTimestampFunction: a Presto date_add timestamp overload that adds on the raw UTC instant under a UTC session and on the session-local wall clock (then converts back to UTC, throwing on a spring-forward gap like toGMT) when the session applies a timezone. Handles the full unit set (sub-day, day/week via a duration add, month/quarter/year via add_calendrical_months) and reuses the int32 value-range check. Dispatch the date_add factory by third-arg type and register the timestamp signature. Value parity in FilterProjectTest; routing in ToCudfSelectionTest.
date_add now runs on the GPU when the third argument is a TIMESTAMP WITH TIME ZONE, preserving each row's zone. Unlike the plain-timestamp path, adding across a spring-forward gap resolves forward instead of failing, matching Presto's addToTimestampWithTimezone. For example:
date_add('day', 1, TIMESTAMP '2024-03-09 02:30 America/Los_Angeles')
lands on the nonexistent local 2024-03-10 02:30 and returns 10:30 UTC rather than raising a daylight-savings-gap error.
Sub-day units (millisecond through hour) add straight to the UTC instant. Day-and-above units add on each row's local wall clock and convert back to UTC. That conversion needs a variant of toUtcTimestamp that keeps the computed instant through a gap instead of throwing: shifting the nonexistent local by the gap size and subtracting the post-transition offset reduces to subtracting the pre-transition offset, which is the offset the local-keyed transition table already stores across the gap. A correctForward flag on OffsetTable::toUtc selects between raising and keeping.
Selective Build Plan
Selective build plan |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Preservation draft for the GPU (cuDF) timezone-aware temporal work stacked on the
cudf-timezone-supportPR. Adds GPU support fordate_truncanddate_addoverTIMESTAMPandTIMESTAMP WITH TIME ZONE, plus the shared per-row multi-zoneprimitives they rely on.
The tip commits (this branch's own work beyond the timezone-support base):
feat(cudf): timezone-aware date_trunc(timestamp) on GPUrefactor(cudf): extract shared TimestampWithTimeZoneColumn helpersfeat(cudf): timezone-aware date_trunc(timestamp with time zone) on GPUfeat(cudf): timezone-aware date_add(unit, value, timestamp) on GPUfeat(cudf): GPU date_add(unit, value, timestamp with time zone)Semantics
date_add/date_trunconTIMESTAMP WITH TIME ZONEpreserve each row's zone key.millisecond..hour) add straight to the UTC instant.date_addacross a spring-forward gap resolves forward instead of throwing,matching Presto's
addToTimestampWithTimezone;date_truncsnaps to localmidnight (never a gap) and keeps the throwing conversion.
Testing
Built and run in the CUDA docker image (
prestobuild/dependency:cio-new):velox_cudf_filter_project_test,velox_cudf_tocudf_selection_test, andvelox_cudf_timezone_function_testall pass.