Support date_bin on timestamps with timezone, properly accounting for Daylight Savings Time

@Abdullahsab3

Is your feature request related to a problem or challenge?

Summary

Given this data in UTC timestamps

> select * from t_utc;
+----------------------+
| column1              |
+----------------------+
| 2024-04-30T21:30:00Z |
| 2024-04-30T22:30:00Z |
| 2024-04-30T23:30:00Z |
| 2024-05-01T00:00:00Z |
| 2024-05-01T00:30:00Z |
| 2024-05-01T10:30:00Z |
| 2024-05-01T20:30:00Z |
+----------------------+
7 row(s) fetched.
Elapsed 0.002 seconds.

How to create the data

-- Dates right around the first of May
create or replace table t AS
VALUES
  ('2024-04-30T21:30:00'),
  ('2024-04-30T22:30:00'),
  ('2024-04-30T23:30:00'),
  ('2024-05-01T00:00:00'),
  ('2024-05-01T00:30:00'),
  ('2024-05-01T10:30:00'),
  ('2024-05-01T20:30:00')
;


create or replace view t_utc as
select column1::timestamp  AT TIME ZONE 'UTC' as "column1"
from t;

I would like to be able to get this output (bin the data by their day in Brussels, properly accounting for daylight savings time).

+---------------------------+---------------------+
| time in Brussels          | date_bin            |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T00:00:00 | <-- in 2024-04-01
| 2024-05-01T00:30:00+02:00 | 2024-05-01T00:00:00 | <-- note this is in 2024-05-01
| 2024-05-01T01:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T00:00:00 |
+---------------------------+---------------------+
7 row(s) fetched.
Elapsed 0.004 seconds.

We can't figure out how to do this at the moment

Details

Broken out from @Abdullahsab3's great ticket #10368

We would like to apply date_bin and have it bin the dates in local time, properly including daylight savings time calculations

For example, given the following data at 12:01AM in the morning on the first day of each month,I would like to bin it into the first day of each month.

create or replace table t AS
VALUES
  ('2024-01-01T00:00:01'),
  ('2024-02-01T00:00:01'),
  ('2024-03-01T00:00:01'),
  ('2024-04-01T00:00:01'),
  ('2024-05-01T00:00:01'),
  ('2024-06-01T00:00:01'),
  ('2024-07-01T00:00:01'),
  ('2024-08-01T00:00:01'),
  ('2024-09-01T00:00:01'),
  ('2024-10-01T00:00:01'),
  ('2024-11-01T00:00:01'),
  ('2024-12-01T00:00:01')
;

This works as expected when there is no timestamp (all timestamps end up in the correct day)

> select arrow_typeof(column1::timestamp) as arrow_type, column1::timestamp, date_bin(interval '1 day', column1::timestamp) as "date_bin" from t;
+-----------------------------+---------------------+---------------------+
| arrow_type                  | t.column1           | date_bin            |
+-----------------------------+---------------------+---------------------+
| Timestamp(Nanosecond, None) | 2024-01-01T23:59:00 | 2024-01-01T00:00:00 | <-- binned correctly
| Timestamp(Nanosecond, None) | 2024-02-01T23:59:00 | 2024-02-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-03-01T23:59:00 | 2024-03-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-04-01T23:59:00 | 2024-04-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-05-01T23:59:00 | 2024-05-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-06-01T23:59:00 | 2024-06-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-07-01T23:59:00 | 2024-07-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-08-01T23:59:00 | 2024-08-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-09-01T23:59:00 | 2024-09-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-10-01T23:59:00 | 2024-10-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-11-01T23:59:00 | 2024-11-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-12-01T23:59:00 | 2024-12-01T00:00:00 |
+-----------------------------+---------------------+---------------------+
12 row(s) fetched.
Elapsed 0.004 seconds.

However, once the timestamp is in a timezone it has problems:

> create or replace view t_timezone 
as 
select column1::timestamp  AT TIME ZONE 'Europe/Brussels' as "column1" 
from t;
0 row(s) fetched.
Elapsed 0.001 seconds.

The timestamps now binned at incorrectly (the are binned based on UTC midnight, not midnight in Brussels) so for example a time in 2024-01-01 goes into a 2023-12-31 bucket:

> select
   arrow_typeof(column1) as arrow_type,
   column1,
   date_bin(interval '1 day', column1) as "date_bin"
  from t_timezone;
+------------------------------------------------+---------------------------+---------------------------+
| arrow_type                                     | column1                   | date_bin                  |
+------------------------------------------------+---------------------------+---------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-01-01T00:00:01+01:00 | 2023-12-31T01:00:00+01:00 | <-- binned in previous day
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-02-01T00:00:01+01:00 | 2024-01-31T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-03-01T00:00:01+01:00 | 2024-02-29T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-04-01T00:00:01+02:00 | 2024-03-31T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-05-01T00:00:01+02:00 | 2024-04-30T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-06-01T00:00:01+02:00 | 2024-05-31T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-07-01T00:00:01+02:00 | 2024-06-30T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-08-01T00:00:01+02:00 | 2024-07-31T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-09-01T00:00:01+02:00 | 2024-08-31T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-10-01T00:00:01+02:00 | 2024-09-30T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-11-01T00:00:01+01:00 | 2024-10-31T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-12-01T00:00:01+01:00 | 2024-11-30T01:00:00+01:00 |
+------------------------------------------------+---------------------------+---------------------------+

The third argument to date_bin allows changing the starting offset for a day, so let's set it to midnight in Europe/Brussels time. This almost works but does not take into account daylight savings time so the timestamps are binned incorrectly during DST:

> select
   arrow_typeof(column1) as arrow_type,
   column1,
   date_bin(interval '1 day', column1, '2020-01-01T00:00:00Z'::timestamp AT TIME ZONE 'Europe/Brussels') as "date_bin"
  from t_timezone;
+------------------------------------------------+---------------------------+---------------------------+
| arrow_type                                     | column1                   | date_bin                  |
+------------------------------------------------+---------------------------+---------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-01-01T00:00:01+01:00 | 2024-01-01T00:00:00+01:00 | <-- correct!
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-02-01T00:00:01+01:00 | 2024-02-01T00:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-03-01T00:00:01+01:00 | 2024-03-01T00:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-04-01T00:00:01+02:00 | 2024-03-31T00:00:00+01:00 | <-- DST! Binned into previous day
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-05-01T00:00:01+02:00 | 2024-04-30T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-06-01T00:00:01+02:00 | 2024-05-31T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-07-01T00:00:01+02:00 | 2024-06-30T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-08-01T00:00:01+02:00 | 2024-07-31T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-09-01T00:00:01+02:00 | 2024-08-31T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-10-01T00:00:01+02:00 | 2024-09-30T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-11-01T00:00:01+01:00 | 2024-11-01T00:00:00+01:00 | <-- DST over, now correct again
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-12-01T00:00:01+01:00 | 2024-12-01T00:00:00+01:00 |
+------------------------------------------------+---------------------------+---------------------------+
12 row(s) fetched.
Elapsed 0.003 seconds.

Describe the solution you'd like

Describe alternatives you've considered

No response

Additional context

No response

@mhilton

The way you can perform this binning in postgres is somewhat paradoxically to convert a timestamp with a timezone back to a timestamp without timezone and then apply date_bin.

The syntax to convert a timestamp to something without timezone is 🤯 : you apply AT TIME ZONE to a timestamp with a timezone and then it has no timezone) as @mhilton notes in #10368 (comment)

postgres=# select pg_typeof('2024-05-21T12:00:00'::timestamp AT TIME ZONE 'America/New_York');
        pg_typeof
--------------------------
 timestamp with time zone
(1 row)

postgres=# select pg_typeof('2024-05-21T12:00:00'::timestamp AT TIME ZONE 'America/New_York' AT TIME ZONE 'Europe/Brussels'
);
          pg_typeof
-----------------------------
 timestamp without time zone. <-- WTF no timezone as the result of AT TIME ZONE
(1 row)

If we cast using arrow_cast back to Timestamp(Nanosecond, None) the binning does appear to work correctly

> create or replace view t_roundtrip as select arrow_cast(column1, 'Timestamp(Nanosecond, None)') as "column1" from t;
0 row(s) fetched.
Elapsed 0.002 seconds.

> select * from t_roundtrip;
+---------------------+
| column1             |
+---------------------+
| 2024-01-01T00:00:01 | <--- timestamps now have no timezone
| 2024-02-01T00:00:01 |
| 2024-03-01T00:00:01 |
| 2024-04-01T00:00:01 |
| 2024-05-01T00:00:01 |
| 2024-06-01T00:00:01 |
| 2024-07-01T00:00:01 |
| 2024-08-01T00:00:01 |
| 2024-09-01T00:00:01 |
| 2024-10-01T00:00:01 |
| 2024-11-01T00:00:01 |
| 2024-12-01T00:00:01 |
+---------------------+
12 row(s) fetched.
Elapsed 0.001 seconds.

> select arrow_typeof(column1) as arrow_type, column1, date_bin(interval '1 day', column1) as "date_bin" from t_roundtrip;
+-----------------------------+---------------------+---------------------+
| arrow_type                  | column1             | date_bin            |
+-----------------------------+---------------------+---------------------+
| Timestamp(Nanosecond, None) | 2024-01-01T00:00:01 | 2024-01-01T00:00:00 | <-- bins are as desired
| Timestamp(Nanosecond, None) | 2024-02-01T00:00:01 | 2024-02-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-03-01T00:00:01 | 2024-03-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-04-01T00:00:01 | 2024-04-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-05-01T00:00:01 | 2024-05-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-06-01T00:00:01 | 2024-06-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-07-01T00:00:01 | 2024-07-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-08-01T00:00:01 | 2024-08-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-09-01T00:00:01 | 2024-09-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-10-01T00:00:01 | 2024-10-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-11-01T00:00:01 | 2024-11-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-12-01T00:00:01 | 2024-12-01T00:00:00 |
+-----------------------------+---------------------+---------------------+
12 row(s) fetched.
Elapsed 0.003 seconds.

Given the statement in the description, here is the best I can come up with using arrow_cast

-- Times in brussels
WITH t_brussels
AS (
  SELECT
    column1 AT TIME ZONE 'Europe/Brussels' as ts -- timestamp in specified timezone
  FROM t_utc
)
SELECT
  ts as "time in Brussels",
  date_bin(
    interval '1 day',
    arrow_cast(ts, 'Timestamp(Nanosecond, None)'),
    '2020-01-01T00:00:00Z'::timestamp
  ) as date_bin
FROM
  t_brussels;

+---------------------------+---------------------+
| time in Brussels          | date_bin            |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T00:00:00 |
| 2024-05-01T00:30:00+02:00 | 2024-04-30T00:00:00 | <-- this is in the wrong bin
| 2024-05-01T01:30:00+02:00 | 2024-04-30T00:00:00 | <-- this in in the wrong bin 
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T00:00:00 |
+---------------------------+---------------------+
7 row(s) fetched.
Elapsed 0.004 seconds.

@mhilton

@mhilton and I agree that if we had the functionality suggested by @Abdullahsab3's on #10368 (comment)

given a UTC timestamp, I would like to have that timestamp in local time of a given timezone

We think we could get the right value out of date_bin.

For example, If we had a function like remove_timezone that behaved like

select remove_timezone("time in Brussels")

+---------------------------+---------------------+
| time in Brussels          | remove_timezone            |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T23:30:00 |
| 2024-05-01T00:30:00+02:00 | 2024-05-01T00:30:00 | <-- timezone removed (no +02:00, but value not adjusted)
| 2024-05-01T01:30:00+02:00 | 2024-05-01T01:30:00 |
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T02:30:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T12:30:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T22:30:00 |
+---------------------------+---------------------+

Note that this is different than arrow_cast("time in Brussels", 'Timestamp(Nanosecond, None)') because arrow_cast will adjust the timestamp values

-- Times in brussels
WITH t_brussels
AS (
  SELECT
    column1 AT TIME ZONE 'Europe/Brussels' as ts -- timestamp in specified timezone
  FROM t_utc
)
SELECT
  ts as "time in Brussels",
  arrow_cast(ts, 'Timestamp(Nanosecond, None)') as arrow_cast
FROM
  t_brussels;

+---------------------------+---------------------+
| time in Brussels          | arrow_cast          |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T21:30:00 |
| 2024-05-01T00:30:00+02:00 | 2024-04-30T22:30:00 | <-- note this is now in 2024-04-30 not 2024-05-01
| 2024-05-01T01:30:00+02:00 | 2024-04-30T23:30:00 |
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T00:30:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T10:30:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T20:30:00 |
+---------------------------+---------------------+
7 row(s) fetched.
Elapsed 0.003 seconds.

My suggested next steps for this ticket:

Someone prototype the "strip_timezone" function as a ScalarUDF and verify that we can in fact we can achieve the expected result from date_bin
If we can achieve the expected results, then file another ticket / proposal with how to integrate this into DataFusion (e.g. strip_timezone function, and/or update the arrow_cast kernel, and/or add syntax like WITHOUT TIME ZONE)

I'd like to work on this issue 🙋‍♀️

The challenge with an approach that relies on stripping the timezone, is you end up back with the ambiguous timestamp issue. The UX of relying on users to strip the timezone is also rather unfortunate. I'm curious as to why we wouldn't want to just make date_bin timezone aware? This would appear to have a lot less potential edge cases

@mhilton

I'm curious as to why we wouldn't want to just make date_bin timezone aware? This would appear to have a lot less potential edge cases

I believe @mhilton had thought about this and maybe has some thoughts

I'm curious as to why we wouldn't want to just make date_bin timezone aware?

I'm not sure that it's obvious how a time-zone aware date_bin would behave. For example On the day in october when many time zones repeat an hour would the two hours count into a single bin? What about if you had hour long bins that start 30 minutes into the hour? Given that the two hours are easily distinguishable the answer is probably not, and if you really want that behviour then stripping time zone information would be a reasonable way to get it.

If we restricted timezone awareness to intervals of unit days then that would be much more reasonable. We could make it so that date_bin understands the number of hours in any given day in a time zone and bins a time accordingly. Anyone needing a strict 24 hour binning could specify the interval in hour units.

zones repeat an hour would the two hours count into a single bin

This would be the intuitive thing for it to do IMO, if people don't want this, they should cast to a timezone that doesn't have DST

To make date_bin timezone aware, there are some edge cases we need to consider when design it:

Daylight Saving Time (DST) Transitions:
- Spring Forward: When the clocks move forward, there is a "missing" hour.
- Fall Back: When the clocks move backward, there is an "extra" hour. For example, in US central time zone, when DST ends at 2:00 AM, the clocks are set back to 1:00 AM. This means that there are two 1:20 AM in a day. If a user do a date_bin with 10 min interval, how to handle the returned data? Aggregate them? Return two sets of data?
Crossing Midnight Boundaries:
- Ensure that timestamps are correctly binned at the right local day and hour boundaries, especially when converting from UTC to a local timezone. This edge case is what this issue is trying to solve.
Timezone Offsets:
- Different timezones have different offsets from UTC, and these can change over time (e.g., due to DST).
- For certain timezones, ensure the offset works correctly at the 30-min mark when DST happens. e.g.:
  - Lord Howe Island Time Zone (LHST) in Australia shifts time by 30 min during DST
  - Nepal (NPT) shifts by 45 min
  - Chatham Islands, New Zealand (CHAST) shifts by 12 hr and 45 min.
- Ensure the function correctly applies the current offset for the given timestamp, considering historical changes in timezone rules. e.g.:
  - America/Sao_Paulo stoped doing DST in 2019
  - Russia set the clocks ahead permanently in 2011, then was reversed in 2014
  - Iceland, Turkey, Egypt, India, and China
Leap Years and Leap Seconds:
- Leap Years: February 29th is handled correctly in leap years.
- Leap Seconds: this is less common. e.g. the most recent one was on Dec. 31, 2016, a leap second was introduced at 23:59:60 UTC
Handling Null and Invalid Timestamps:
- Ensure that null or invalid timestamps are handled gracefully, either by ignoring them or providing a default bin. e.g. '2021-03-28T02:30:00' AT TIME ZONE 'Europe/Brussels' does not exist
Time Precision:
- Ensure that the function handles different precisions of timestamps correctly (e.g., seconds, milliseconds, nanoseconds).

Fall Back: When the clocks move backward, there is an "extra" hour. For example, in US central time zone, when DST ends at 2:00 AM, the clocks are set back to 1:00 AM. This means that there are two 1:20 AM in a day. If a user do a date_bin with 10 min interval, how to handle the returned data? Aggregate them? Return two sets of data?

I want to highlight this open-ended question and hear how others think

Thanks for filing the ticket and for all the detailed explanations! very enriching

I wonder whether the Postgres behavior is actually that bad. Though it looks weird, it still is generic enough to make it widely applicable. The issue with making date_bin specifically timezone-aware in my opinion is the fact that future or similar grouping functions (or rolling windows functions) will also have to be implemented in a timezone-aware way, accounting for similar pitfalls and edge cases. The same problem will also occur in other functionalities; for example if you would like to return local time. I personally think that the problem might be better tackled as an additional time functionality, which may be the same way that Postgres does it.

The postgres way of converting UTC to local time of a given timezone is:

select '2024-05-21T12:00:00Z'::timestamp AT TIME ZONE 'UTC' AT TIME ZONE 'Europe/Brussels';

timezone
--
2024-05-21 14:00:00

As already mentioned by Andrew, the AT TIME ZONE operator in postgres converts a timezone-aware timestamp to local time (with no offset or timezone information), and local time to a timezone-aware timestamp. Though the overloaded functionalities of the AT TIME ZONE operator in Postgres are weird, they are definitely usable in my opinion. I also like the idea of having a separate to_local_time/strip_timezone function, though this would break Datafusions compatibility with Postgres.

@appletreeisyellow

Very nice! Thanks for verifying @appletreeisyellow. I was able to do some minimal validations today on our systems and it seems indeed to be binning correctly. Happy to see this as an alternative to what I was doing as a fix for this issue

My main concerns with the use of to_char are:

Performance: Not super familiar with Datafusion internals, but I assume to_char will parse the given timestamp in the provided format, and return a string, which will be coerced to a (localtime) timestamp (which I assume includes another parsing step).
Probably not a problem and I am just overthinking it, but I wonder whether %F %X will always produce the correct localtime. My concern is mostly the %F part

@appletreeisyellow

@appletreeisyellow and I just had a chat about what to do about this ticket. It is great that we seem to have found a workaround there are at least two potential issues we see with the workaround:

It is non obvious and likely would not be obvious to other future users
It may have performance issues, as @Abdullahsab3 notes in Support date_bin on timestamps with timezone, properly accounting for Daylight Savings Time #10602 (comment)

We think it would be great to improve DataFusion so future users (both of InfluxDB and other systems) who use date_bin have fast queries and are not confused.

Ideal Behavior

We think the ideal user experience would be as @tustvold suggests for date_bin to be timezone aware and just "do the right thing and bin dates respecting the timezone of the timestamp".

While this might be different than what postgres does, the underlying treatment if timestamps is already different, so I don't think we need to be quirk for quirk compatible.

Proposal

Thus we propose @appletreeisyellow will try and complete her prototype work to make date_bin do the intuitive thing with timestamps with timezones

It is likely we can't devote a large amount of time to this project, so if the implementation is taking too long we may have to abandon this effort or go with a less optimal route (e.g. a function to strip off timezones like Postgres)

I filed apache/arrow-rs#5826 to document more precisely what is going on with casting

I wrote up a high-level doc on how to implement the date_bin with timezone awareness. If you cannot access the document, feel free to request access and I'll give you the permission to view/comment.

Another thing that needs to be taken into consideration: I think the returned timestamp from date_bin needs to be in the timezone that was provided as an argument (in the internal Arrow represenation). For example:
The returned timestamp is in correct local time, but the underlying Arrow kernel has no timezone information about it (which I think means that the timestamp will be interpreted as being in UTC)

❯ select to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X')::timestamp;
+----------------------------------------------------+
| to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X")) |
+----------------------------------------------------+
| 2024-04-30T04:00:00                                |
+----------------------------------------------------+
1 row in set. Query took 0.002 seconds.

❯ select arrow_typeof(to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X')::timestamp);
+------------------------------------------------------------------+
| arrow_typeof(to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X"))) |
+------------------------------------------------------------------+
| Timestamp(Nanosecond, None)                                      |
+------------------------------------------------------------------+
1 row in set. Query took 0.003 seconds.

When we add the timezone information:

❯ select to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X') at time zone 'Europe/Brussels';
+----------------------------------------------------+
| to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X")) |
+----------------------------------------------------+
| 2024-04-30T04:00:00+02:00                          |
+----------------------------------------------------+
1 row in set. Query took 0.001 seconds.

❯ select arrow_typeof(to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X') at time zone 'Europe/Brussels');
+------------------------------------------------------------------+
| arrow_typeof(to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X"))) |
+------------------------------------------------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels"))                   |
+------------------------------------------------------------------+
1 row in set. Query took 0.001 seconds.

The timezone information there is important, as the timestamp will be otherwise interpreted by external integrations (such as Grafana) as being in UTC.

Note that applying at time zone 'Europe/Brussels' on the results of to_char will never fail during daylight savings hours, as the returned value from to_char is always in localtime, meaning that the 'missing/additional hour' should not be an issue for the at time zone operator.

TLDR; if you're using the to_char workaround now for localisation, you should add the timezone information to the results

I think the returned timestamp from date_bin needs to be in the timezone that was provided as an argument (in the internal Arrow represenation)

Thanks for highlighting it. Yes, this is the expected behavior. The timezone representation of the returned timestamp will be the same as the input timestamp. For example:

if the input doesn't have a timezone, then the date_bin returns timezone as None:

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20Z'));
+---------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20Z"))) |
+---------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, None)                                                                       |
+---------------------------------------------------------------------------------------------------+

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20'));
+--------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20"))) |
+--------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, None)                                                                      |
+--------------------------------------------------------------------------------------------------+

if the input has a timezone in UTC, then the date_bin returns timezone as Some('UTC')

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20Z' AT TIME ZONE 'UTC'));
+---------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20Z"))) |
+---------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, Some("UTC"))                                                                |
+---------------------------------------------------------------------------------------------------+

if the input has a timezone in Europe/Brussels, then the date_bin returns timezone as Some('Europe/Brussels')

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20Z' AT TIME ZONE 'Europe/Brussels'));
+---------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20Z"))) |
+---------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels"))                                                    |
+---------------------------------------------------------------------------------------------------+

I probably missed some prior discussion. Do we want date_bin syntax?
Can we support more standard date_trunc('day', timestamp_tz)?

I think we already have date_trunc
docs: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#date-trunc
code:

datafusion/datafusion/functions/src/datetime/date_trunc.rs

Line 361 in dce77db

fn general_date_trunc(

I think the difference between date_trunc and date_bin is that date_bin can bin to dates that are not aligned with the start/stop of the interval (e.g. you could bin days at noon rather than at midnight)

However it is a good question if date_trunc would be a better way to get the desired behavior

@appletreeisyellow

Update: @appletreeisyellow merged an workaround -- to_local_time in #11401

She has no further current concrete plans to work on this ticket at this time

FYI - I came across the Jiff crate the other day that looks to have some nice support for tz aware calendar handling. From the benchmarks I've run it's not quite as quick as chrono for some things and about equivalent to others. I would recommend anyone who wants to tackle this issue to have a close look at it to see if it solves some of the intricacies around tz conversion and DST arithmetic and rounding.

Author of Jiff here. I don't have a ton of context on the specific problem in this issue, but I'd be happy to field questions about DST safe arithmetic. Jiff in particular is heavily inspired by Temporal, and Temporal (and Jiff) both intend to match RFC 5545 for duration handling.

(It's still early days and I think there is a fair bit of room for improvement for Jiff in terms of perf. Both in micro-optimizing its existing code and in adding new APIs to facilitate faster ops when the use case demands it.)

@alamb

Two data points for this thread. to_local_time (#11401) was merged here as a workaround, and the issue was deliberately left open for native support, so this is not a request to close.

1. The to_local_time workaround is correct across DST transitions, including the repeated hour. The verification earlier in this thread used to_char, on a monthly dataset that crosses transitions without landing on them. I checked to_local_time itself on DataFusion 55.0.0, grouping UTC instants by local day in Europe/Brussels:

SET datafusion.execution.time_zone = 'UTC';
SELECT date_bin(INTERVAL '1 day', to_local_time(column1 AT TIME ZONE 'Europe/Brussels')) AS local_day, count(*) AS n
FROM t GROUP BY 1 ORDER BY 1;

dataset	local_day	n
the `t_utc` dataset from the issue body	`2024-04-30`	1
	`2024-05-01`	6
spring forward, 2024-03-31 (a 23-hour local day)	`2024-03-30`	1
	`2024-03-31`	4
	`2024-04-01`	1
fall back, 2024-10-27 (a 25-hour local day; 02:00–03:00 happens twice)	`2024-10-27`	5
	`2024-10-28`	1

The first rows match the output requested in the issue body. In the fall-back set, both instants that read 02:30 locally (one at +02:00, one at +01:00) bin to 2024-10-27.

2. #25165 would make the PostgreSQL spelling work, which addresses the discoverability concern raised here. @alamb's first comment on this issue notes that in PostgreSQL you get local-day bins by using AT TIME ZONE to turn a timestamptz back into a timestamp, then applying date_bin. DataFusion cannot express that today, because AT TIME ZONE on a timezone-aware value returns a timezone-aware value (#12218). #25165 changes that. With it applied,

SELECT date_bin(INTERVAL '1 day', column1 AT TIME ZONE 'Europe/Brussels') AS local_day, count(*) AS n
FROM t GROUP BY 1 ORDER BY 1;

gives byte-identical results to the to_local_time spelling above for all three datasets, and the existing to_local_time spelling keeps working unchanged. Users could then write the idiom they already know from PostgreSQL instead of having to discover to_local_time.

That still is not a timezone-aware date_bin, which is the remaining ask of this issue, so it should stay open. #25165 is under review and has a known limitation with CASE inputs, described at the top of that PR.

Related: #25167 (date_bin and date_trunc disagree on timezone-aware input) and #25168 (aligning date_bin to local midnight with an explicit origin drifts by an hour across a DST transition).

Seeing as there has been some activity around this recently I've been trying to think about what reasonable semantics should be for a date_bin that adjusts for DST changes. These are some suggestions designed to promote discussion rather than dictate policy.

1. `date_bin`s should never jump backwards

A naive implementation that simply shifts to the timestamp to local time, performs the data_bin and then shifts back the local time using the timezone offset can produce surprising results where the bins overlap the discontinuity. Consider the following times when used with such a naive implementation of date_bin(INTERVAL '1 HOUR', t, '1970-01-01T00:30:00Z'):

+----------------------+------------------------+------------------------+--------------------------+
| UTC                  | Europe/London          | date_bin (timestamp)   | date_bin (timestamp UTC) |
+----------------------+------------------------+------------------------+--------------------------+
| 2026-10-24T23:30:00Z | 2026-10-25T00:30:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-24T23:45:00Z | 2026-10-25T00:45:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T00:00:00Z | 2026-10-25T01:00:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T00:15:00Z | 2026-10-25T01:15:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T00:30:00Z | 2026-10-25T01:30:00+01 | 2026-10-25T01:30:00+01 | 2026-10-25T00:30:00Z     |
| 2026-10-25T00:45:00Z | 2026-10-25T01:45:00+01 | 2026-10-25T01:30:00+01 | 2026-10-25T00:30:00Z     |
| 2026-10-25T01:00:00Z | 2026-10-25T01:00:00+00 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T01:15:00Z | 2026-10-25T01:15:00+00 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T01:30:00Z | 2026-10-25T01:30:00+00 | 2026-10-25T01:30:00+00 | 2026-10-25T01:30:00Z     |
| 2026-10-25T01:45:00Z | 2026-10-25T01:30:00+00 | 2026-10-25T01:30:00+00 | 2026-10-25T01:30:00Z     |
+----------------------+------------------------+------------------------+--------------------------+

As can be seen around the discontinuity it is possible for an absolute timestamp to be put in a bin earlier than a smaller absolute timestamp. In my opinion a date_bin implementation should be written to avoid this happening. In this particular case I think it would be better for a DST aware date_bin to follow how UTC would bin the values:

+----------------------+------------------------+------------------------+--------------------------+
| UTC                  | Europe/London          | date_bin (timestamp)   | date_bin (timestamp UTC) |
+----------------------+------------------------+------------------------+--------------------------+
| 2026-10-24T23:30:00Z | 2026-10-25T00:30:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-24T23:45:00Z | 2026-10-25T00:45:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T00:00:00Z | 2026-10-25T01:00:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T00:15:00Z | 2026-10-25T01:15:00+01 | 2026-10-25T00:30:00+01 | 2026-10-24T23:30:00Z     |
| 2026-10-25T00:30:00Z | 2026-10-25T01:30:00+01 | 2026-10-25T01:30:00+01 | 2026-10-25T00:30:00Z     |
| 2026-10-25T00:45:00Z | 2026-10-25T01:45:00+01 | 2026-10-25T01:30:00+01 | 2026-10-25T00:30:00Z     |
| 2026-10-25T01:00:00Z | 2026-10-25T01:00:00+00 | 2026-10-25T01:30:00+01 | 2026-10-25T00:30:00Z     |
| 2026-10-25T01:15:00Z | 2026-10-25T01:15:00+00 | 2026-10-25T01:30:00+01 | 2026-10-25T00:30:00Z     |
| 2026-10-25T01:30:00Z | 2026-10-25T01:30:00+00 | 2026-10-25T01:30:00+00 | 2026-10-24T01:30:00Z     |
| 2026-10-25T01:45:00Z | 2026-10-25T01:30:00+00 | 2026-10-25T01:30:00+00 | 2026-10-24T01:30:00Z     |
+----------------------+------------------------+------------------------+--------------------------+

2. An `origin_timestamp` parameter without a timezone offset acts as if it is in the binned timestamp's timezone

Most implementations I can think of would do this for free anyway, but it is worth stating explicitly. If the origin_timestamp does not explicitly say what time zone it is in then it is treated as if it is in the timezone of the value being binned. For example when binning timestamps in the American/Los_Angeles timezone then an origin timestamp of 1970-01-01T00:30:00 works as if it were the absolute value 1970-01-01T00:30:00-08:00.

The default origin_timestamp should be defined to be 1970-01-01T00:00:00 in local time, which is the start of the year 1970 in that timezone. So for Australia/Adelaide this will map to 1970-01-01T00:00:00+09:30 or 1969-12-31T14:30:00Z.

If the specified origin_timestamp is undefined or ambiguous in the binned timestamp's timezone then I think date_bin should output an error.

3. A day in a timezone starts at the first instance that reports the date as being that day

This might seem self-evident, but it is a looser definition than days start at midnight. This in necessary because historically there have been timezones where midnight does not exist in local time. For example in America/Sao_Paulo the timestamp 2018-10-03T23:59:59.999999999-03 jumps straight to 2018-10-04T01:00:00.000000000-02.

A similar rule needs to apply to the start of months. In the same timezone 2015-10-31T23:59:59:.999999999-03 jumped straight to 2015-11-01T01:00:00.000000000-02.

4. The value returned from `date_bin` cannot be after the timestamp being binned

The timestamp value output from date_bin should be lower than or equal to the timestamp being binned. This is another one that might be obvious but the arithmetic can get tricky when one starts dealing with negative values.

5. Intervals adjusted for daylight savings should not be adjusted further than the offset change

Where an interval stretches over a discontinuity then the interval may be grown or shrunk to accommodate the discontinuity. Effort should be made to avoid an interval's length being further away from requested than is needed for the daylight savings adjustment.

6. Using an `expression` with a timezone is sufficient to opt in to DST adjustment

I hold this loosely as it might give confusing results for existing date_bin users. However the only alternatives I can think of either restrict the DST adjustment to intervals specified in months or days, or require an additional parameter. Seeing as changing timestamp arrays between two defined timezones is simply a metadata change it seems reasonable that users can cast to Timestamp(ns, Some("UTC")) if they want to avoid DST adjustment.

I haven't actually put much thought into how one would implement all of this. I'd be interested to hear any other's thoughts.

I think the 6 principles sound good to me -- maybe we should also do some investigation into other implementations of date_bin (I know you say postgres is not a good example, but has any other system figured out how to do this nicely?)

Navigation Menu

Repository navigation

Support `date_bin` on timestamps with timezone, properly accounting for Daylight Savings Time #10602

Is your feature request related to a problem or challenge?

Summary

Details

Describe the solution you'd like

Describe alternatives you've considered

Additional context

4 remaining items

Ideal Behavior

Proposal

1. `date_bin`s should never jump backwards

2. An `origin_timestamp` parameter without a timezone offset acts as if it is in the binned timestamp's timezone

3. A day in a timezone starts at the first instance that reports the date as being that day

4. The value returned from `date_bin` cannot be after the timestamp being binned

5. Intervals adjusted for daylight savings should not be adjusted further than the offset change

6. Using an `expression` with a timezone is sufficient to opt in to DST adjustment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support date_bin on timestamps with timezone, properly accounting for Daylight Savings Time #10602

Description

Is your feature request related to a problem or challenge?

Summary

Details

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

alamb commented on May 21, 2024

alamb commented on May 21, 2024

alamb commented on May 21, 2024

alamb commented on May 21, 2024

alamb commented on May 21, 2024

appletreeisyellow commented on May 21, 2024

tustvold commented on May 22, 2024

alamb commented on May 22, 2024

mhilton commented on May 23, 2024

tustvold commented on May 23, 2024

appletreeisyellow commented on May 23, 2024

appletreeisyellow commented on May 23, 2024

Abdullahsab3 commented on May 25, 2024

4 remaining items

Abdullahsab3 commented on May 30, 2024

alamb commented on May 31, 2024

Ideal Behavior

Proposal

alamb commented on May 31, 2024

appletreeisyellow commented on Jun 4, 2024

Abdullahsab3 commented on Jun 5, 2024

appletreeisyellow commented on Jun 6, 2024

findepi commented on Jul 3, 2024

alamb commented on Jul 5, 2024

alamb commented on Jul 15, 2024

Omega359 commented on Jul 23, 2024

BurntSushi commented on Jul 27, 2024

adriangb commented on Sep 11, 2026

mhilton commented on Sep 26, 2026

1. date_bins should never jump backwards

2. An origin_timestamp parameter without a timezone offset acts as if it is in the binned timestamp's timezone

3. A day in a timezone starts at the first instance that reports the date as being that day

4. The value returned from date_bin cannot be after the timestamp being binned

5. Intervals adjusted for daylight savings should not be adjusted further than the offset change

6. Using an expression with a timezone is sufficient to opt in to DST adjustment

alamb commented on Sep 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Support `date_bin` on timestamps with timezone, properly accounting for Daylight Savings Time #10602

1. `date_bin`s should never jump backwards

2. An `origin_timestamp` parameter without a timezone offset acts as if it is in the binned timestamp's timezone

4. The value returned from `date_bin` cannot be after the timestamp being binned

6. Using an `expression` with a timezone is sufficient to opt in to DST adjustment