The optimizer no longer automatically rewrites query plans, so aggregates are computed directly from the models instead of from reconstructed data points, as Apache DataFusion creates the following query plan instead of the expected. For both physical plans, the indentation means that an operator is created per field, while only a single operator is created for those that are not indented. This is even though GridStream specifies that its input must be sorted by time and that TimeSeriesTable::new_data_source_exec() specifies that DataSourceExec returns a stream of segments with that specific ordering. Removing the ordering requirement from GridExec and SortedJoinExec fixed the problem, but also removed the guarantee that the segments are provided in the correct order. Also, if the requirement is only removed from GridExec, the sorting is just added between GridExec and SortedJoinExec. Finally, it is unclear if it is worth fixing this until issue #299 is closed.
Current Behaviour
AggregateExec: mode=Final, gby=[], aggr=[sum(time_series_table.wind_speed)]
CoalescePartitionsExec
AggregateExec: mode=Partial, gby=[], aggr=[sum(time_series_table.wind_speed)]
RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1
SortedJoinExec
GridExec: limit=None
SortPreservingMergeExec: [start_time@1 ASC NULLS LAST]|
SortExec: expr=[start_time@1 ASC NULLS LAST], preserve_partitioning=[true]
DataSourceExec: ...
Expected Behaviour
AggregateExec: mode=Final, gby=[], aggr=[sum(time_series_table.wind_speed)]
CoalescePartitionsExec
AggregateExec: mode=Partial, gby=[], aggr=[sum(time_series_table.wind_speed)]
RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1
SortedJoinExec
GridExec: limit=None
DataSourceExec: ...
The optimizer no longer automatically rewrites query plans, so aggregates are computed directly from the models instead of from reconstructed data points, as Apache DataFusion creates the following query plan instead of the expected. For both physical plans, the indentation means that an operator is created per field, while only a single operator is created for those that are not indented. This is even though
GridStreamspecifies that its input must be sorted by time and thatTimeSeriesTable::new_data_source_exec()specifies thatDataSourceExecreturns a stream of segments with that specific ordering. Removing the ordering requirement fromGridExecandSortedJoinExecfixed the problem, but also removed the guarantee that the segments are provided in the correct order. Also, if the requirement is only removed fromGridExec, the sorting is just added betweenGridExecandSortedJoinExec. Finally, it is unclear if it is worth fixing this until issue #299 is closed.Current Behaviour
Expected Behaviour