Problem
After the DataFusion 55 optimizer changes, a ProjectionExec can be placed above a fetch-bearing operator. ProjectionExec preserves row count, but the distributed fetch pushdown currently does not look through it. As a result, plans such as ClickBench q17 gain an unnecessary per-task LocalLimitExec.
Desired behavior
When determining whether to propagate a fetch into a NetworkCoalesceExec input stage, traverse through a ProjectionExec and evaluate the fetch on its child. Preserve the existing handling for GlobalLimitExec, including its skip adjustment.
Acceptance criteria
- A fetch separated by a
ProjectionExec is handled without adding the redundant LocalLimitExec.
- Add a focused regression test for this plan shape.
Context
Follow-up from #540 (comment)
Problem
After the DataFusion 55 optimizer changes, a
ProjectionExeccan be placed above a fetch-bearing operator.ProjectionExecpreserves row count, but the distributed fetch pushdown currently does not look through it. As a result, plans such as ClickBench q17 gain an unnecessary per-taskLocalLimitExec.Desired behavior
When determining whether to propagate a fetch into a
NetworkCoalesceExecinput stage, traverse through aProjectionExecand evaluate the fetch on its child. Preserve the existing handling forGlobalLimitExec, including its skip adjustment.Acceptance criteria
ProjectionExecis handled without adding the redundantLocalLimitExec.Context
Follow-up from #540 (comment)