Repository navigation
Branch 55 - #167
Draft
gene-bordegaray wants to merge 888 commits into
Draft
Branch 55#167gene-bordegaray wants to merge 888 commits into
gene-bordegaray wants to merge 888 commits into
Conversation
…t/datafusion-wasm-app (apache#23866) Bumps [ws](https://github.com/websockets/ws) from 8.18.2 to 8.21.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/websockets/ws/releases">ws's releases</a>.</em></p> <blockquote> <h2>8.21.1</h2> <h1>Bug fixes</h1> <ul> <li>Empty fragments are now counted toward the limit (a2f4e7c0).</li> <li>The default values of the <code>maxBufferedChunks</code> and <code>maxFragments</code> options have been reduced (f197ac65).</li> </ul> <h2>8.21.0</h2> <h1>Features</h1> <ul> <li>Introduced the <code>maxBufferedChunks</code> and <code>maxFragments</code> options (2b2abd45).</li> </ul> <h1>Bug fixes</h1> <ul> <li>Fixed a remote memory exhaustion DoS vulnerability (2b2abd45).</li> </ul> <p>A high volume of tiny fragments and data chunks could be sent by a peer, using modest network traffic, to crash a <code>ws</code> server or client due to OOM.</p> <pre lang="js"><code>import { WebSocket, WebSocketServer } from 'ws'; <p>const wss = new WebSocketServer({ port: 0 }, function () { const data = Buffer.alloc(1); const options = { fin: false }; const { port } = wss.address(); const ws = new WebSocket(<code>ws://localhost:${port}</code>);</p> <p>ws.on('open', function () { (function send() { ws.send(data, options, function (err) { if (err) return; send(); }); })(); });</p> <p>ws.on('error', console.error); ws.on('close', function (code, reason) { console.log(<code>client close - code: ${code} reason: ${reason.toString()}</code>); }); });</p> <p>wss.on('connection', function (ws) { ws.on('error', console.error); ws.on('close', function (code, reason) { console.log(<code>server close - code: ${code} reason: ${reason.toString()}</code>); }); }); </code></pre></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/websockets/ws/commit/ae1de54330cef77e487548890fabfeb9aae1d83d"><code>ae1de54</code></a> [dist] 8.21.1</li> <li><a href="https://github.com/websockets/ws/commit/8e9511b86b3fc6deebbd97dd9af7c9056deea8d1"><code>8e9511b</code></a> [ci] Trust Coveralls Homebrew tap</li> <li><a href="https://github.com/websockets/ws/commit/f197ac65140920bdcecdab74bfc69c2d7858e55d"><code>f197ac6</code></a> [fix] Lower default values of <code>maxBufferedChunks</code> and <code>maxFragments</code></li> <li><a href="https://github.com/websockets/ws/commit/8df8265c2f63fd44af3193a98e23cf38888cd991"><code>8df8265</code></a> [ci] Update actions/checkout action to v7</li> <li><a href="https://github.com/websockets/ws/commit/a2f4e7c046c2112bbce6fef39a083dac77d6f0d2"><code>a2f4e7c</code></a> [fix] Count empty fragments toward the limit (<a href="https://redirect.github.com/websockets/ws/issues/2329">#2329</a>)</li> <li><a href="https://github.com/websockets/ws/commit/e79f912cb3f492ae04c28feb9459a209e186b0ad"><code>e79f912</code></a> [pkg] Approve install scripts for bufferutil and utf-8-validate</li> <li><a href="https://github.com/websockets/ws/commit/4ea355d6d3069394994f82ca1b6d38c32ba208fb"><code>4ea355d</code></a> [doc] Document 32-bit signed integer coercion for option values</li> <li><a href="https://github.com/websockets/ws/commit/2120f4c8c625a76316792680a231496e1b615252"><code>2120f4c</code></a> [example] Remove uuid dependency</li> <li><a href="https://github.com/websockets/ws/commit/4c534a6b8a5224a563af116e85c6ced7d4ca60cf"><code>4c534a6</code></a> [security] Add latest vulnerability to SECURITY.md</li> <li><a href="https://github.com/websockets/ws/commit/bca91adf15677e47dbe4f959653452727be28b94"><code>bca91ad</code></a> [dist] 8.21.0</li> <li>Additional commits viewable in <a href="https://github.com/websockets/ws/compare/8.18.2...8.21.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…datafusion/wasmtest/datafusion-wasm-app (apache#23865) Bumps [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware) from 2.0.9 to 2.0.10. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/chimurai/http-proxy-middleware/releases">http-proxy-middleware's releases</a>.</em></p> <blockquote> <h2>v2.0.10-beta.0</h2> <h2>What's Changed</h2> <ul> <li>fix: harden proxy-table matching to prevent routing bypass by <a href="https://github.com/G-Rath"><code>@G-Rath</code></a> in <a href="https://redirect.github.com/chimurai/http-proxy-middleware/pull/1268">chimurai/http-proxy-middleware#1268</a></li> <li>ci(github-actions): update publish.yml by <a href="https://github.com/chimurai"><code>@chimurai</code></a> in <a href="https://redirect.github.com/chimurai/http-proxy-middleware/pull/1270">chimurai/http-proxy-middleware#1270</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/G-Rath"><code>@G-Rath</code></a> made their first contribution in <a href="https://redirect.github.com/chimurai/http-proxy-middleware/pull/1268">chimurai/http-proxy-middleware#1268</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/chimurai/http-proxy-middleware/compare/v2.0.9...v2.0.10-beta.0">https://github.com/chimurai/http-proxy-middleware/compare/v2.0.9...v2.0.10-beta.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/chimurai/http-proxy-middleware/blob/v2.0.10/CHANGELOG.md">http-proxy-middleware's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v2.0.10">v2.0.10</a></h2> <ul> <li>fix(router): harden proxy-table matching (exact host for host+path keys, prefix-only path matching) to prevent routing bypass</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/chimurai/http-proxy-middleware/commit/f0be839066ba14f2598c6c1aef10ff54b020babb"><code>f0be839</code></a> chore(package.json): v2.0.10 (<a href="https://redirect.github.com/chimurai/http-proxy-middleware/issues/1271">#1271</a>)</li> <li><a href="https://github.com/chimurai/http-proxy-middleware/commit/19c860de9854988351c4c9bf5fe79687942e1833"><code>19c860d</code></a> ci(github-actions): update publish.yml (<a href="https://redirect.github.com/chimurai/http-proxy-middleware/issues/1270">#1270</a>)</li> <li><a href="https://github.com/chimurai/http-proxy-middleware/commit/d0f7d6368ac9cf7dac5638c466d820370a4ae98f"><code>d0f7d63</code></a> fix: harden proxy-table matching to prevent routing bypass (<a href="https://redirect.github.com/chimurai/http-proxy-middleware/issues/1268">#1268</a>)</li> <li>See full diff in <a href="https://github.com/chimurai/http-proxy-middleware/compare/v2.0.9...v2.0.10">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~GitHub%20Actions">GitHub Actions</a>, a new releaser for http-proxy-middleware since your current version.</p> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
add badge; easier to see at a glance what the coverage is, and also easier to access our codecov page <img width="789" height="145" alt="image" src="https://github.com/user-attachments/assets/ed44f56c-8daa-48c7-809a-679823084e0f" />
## Rationale for this change Logical `CASE` nullability unwraps null preserving casts before analyzing guarded branches, but physical `CASE` nullability did not. Type coercion could therefore produce conflicting schemas and cause valid aggregation queries to fail during planning --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…driven optimizations (apache#23821) ## Which issue does this PR close? This PR adds test coverage rather than closing an issue. It documents the current behavior of these bugs so that fixing them shows up as a test change: - apache#23634 - apache#23818 - apache#23819 - apache#23820 ## Rationale for this change While reviewing apache#23636 @neilconway and I kept coming up with more examples of bad plans, and it was not obvious which of the surrounding wrong answers were pre-existing and which the PR introduced. ## What changes are included in this PR? A new `datafusion/sqllogictest/test_files/functional_dependencies.slt` with one section per consumer of functional dependencies: 1. `ReplaceDistinctWithAggregate` — removing `DISTINCT` 2. `eliminate_duplicated_expr` — dropping trailing `ORDER BY` keys 3. `optimize_projections` — dropping `GROUP BY` expressions 4. `add_group_by_exprs_from_dependencies` — selecting non-grouped columns 5. `GROUP BY` derived keys on the NULL-padded side of an outer join Cases that currently return wrong answers are labelled `BUG` with the expected result and a link to the issue: | Case | Symptom | Issue | | --- | --- | --- | | 1.2 | `DISTINCT` over a nullable `UNIQUE` column returns both `NULL` rows | apache#23634 | | 2.2 | `ORDER BY x, y` drops the `y` key, so the `NULL` rows come back unordered | apache#23818 | | 3.2 | `GROUP BY x, y` drops `y`, merging the two `NULL` groups and losing a row | apache#23819 | | 4.2 | `SELECT x, y ... GROUP BY x` returns two rows for the `x = NULL` group | apache#23820 | ## Are these changes tested? CI ## Are there any user-facing changes? No. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…3679) ## Which issue does this PR close? - Closes: N/A ## Rationale for this change There are a reasonable number of places where we have functions marked as `async` that don't need to be so. Fix this by enabling the `unused_async` lint, and fixing up the resulting breakage. There are a handful of spots that need an `expect(clippy::unused_async)` -- mostly mocks of `async` methods and example code. ## What changes are included in this PR? * Enable `unused_async` lint * Remove unnecessary `async` annotations in a bunch of places * Add `expect(clippy::unused_async)` where necessary ## Are these changes tested? Yes, covered by existing tests (no behavioral change expected). ## Are there any user-facing changes? Yes: this PR updates a few public APIs: - `datafusion_cli::command::OutputFormat::execute` - `datafusion::test_util::parquet::TestParquetFile::create_scan` - `datafusion_datasource_csv::file_format::CsvFormat::read_to_delimited_chunks_from_stream` - `datafusion_substrait::serializer::deserialize_bytes` Migration is mostly straightforward (e.g., removing `await` from calling code).
…fusion/wasmtest/datafusion-wasm-app (apache#23868) Bumps [webpack-dev-server](https://github.com/webpack/webpack-dev-server) from 5.2.6 to 6.0.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/webpack/webpack-dev-server/releases">webpack-dev-server's releases</a>.</em></p> <blockquote> <h2>v6.0.0</h2> <h3>Major Changes</h3> <ul> <li> <p>Bump Express to v5. See the <a href="https://expressjs.com/en/guide/migrating-5.html">Express 5 migration guide</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Bump the <code>webpack</code> peer dependency range from <code>^5.0.0</code> to <code>^5.101.0</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Drop support for Node.js < 22.15.0. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Convert the source to native ES modules. The package keeps <code>"type": "module"</code> and now exposes both an ESM and a CommonJS build via the <code>exports</code> field: ESM consumers import the native <code>lib/</code>, while CommonJS consumers <code>require()</code> a transpiled <code>dist/</code> build, allowing the package to be consumed from both ESM and CommonJS without relying on <code>require(ESM)</code> for CommonJS consumers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5674">#5674</a>)</p> </li> <li> <p>Remove CLI flags. Use the <code>serve</code> command from <code>webpack-cli</code> together with a configuration file or the programmatic API instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>internalIP</code> and <code>internalIPSync</code> static methods from <code>Server</code>. Resolve the local IP yourself if you need it. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>bypass</code> option from proxy configuration. Use the <code>router</code> or <code>context</code> options provided by <code>http-proxy-middleware</code> instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove SockJS support. The <code>webSocketServer</code> option no longer accepts <code>"sockjs"</code>; use the default <code>"ws"</code> transport instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>spdy</code> dependency. Use the built-in <code>node:http2</code> module via the <code>server</code> option for HTTP/2 support. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>http-proxy-middleware</code> to v4. See the <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v3.0.0">http-proxy-middleware v3 release notes</a> and <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v4.0.0">v4 release notes</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>webpack-dev-middleware</code> to v8 and sync <code>originalUrl</code> for middleware compatibility. <code>server.middleware.getFilenameFromUrl()</code> is now asynchronous and resolves to <code>{ filename, extra: { stats, outputFileSystem } }</code>. See the <a href="https://github.com/webpack/webpack-dev-middleware/releases/tag/v8.0.0">webpack-dev-middleware v8 release notes</a> for details. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Minor Changes</h3> <ul> <li> <p>Add plugin support. <code>webpack-dev-server</code> can now be used as a webpack plugin, integrating with the compiler lifecycle without explicitly passing a compiler, preventing multiple server starts on recompilation, ensuring clean shutdown, and supporting <code>MultiCompiler</code> setups with multiple independent plugin servers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Enable the compression middleware for HTTP/2 connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>colorette</code> dependency in favor of native ANSI styling. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>chokidar</code> to v5 and extend <code>watchFiles.options.ignored</code> to support glob string patterns via <code>tinyglobby</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use <code>compiler.platform</code> to determine the target environment instead of inspecting the resolved <code>target</code> string. Universal targets (<code>"universal"</code> or <code>["web", "node"]</code>, where <code>compiler.platform.universal</code> is <code>true</code> since webpack <code>5.108.0</code>) are treated as web targets so the client runtime is injected. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use the WHATWG <code>URL</code> API instead of the deprecated <code>url.parse</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Patch Changes</h3> <ul> <li> <p>Bump production dependencies, notably <code>open</code> to v11 and <code>p-retry</code> to v8. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Reject cross-site requests to the internal <code>open-editor</code> and <code>invalidate</code> endpoints. They performed state-changing actions (opening a file in the editor, forcing a recompilation) on any GET request, so a page the developer visited could trigger them. They now require a same-origin request, validated via <code>Sec-Fetch-Site</code> with an <code>Origin</code>/<code>Host</code> fallback. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5691">#5691</a>)</p> </li> <li> <p>Treat loopback aliases (<code>127.0.0.1</code>, <code>::1</code>, <code>localhost</code>) as equivalent in <code>isSameOrigin</code> so the WebSocket client does not reject valid same-origin connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Migrate the test suite from Jest to <code>node:test</code> and set up the jsdom environment. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/webpack/webpack-dev-server/blob/main/CHANGELOG.md">webpack-dev-server's changelog</a>.</em></p> <blockquote> <h2>6.0.0</h2> <h3>Major Changes</h3> <ul> <li> <p>Bump Express to v5. See the <a href="https://expressjs.com/en/guide/migrating-5.html">Express 5 migration guide</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Bump the <code>webpack</code> peer dependency range from <code>^5.0.0</code> to <code>^5.101.0</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Drop support for Node.js < 22.15.0. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Convert the source to native ES modules. The package keeps <code>"type": "module"</code> and now exposes both an ESM and a CommonJS build via the <code>exports</code> field: ESM consumers <code>import</code> the native <code>lib/</code>, while CommonJS consumers <code>require()</code> a transpiled <code>dist/</code> build — so the package works from both ESM and CommonJS, including environments where <code>require(ESM)</code> is not supported. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove CLI flags. Use the <code>serve</code> command from <code>webpack-cli</code> together with a configuration file or the programmatic API instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>internalIP</code> and <code>internalIPSync</code> static methods from <code>Server</code>. Resolve the local IP yourself if you need it. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>bypass</code> option from proxy configuration. Use the <code>router</code> or <code>context</code> options provided by <code>http-proxy-middleware</code> instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove SockJS support. The <code>webSocketServer</code> option no longer accepts <code>"sockjs"</code>; use the default <code>"ws"</code> transport instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>spdy</code> dependency. Use the built-in <code>node:http2</code> module via the <code>server</code> option for HTTP/2 support. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>http-proxy-middleware</code> to v4. See the <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v3.0.0">http-proxy-middleware v3 release notes</a> and <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v4.0.0">v4 release notes</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>webpack-dev-middleware</code> to v8 and sync <code>originalUrl</code> for middleware compatibility. <code>server.middleware.getFilenameFromUrl()</code> is now asynchronous and resolves to <code>{ filename, extra: { stats, outputFileSystem } }</code>. See the <a href="https://github.com/webpack/webpack-dev-middleware/releases/tag/v8.0.0">webpack-dev-middleware v8 release notes</a> for details. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Minor Changes</h3> <ul> <li> <p>Add plugin support. <code>webpack-dev-server</code> can now be used as a webpack plugin, integrating with the compiler lifecycle without explicitly passing a compiler, preventing multiple server starts on recompilation, ensuring clean shutdown, and supporting <code>MultiCompiler</code> setups with multiple independent plugin servers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Enable the compression middleware for HTTP/2 connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>colorette</code> dependency in favor of native ANSI styling. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>chokidar</code> to v5 and extend <code>watchFiles.options.ignored</code> to support glob string patterns via <code>tinyglobby</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use <code>compiler.platform</code> to determine the target environment instead of inspecting the resolved <code>target</code> string. Universal targets (<code>"universal"</code> or <code>["web", "node"]</code>, where <code>compiler.platform.universal</code> is <code>true</code> since webpack <code>5.108.0</code>) are treated as web targets so the client runtime is injected. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use the WHATWG <code>URL</code> API instead of the deprecated <code>url.parse</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Patch Changes</h3> <ul> <li> <p>Bump production dependencies, notably <code>open</code> to v11 and <code>p-retry</code> to v8. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Reject cross-site requests to the internal <code>open-editor</code> and <code>invalidate</code> endpoints. They performed state-changing actions (opening a file in the editor, forcing a recompilation) on any GET request, so a page the developer visited could trigger them. They now require a same-origin request, validated via <code>Sec-Fetch-Site</code> with an <code>Origin</code>/<code>Host</code> fallback. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5691">#5691</a>)</p> </li> <li> <p>Treat loopback aliases (<code>127.0.0.1</code>, <code>::1</code>, <code>localhost</code>) as equivalent in <code>isSameOrigin</code> so the WebSocket client does not reject valid same-origin connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Migrate the test suite from Jest to <code>node:test</code> and set up the jsdom environment. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/webpack/webpack-dev-server/commit/05cb7921b2cd216f8caa74d2bdcc39b3f0d05bea"><code>05cb792</code></a> chore(release): new release (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5692">#5692</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/a451839256733b37c8b71d15b9e22a55db3290d1"><code>a451839</code></a> fix: handle middleware teardown in plugin mode (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5703">#5703</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/c2d23a7e9f8562c634b3f3cabf8cee18123a402b"><code>c2d23a7</code></a> fix: load ESM-only dependencies with native import() in the CommonJS build (#...</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/ba54764c65464b43ab695ab8e7f20ca8e6c92e8d"><code>ba54764</code></a> fix: reject cross-site requests to open-editor and invalidate endpoints (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5691">#5691</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/2b369b39d537331fc44df3d3c1d62f2ac4598016"><code>2b369b3</code></a> fixup!</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/08a0ea7e92b2f954d2ab88e4b9669a45108cf0c8"><code>08a0ea7</code></a> fix: ensure undefined options default to an empty object in Server constructor</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/797b9e7d780eb5202895021bbd47253774a0135d"><code>797b9e7</code></a> fix: handle undefined options in Server constructor</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/e90221cd422dbaa71b3a870538be1c88caf98361"><code>e90221c</code></a> feat: plugin support (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5650">#5650</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/4c351e1b45f6a0bcc5fb3ef1e6bfad2d0c4fdd8b"><code>4c351e1</code></a> feat: support universal platform as a web target (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5690">#5690</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/2236aa4b490a0b6abfd9e234beb3da5a32da91e7"><code>2236aa4</code></a> chore: update http-proxy-middleware to version 4.1.1 and add tests for pathRe...</li> <li>Additional commits viewable in <a href="https://github.com/webpack/webpack-dev-server/compare/v5.2.6...v6.0.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? - Closes apache#23507. ## Rationale for this change Part of epic apache#23494. Moves `HashJoinExec` protobuf serialization from central dispatch. ## What changes are included in this PR? Add protobuf serialization and deserialization to the `HashJoinExec` physical plan implementation and deprecate the corresponding central proto methods. The wire format remains unchanged. ## Are these changes tested? Yes, existing HashJoin round-trip tests cover this plan. ## Are there any user-facing changes? Existing methods are deprecated with no immediate API change.
…che#23825) Closes apache#23514. Part of apache#23494. Migrate `AsyncFuncExec` proto encode/decode into the plan itself via `try_to_proto` / `try_from_proto`, removing its central-arm handling in `physical_plan/mod.rs`. ## Rationale `datafusion-proto` currently downcasts `AsyncFuncExec` in the central encode match and rebuilds it inline on decode. apache#23495 introduced self-serializing hooks so each plan owns its own wire format. This moves `AsyncFuncExec` onto that pattern. ## Changes - Added `try_to_proto` / `AsyncFuncExec::try_from_proto`, wired into the decode dispatch - Removed the central encode downcast branch - Old helper methods kept as `#[deprecated]` stubs, per existing convention - Wire format unchanged ## Testing Existing `roundtrip_async_func_exec` integration test now exercises the new hooks (old path deleted, so it's the only path left). `cargo fmt` + `cargo clippy --all-targets --features proto -- -D warnings` clean. ## User-facing changes No Co-authored-by: Matthew Patton <matthewpatton@macbookpro.mynetworksettings.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23513. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Window plans still relied on centralized protobuf dispatch, keeping serialization separate from the execution plans that own their state. Both window executors share one protobuf variant. Decoding must inspect `input_order_mode` before selecting the concrete plan. Add plan-local encoders for both executors and a decoder for the shared node. Keep the legacy helpers as deprecated delegates while preserving window frames and UDF payloads on the existing wire format. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23511. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Physical plan protobuf serialization is being moved from the central dispatch module into each `ExecutionPlan` implementation. Co-locating this logic with the corresponding execution plan makes the serialization code easier to maintain and incrementally reduces the central downcast chain. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Implement plan-local protobuf serialization and deserialization for `ExplainExec`. - Implement plan-local protobuf serialization and deserialization for `AnalyzeExec`. - Remove both plans from the live central serialization dispatch. - Retain the deprecated compatibility methods as delegates to the new implementations. - Preserve the existing protobuf wire format. - Strengthen roundtrip tests to cover all stored fields, including every `StringifiedPlan` variant, metric categories, and explain formats. Each execution plan migration is kept in a separate commit. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. The following checks pass: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - Full extended workspace test suite - `cargo test -p datafusion-proto --test proto_integration` The proto integration suite passes all 209 tests after rebasing onto the latest `main`. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> No. This is an internal refactor and does not change the protobuf wire format or user-facing behavior. --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23512. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Aggregate protobuf conversion lived in the central proto crate, which prevented AggregateExec from owning its function-codec serialization. Move encoding and decoding into AggregateExec and keep the deprecated helpers as delegates so existing callers retain wire-compatible behavior. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
…nparser (apache#23789) ## Which issue does this PR close? - Closes apache#23668. ## Rationale for this change - When printing a GROUP BY query, the unparser sometimes wraps the aggregate's input in an inner subquery (`... FROM (SELECT ...)`). - Table aliases like `cs` only exist inside that subquery. Any clause outside it (SELECT, GROUP BY, HAVING, QUALIFY, ORDER BY) must use the subquery's output columns, not the aliases, so the unparser drops the alias. - Each clause does that dropping on its own, so it is easy to miss one. - ORDER BY on an aggregate that is not in the SELECT list was missed: it printed `ORDER BY round(sum("cs"."total_revenue"), 2)`, while the SELECT list in the same query correctly printed `sum("total_revenue")`. - DataFusion reads that SQL back fine, but stricter databases reject it because `cs` is out of scope there. ## What changes are included in this PR? I moved the rule into one helper, `UnparserAggScope`, that checks once per aggregate whether the input is an inner subquery and then prepares expressions for each clause. SELECT, GROUP BY, HAVING, QUALIFY, and both ORDER BY paths now go through it, which fixes the ORDER BY case. The window-over-aggregate path is left as-is with a comment: it is only reachable from hand-built plans, since a window in SQL always sits inside a SELECT that already drops the alias. Only ORDER BY output changes; every already-correct case stays the same. ## Are these changes tested? Yes. New tests in `datafusion/core/tests/sql/unparser.rs` cover the inner-subquery shape for a window sorting by an aggregate, ORDER BY on an unselected aggregate (the fixed case), and a top-level ORDER BY (the second sort path). Existing unparser tests and the TPC-H/Clickbench roundtrips still pass. ## Are there any user-facing changes? ORDER BY over an aggregate is now printed without an out-of-scope table qualifier, so the generated SQL is valid for stricter databases. No API changes.
…che#22980) ## Which issue does this PR close? * Part of apache#20835 ## Rationale for this change `FixedSizeList` containing `Struct` values was not handled by the existing recursive nested adaptation logic used for schema evolution. As a result, planner-time compatibility checks, nested cast detection, and runtime casting did not support additive struct evolution within `FixedSizeList` containers. This change adds `FixedSizeList` support and verifies planner/runtime parity so that planning allows exactly the cases runtime can adapt while continuing to reject incompatible schema changes. ## What changes are included in this PR? * Extend `cast_column` to support recursive casting of `FixedSizeList` values when source and target list sizes match. * Add `FixedSizeList` handling to: * `requires_nested_struct_cast` * `validate_data_type_compatibility` * Implement recursive casting of nested `Struct` values contained in `FixedSizeList`. * Preserve planner/runtime parity by validating child type compatibility before runtime fallback logic is applied. * Add handling for null-parent `FixedSizeList` entries by masking hidden child values before retrying casts, avoiding failures caused by semantically inaccessible child data. * Refactor list and list-view casting helpers to use Arrow `AsArray` accessors. ## Are these changes tested? Yes. The following tests were added: * `test_cast_fixed_size_list_struct` * `test_validate_fixed_size_list_struct_compatibility` * `test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected` * `test_validate_fixed_size_list_struct_size_mismatch_rejected` * `test_cast_fixed_size_list_struct_all_null` * `test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type` * `test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected` * `test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent` Existing coverage in `test_requires_nested_struct_cast` was also extended to include `FixedSizeList` cases. These tests cover: * Additive nullable nested-field evolution * All-null and partially null list cases * Incompatible nested type changes * Non-nullable field addition rejection * Planner/runtime parity validation ## Are there any user-facing changes? No user-facing changes. This is an internal enhancement to nested schema adaptation and casting behavior for `FixedSizeList<Struct>` types. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
… spark (apache#23766) ## Which issue does this PR close? N/A ## Rationale for this change Hex encoding was implemented six times across the workspace, at two different levels of optimization. Three copies used a fast byte-pair lookup table: - `datafusion/spark/src/function/math/hex.rs` - `datafusion/functions/src/string/to_hex.rs` - `datafusion/functions/src/encoding/inner.rs` (via the `hex` crate) The other three used a slower nibble-at-a-time loop pushing one character at a time: - `datafusion/functions/src/crypto/md5.rs` - `datafusion/spark/src/function/hash/sha1.rs` - `datafusion/spark/src/function/hash/sha2.rs` Beyond the duplication, this split meant the digest functions were paying for a slower encoder than the one already sitting elsewhere in the tree. Consolidating on a single implementation removes the duplication and moves `md5`, `sha1`, and `sha2` onto the fast path. ## What changes are included in this PR? A new `datafusion_common::utils::hex` module holds the only hex encoder in the workspace: ```rust pub enum HexCase { Lower, Upper } pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>); pub fn encode_bytes(bytes: &[u8], case: HexCase) -> String; pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]); pub fn encode_u64(v: u64, case: HexCase, buf: &mut [u8; 16]) -> &[u8]; ``` Four entry points rather than one because the call sites have genuinely different output needs, and forcing them through a single shape would cost an allocation somewhere: `to_hex` writes straight into a `StringArray` values buffer, Spark's `hex` appends to a reused scratch `Vec`, `encode` writes into a pre-sized slice, and the digest functions want an owned `String`. Migrated call sites, all producing bit-identical output: | File | Change | | --- | --- | | `functions/src/string/to_hex.rs` | local table and two write helpers deleted; eight trait impls collapsed to two macros | | `spark/src/function/math/hex.rs` | two nibble tables, two lookup tables, `build_hex_lookup` and `hex_int64` deleted | | `functions/src/crypto/md5.rs` | local table and `hex_encode` deleted | | `spark/src/function/hash/sha1.rs` | local table and inline nibble loop deleted | | `spark/src/function/hash/sha2.rs` | local table and `hex_encode` deleted; eight call sites migrated | | `functions/src/encoding/inner.rs` | both encode sites moved off the `hex` crate | Deliberately left alone: - `hex::decode` / `hex::decode_to_slice` in `encoding/inner.rs`, and Spark's `unhex`. The decode direction has different semantics — Spark's `unhex` left-pads odd-length input, the `hex` crate does not — so unifying it is a separate question. The `hex` dependency stays for those. - `ScalarValue`'s binary `Display` impl in `common/src/scalar/mod.rs`, which writes to a `fmt::Formatter` rather than a byte buffer, and is a cold path. Three details worth a reviewer's attention: - The two ancestors of `encode_u64` disagreed on zero: `to_hex` wrote `'0'` into the caller's buffer, Spark's `hex_int64` returned a `'static` `b"0"` that never touched it. The shared version always writes into the buffer and returns a subslice of it, so the lifetime is uniform. Both callers still produce `"0"`. - Spark's `hex_encode_bytes` guards large binary input with `checked_mul(2)` + `try_reserve`, returning a `DataFusionError` rather than aborting on allocation failure. That guard stays at the call site; `encode_bytes_into` performs no reservation of its own, so behaviour is unchanged. - The encoders are `#[inline(always)]`, not `#[inline]`. They are called once per row from two other crates, and plain `#[inline]` left them out-of-line across the crate boundary. Measured cost of that: +3% on Spark's byte paths and up to +18% on `to_hex`'s i32 path, i.e. the refactor was a net regression on those benchmarks until the attribute changed. ## Are these changes tested? Every migrated function keeps its existing unit and sqllogictest coverage, which is what pins bit-identical output — in particular `spark/hash/sha1.slt` and `sha2.slt` assert concrete digest strings for all four SHA-2 bit lengths across both the scalar and array paths, and `expr.slt` covers `to_hex` and `md5`. New unit tests in `common/src/utils/hex.rs` cover zero, `u64::MAX`, single-nibble values, the odd/even digit-count boundary, two's complement of negative input, empty input, all 256 byte values in both cases, appending into a non-empty buffer, and that a reused scratch buffer never leaks stale digits between calls. Tests cross-check against `format!("{:x}")` rather than restating the implementation. Two Spark tests changed. `test_hex_int64` now drives `hex_encode_int64` instead of the deleted private `hex_int64`, keeping all ten cases including `i64::MIN`, `-1`, and the uppercase expectations. `test_hex_lookup_table_covers_all_bytes` was deleted — it only cross-checked the raw lookup tables, and `encode_bytes_covers_every_byte_value` now does that exhaustively through the public API. A test was added for Spark's lowercase byte path, which previously had no coverage. ### Benchmarks Criterion, `apache/main` @ `eef101769` as baseline. Median of the reported change interval. `datafusion/functions/benches/to_hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `i32_random` | −13.8% | −12.8% | −10.2% | | `i64_random` | −8.8% | −9.5% | −8.1% | | `i64_large_values` | −9.9% | −9.8% | −7.9% | `scalar_i32` −2.3%, `scalar_i64` −1.8%. `datafusion/spark/benches/sha2.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `array_binary_256` | −20.7% | −18.2% | −19.1% | | `array_scalar_binary_256` | −14.6% | −13.2% | −13.1% | `scalar/size=1` −3.3%. `datafusion/spark/benches/hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `hex_int64` | −2.9% | −3.6% | −3.5% | | `hex_int64_dict` | −3.2% | −1.5% | −0.2% | | `hex_utf8` | −0.2% | −0.3% | +0.7% | | `hex_binary` | −0.4% | −0.6% | −0.3% | The `hex_utf8` and `hex_binary` paths already used the byte-pair table before this PR, so they are expected to be flat; they are. `datafusion/functions/benches/crypto.rs`: `md5_array` −4.0%, `md5_scalar` −3.7%. The `sha224` / `sha256` / `sha384` / `sha512` cases in that file range from −0.1% to +2.2%, but they exercise `crypto/basic.rs`, which this PR does not modify and which contains no hex encoding — those numbers are run-to-run variance, not an effect of this change. No number is quoted for Spark `sha1`: it has no benchmark, and its change is the same substitution applied to `md5` and `sha2`. ## Are there any user-facing changes? No behaviour change — all migrated functions produce byte-identical output. `datafusion_common::utils::hex` is new public API on `datafusion-common`. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…t_batch` memory (apache#23873) ## Which issue does this PR close? - related to apache#23716 ## Rationale for this change It adds test coverage for two gaps found while reviewing apache#23716. ## What changes are included in this PR? Tests only, no functional change. - Dictionary inputs - memory usage on retract (make sure memory is released) Note I moved `array_agg` cases out `aggregate.slt` as it is already more than 9k lines long ## Are these changes tested? They are only tests ## Are there any user-facing changes? No. Tests only, no public API changes. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…3881) ## Which issue does this PR close? N/A ## Rationale for this change Two Spark functions allocated a `String` per row purely to render a small, bounded amount of text: - `bin` called `format!("{value:b}")` for every row. - `char` called `ch.to_string()` for every row — a heap allocation for a single character. In both cases the output has a known upper bound (64 binary digits for an `i64`, 4 bytes for a UTF-8 character), so the rendering fits in a stack buffer and the result can be appended straight to a pre-sized builder. ## What changes are included in this PR? `math/bin.rs`: - `spark_bin` now writes digits right-aligned into a caller-supplied `[u8; 64]` and returns a `&str` borrowed from it, instead of returning an owned `String`. - The `collect::<StringArray>()` becomes an explicit loop over a `StringBuilder::with_capacity`, sized at 8 digits per row. - Negative values still render as their two's-complement bit pattern, matching `{:b}`. The digit loop is a `loop`, not a `while`, so zero renders as `"0"` rather than the empty string. `string/char.rs`: - `ch.to_string()` becomes `ch.encode_utf8(&mut encoded)` against a `[u8; 4]` hoisted out of the loop. Output is unchanged in both cases. ## Are these changes tested? Existing coverage pins the behaviour. `spark/math/bin.slt` asserts concrete output for the cases the rewrite had to get right: zero, negative values, `i64::MIN` (`-9223372036854775808`), `i64::MAX`, and `-2147483648` / `-32768` widened from narrower integer types. `spark/string/char.slt` covers the negative-input empty string, the null path, and characters on both sides of the ASCII boundary (`char(256)` and above wrap via `% 256`). All 119 `spark/math` and `spark/string` sqllogictest files pass, along with the 258 `datafusion-spark` unit tests. The `bin` benchmark used below, `datafusion/spark/benches/bin.rs`, is added separately in apache#23882 so the baseline can be measured on `main` before this change lands. It covers 1024 and 8192 rows with 20% nulls over two value distributions: small values that render to a handful of digits, and full-range values that render to the maximum 64. `char` already had `datafusion/spark/benches/char.rs` on `main`, so no benchmark change is needed for it. ### Benchmarks Criterion, `apache/main` @ `f1ab86dad` as baseline. Median of the reported change interval. | Benchmark | 1024 | 8192 | | --- | --- | --- | | `bin/small` | −75.7% | −73.7% | | `bin/wide` | −47.8% | −49.8% | `char` (1024 rows): −76.1%. ## Are there any user-facing changes? No. Both functions produce byte-identical output; this is purely an allocation change.
…ts enabled (apache#23848) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23847 . ## Rationale for this change Fixes a correctness bug, not sure when that config used/desirable. This is another thing I ran into during apache#21585 ## What changes are included in this PR? 1. New SLT test 2. Fix for bug in `datafusion-optimizer` ## Are these changes tested? Existing tests, and a new SLT test verifying the config change doesn't affect the query result and basic unit test. ## Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
…lter (apache#23901) ## Which issue does this PR close? - Closes apache#23900. ## Rationale for this change `push_down_filter` infers equi-key predicates across a join's ON keys and pushes them to the opposite side. For a null-aware join (the `LeftAnti` join produced by `NOT IN` with a nullable subquery), an outer predicate on the left key like `outer.id > 5` is rewritten to `sub.id > 5` and pushed onto the subquery input. Since the inferred predicate must be null-rejecting to be pushed, this drops the subquery's NULL rows and breaks the three-valued `NOT IN` semantics — a NULL in the subquery key must reach the join so the result is empty. Same class of bug as apache#23848, in a different rule. ## What changes are included in this PR? - Skip predicate inference in `infer_join_predicates` when `join.null_aware` is set (mirrors the apache#23848 guard on `FilterNullJoinKeys`). - A `push_down_filter` unit test asserting no predicate is inferred onto the subquery side of a null-aware `LeftAnti` join. - SLT coverage for the failing query, plus a `prefer_hash_join = false` / multi-partition variant. ## Are these changes tested? Yes — new unit test (verified it fails without the guard) and SLT cases. The full optimizer lib suite passes. ## Are there any user-facing changes? No, aside from the correctness fix.
## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#23311](apache#23311). - Next in stack: apache#23015. - Extracted from apache#19390. ## Rationale for this change For very small `IN` lists, building or probing a hash table can be more work than just comparing the input value with each constant. For example, for `x IN (10, 20, 30)`, the fast path can behave like: ```text x == 10 OR x == 20 OR x == 30 ``` Because the list is tiny, those comparisons are cheap. The implementation stores the constants in a fixed-size array and checks them with a compact comparison chain. “Branchless” here means the comparisons are combined without stopping at the first match. That can be faster for these small fixed-width lists because the CPU gets a predictable sequence of simple operations instead of hash-table setup and probe logic. For primitive values that are not already plain unsigned integers, this PR keeps the logical Arrow type explicit and uses a matching same-width comparison representation only inside the branchless filter. For example, `Float16` uses `UInt16` storage, `Float32` uses `UInt32` storage, and `TimestampNanosecond` uses `UInt64` storage. `Decimal128` and `IntervalMonthDayNano` use their own 16-byte native representation. This preserves bit-pattern equality while relying on Arrow's native primitive compatibility rules: timestamp timezone metadata and Decimal128 precision/scale metadata may differ, while incompatible primitive representations remain rejected. ## What changes are included in this PR? - Adds a const-generic `BranchlessFilter` for small primitive `IN` lists. - Adds thresholds for when this path is used: - up to 16 values for 1-byte types - up to 8 values for 2-byte types - up to 32 values for 4-byte types - up to 16 values for 8-byte types - up to 4 values for 16-byte types - Keeps dispatch concrete and explicit in `strategy.rs`. - Maps each optimized logical type to the comparison representation used by the branchless filter: - `Int8` -> `UInt8` - `Int16`, `Float16` -> `UInt16` - `Int32`, `Float32`, `Date32`, `Time32` -> `UInt32` - `Int64`, `Float64`, `Date64`, `Time64`, `Timestamp`, `Duration` -> `UInt64` - `Decimal128`, `IntervalMonthDayNano` -> their native 16-byte representation - Leaves larger 1-byte and 2-byte lists on the existing bitmap filters. - Leaves larger 4-byte and 8-byte lists on the existing hash/generic paths. - Leaves wider primitive types such as `Decimal256` and unsupported complex types on the generic path. - Keeps the same `IN` / `NOT IN` null behavior as the rest of the stack. - Adds focused coverage for branchless null handling, signed boundary values, slices, Float16/Float32/Float64 bit patterns, compatible timestamp/Decimal128 metadata, incompatible timestamp units, IntervalMonthDayNano values, and same-width wrong-type probe rejection. ## Are these changes tested? Yes. - `cargo fmt --all -- --check` - `cargo test -p datafusion-physical-expr expressions::in_list --lib` - `cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This is an internal performance optimization only. ## Local benchmark snapshot Built and run with `release-nonlto`, filtered to the relevant small primitive-list rows: ```bash cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline> ``` Filters used: `narrow_integer`, `primitive/i32/small_list`, `primitive/i64/small_list`, `f32/small_list`, `timestamp_ns/small_list`, and `interval_month_day_nano/small_list`. Method: directly compared Criterion's raw sample minima (`min(time / iterations)`) from `sample.json`. Lower is better; changes within +/-5% are treated as noise. Compared baselines: [apache#23311](apache#23311) -> [apache#23014](apache#23014) Relevant scope: small primitive-list rows. Summary: 39 relevant rows, 28 faster, 0 slower, 11 within +/-5%. Largest relevant deltas: | Benchmark | Before | After | Change | |---|---:|---:|---:| | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | <details> <summary>Full relevant table (39 rows)</summary> | Benchmark | Before | After | Change | |---|---:|---:|---:| | `narrow_integer/u8/list=4/match=0%` | 3.86 us | 2.79 us | -27.8% (1.38x faster) | | `narrow_integer/u8/list=4/match=50%` | 3.84 us | 2.78 us | -27.7% (1.38x faster) | | `narrow_integer/u8/list=16/match=0%` | 3.88 us | 3.85 us | -0.8% (within noise) | | `narrow_integer/u8/list=16/match=50%` | 3.84 us | 3.86 us | +0.5% (within noise) | | `narrow_integer/i16/list=4/match=0%` | 3.93 us | 3.18 us | -19.1% (1.24x faster) | | `narrow_integer/i16/list=4/match=50%` | 3.92 us | 3.16 us | -19.5% (1.24x faster) | | `narrow_integer/i16/list=64/match=0%` | 3.96 us | 3.82 us | -3.5% (within noise) | | `narrow_integer/i16/list=64/match=50%` | 3.91 us | 3.80 us | -2.9% (within noise) | | `narrow_integer/i16/list=256/match=0%` | 3.90 us | 3.81 us | -2.5% (within noise) | | `narrow_integer/i16/list=256/match=50%` | 3.97 us | 3.81 us | -4.1% (within noise) | | `narrow_integer/f16/list=4/match=0%` | 3.87 us | 3.16 us | -18.5% (1.23x faster) | | `narrow_integer/f16/list=4/match=50%` | 3.94 us | 3.15 us | -20.2% (1.25x faster) | | `narrow_integer/f16/list=64/match=0%` | 3.87 us | 3.84 us | -0.6% (within noise) | | `narrow_integer/f16/list=64/match=50%` | 3.93 us | 3.85 us | -1.9% (within noise) | | `narrow_integer/f16/list=256/match=0%` | 3.90 us | 3.84 us | -1.5% (within noise) | | `narrow_integer/f16/list=256/match=50%` | 3.87 us | 3.91 us | +1.2% (within noise) | | `nulls/narrow_integer/u8/list=16/match=50%/nulls=20%` | 3.92 us | 4.02 us | +2.5% (within noise) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i32/small_list/list=32/match=0%` | 16.34 us | 13.33 us | -18.5% (1.23x faster) | | `primitive/i32/small_list/list=32/match=50%` | 31.17 us | 13.31 us | -57.3% (2.34x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `primitive/i64/small_list/list=16/match=0%` | 16.34 us | 11.93 us | -27.0% (1.37x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `f32/small_list/list=32/match=0%` | 22.05 us | 13.43 us | -39.1% (1.64x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `timestamp_ns/small_list/list=16/match=0%` | 19.73 us | 12.07 us | -38.8% (1.63x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `interval_month_day_nano/small_list/list=4/match=0%` | 20.12 us | 13.20 us | -34.4% (1.52x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | </details> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? - Closes apache#23770 - Part of apache#15914 ## Rationale for this change Spark provides [`hypot(expr1, expr2)`](https://spark.apache.org/docs/latest/api/sql/#hypot), which returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or underflow. It was not yet implemented in `datafusion-spark` — only an auto-generated test stub existed at `spark/math/hypot.slt` with its query commented out. ## What changes are included in this PR? - Add `SparkHypot` (implementing `ScalarUDFImpl`) in `datafusion/spark/src/function/math/hypot.rs`, backed by Rust's `f64::hypot` — the same overflow-safe algorithm as Java/Spark's `Math.hypot`. - Register it in `datafusion/spark/src/function/math/mod.rs`. - Enable the `hypot.slt` sqllogictest. The signature is `exact(Float64, Float64) -> Float64`, following the `datafusion-spark` convention of only accepting types Spark supports. Computation uses the Arrow `binary` kernel so NULL in either argument propagates to a NULL result, matching Spark. ## Are these changes tested? Yes — `datafusion/sqllogictest/test_files/spark/math/hypot.slt` covers: - scalar Pythagorean triples (`hypot(3, 4)` → 5, `hypot(5, 12)` → 13), - double inputs, - NULL propagation when either argument is NULL, - the array path (including a NULL row), - overflow-safety: `hypot(3e200, 4e200)` stays finite, whereas a naive `sqrt(a^2 + b^2)` would overflow to `Infinity`. ## Are there any user-facing changes? Yes — adds the Spark-compatible `hypot` scalar function to `datafusion-spark`. No breaking changes to public APIs.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of apache#23393 . ## Rationale for this change `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` only reported the stack size of their `ScalarValue` field plus its heap payload, ignoring the memory held by the underlying `MovingMin` / `MovingMax` sliding-window buffers. For windowed `MIN`/`MAX` over string or list data, the two per-element stacks can hold megabytes of `ScalarValue` payload that the memory pool was never told about, understating accumulator memory usage. ## What changes are included in this PR? - Add a private `heap_size(elem_heap)` method to `MovingMin<T>` and `MovingMax<T>` that reports the two stack buffers' capacity in bytes plus each stored element's heap payload. - Factor the shared implementation into a `moving_stacks_heap_size` free helper so the two types stay in sync. - Include the buffer bytes in `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` via the new method. ## Are these changes tested? Yes. Two new unit tests in `datafusion/functions-aggregate/src/min_max.rs`: - `moving_min_max_heap_size_i32` — fixed-width `T`, verifies buffer-only accounting with and without pushed elements. - `moving_min_max_heap_size_counts_elems` — `T = String`, verifies each of the two slots in a `(T, T)` pair contributes independently to the heap payload (mirroring the two independent `Clone`s made by `push`).
## Which issue does this PR close? - Related to apache#21882 (comment). ## Rationale for this change PR apache#21882 adds custom spill file support. Having an example showing how a downstream application can use this new API will help make sure the API is good enough for our needs ## What changes are included in this PR? This PR adds a small example showing how users can back spill files with an `ObjectStore`, using a local object store for a runnable example while keeping the implementation applicable to remote stores. ## Are these changes tested? Yes by CI ## Are there any user-facing changes? Yes. This adds a new example for configuring ObjectStore-backed spill files.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #N/A. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Since DataFusion doesn't typically guartantee wire format compatibility, I remove the backward compatiblity shim that introduced in PR apache#23189 FYI: apache#23189 (comment) ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> DataFusion does not guarantee serialized plans across versions. Keeping `partitioned_by_file_group` therefore leaves a dead schema field and decoder path after `output_partitioning` became the source of truth. Reserve the old field number and name to prevent future reuse. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Yes Signed-off-by: Jiawei Zhao <Phoenix500526@163.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…llable build key (apache#23173) ## Which issue does this close? - Closes apache#23126. ## Rationale for this change `x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build = outer `x`, probe = subquery). Join dynamic filter pushdown pushes a bounds + membership filter, built from the build keys, onto the probe scan. That filter can prune every probe row. A null-aware `LeftAnti` reads an empty probe as a genuinely-empty subquery, so it emits build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is UNKNOWN, not TRUE. The result is scan-dependent, so it's a silent correctness bug. A `VALUES` scan ignores the pushed filter and stays correct; a parquet scan applies it and is wrong. apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side NULL. ## What changes are included in this PR? Skip join dynamic filter pushdown for a null-aware anti join when the build key can be NULL. The build-side NULL emission depends on whether the probe is truly empty, which the pushed filter can change by emptying it. A NOT NULL build key has no such NULL, so it keeps the pushdown. The check is static: a schema-nullable build key disables the pushdown even when the data contains no NULLs. A runtime alternative (keep the pushdown and neutralize the filter only when the build actually holds a NULL key) would restore the optimization for those cases. I'd leave that as a follow-up. ## Are these changes tested? Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side NULL, a non-matching parquet probe) and asserts the single correct row. Without the change it returns the extra NULL. In addition, unit tests pin both directions of the guard: a nullable build key rejects the pushdown and a NOT NULL build key keeps it. ## Are there any user-facing changes? `NOT IN` over a parquet (or otherwise prunable) scan with a nullable outer key now returns correct results. Such joins lose the dynamic filter pushdown.
…cimal) (apache#23631) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> N/A ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> I noticed there were some subtle errors with how decimals were handled in scalar values, and also opportunity to remove power calls in favour of precomputed constant tables. Also filling out some other missing support. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> I recommend looking at the commits as they are self contained with detailed messages for each. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…lter pushdown enabled (apache#23638) ## Which issue does this PR close? - Closes apache#23531. ## Rationale for this change `input_file_name()` is `FileSource` dependent like `file_row_index()`, and therefore shouldn't be pushed down into a filter. ## What changes are included in this PR? `PushdownChecker` now handles both UDFs consistently. If we keep adding this sort of metadata functions, we might want a better API to detect them, but for now I think this is a reasonable approach that isn't very invasive. ## Are these changes tested? Additional SLT test that verifies that both function behave correctly with pushdown either enabled or disabled. ## Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
…xpr::check_bigger_cast (apache#23808) (apache#23809) ## Which issue does this PR close? - Closes apache#23808. ## Rationale for this change ref. apache#23807 `CastExpr::check_bigger_cast` is used to determine whether a cast is a widening (order-preserving) conversion. Currently, it classifies `Int32 -> Float32`, `UInt32 -> Float32`, `Int64 -> Float64`, and `UInt64 -> Float64` as widening casts. However, integer-to-float conversions for 32-bit and 64-bit integers lose precision when values exceed the mantissa bit limit (24 bits for `Float32`, 53 bits for `Float64`). For example: `16_777_216_i32 as f32 == 16_777_217_i32 as f32` (both yield 16777216.0f32). Because distinct integer inputs can collapse to the same float output, these casts are not strictly 1-to-1 (injective) and can break suffix sort key ordering in multi-column ordering analysis (e.g. `[CAST(a AS Float32), b]`). ## What changes are included in this PR? - Updated `CastExpr::check_bigger_cast` to exclude precision-losing integer-to-float conversions (`Int32/UInt32 -> Float32` and `Int64/UInt64 -> Float64`). - Added unit test `test_check_bigger_cast_precision_loss` to verify precision-losing casts return `false` while exact conversions (`Int16 -> Float32`, `Int32 -> Float64`, etc.) continue to return `true`. ## Are these changes tested? Yes, new unit test `test_check_bigger_cast_precision_loss` in `cast.rs`. ## Are there any user-facing changes? No breaking API changes. Internal optimizer behavior fix.
…ossible (apache#23702) ## Which issue does this PR close? N/A ## Rationale for this change `SortPreservingMergeStream` is a little complex, so add some guiding comments and make it as textbook-like as possible ## What changes are included in this PR? Added comments, reorder code While this was done this also fixed couple of bugs due to how it work: 1. leftover drain was not counted in the `elapsed_compute` 2. `limit(0)` returns 0 rows and not 1 ## Are these changes tested? Existing tests ## Are there any user-facing changes? Not API ones. `limit(0)` now returns 0 rows
## Which issue does this PR close? - Closes apache#21428. ## Rationale for this change This PR adds `BuildHasher`-based variants for `hash_utils` so callers can compute row hashes with a caller-provided hash builder instead of always using DataFusion's default `RandomState`. The main constraint is performance: `with_hashes` is a hot path, especially for string, dictionary, and nested array hashing. A previous version in apache#21429 caused measurable regressions in the default `RandomState` path, for example `large_utf8: single, no nulls` regressed from roughly `26.7us` to `36.3us`, and `large_utf8: multiple, no nulls` from roughly `112us` to `127us`. This version keeps the default path performance-oriented by avoiding a fully generic `BuildHasher` rewrite of the existing hot loops. ## What changes are included in this PR? This PR adds: - `with_hashes_with_hasher` - `create_hashes_with_hasher` - custom-hasher implementations for primitive, string, binary, byte-view, dictionary, and nested arrays - tests covering custom hashers, multi-column hashing, and dictionary equivalence The implementation intentionally uses a hybrid design: - Default `RandomState` leaf hot paths remain specialized. - Custom `BuildHasher` leaf paths live separately in `hash_utils/build_hasher.rs`. - Nested/structural logic is shared through an internal child-hashing adapter, so struct/list/map/union/run/dictionary behavior does not need to be broadly duplicated. The trade-off is that there is still some duplication for primitive/string/binary leaf loops. That duplication is intentional: those are the hottest loops, and keeping them separate prevents the existing `RandomState` path from becoming generic over `BuildHasher` or being perturbed by the custom-hasher implementation. ## Are these changes tested? Yes. --------- Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
…lan` API with `replace_children` (apache#24296) Backport of apache#23903 to `branch-55`. Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Additional commits were added into `branch-55` so a new changelog was generated.
…ache#24345) (apache#24367) This is a backport of apache#24345 for the release 55.0.0.
…row groups (apache#24352) (apache#24368) Backport of apache#24354 to `branch-55` for the 55.0.0 release, per @timsaucer's request in apache#22393. ## Which issue does this PR close? - Backports the fix for apache#24352 (wrong TopK results from re-reading already-delivered row groups). ## Rationale apache#24352 is a **silent wrong-results** bug: with `pushdown_filters=true` + TopK dynamic filter pushdown (both on by default), a row group whose post-predicate selection is empty is finished by arrow-rs without handing back a reader, so DataFusion's `rg_plan` trails the decoder frontier by one and a later runtime prune rebuilds the decoder from a stale plan — re-reading an already-delivered row group, duplicating rows and dropping the true top-k tail. No error is raised. This is a clean cherry-pick of the squashed apache#24354 commit (`574fe67`); it applies to `branch-55` without conflicts. ## What changes are included? `push_decoder.rs`: sync `rg_plan` to the decoder frontier via `peek_next_row_group()` before each runtime prune/rebuild (gated on `row_group_pruner.is_some()` so ordinary scans pay nothing), with a defensive `internal_err!` if the frontier diverges from the plan. Plus the slt + rust regression tests from apache#24354. cc @timsaucer @alamb @adriangb
…ection is live (apache#24355) (apache#24374) Backport of apache#24359 to `branch-55` for the 55.0.0 release, per @timsaucer's request in apache#22393. Stacks cleanly on the already-merged apache#24368 (apache#24354 backport). ## Which issue does this PR close? - Backports the fix for apache#24355 — a second, independent silent wrong-results bug in the same parquet dynamic row-group pruning path as apache#24352. ## Rationale With `pushdown_filters=true` + a TopK dynamic filter, the runtime row-group pruner rebuilds the push decoder via `into_builder().with_row_groups(...)`, which drops row groups **without slicing** the carried flat page-index `RowSelection` to match — a dropped RG's selectors are then applied to the next surviving RG, silently returning wrong rows (no error). The fix declines to build the runtime `RowGroupPruner` when a row selection is present (correctness over the pruning optimization); the proper fix that keeps both is tracked upstream in apache/arrow-rs#10624 / apache#24358. ## Notes - Clean cherry-pick of apache#24359 onto `branch-55` (which now has apache#24354 via apache#24368). No conflicts. - apache#24359 is **approved** on `main` and pending merge; opening this now so it can ride RC3. - Verified locally on this branch: the full `dynamic_row_group_pruning` rust module (9/9) and `dynamic_row_group_pruning.slt` pass; clippy clean. cc @timsaucer @alamb @adriangb
…d/prepend/replace - apache#24365 (apache#24377) This is a back port of apache#24365 into `branch-55`
New commits were added to branch-55 so the changelog is updated.
## Which issue does this PR close? - Relates to apache#22989 but does not close it. More types are coming. ## Rationale for this change - Support the Arrow type `Dictionary` for `approx_distinct` ## What changes are included in this PR? - Enable `HLLAccumulator` and `HllGroupsAccumulator` to support `Dictionary` - Tests ## Are these changes tested? Yes ## Are there any user-facing changes? Yes, `approx_distinct` supports now `Dictionary` but no breaking changes.
…anch-55 [Cherry-pick] Add support for dictionary for approx_distinct (apache#24646)
…r-defined types (apache#25015) ## Which issue does this PR close? N/A ## Rationale for this change `consume_user_defined_type` maps a Substrait user-defined type straight to a bare `DataType`, so a consumer has no way to mark the resulting `Field` for a downstream reader to recover semantic type lost in that mapping. For example, a UDT representing JSON is commonly mapped to plain `Utf8`, which is then indistinguishable from a real string column once it reaches a consumer that only sees the Arrow schema (e.g. over Arrow Flight, with no access to the original plan). ## What changes are included in this PR? - Adds `consume_user_defined_type_metadata`, a second, additive extension point on `SubstraitConsumer` defaulting to `Ok(None)`. - Applies its result to the `Field` built in `from_substrait_struct_type`, via `Field::with_metadata`. - Existing consumers are unaffected: the default keeps today's behavior exactly (no metadata attached). ## Are these changes tested? Yes, two new unit tests: a consumer that supplies metadata for a `UserDefined` field gets it attached to the field (data type unchanged), and a consumer returning `None` (the default) attaches nothing. ## Are there any user-facing changes? No behavior change for existing consumers. This is a new, optional trait method with a default implementation. (cherry picked from commit d682553)
#183) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - works towards closing apache#22682. - replacement for apache#21765 - - apache#21765 (review) ## Rationale for this change This PR introduces a specialized `GroupColumn` implementation for dictionary-typed columns inside `GroupValuesColumn`, allowing dictionary columns to participate in the columnar, vectorized aggregation path instead of the row-based fallback. **The Implementation is only about 175+ lines of code**. the remaining LOC is adding extensive test at the `GroupColumn` trait level as well as testing the `GroupValuesColumn` GroupValues trait and how it inter-opts with multi-dictionary group by's. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Adds a `DictionaryGroupValueBuilder` struct implementing the `GroupColumn` trait for `Dictionary`-typed group-by columns, supporting a configurable subset of value types - Extends the type-check gate in `GroupValuesColumn::try_new` (the `matches!` block) to accept `Dictionary(_, value_type)` where `value_type` is already supported. - Adds schema-level support so emitted dictionary group key columns round-trip through the output schema correctly - [removes casting ](https://github.com/apache/datafusion/blob/9e8dd76d6deb6736c51962d9c97e04be4e3f1fc9/datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs#L1200)thats done for each dictionary array in `emit` <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? yes. a majority of this PR is test <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? no. this is a pure perf boost for users. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> (cherry picked from commit c1b39bd) Co-authored-by: RIchard Baah <137434454+Rich-T-kid@users.noreply.github.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23921. - this PR was created on top of apache#24416, so once that is merged in we can rebase this branch ## Rationale for this change append_val is called once per new group on the scalar (streaming) code path. Previously it hashed a single-element slice of the dictionary values array on every call, paying create_hashes fixed overhead 65k times for a high-cardinality batch. Caching the full values array hash keyed on Arc::ptr_eq collapses that to one vectorized hash pass per batch, yielding a 2× speedup on the all-unique case with no measurable regression elsewhere. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> ## What changes are included in this PR? - DictionaryGroupValuesColumn gains a cached_values: Option<ArrayRef> field - append_val: on a cache miss (Arc::ptr_eq fails), hashes the entire dict.values() array into val_hashes and stores the Arc; on a hit, reuses val_hashes[val_idx] directly — eliminating the per-call slice(val_idx, 1) allocation and create_hashes call <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? nothing logically changed. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? no <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> --------- (cherry picked from commit 66a901f) Co-authored-by: RIchard Baah <137434454+Rich-T-kid@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…he#25185) (#185) ## Which issue does this PR close? - Closes #. ## Rationale for this change `vectorized_append` is the path taken when we have non streaming hash aggregations (when there is no ordering in the `group by`), right now it re hashed the entire dictionary values on every batch unlike append_val which that only does this if the values array was already seen before. ## What changes are included in this PR? `vectorized_append` now reuses the cached value hashes when a batch carries the same dictionary values array as the previous one (this is the common case of a repartition or filter), instead of re hashing the whole dictionary and re-initialising the map on every batch. This extends the existing `Arc::ptr_eq` cache from apache#24418, which only covered the scalar `append_val` path. ## What is the testing strategy for this PR? All dict tests pass, I have a PR for a benchmark that exercises this path -> apache#25198 ## Are there any user-facing changes? no, this is a pure perf PR (cherry picked from commit a0631ed)
…apache#25418) (#186) ## Which issue does this PR close? - Closes apache#25411 ## Rationale for this change `create_physical_expr` required an `Expr::LambdaVariable`'s field to equal the planning schema's field and returned a plan error otherwise. The two can legitimately differ when a plan producer records a type that Substrait cannot express, such as dictionary encoding or a string view (Utf8view), since the recorded field and the field derived by `lambda_parameters` come from independent sources. ## What changes are included in this PR? Cast the variable to the recorded type instead, so the declared type is enforced rather than asserted. This is the same approach already used for `ScalarFunction` arguments, which the `TypeCoercion` analyzer casts to their coerced types instead of requiring them to match. Incompatible types still fail, as a cast error. ## What is the testing strategy for this PR? I added - Unit tests covering dictionary-encoded and `Utf8View` list elements with the lambda variable declared as plain `Utf8`, plus plan-shape tests pinning that the cast is inserted on a mismatch and omitted when the types already agree. - A Substrait integration test with a plan whose read schema declares a dictionary-encoded list element while the lambda carries its own plain `Utf8` parameter type. ## Are there any user-facing changes? No, this is a bug fix (cherry picked from commit 5da95ac)
…pache#25259) (#187) - Closes apache#25244. - Closes apache#25262. - Closes apache#25263. - Closes apache#25264. - Closes apache#21246. - Closes apache#25296. Enabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` can silently drop matching rows when the probe side of a join contains several columns with the same name, for example `a.id` and `b.id` from a nested join. Physical filter pushdown resolved a pushed filter's columns in the child schema by name, so a predicate on the second `id` column was rewritten to the first `id` column, which holds a different value. The same name-based mapping also affects TopK dynamic filters under the default configuration (apache#25296). For a nested join of orders and payments that both expose `amount`, `ORDER BY p.amount LIMIT 1` can push the payment threshold onto `orders.amount`. Later matching orders are incorrectly pruned, returning `(100, 30)` instead of `(300, 10)`. Every operator that forwards parent filters had the same weakness, in slightly different forms: - `HashJoinExec` computed which output columns belong to each side by position, but then resolved the child column by name. - `FilterExec`, `SortExec`, `RepartitionExec`, `CoalesceBatchesExec`, `UnionExec` and similar nodes used the generic name lookup even though their output positions equal their input positions. - `FilterExec` with an embedded projection resolved unconsumed parent filters back into input coordinates by name. - `ProjectionExec` looked up output aliases by name to find the expression to substitute, so two outputs aliased `id` both mapped to the first. - `AggregateExec` restricted pushdown to grouping positions but resolved the input column by name. The join-filter regression queries return one row with dynamic filtering disabled and zero rows with it enabled across these shapes. The `FilterExec`, `ProjectionExec` and `AggregateExec` cases were found while auditing the remaining name-based paths for apache#25244; they share the root cause, so this PR fixes them together. The TopK case in apache#25296 is fixed by the same positional mapping. Built-in operators now remap filter columns by position. The deprecated public API retains its historical name-based resolution for compatibility. - `FilterRemapper` uses a `ColumnMapping` enum: `Identity` preserves positions and checks that names match; `Explicit` uses a caller-supplied parent-output to child-input mapping and allows names to differ. - `ChildFilterDescription::from_child` now maps by position. New `ChildFilterDescription::from_child_with_column_mapping` takes an explicit `HashMap<usize, usize>`. - `HashJoinExec` builds the explicit mapping from its `column_indices` and output projection. For semi joins, output join keys on the emitted side are mapped to the paired key on the other side, which also supports differently named keys. - `FilterExec` maps through its embedded projection in both pushdown phases and when folding unconsumed parent filters back into its predicate. - `ProjectionExec` substitutes the expression at each output position instead of looking the alias up in the output schema. - `AggregateExec` maps each grouping output position to the input column that grouping expression reads; non-column grouping expressions are not forwarded, as before. - `ChildFilterDescription::from_child_with_allowed_indices` is kept as a deprecated wrapper that preserves name-based resolution to the first matching child field. It translates allowed parent column references into an explicit positional mapping; new callers should supply positions directly to avoid ambiguous duplicate names. - `dynamic_filter_pushdown_config.slt` gains four regression queries over two small Parquet tables, run once with join dynamic filtering disabled and once enabled, asserting identical rows: nested joins with `RepartitionExec` between them (the query from apache#25244), a `FilterExec` with an embedded projection, a `ProjectionExec` with duplicate aliases, and an `AggregateExec` grouping on same-named columns. All four return zero rows on `main` with dynamic filtering enabled. - `dynamic_filter_pushdown_config.slt` also covers apache#25296 using nested joins of orders, payments and customers, with one row per batch and one row per orders Parquet row group. The query returns `(300, 10)` with TopK dynamic filtering disabled, enabled, and enabled together with Parquet filter pushdown. On `main` at `85d4cbb0a9`, both enabled cases reproduce the incorrect `(100, 30)` result; all three pass on this branch. - `datafusion/core/tests/physical_optimizer/filter_pushdown.rs` gains focused tests that call `gather_filters_for_pushdown` directly on `RepartitionExec`, `HashJoinExec` (duplicate child columns with and without projection, both semi join directions with differently named keys, and a semi join whose key is not a plain column), `FilterExec` with a projection in both phases, `ProjectionExec` with duplicate aliases, and `AggregateExec` with reordered same-named grouping columns. Further tests cover the deprecated `from_child_with_allowed_indices` wrapper preserving the first name match, accepting allowed parent indices outside the child schema, and rejecting unresolvable names, the identity mapping rejecting a column whose name differs from the child field at that position, and `FilterExec` rejecting an unconsumed parent filter outside its projection. Passed locally on the updated implementation: - `cargo fmt --all` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` - `cargo test --profile ci -p datafusion --test core_integration physical_optimizer::filter_pushdown` — 70 tests passed. - `cargo test --profile ci --test sqllogictests -- dynamic_filter_pushdown_config.slt` — passed. - The isolated apache#25296 regression fails on `main` in both TopK-enabled configurations with the expected wrong-result mismatch. Queries that push join or TopK dynamic filters through operators with duplicate column names now return the correct rows, including the default-configuration TopK query in apache#25296. API change in `datafusion-physical-plan`: `ChildFilterDescription::from_child_with_allowed_indices` is deprecated in favour of `from_child` and the new `from_child_with_column_mapping`; the deprecated function preserves its previous name-based behavior. Callers should migrate to explicit positions because duplicate names make name resolution ambiguous. `ChildFilterDescription::from_child` also resolves by position, which only affects callers that used it on a node whose output positions differ from its child's. (cherry picked from commit b376290) Co-authored-by: Huaijin <haohuaijin@gmail.com>
…#25977) (#188) <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> Related to apache#23237 and [feat(metric): Add output skewness metric to detect skewed plans easier](apache#21211) - Closes #. I think it would be useful to add the skewness metric for nodes that execute in partitioned mdoe -- right now the only node that includes that metric in the explain analyze is DataSourceExec for parquet. This is specially useful to check how many partitions were idle during Aggregations or HashJoins for example. The PR adds a new metric `output_rows_skew` in the Explain Analyze output for the following nodes: - RepartitionExec - HashJoinExec - AggregateExec - BoundedWindowAggExec - WindowAggExec Note that for a plan that has a Parquet DataSource the skewness might not be the same as for the rest of the nodes, for example we might have the following scenario: ``` DataSourceExec [100, 100, 100, 100] → 0% skewness, data is even FilterExec [100, 0, 0, 0] → 100% (filter only matches in partition 0) AggregateExec [ 3, 0, 0, 0] → 100% (Partial: groups, not rows) RepartitionExec Hash [1, 1, 1, 0] → ~11% (3 groups re-spread by key) ``` I added a Explain Analyze test Yes, a new metric will show in the explain analyze plan for the nodes mentioned above. (cherry picked from commit c2baac5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?