From ecd6c42b0042aa450dcd06eb379aead40a7f11b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Efe=20G=C3=B6kdemir?= Date: Tue, 6 Oct 2026 01:51:31 +0300 Subject: [PATCH 1/4] docs: add DataFusion security guidance --- docs/source/index.rst | 1 + docs/source/library-user-guide/index.md | 3 + .../library-user-guide/securing-datafusion.md | 72 +++++++++++++++++++ 3 files changed, 76 insertions(+) create mode 100644 docs/source/library-user-guide/securing-datafusion.md diff --git a/docs/source/index.rst b/docs/source/index.rst index ea6ebb74c08b1..c994ae65249e6 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -137,6 +137,7 @@ To get started, see :caption: Library User Guide library-user-guide/index + library-user-guide/securing-datafusion library-user-guide/upgrading/index library-user-guide/extensions library-user-guide/using-the-sql-api diff --git a/docs/source/library-user-guide/index.md b/docs/source/library-user-guide/index.md index fd126a1120edf..0dffc279ab06a 100644 --- a/docs/source/library-user-guide/index.md +++ b/docs/source/library-user-guide/index.md @@ -29,6 +29,9 @@ for details on how to contribute to DataFusion. If you haven't reviewed the [architecture section in the docs][docs], it's a useful place to get the lay of the land before starting down a specific path. +For guidance on running DataFusion with SQL from untrusted users, see +[Securing DataFusion](securing-datafusion.md). + DataFusion is designed to be extensible at all points, including - [x] User Defined Functions (UDFs) diff --git a/docs/source/library-user-guide/securing-datafusion.md b/docs/source/library-user-guide/securing-datafusion.md new file mode 100644 index 0000000000000..7135d584f806f --- /dev/null +++ b/docs/source/library-user-guide/securing-datafusion.md @@ -0,0 +1,72 @@ + + +# Securing DataFusion + +DataFusion is an embedded query engine, not an authorization boundary. If an +application accepts SQL from users, the application is responsible for deciding +which data and operations each user may access. The settings below can reduce +what a query can do, but they do not replace application authorization or +operating-system isolation. + +## Restrict SQL statements + +[`SQLOptions`] allows an application to reject classes of SQL statements when +creating a `DataFrame`. DDL, DML, and other statements are all allowed by +default. Disable the classes that the application does not need and pass the +options to `SessionContext::sql_with_options` for every user-provided query: + +```rust +use datafusion::prelude::*; + +let options = SQLOptions::new() + .with_allow_ddl(false) + .with_allow_dml(false) + .with_allow_statements(false); + +let dataframe = ctx.sql_with_options(sql, options).await?; +``` + +These checks reject statement types such as `CREATE TABLE`, `INSERT`, and +`SET`; they do not decide which tables or rows a user is authorized to read. +Expose only the appropriate catalogs and tables to each user, and enforce +application-specific access rules separately. + +## Limit file access + +[`SessionContext::enable_url_table()`] is an opt-in feature that lets SQL query +local files by path. Leave it disabled when users should only query tables +registered by the application. If it is needed, run DataFusion with filesystem +permissions limited to the files the application intends to expose. + +## Set query memory limits + +The `datafusion.runtime.memory_limit` setting defaults to `NULL` (no configured +query memory limit). Set an appropriate limit for the workload using the +[runtime configuration settings](../user-guide/configs.md#runtime-configuration-settings). +This limits memory used by DataFusion's query execution memory pool; use +process- or container-level resource limits as well when a hard bound on total +application memory is required. + +Also review the capabilities of custom table providers, functions, and other +extensions registered by the application: they determine which external data +and operations queries can reach. + +[sqloptions]: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SQLOptions.html +[sessioncontext::enable_url_table()]: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html#method.enable_url_table From 4ac3bf1edf4c0ea43e5b2f1222214334e5822fb1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Efe=20G=C3=B6kdemir?= Date: Thu, 8 Oct 2026 01:21:14 +0300 Subject: [PATCH 2/4] docs: describe spill storage limits for untrusted SQL --- docs/source/library-user-guide/securing-datafusion.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/source/library-user-guide/securing-datafusion.md b/docs/source/library-user-guide/securing-datafusion.md index 7135d584f806f..648c47ad3eff0 100644 --- a/docs/source/library-user-guide/securing-datafusion.md +++ b/docs/source/library-user-guide/securing-datafusion.md @@ -64,6 +64,16 @@ This limits memory used by DataFusion's query execution memory pool; use process- or container-level resource limits as well when a hard bound on total application memory is required. +## Bound spill storage + +When an execution operator supports spilling, DataFusion may write intermediate +query data to temporary files under memory pressure. A query memory limit does +not limit this disk usage. Set `datafusion.runtime.temp_directory` to a +controlled location and `datafusion.runtime.max_temp_directory_size` to cap +DataFusion's temporary-file directory size (the default is `100G`). For +untrusted SQL workloads, apply appropriate filesystem permissions and storage +limits to that location as well. + Also review the capabilities of custom table providers, functions, and other extensions registered by the application: they determine which external data and operations queries can reach. From 8a146c79fc72ab4c7e693bb9888c68807c186375 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Efe=20G=C3=B6kdemir?= Date: Thu, 8 Oct 2026 13:20:47 +0300 Subject: [PATCH 3/4] docs: refine DataFusion security guidance --- .../library-user-guide/securing-datafusion.md | 38 +++++++------------ 1 file changed, 13 insertions(+), 25 deletions(-) diff --git a/docs/source/library-user-guide/securing-datafusion.md b/docs/source/library-user-guide/securing-datafusion.md index 648c47ad3eff0..838fac1f91d18 100644 --- a/docs/source/library-user-guide/securing-datafusion.md +++ b/docs/source/library-user-guide/securing-datafusion.md @@ -19,18 +19,17 @@ # Securing DataFusion -DataFusion is an embedded query engine, not an authorization boundary. If an -application accepts SQL from users, the application is responsible for deciding -which data and operations each user may access. The settings below can reduce -what a query can do, but they do not replace application authorization or -operating-system isolation. +As described in the [DataFusion security policy](../../../SECURITY.md), the end +application is responsible for security decisions. The settings below can help +control what a query can do. ## Restrict SQL statements [`SQLOptions`] allows an application to reject classes of SQL statements when -creating a `DataFrame`. DDL, DML, and other statements are all allowed by -default. Disable the classes that the application does not need and pass the -options to `SessionContext::sql_with_options` for every user-provided query: +creating a `DataFrame`. DDL (such as [`CREATE TABLE`](../user-guide/sql/ddl.md)) +and DML (such as [`INSERT`](../user-guide/sql/dml.md)) are allowed by default. +Disable the classes that the application does not need and pass the options to +[`SessionContext::sql_with_options`] for every user-provided query: ```rust use datafusion::prelude::*; @@ -43,11 +42,6 @@ let options = SQLOptions::new() let dataframe = ctx.sql_with_options(sql, options).await?; ``` -These checks reject statement types such as `CREATE TABLE`, `INSERT`, and -`SET`; they do not decide which tables or rows a user is authorized to read. -Expose only the appropriate catalogs and tables to each user, and enforce -application-specific access rules separately. - ## Limit file access [`SessionContext::enable_url_table()`] is an opt-in feature that lets SQL query @@ -57,8 +51,7 @@ permissions limited to the files the application intends to expose. ## Set query memory limits -The `datafusion.runtime.memory_limit` setting defaults to `NULL` (no configured -query memory limit). Set an appropriate limit for the workload using the +Set an appropriate `datafusion.runtime.memory_limit` for the workload using the [runtime configuration settings](../user-guide/configs.md#runtime-configuration-settings). This limits memory used by DataFusion's query execution memory pool; use process- or container-level resource limits as well when a hard bound on total @@ -67,16 +60,11 @@ application memory is required. ## Bound spill storage When an execution operator supports spilling, DataFusion may write intermediate -query data to temporary files under memory pressure. A query memory limit does -not limit this disk usage. Set `datafusion.runtime.temp_directory` to a -controlled location and `datafusion.runtime.max_temp_directory_size` to cap -DataFusion's temporary-file directory size (the default is `100G`). For -untrusted SQL workloads, apply appropriate filesystem permissions and storage -limits to that location as well. - -Also review the capabilities of custom table providers, functions, and other -extensions registered by the application: they determine which external data -and operations queries can reach. +query data to temporary files under memory pressure. Use +`datafusion.runtime.temp_directory` and +`datafusion.runtime.max_temp_directory_size` to configure the location and size +of that temporary storage; see the [runtime configuration settings](../user-guide/configs.md#runtime-configuration-settings). [sqloptions]: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SQLOptions.html +[sessioncontext::sql_with_options]: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html#method.sql_with_options [sessioncontext::enable_url_table()]: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html#method.enable_url_table From 6be87571b7dcb182d1cb031896d15ff160de4218 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Efe=20G=C3=B6kdemir?= Date: Thu, 8 Oct 2026 15:46:11 +0300 Subject: [PATCH 4/4] docs: fix security policy link in guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Efe Gökdemir --- docs/source/library-user-guide/securing-datafusion.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/library-user-guide/securing-datafusion.md b/docs/source/library-user-guide/securing-datafusion.md index 838fac1f91d18..968310a3e2598 100644 --- a/docs/source/library-user-guide/securing-datafusion.md +++ b/docs/source/library-user-guide/securing-datafusion.md @@ -19,9 +19,9 @@ # Securing DataFusion -As described in the [DataFusion security policy](../../../SECURITY.md), the end -application is responsible for security decisions. The settings below can help -control what a query can do. +As described in the [DataFusion security policy](https://github.com/apache/datafusion/blob/main/SECURITY.md), +the end application is responsible for security decisions. The settings below +can help control what a query can do. ## Restrict SQL statements