diff --git a/README.md b/README.md new file mode 100644 index 0000000..8aaaa4a --- /dev/null +++ b/README.md @@ -0,0 +1,95 @@ + + +# Apache Iceberg DataFusion Integration + +This repository connects [Apache Iceberg](https://iceberg.apache.org/) tables to +[Apache DataFusion](https://datafusion.apache.org/). The `datafusion-iceberg` +crate provides table providers for querying Iceberg data and for inserting rows +through an Iceberg catalog. + +## Workspace + +- [`crates/datafusion`](crates/datafusion): the DataFusion integration library. +- [`crates/sqllogictest`](crates/sqllogictest): SQL logic tests for the integration. +- [`crates/playground`](crates/playground): a command-line SQL playground backed + by DataFusion and Iceberg catalogs. + +## Query an existing table + +Register `IcebergTableProviderFactory` in a DataFusion session, then point an +external table at an existing Iceberg metadata file: + +```rust +use std::sync::Arc; + +use datafusion::execution::session_state::SessionStateBuilder; +use datafusion::prelude::SessionContext; +use datafusion_iceberg::IcebergTableProviderFactory; + +async fn query_table() -> datafusion::error::Result<()> { + let mut state = SessionStateBuilder::new().with_default_features().build(); + state.table_factories_mut().insert( + "ICEBERG".to_string(), + Arc::new(IcebergTableProviderFactory::new()), + ); + let ctx = SessionContext::new_with_state(state); + + ctx.sql( + "CREATE EXTERNAL TABLE trips STORED AS ICEBERG \ + LOCATION '/absolute/path/to/table/metadata/v1.metadata.json'", + ) + .await? + .collect() + .await?; + + let batches = ctx + .sql("SELECT * FROM trips LIMIT 10") + .await? + .collect() + .await?; + println!("{batches:?}"); + Ok(()) +} +``` + +Replace the metadata path with one from an existing table whose data files are +accessible to the process. External-table registration reads an existing table; +it does not create one. For catalog-backed access and inserts, register an +`IcebergCatalogProvider` with a configured Iceberg `Catalog`. See the +[integration tests](crates/datafusion/tests/integration_datafusion_test.rs) for +examples. + +The workspace uses a pinned `iceberg-rust` Git revision. Applications that +also depend on Iceberg crates should use the same revision shown in +[`Cargo.toml`](Cargo.toml) so Cargo uses one Iceberg crate source. + +## Development + +The repository's [`rust-toolchain.toml`](rust-toolchain.toml) selects the Rust +toolchain used by CI. From the repository root, run: + +```sh +cargo fmt --all -- --check +cargo clippy --workspace --locked --all-targets -- -D warnings +cargo test --workspace --locked +``` + +This project is licensed under the Apache License, Version 2.0. See +[`LICENSE`](LICENSE) and [`NOTICE`](NOTICE).