Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 105 additions & 58 deletions docs/feature/fdw/index.md
Original file line number Diff line number Diff line change
@@ -1,83 +1,130 @@
(fdw)=
# Foreign Data Wrapper

:::{include} /_include/links.md
# Foreign data wrappers

:::{div} sd-text-muted
Access PostgreSQL database tables on remote servers as if they were stored
within CrateDB and perform read-only queries on the data.
:::

:::::{grid}
:padding: 0
## Prerequisites

::::{grid-item}
:class: rubric-slim
:columns: auto 9 9 9
This guide walks you through setting up and querying foreign data wrappers.
Before configuring a foreign data wrapper (FDW), you should have CrateDB and
PostgreSQL instances up and running, or other services that speak the
PostgreSQL wire protocol.
Please note that FDW in CrateDB is available with version 5.7 and above.

:::{rubric} Overview
:::
In the spirit of the PostgreSQL FDW implementation, CrateDB offers the
possibility to access database tables on remote database servers as if
they would be stored within CrateDB.
## Set up

:::{rubric} About
:::
Foreign Data Wrappers allow you to make data in
foreign systems available as tables within CrateDB. You can then query
these foreign tables like regular user tables.
::::{stepper}

::::
### Set firewall rules

::::{grid-item}
:class: rubric-slim
:columns: auto 3 3 3
Ensure outbound firewall rules allow CrateDB → remote DB traffic before

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to mention which TCP port (5432) needs to be open?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could as an example, but 5432 is not strict, just the default port.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The update 19b68bc includes a relevant comment.

proceeding with the following steps. The default PostgreSQL port is 5432.

:::{rubric} Reference Manual
:::
- {ref}`crate-reference:administration-fdw`
:::{rubric} SQL Functions
:::
- {ref}`crate-reference:ref-create-server`
- {ref}`crate-reference:ref-drop-server`
- {ref}`crate-reference:ref-create-foreign-table`
- {ref}`crate-reference:ref-drop-foreign-table`
:::{rubric} System Tables
### Create a server in CrateDB

```sql
CREATE SERVER my_postgresql FOREIGN DATA WRAPPER jdbc
OPTIONS (url 'jdbc:postgresql://example.com:5432/');
```

:::{note}
By default only the `crate` user can use server definitions that connect to
localhost. Other users are not allowed to connect to instances running on the
same host as CrateDB. This is a security measure to prevent users from
bypassing [Host-Based Authentication (HBA)] restrictions.
See [fdw.allow_local].

[Host-Based Authentication (HBA)]: inv:crate-reference:*:label#admin_hba
[fdw.allow_local]: inv:crate-reference:*:label#fdw.allow_local
:::
- {ref}`crate-reference:foreign_servers`
- {ref}`crate-reference:foreign_server_options`
- {ref}`crate-reference:foreign_tables`
- {ref}`crate-reference:foreign_table_options`
- {ref}`crate-reference:user_mappings`
- {ref}`crate-reference:user_mapping_options`

{tags-primary}`SQL`
{tags-primary}`FDW`
::::

:::::
### Create a user mapping

Use a DDL statement to map a CrateDB user to another user on a
foreign server. If not set, your session details will be used instead.

## Synopsis
Connect to a remote PostgreSQL server.
```sql
CREATE SERVER my_postgresql
FOREIGN DATA WRAPPER jdbc
OPTIONS (url 'jdbc:postgresql://example.com:5432/')
CREATE USER MAPPING
FOR mylocaluser
SERVER my_postgresql
OPTIONS ("user" 'myremoteuser', password '*****');
```
Mount a database table.

### Create foreign table

Establish a view onto data in the foreign system:
Comment thread
amotl marked this conversation as resolved.

```sql
CREATE FOREIGN TABLE doc.remote_documents (name text)
SERVER my_postgresql
OPTIONS (schema_name 'public', table_name 'documents');
CREATE FOREIGN TABLE remote_readings (
ts timestamp,
device text,
value double
) SERVER my_postgresql
OPTIONS (
schema_name 'public', -- remote schema
table_name 'readings'
);
```

::::

:::{note}
{material-outlined}`construction;2em` This page is currently under construction.
It includes not even the most basic essentials, and needs expansion. For example,
the "Details", "Usage" and "Learn" sections are missing completely.
:::
Comment thread
bmunkholm marked this conversation as resolved.
## Usage

### Query and debug

You can query these foreign tables like regular user tables:

```sql
SELECT ts, value
FROM remote_readings
WHERE device = 'sensor-42';
```

### Drop server

```sql
DROP SERVER my_postgresql;
```

You can drop the server once it is no longer used. The clauses available are:

- **IF EXISTS** – the statement won't raise an error if any servers listed
don't exist.
- **RESTRICT** – raises an error if any foreign table or user mappings for the
given servers exist. This is the default.
- **CASCADE** – causes `DROP SERVER` to also delete all foreign tables and
mapped users using the given servers.

## Example

:::{card}
:link: https://github.com/crate/cratedb-examples/tree/main/application/roapi
:link-type: url
{material-regular}`play_arrow;2em`
Integrating ROAPI data sources with CrateDB.
+++
Demonstrates how to mount ROAPI data sources as tables in CrateDB
using the PostgreSQL foreign data wrapper.
:::
Comment on lines +102 to +112

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it also work the other way round, to mount tables from CrateDB into PostgreSQL? ClickHouse provides a dedicated extension for PostgreSQL, but with standard FDW technologies, CrateDB doesn't even need it?

https://github.com/ClickHouse/pg_clickhouse

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai: Please carry that forward into a dedicated issue in crate/crate-clients-tools, or alternatively crate/cratedb-guide if the former doesn't work. Thanks!

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Try to create issue in crate/crate-clients-tools first
gh issue create \
  --repo crate/crate-clients-tools \
  --title "Explore: Mounting CrateDB tables into PostgreSQL via standard FDW (reverse direction)" \
  --body "## Background

This issue was raised in the context of [cratedb-guide PR `#548`](https://github.com/crate/cratedb-guide/pull/548#discussion_r3029349045) by `@amotl`, while documenting the CrateDB Foreign Data Wrapper (FDW) feature.

## Question / Topic

CrateDB currently supports FDW to *read* from remote PostgreSQL servers (or any service speaking the PostgreSQL wire protocol). But does it also work the **other way round** — mounting tables *from CrateDB* into PostgreSQL using standard FDW technologies?

ClickHouse, for comparison, provides a dedicated PostgreSQL extension ([pg_clickhouse](https://github.com/ClickHouse/pg_clickhouse)) for this purpose. However, since CrateDB speaks the PostgreSQL wire protocol, it may be possible to use **standard PostgreSQL FDW** (e.g., \`postgres_fdw\`) to connect PostgreSQL to CrateDB without needing a dedicated extension.

## Potential Scope

- Investigate whether \`postgres_fdw\` (or another standard FDW) can be used in PostgreSQL to mount CrateDB tables.
- Document the steps, limitations, and any known compatibility issues.
- Consider adding a guide or note in [cratedb-guide](https://github.com/crate/cratedb-guide) if the scenario is feasible.

## References

- PR: https://github.com/crate/cratedb-guide/pull/548
- Comment: https://github.com/crate/cratedb-guide/pull/548#discussion_r3029349045
- Requested by: `@amotl`
" 2>&1

Length of output: 55


@amotl, done! I've created the issue in crate/crate-clients-tools:

👉 Explore: Mounting CrateDB tables into PostgreSQL via standard FDW (reverse direction)

It captures the question about whether standard PostgreSQL FDW (e.g., postgres_fdw) can be used to mount CrateDB tables into PostgreSQL — without needing a dedicated extension like ClickHouse's pg_clickhouse — given that CrateDB speaks the PostgreSQL wire protocol. It also includes backlinks to this PR and the originating comment.


🧠 Learnings used
Learnt from: amotl
Repo: crate/cratedb-guide PR: 385
File: docs/connect/java.md:48-51
Timestamp: 2025-10-08T01:34:18.867Z
Learning: CrateDB JDBC driver uses the `jdbc:crate://` protocol scheme but communicates via the PostgreSQL wire protocol on port 5432, just like the PostgreSQL JDBC driver (`jdbc:postgresql://`). Do not confuse the `jdbc:crate://` scheme with other protocol schemes like `crate://` (used by SQLAlchemy dialect for CrateDB) or assume it uses the HTTP endpoint on port 4200.

Learnt from: amotl
Repo: crate/cratedb-guide PR: 408
File: docs/connect/elixir.md:18-22
Timestamp: 2025-10-16T10:58:18.401Z
Learning: When connecting to CrateDB using PostgreSQL-compatible drivers like Postgrex (Elixir), the `database` parameter handling may differ from standard PostgreSQL usage because CrateDB has no notion of databases like PostgreSQL does. CrateDB uses schemas to separate concerns instead. The `database` parameter might need to be omitted or handled differently to avoid connection issues.

Learnt from: amotl
Repo: crate/cratedb-guide PR: 0
File: :0-0
Timestamp: 2025-10-06T13:45:25.891Z
Learning: For the cratedb-guide repository, general discussions and broader documentation concerns should be added to the existing runsheet/tracking ticket (e.g., GH-227) rather than creating separate issues.

Learnt from: amotl
Repo: crate/cratedb-guide PR: 0
File: :0-0
Timestamp: 2025-10-06T16:40:16.322Z
Learning: In the crate/cratedb-guide repository, documentation is organized by content type: files in `docs/feature/`, `docs/integrate/`, and `docs/ingest/` are reference material designed for quick access, while `docs/start/` contains learning path content. Reference material sections use top-right reference layouts for efficient navigation.

Learnt from: amotl
Repo: crate/cratedb-guide PR: 395
File: docs/install/cloud/azure/terraform.md:108-127
Timestamp: 2025-10-10T17:58:26.545Z
Learning: In the crate/cratedb-guide repository, during documentation format conversions (e.g., from reStructuredText to MyST/Markdown), markdownlint warnings such as MD053 (unused reference-style links) should be ignored and not flagged as issues.

Learnt from: amotl
Repo: crate/cratedb-guide PR: 406
File: docs/connect/go.md:110-137
Timestamp: 2025-10-20T21:49:18.785Z
Learning: In the cratedb-guide repository, documentation examples for connecting to CrateDB (e.g., Go examples in docs/connect/go.md) should be kept minimal and focused on demonstrating basic functionality, rather than including comprehensive error handling. Full examples with proper error handling are maintained separately in the cratedb-examples repository.

Learnt from: amotl
Repo: crate/cratedb-guide PR: 263
File: docs/integrate/kafka/docker-python.md:112-120
Timestamp: 2025-08-22T18:11:12.776Z
Learning: In documentation and tutorial repositories like cratedb-guide, code examples should prioritize simplicity, clarity, and educational value over production-ready features. Comprehensive error handling, extensive validation, and other production concerns can distract from the main learning objectives and make examples harder to follow. Review suggestions should focus on correctness and clarity rather than production hardening.

Learnt from: bmunkholm
Repo: crate/cratedb-guide PR: 340
File: docs/home/index.md:84-97
Timestamp: 2025-09-25T19:31:54.320Z
Learning: In the CrateDB Guide docs (MyST), the CrateDB Cloud card on the homepage should link to `getting-started` using `:link-type: ref` instead of the previous `cloud:index` intersphinx target. This change was implemented in PR `#340` to direct users to the getting started section rather than directly to the Cloud documentation.


:::{seealso}
**Product:**
[Relational Database]
**Reference manual:** {ref}`Foreign data wrappers <crate-reference:administration-fdw>`
<br>
**SQL Functions:**
{ref}`crate-reference:ref-create-server`
• {ref}`crate-reference:ref-drop-server`
• {ref}`crate-reference:ref-create-foreign-table`
• {ref}`crate-reference:ref-drop-foreign-table`
<br>
**System Tables:**
{ref}`crate-reference:foreign_servers`
• {ref}`crate-reference:foreign_server_options`
• {ref}`crate-reference:foreign_tables`
• {ref}`crate-reference:foreign_table_options`
• {ref}`crate-reference:user_mappings`
• {ref}`crate-reference:user_mapping_options`
:::
Comment thread
bmunkholm marked this conversation as resolved.