Add /refresh endpoint

[zfarrell]

Summary

• Replaces /connections/{id}/discover with unified POST /refresh for schema + data refresh
• Adds versioned cache writes (atomic swaps) and deferred deletion with a background worker
• Adds refresh warnings and better error mapping for not-found tables

Details

• New refresh request/response models with optional warnings
• Storage refactor: versioned directories + cache write handle (filesystem + S3)
• Catalog support for pending deletions with retry tracking
• Extensive new tests for refresh behavior, storage, and deletion flow

Breaking Changes

• Removes /connections/{id}/discover; use POST /refresh instead

API

The /refresh endpoint provides a unified interface for refreshing schema metadata and cached data. It supports multiple operation modes depending on the parameters provided.

Endpoint

POST /refresh
Content-Type: application/json

Request Parameters

Parameter	Type	Required	Description
connection_id	string	No	External connection ID to target. If omitted, operates on all connections.
schema_name	string	No	Database schema name. Required when targeting a specific table.
table_name	string	No	Table name. Requires schema_name and connection_id.
data	boolean	No	If true, refreshes cached parquet data. If false (default), refreshes schema metadata only.
include_uncached	boolean	No	When data=true on a connection-wide refresh, also sync tables that aren't already cached. Default: false.

Operation Modes

1. Schema Refresh: All Connections

Discovers tables from all configured connections and updates the catalog.

POST /refresh
{}

Response:

{
  "connections_refreshed": 2,
  "connections_failed": 0,
  "tables_discovered": 15,
  "tables_added": 3,
  "tables_modified": 1
}

If any connections fail (e.g., due to missing credentials), the operation continues with remaining connections and reports failures:

{
  "connections_refreshed": 1,
  "connections_failed": 1,
  "tables_discovered": 10,
  "tables_added": 2,
  "tables_modified": 0,
  "errors": [
    {
      "connection_id": "broken_conn",
      "error": "Failed to connect: missing credentials"
    }
  ]
}

2. Schema Refresh: Single Connection

Discovers tables from a specific connection.

POST /refresh
{
  "connection_id": "my_postgres"
}

Response:

{
  "connections_refreshed": 1,
  "connections_failed": 0,
  "tables_discovered": 8,
  "tables_added": 1,
  "tables_modified": 0
}

3. Data Refresh: Single Table

Re-fetches data for a specific table and updates the parquet cache.

POST /refresh
{
  "connection_id": "my_postgres",
  "schema_name": "public",
  "table_name": "orders",
  "data": true
}

Response:

{
  "connection_id": "my_postgres",
  "schema_name": "public",
  "table_name": "orders",
  "rows_synced": 15000,
  "duration_ms": 1234
}

Returns 404 if the table doesn't exist in the catalog.

4. Data Refresh: All Tables in Connection

Re-fetches data for all cached tables in a connection.

POST /refresh
{
  "connection_id": "my_postgres",
  "data": true
}

Response:

{
  "connection_id": "my_postgres",
  "tables_refreshed": 5,
  "tables_failed": 1,
  "total_rows": 50000,
  "duration_ms": 5678,
  "errors": [
    {
      "schema_name": "public",
      "table_name": "broken_table",
      "error": "Query failed: relation does not exist"
    }
  ]
}

By default, only tables that already have cached data are refreshed. To also sync uncached tables:

POST /refresh
{
  "connection_id": "my_postgres",
  "data": true,
  "include_uncached": true
}

Warnings

Non-fatal issues (like failed cleanup of old cache files) are reported in a warnings array without failing the operation:

{
  "connection_id": "my_postgres",
  "schema_name": "public",
  "table_name": "orders",
  "rows_synced": 15000,
  "duration_ms": 1234,
  "warnings": [
    {
      "schema_name": "public",
      "table_name": "orders",
      "message": "Failed to schedule deletion of old cache: disk full"
    }
  ]
}

The warnings field is omitted when empty.

Invalid Combinations

Request	Error
data=true without connection_id	400 Bad Request - data refresh requires connection_id
schema_name without table_name (schema=false)	400 Bad Request - schema-level refresh not supported
schema_name without table_name (data=true)	400 Bad Request - data refresh with schema requires table_name
table_name without schema_name	400 Bad Request - table_name requires schema_name
schema_name or table_name with schema refresh	400 Bad Request - schema refresh cannot target specific tables

Error Responses

• 400 Bad Request - Invalid parameter combination
• 404 Not Found - Connection or table not found
• 500 Internal Server Error - Unexpected server error