docs(v1.1): Enable Iceberg REST Catalog — milestone PR description

.claude/commands/test-rest-catalog.md

@@ -0,0 +1,279 @@
+---
+name: test-rest-catalog
+description: Deploy a REST catalog, register it as a RESTCATALOG source in Dremio, and validate read operations
+argument-hint: "<catalog-name> (e.g., nessie, lakekeeper, polaris, gravitino, unity)"
+allowed-tools:
+  - Read
+  - Write
+  - Edit
+  - Glob
+  - Grep
+  - Bash
+  - Task
+  - AskUserQuestion
+  - WebSearch
+  - WebFetch
+---
+
+<objective>
+Automated end-to-end validation of the Dremio RESTCATALOG plugin against any Iceberg REST Catalog server.
+
+**What this does:**
+1. Research deployment of the specified REST catalog
+2. Create a Docker Compose (or alternative) to run it locally
+3. Seed test data (namespace + table with sample rows)
+4. Obtain a Dremio admin token
+5. Register the catalog as a RESTCATALOG source in Dremio
+6. Validate: source health, namespace browsing, table listing, SELECT query
+7. Report pass/fail results
+
+**Prerequisites:**
+- Dremio must be running locally on port 9047
+- Docker must be available
+- Python 3 with `pyiceberg`, `pyarrow`, and `s3fs` installed (for data seeding)
+</objective>
+
+<context>
+**Argument:** $ARGUMENTS — the REST catalog name to test (e.g., "nessie", "lakekeeper", "polaris", "gravitino", "unity")
+
+If no argument given, ask the user which catalog to test.
+
+**Known catalog recipes:**
+
+### Nessie
+- Image: `ghcr.io/projectnessie/nessie:latest`
+- Port: 19120
+- Iceberg REST endpoint: `http://localhost:19120/iceberg/`
+- Needs: MinIO for storage (S3-compatible)
+- Auth: disabled for testing (`nessie.server.authentication.enabled=false`)
+- Warehouse config is server-side (`nessie.catalog.warehouses.warehouse.location=s3://warehouse/`)
+- S3 creds configured server-side via secrets indirection pattern
+
+### Lakekeeper
+- Image: `quay.io/lakekeeper/catalog:latest-main`
+- Port: 8181
+- Iceberg REST endpoint: `http://localhost:8181/catalog/`
+- Needs: PostgreSQL + MinIO
+- Requires bootstrap: accept terms + create warehouse via management API
+- Warehouse name configured via management API after startup
+
+### Polaris (Apache, formerly Snowflake)
+- Image: `apache/polaris:latest` (or `polarisoss/polaris:latest`)
+- Port: 8181
+- Iceberg REST endpoint: `http://localhost:8181/api/catalog`
+- Needs: MinIO for storage
+- Requires bootstrap: create principal, catalog, grants via management API
+- OAuth2 client credentials auth
+
+### Gravitino (Apache)
+- Image: `apache/gravitino:latest`
+- Port: 8090
+- Iceberg REST endpoint: `http://localhost:9001/iceberg/`
+- Gravitino acts as meta-catalog; Iceberg REST is one backend
+- Needs: separate catalog registration via Gravitino API
+
+### Unity Catalog (Databricks)
+- Image: `unitycatalog/unitycatalog:latest`
+- Port: 8080
+- Iceberg REST endpoint: `http://localhost:8080/api/2.1/unity-catalog/iceberg`
+- Local filesystem storage (no MinIO needed for basic test)
+
+**Dremio RESTCATALOG source config pattern:**
+- `restEndpointUri`: the Iceberg REST endpoint URL
+- `propertyList`: catalog properties including `fs.s3a.*` for S3 storage access
+- `secretPropertyList`: for sensitive properties like `rest.token`
+- Static `fs.s3a.*` credentials needed (Dremio doesn't propagate vended credentials)
+</context>
+
+<process>
+
+## Step 1: Determine Target Catalog
+
+If `$ARGUMENTS` is empty or not recognized, ask:
+
+```
+AskUserQuestion(
+  header="Catalog",
+  question="Which Iceberg REST Catalog do you want to test?",
+  options: [
+    "Nessie (Recommended)" — "Well-tested, simple setup, Iceberg REST at /iceberg/",
+    "Lakekeeper" — "Native Iceberg REST spec, needs PostgreSQL + MinIO",
+    "Polaris" — "Apache Polaris, OAuth2 auth, needs bootstrap",
+    "Other" — "Provide Docker image and endpoint details"
+  ]
+)
+```
+
+Set `CATALOG_NAME` from argument or selection.
+
+## Step 2: Research Deployment
+
+**If catalog is in the known recipes above**, use those details directly.
+
+**If catalog is unknown or "Other":**
+1. Use WebSearch to find: `"<catalog-name> iceberg rest catalog docker" site:github.com OR site:hub.docker.com`
+2. Identify: Docker image, port, Iceberg REST endpoint path, required dependencies
+3. Present findings and confirm with user before proceeding
+
+## Step 3: Create Infrastructure
+
+Create a working directory at `/tmp/dremio-test-<catalog-name>/`.
+
+Generate `docker-compose.yml` based on the catalog recipe:
+- Include MinIO if the catalog needs S3 storage
+- Include PostgreSQL if the catalog needs it
+- Use healthchecks where possible
+- Avoid port conflicts with existing services (check with `docker ps` first)
+
+Generate `seed-data.py` using PyIceberg:
+```python
+catalog = load_catalog("<name>", **{
+    "type": "rest",
+    "uri": "<iceberg-rest-endpoint>",
+    # S3 creds if needed for client-side writes
+})
+catalog.create_namespace("testns")
+table = catalog.create_table("testns.products", schema=...)
+table.append(data)  # 5 sample rows
+```
+
+## Step 4: Start and Seed
+
+```bash
+cd /tmp/dremio-test-<catalog-name>
+docker compose up -d
+# Wait for service health
+# Run any bootstrap steps (e.g., Lakekeeper warehouse creation, Polaris principal setup)
+python3 seed-data.py
+```
+
+## Step 5: Get Dremio Token
+
+```bash
+# Ask user for credentials if not known
+TOKEN=$(curl -s http://localhost:9047/apiv2/login \
+  -H "Content-Type: application/json" \
+  -d '{"userName":"<user>","password":"<pass>"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
+```
+
+If login fails, ask user for correct Dremio credentials:
+```
+AskUserQuestion(header="Dremio Auth", question="What are your Dremio admin credentials?", ...)
+```
+
+## Step 6: Register Source in Dremio
+
+```bash
+curl -s -X POST "http://localhost:9047/api/v3/catalog" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "entityType": "source",
+    "name": "<catalog-name>_rest",
+    "type": "RESTCATALOG",
+    "config": {
+      "restEndpointUri": "<iceberg-rest-endpoint>",
+      "propertyList": [
+        // fs.s3a.* properties for MinIO/S3 access
+      ],
+      "secretPropertyList": [
+        // rest.token if OAuth2 needed
+      ],
+      "enableAsync": false,
+      "isCachingEnabled": true,
+      "maxCacheSpacePct": 100
+    }
+  }'
+```
+
+Verify response shows `"state": {"status": "good"}`.
+
+If source creation fails:
+- Check Dremio coordinator logs: `docker logs dremio 2>&1 | tail -30` or local logs
+- Common issues: endpoint not reachable, auth failure, missing catalog properties
+- Report error and suggest fix
+
+## Step 7: Validate Read Operations
+
+Run these checks sequentially and collect results:
+
+### 7a. Source Health
+```bash
+curl -s "http://localhost:9047/api/v3/catalog/by-path/<source-name>" \
+  -H "Authorization: Bearer $TOKEN" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['state']['status'])"
+```
+Expected: `good`
+
+### 7b. Namespace Browsing
+```bash
+curl -s "http://localhost:9047/api/v3/catalog/by-path/<source-name>" \
+  -H "Authorization: Bearer $TOKEN" | python3 -c "
+import sys,json; d=json.load(sys.stdin)
+for c in d.get('children',[]): print(f\"{c['type']}: {'/'.join(c['path'])}\")"
+```
+Expected: `testns` appears as CONTAINER/FOLDER
+
+### 7c. Table Listing
+```bash
+curl -s "http://localhost:9047/api/v3/catalog/by-path/<source-name>/testns" \
+  -H "Authorization: Bearer $TOKEN" | python3 -c "
+import sys,json; d=json.load(sys.stdin)
+for c in d.get('children',[]): print(f\"{c['type']}: {'/'.join(c['path'])}\")"
+```
+Expected: `products` appears as DATASET
+
+### 7d. SELECT Query
+```bash
+JOB=$(curl -s -X POST "http://localhost:9047/api/v3/sql" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"sql": "SELECT * FROM <source-name>.testns.products"}')
+JOB_ID=$(echo "$JOB" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
+# Poll until COMPLETED/FAILED
+# Fetch results
+curl -s "http://localhost:9047/api/v3/job/$JOB_ID/results" -H "Authorization: Bearer $TOKEN"
+```
+Expected: 5 rows with columns id (BIGINT), name (VARCHAR), city (VARCHAR)
+
+## Step 8: Report Results
+
+Present results as a table:
+
+```
+## Test Results: <catalog-name> via RESTCATALOG
+
+| Check               | Status  | Details                    |
+|---------------------|---------|----------------------------|
+| Docker stack        | PASS    | All containers healthy     |
+| Data seeding        | PASS    | testns.products: 5 rows    |
+| Source creation      | PASS    | State: good                |
+| Namespace browsing   | PASS    | testns visible             |
+| Table listing        | PASS    | products visible           |
+| SELECT query         | PASS    | 5 rows returned            |
+
+Infrastructure: /tmp/dremio-test-<catalog-name>/
+Teardown: docker compose -f /tmp/dremio-test-<catalog-name>/docker-compose.yml down
+```
+
+If any check failed, report the failure with error details and suggest troubleshooting steps.
+
+</process>
+
+<error_handling>
+
+**Docker not available:** Tell user to install Docker and retry.
+**Port conflict:** Check `docker ps` and `ss -tlnp` for conflicts. Use alternative ports.
+**Dremio not running:** Tell user to start Dremio first.
+**PyIceberg not installed:** Run `pip install pyiceberg pyarrow s3fs` and retry.
+**Seed data fails:** Check if catalog is healthy, storage is accessible. Show logs.
+**Source creation fails:** Check Dremio logs, verify endpoint reachability from host.
+**SELECT fails:** Check fs.s3a.* properties, hostname resolution, storage connectivity.
+
+</error_handling>
+
+<cleanup>
+The Docker stack is left running for the user to explore. Remind them:
+```
+Teardown: docker compose -f /tmp/dremio-test-<catalog-name>/docker-compose.yml down -v
+```
+</cleanup>

.planning/MILESTONES.md

@@ -28,3 +28,31 @@
 
 ---
 
+
+## v1.1 Enable Iceberg REST Catalog (Shipped: 2026-02-20)
+
+**Phases completed:** 2 phases, 3 plans, 4 tasks
+**Code files modified:** 3 (1 modified, 2 created)
+**Lines of code:** 120 insertions (Java + JSON + SVG)
+**Timeline:** 1 day (2026-02-20)
+**Git range:** 37b035f80..079b07017
+
+**Delivered:** Iceberg REST Catalog source type wired into Dremio OSS, enabling read-only connectivity to any Iceberg REST Catalog server (Lakekeeper, Nessie, Polaris) through standard SQL.
+
+**Key accomplishments:**
+- Added `@SourceType(value="RESTCATALOG")` annotation making the Iceberg REST Catalog plugin discoverable by Dremio's connection scanner
+- Created `restcatalog-layout.json` with 3-tab UI form covering all 8 config fields (endpoint, namespaces, properties, credentials, caching)
+- Validated end-to-end against Lakekeeper: source creation, namespace browsing, table listing, SELECT queries
+- Validated OAuth2 bearer token authentication via `rest.token` catalog property
+- Documented credential vending gap and established portable static `fs.s3a.*` workaround
+- Validated plugin against both Lakekeeper and Nessie REST catalogs
+
+### Known Gaps
+
+- **CONN-03 (partial):** Credential vending from Lakekeeper `loadTable()` not propagated through DremioFileIO. Static `fs.s3a.*` credentials workaround validated. Fix point: `AbstractRestCatalogAccessor.getTableHandleInternal()`. Works for long-lived creds; fails for IAM/STS short-lived tokens.
+- **hasAccessPermission() no-op:** All Dremio users have full read access to all REST catalog tables. RBAC integration deferred to future milestone.
+
+**Archives:** `milestones/v1.1-ROADMAP.md`, `milestones/v1.1-REQUIREMENTS.md`, `milestones/v1.1-MILESTONE-AUDIT.md`
+
+---
+

.planning/PROJECT.md

@@ -1,12 +1,12 @@
-# Dremio OSS Naive RBAC
+# Dremio OSS Enhancements
 
 ## What This Is
 
-A role-based access control system for Dremio OSS that controls who can SELECT views (VDS), CREATE OR REPLACE views, and EXECUTE user-defined functions. Built on top of Dremio's existing catalog infrastructure, wiring up the previously no-op `validatePrivilege()` to enforce real permission checks backed by RocksDB. Manageable via SQL DDL, REST API, and observable through system tables.
+Enterprise-grade enhancements for Dremio OSS. v1.0 delivered deny-by-default RBAC for views and UDFs. v1.1 enabled the Iceberg REST Catalog source type, allowing Dremio OSS to connect to external Iceberg REST catalog servers (Lakekeeper, Nessie, Polaris) and query tables through standard SQL. Validated against both Lakekeeper and Nessie.
 
 ## Core Value
 
-Users can only access views and UDFs they've been explicitly granted access to, with deny-by-default policy and admin bypass — closing the open-access gap in Dremio OSS.
+Make Dremio OSS a production-capable data lakehouse query engine by closing critical gaps in access control and catalog connectivity.
 
 ## Requirements
 
@@ -28,12 +28,15 @@ Users can only access views and UDFs they've been explicitly granted access to,
 - ✓ Wire up existing GRANT/REVOKE SQL DDL (no longer throws UnsupportedError in OSS) — v1.0
 - ✓ REST API endpoints for role and grant management (9 endpoints at /api/v3/rbac) — v1.0
 - ✓ System tables populated: sys.roles, sys.privileges, sys.membership — v1.0
+- ✓ Iceberg REST Catalog source type discoverable and creatable via UI/API — v1.1
+- ✓ Read-only operations: browse namespaces, list tables, SELECT from tables — v1.1
+- ✓ Validated end-to-end against Lakekeeper and Nessie — v1.1
 
 ### Active
 
-<!-- Current scope. Building toward these. -->
+<!-- Next milestone scope. -->
 
-(None yet — define in next milestone)
+(None yet — define with `/gsd:new-milestone`)
 
 ### Out of Scope
 
@@ -48,25 +51,19 @@ Users can only access views and UDFs they've been explicitly granted access to,
 - Ownership transfer (GRANT OWNERSHIP) — not needed for naive model
 - Source-level or space-level permissions — out of scope; focus is on VDS and UDFs
 - Offline mode — real-time catalog enforcement is the model
+- Credential vending propagation — DremioFileIO uses static Hadoop Config; static creds workaround sufficient for v1.1
 
 ## Context
 
-Shipped v1.0 with ~4,577 LOC Java across 69 files.
-Tech stack: Java, Proto3, RocksDB KV stores, Jersey/JAX-RS REST, Dremio CatalogImpl enforcement.
-All RBAC code lives in `com.dremio.exec.rbac` package (sabot/kernel module).
+**v1.0 RBAC:** Shipped with ~4,577 LOC Java across 69 files. All RBAC code in `com.dremio.exec.rbac` package.
 
-Post-v1.0 audit identified and fixed 2 enforcement bypass paths (bulkGetTables, AT-specifier) and v2 API visibility gaps.
-Known v1.0 limitation: catalog visibility pagination may return fewer items than requested when RBAC filters are active.
-Build caveat: Maven build requires Java 21 (enforcer [21,22) range); proto verified with protoc 3.6.0 directly.
+**v1.1 Iceberg REST Catalog:** `@SourceType(value="RESTCATALOG")` added to `RestIcebergCatalogPluginConfig`, `restcatalog-layout.json` created with 3-tab UI form, `RESTCATALOG.svg` icon at classpath root. 120 LOC across 3 files. Validated against Lakekeeper and Nessie. Static `fs.s3a.*` credentials needed as workaround for credential vending gap.
 
-### v2 candidates (from requirements backlog)
-- WITH GRANT OPTION (delegate privilege granting)
-- REVOKE CASCADE
-- Container grants (space/folder-level)
-- INFORMATION_SCHEMA filtering
-- Audit logging for RBAC DDL operations
-- Migration tooling for existing deployments
-- DACSecurityContext.isUserInRole() real implementation
+Build caveat: Maven build requires Java 21 (enforcer [21,22) range).
+
+### Future candidates
+- RBAC: WITH GRANT OPTION, REVOKE CASCADE, container grants, INFORMATION_SCHEMA filtering, audit logging, privilege caching
+- Iceberg REST Catalog: write operations, credential vending fix, RBAC integration, auth config validation, multi-catalog validation (Polaris, Unity, Gravitino)
 
 ## Constraints
 
@@ -90,6 +87,9 @@ Build caveat: Maven build requires Java 21 (enforcer [21,22) range); proto verif
 | Role IDs = slugified names (not UUIDs) | Human-readable keys, immutable (no rename support) | ✓ Good — simple lookup |
 | No privilege caching in v1 | Hit KV store every hasPrivilege() call; simplicity over performance | ⚠️ Revisit — may need caching at scale |
 | DDL works when RBAC flag is OFF | Admins set up roles/grants before enabling enforcement | ✓ Good — enables staged rollout |
+| @SourceType on concrete class only | ConnectionReaderImpl.getCandidateSources() skips abstract classes | ✓ Good — scanner finds it correctly |
+| Static fs.s3a.* credentials workaround | DremioFileIO discards vended credentials from loadTable(); static Hadoop Config is the only path | ⚠️ Revisit — need credential vending propagation for IAM/STS |
+| Validate against multiple REST catalogs | Nessie + Lakekeeper confirms plugin is spec-compliant, not server-specific | ✓ Good — portable across implementations |
 
 ---
-*Last updated: 2026-02-19 after v1.0 milestone*
+*Last updated: 2026-02-20 after v1.1 milestone shipped*