docs(v1.1): Enable Iceberg REST Catalog — milestone PR description - #2
Conversation
Synthesizes findings from parallel research (STACK, FEATURES, ARCHITECTURE, PITFALLS) into SUMMARY.md. Key conclusion: v1.1 is a two-artifact wiring task (@SourceType annotation + restcatalog-layout.json) followed by end-to-end validation against Lakekeeper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add @SourceType(value="RESTCATALOG", label="Iceberg REST Catalog", uiConfig="restcatalog-layout.json") to RestIcebergCatalogPluginConfig - Add import for com.dremio.exec.catalog.conf.SourceType - Create restcatalog-layout.json at classpath root with 3-tab form: General (endpoint URI, namespace filter), Catalog Properties (propertyList, secretPropertyList), Advanced Options (enableAsync, caching) - All 8 config fields wired with correct config. propName prefixes
- Copy dac/ui-lib/icons/dremio/sources/RESTCATALOG.svg to plugins/icebergcatalog/src/main/resources/RESTCATALOG.svg - SourceTypeTemplate loads icon via classloader at classpath root; placing here makes the API source type descriptor return a non-null icon field
- Create 07-01-SUMMARY.md documenting @SourceType annotation, restcatalog-layout.json, and RESTCATALOG.svg delivery - Update STATE.md: Phase 7 plan 01 complete, progress to ~70%, decisions recorded
Two plans for end-to-end validation against Lakekeeper: - 08-01: Rebuild plugin JAR, Docker infrastructure, test data seeding - 08-02: Source creation, read operations, OAuth2 auth, credential vending Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Confirmed plugins/icebergcatalog/target/ JAR (2026-02-20) contains restcatalog-layout.json and RESTCATALOG.svg from Phase 7 - Copied JAR to distribution/server/target/.../jars/ replacing stale Feb 18 version - Added deploy-plugin-jar.sh script documenting the cp command for reproducibility
…, test data seeded Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…passed CONN-01: Source reaches GOOD state READ-01: Namespace testns browsable READ-02: Table users listed READ-03: SELECT returns 10 rows CONN-02: rest.token in secretPropertyList works CONN-03: Static fs.s3a.* creds workaround validated Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… icon
getSourceIcon() now returns sources/{TYPE} for all non-Nessie sources,
resolving per-source SVGs (e.g. RESTCATALOG.svg, S3.svg) in the source
list and catalog tree. Status variants (-bad, -degraded) still use
generic entity icons since source-specific variants don't exist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delivered: Iceberg REST Catalog source type wired into Dremio OSS, enabling read-only connectivity to any Iceberg REST catalog server (Lakekeeper, Nessie) through standard SQL. Key accomplishments: - @SourceType annotation on RestIcebergCatalogPluginConfig - restcatalog-layout.json with 3-tab UI form - E2E validated: source creation, namespace browsing, SELECT queries - Validated against both Lakekeeper and Nessie Archives: milestones/v1.1-ROADMAP.md, milestones/v1.1-REQUIREMENTS.md Known gap: CONN-03 credential vending (workaround documented) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot wasn't able to review any files in this pull request.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 31 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export const getSourceIcon = (sourceType: string) => { | ||
| if (NESSIE === sourceType) { | ||
| return "entities/nessie-source"; | ||
| } else { | ||
| return "entities/datalake-source"; | ||
| } | ||
| return `sources/${sourceType}`; | ||
| }; |
There was a problem hiding this comment.
getSourceIcon() now assumes every source type has a matching icon id "sources/${sourceType}". That’s not true for at least "SAMPLE_SOURCE": the shipped icon id is "sources/SampleSource" (and there is no "sources/SAMPLE_SOURCE"), so Azure sample sources will render with a missing icon. Consider adding a small mapping/normalization layer (e.g., SAMPLE_SOURCE -> SampleSource) and/or a safe fallback to the generic datalake icon when no source icon exists.
| const getIconForSource = (source: SourceCatalogObject) => { | ||
| switch (source.type) { | ||
| case "NESSIE": | ||
| return <dremio-icon name="entities/nessie-source" class="h-3 w-3" />; | ||
| default: | ||
| return <dremio-icon name="entities/datalake-source" class="h-3 w-3" />; | ||
| return ( | ||
| <dremio-icon name={`sources/${source.type}`} class="h-3 w-3" /> | ||
| ); |
There was a problem hiding this comment.
Catalog tree source icons now render as sources/${source.type} directly. This bypasses existing special handling in SourceIcon (e.g., ADX uses an fallback due to gradients/adblocker issues) and also misses any sourceType->iconId normalization (e.g., SAMPLE_SOURCE). Consider reusing the shared getSourceIcon()/SourceIcon rendering path here (or replicating the necessary special-cases) to avoid broken icons for ADX/NETEZZA/sample sources.
| VERSION="26.0.5-202509091642240013-f5051a07" | ||
| DIST_NAME="dremio-community-${VERSION}" | ||
|
|
||
| SRC_JAR="${REPO_ROOT}/plugins/icebergcatalog/target/dremio-icebergcatalog-plugin-${VERSION}.jar" |
There was a problem hiding this comment.
This script hard-codes the Dremio version/build string, which will go stale quickly and make the script fail in future builds. Consider deriving VERSION from the repo (e.g., Maven project.version), accepting it as an argument, or using a glob to locate the built JAR in target/ and the distribution jars/ directory.
| VERSION="26.0.5-202509091642240013-f5051a07" | |
| DIST_NAME="dremio-community-${VERSION}" | |
| SRC_JAR="${REPO_ROOT}/plugins/icebergcatalog/target/dremio-icebergcatalog-plugin-${VERSION}.jar" | |
| # Optional: pass the Dremio version as the first argument. If omitted, derive it | |
| # from the built icebergcatalog plugin JAR in the Maven target/ directory. | |
| VERSION="${1:-}" | |
| if [ -z "${VERSION}" ]; then | |
| PLUGIN_TARGET_DIR="${REPO_ROOT}/plugins/icebergcatalog/target" | |
| PLUGIN_JAR_GLOB="${PLUGIN_TARGET_DIR}/dremio-icebergcatalog-plugin-*.jar" | |
| # Use globbing to locate the built plugin JAR and extract VERSION from its name. | |
| shopt -s nullglob | |
| plugin_jars=( ${PLUGIN_JAR_GLOB} ) | |
| shopt -u nullglob | |
| if [ "${#plugin_jars[@]}" -eq 0 ]; then | |
| echo "ERROR: No plugin JARs found matching ${PLUGIN_JAR_GLOB}" | |
| echo "Run: mvn package -pl plugins/icebergcatalog -DskipTests" | |
| exit 1 | |
| fi | |
| SRC_JAR="${plugin_jars[0]}" | |
| plugin_jar_name="$(basename "${SRC_JAR}")" | |
| VERSION="${plugin_jar_name#dremio-icebergcatalog-plugin-}" | |
| VERSION="${VERSION%.jar}" | |
| else | |
| SRC_JAR="${REPO_ROOT}/plugins/icebergcatalog/target/dremio-icebergcatalog-plugin-${VERSION}.jar" | |
| fi | |
| DIST_NAME="dremio-community-${VERSION}" |
| ### Blockers/Concerns | ||
|
|
||
| - [Build]: Maven build requires Java 21 (enforcer [21,22) range); only Java 11/17 available. Full Maven compile blocked until Java 21 JDK is installed. | ||
| - [Build]: Maven build requires Java 21 (enforcer [21,22) range); only Java 11/17 available. |
There was a problem hiding this comment.
STATE.md says the Maven build is blocked because only Java 11/17 are available, but the Phase 8 research doc in this PR states Java 21 is already installed/available. Please reconcile these statements (either update STATE.md or adjust the research doc) so the project state is unambiguous for future readers.
| - [Build]: Maven build requires Java 21 (enforcer [21,22) range); only Java 11/17 available. | |
| None — Java 21 is now available; Maven build is no longer blocked. |
| import com.dremio.exec.catalog.conf.SourceType; | ||
| import io.protostuff.Tag; | ||
| import java.util.List; | ||
| import javax.inject.Provider; | ||
|
|
||
| @SourceType(value = "RESTCATALOG", label = "Iceberg REST Catalog", uiConfig = "restcatalog-layout.json") | ||
| public class RestIcebergCatalogPluginConfig extends IcebergCatalogPluginConfig { |
There was a problem hiding this comment.
PR metadata/description says this is a documentation-only PR, but this change adds production code/resources (e.g., new @SourceType annotation and UI/layout/icon wiring). Please update the PR description to reflect the code changes (or split into a docs-only PR + a code PR) so reviewers and release notes aren’t misled.
Updates the PR description for the v1.1 milestone to a structured, information-dense summary covering all phases, acceptance criteria, key files, and known gaps for the Iceberg REST Catalog feature.
Changes
Original prompt
Context
This is a documentation-only PR: create a pull request from branch
milestones/enable_iceberg_rest_catalogintodevelop. Do not modify any source code. The only task is to open the PR with a rich, accurate description summarising all the activities carried out in themilestones/enable_iceberg_rest_catalogmilestone (v1.1).Pull Request Description to write
Write the PR body in Markdown. It must cover all of the phases and commits found in the milestone branch, as reconstructed below from the commit history. The description should be structured as follows:
🎯 Summary
This PR merges milestone v1.1 — Enable Iceberg REST Catalog into
develop.It wires a brand-new Iceberg REST Catalog source type into Dremio OSS, enabling read-only SQL connectivity to any Iceberg REST Catalog server (Lakekeeper, Nessie) without modifying upstream Dremio Java code beyond the plugin boundary.
✅ Acceptance Criteria — all passed
testnsbrowsable in UIuserslisted under namespaceSELECT * FROM ...returns 10 rowsrest.tokenlisted insecretPropertyListfs.s3a.*credentials workaround validated🗂️ Milestone phases completed
Phase 1–6 — Foundation (carried over from upstream branch)
Upstream Dremio OSS codebase was used as the base. Phases 1–6 covered repository analysis, architecture scoping, plugin framework understanding, REST catalog Java API research, initial
RestIcebergCatalogPluginConfigandRestIcebergCatalogPluginimplementation, and iterative compilation fixes.Phase 7 — Plugin Wiring (
07-plugin-wiring)@SourceType("RESTCATALOG")annotation toRestIcebergCatalogPluginConfigrestcatalog-layout.json(3-tab UI form: General, Advanced, Metadata) in theicebergcatalogplugin resourcesRESTCATALOG.svgsource icon at the classpath rootKey commits:
0f6b172—docs(07-plugin-wiring): phase plan creation@SourceTypeannotation and UI form JSONPhase 8 — End-to-End Validation (
08-end-to-end-validation)Plan 08-01 — Infrastructure Setup
icebergcatalogplugin JAR and deployed to Dremio distribution (replaced stale Feb 18 JAR)demo)testns, tableusers(10 rows: id, name, city)RESTCATALOGsource type discoverabledeploy-plugin-jar.shreproducibility scriptKey commits:
95f4e7e—chore(08-01): deploy Phase 7 plugin JAR to distribution9873c3e—docs(08-01): infrastructure setup complete — Lakekeeper stack running, test data seededPlan 08-02 — Validation
testnsbrowsable, tableuserslisted,SELECT *returns 10 rowsrest.tokenmasked in secretPropertyListfs.s3a.*credentials workaround workingKey commits:
d8649dc—docs(08-02): end-to-end validation complete — all 6 success criteria passedPhase 8 finalisation
e7ef756—fix(ui):getSourceIcon()now returnssources/{TYPE}for all non-Nessie sources, resolving per-source SVGs (e.g.RESTCATALOG.svg,S3.svg) in source list and catalog tree079b070—docs(phase-8): complete phase execution — v1.1 milestone finishedd2b694a—chore: complete v1.1 Enable Iceberg REST Catalog milestone (final summary commit)📁 Key files introduced / modified
plugins/icebergcatalog/src/main/java/.../RestIcebergCatalogPluginConfig.java@SourceType("RESTCATALOG")annotation addedplugins/icebergcatalog/src/main/resources/restcatalog-layout.jsonplugins/icebergcatalog/src/main/resources/RESTCATALOG.svgdac/ui/src/utils/iconUtils.js(or similar)getSourceIcon()returnssources/{TYPE}for non-Nessie sources.planning/phases/08-end-to-end-validation/deploy-plugin-jar.sh.planning/phases/08-end-to-end-validation/08-01-SUMMARY.md.planning/STATE.mdmilestones/v1.1-ROADMAP.mdThis pull request was created from Copilot chat.
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.