Manifest decoding does not support lower-case values for `file_format`, making data files from some writers unreadable

[wroever]

Apache Iceberg version

main (development)

Please describe the bug 🐞

Version

Observed on main at 87789102577c (2026-09-01) using DuckDB v1.5.5 as writer.

Summary

Reading a table whose Avro manifests specify a lowercase file_format for data files (e.g. parquet) fails on the first data file with the following error:

not implemented: only parquet format is implemented, got parquet

This happens via the ordinary read path: LoadTable followed by tbl.Scan().

The Iceberg spec uses lowercase strings as illustrative example values for file_format, so so I don't think this is reasonably characterized as a DuckDB quirk/bug.

Meanwhile, unlike iceberg-go, both pyiceberg and the Java reference implementation appear to be case-agnostic with respect to file_format (Pyiceberg has a test case specifically intended to ensure case-insensitivity when scanning). iceberg-go ought to behave similarly to ensure portability of tables written by other writers.

Steps to reproduce

1. Start a REST catalog over a local directory.

mkdir -p /tmp/icewh
docker run -d --name ice-rest -p 8181:8181 \
  -v /tmp/icewh:/tmp/icewh \
  -e CATALOG_WAREHOUSE=file:///tmp/icewh \
  apache/iceberg-rest-fixture:latest

2. Create and populate a table with DuckDB.

INSTALL iceberg; LOAD iceberg;
ATTACH 'file:///tmp/icewh' AS cat (
    TYPE ICEBERG, ENDPOINT 'http://localhost:8181', AUTHORIZATION_TYPE 'none'
);
CREATE SCHEMA cat.ns;
CREATE TABLE cat.ns.t (id BIGINT);
INSERT INTO cat.ns.t VALUES (1);

3. Confirm the manifest records a lowercase file_format

SELECT file_format FROM iceberg_metadata(cat.ns.t); -- "parquet" expected

4. Scan the table with iceberg-go:

cat, _ := rest.NewCatalog(ctx, "cat", "http://localhost:8181",
    rest.WithWarehouseLocation("file:///tmp/icewh"))

tbl, err := cat.LoadTable(ctx, table.Identifier{"ns", "t"})
// succeeds: table metadata and manifest lists parse fine

_, recs, err := tbl.Scan().ToArrowRecords(ctx)
for rec, err := range recs {
    if err != nil {
        log.Fatal(err) // not implemented: only parquet format is implemented, got parquet
    }
    _ = rec
}

Details on Java/Python behavior

Both the Java and Python reference implementations avoid problems here by normalizing `file_format` casing on decode. `pyiceberg` has a test case specifically to ensure case-insensitivity.

Java — BaseFile.internalSet routes the raw Avro value through FileFormat.fromString:

// core/src/main/java/org/apache/iceberg/BaseFile.java
case 2:
  this.format = FileFormat.fromString(value.toString());
  return;

// api/src/main/java/org/apache/iceberg/FileFormat.java
public static FileFormat fromString(String fileFormat) {
  Preconditions.checkArgument(null != fileFormat, "Invalid file format: null");
  try {
    return FileFormat.valueOf(fileFormat.toUpperCase(Locale.ROOT));
  } catch (IllegalArgumentException e) {
    throw new IllegalArgumentException(
      String.format("Invalid file format: %s", fileFormat), e);
  }
}

Python — FileFormat is a str enum with a _missing_ hook that uppercases before matching:

# pyiceberg/manifest.py
class FileFormat(str, Enum):
    AVRO = "AVRO"
    PARQUET = "PARQUET"
    ORC = "ORC"
    PUFFIN = "PUFFIN"

    @classmethod
    def _missing_(cls, value: object) -> None | str:
        for member in cls:
            if member.value == str(value).upper():
                return member
        return None

[wroever]

Will take a stab at a PR for this.