feat: simplified REST catalog CLI with partition pruning and compaction

[smithclay]

feat: simplified REST catalog CLI with partition pruning and compaction

Summary

This PR introduces a major enhancement to icepick with a simplified CLI, partition pruning, and table compaction support.

Key Features

1. Simplified REST Catalog CLI (--catalog-url + --token)

The CLI now requires just two inputs to connect to any Iceberg REST catalog:

• --catalog-url: The base URL of the catalog
• --token: Bearer token for authentication

Everything else (prefix, storage credentials) is derived from the Iceberg REST API's /v1/config endpoint.

Example:

icepick --catalog-url "https://catalog.cloudflarestorage.com/account/bucket" \
        --token "your-token" \
        table list --namespace default

2. Full CLI Implementation

New commands:

• catalog info - Show catalog connection status
• namespace list - List namespaces
• table list --namespace <ns> - List tables in a namespace
• table info <ns.table> - Show table schema and metadata
• table files <ns.table> - List data files
• compact <ns.table> - Compact table data files

Output formats: --output text (default) or --output json

3. Partition Pruning for Table Scans

Tables can now be scanned with filter predicates that prune partitions:

let scan = table.scan()
    .filter("date >= '2024-01-01' AND region = 'us-east-1'")
    .build()?;

Supports:

• Comparison operators: =, !=, <, <=, >, >=
• Logical operators: AND, OR, NOT
• IN lists and IS NULL / IS NOT NULL
• Identity, year, month, day, hour transforms

4. Table Compaction

Compact small files into larger ones:

use icepick::compact::{compact_table, CompactOptions};
 
let options = CompactOptions::new()
    .with_target_file_size(128 * 1024 * 1024)  // 128MB
    .with_min_files_to_compact(5);
 
let result = compact_table(&catalog, &table_id, options).await?;
println!("Compacted {} files into {}", result.input_files, result.output_files);

Technical Changes

• IcebergRestCatalog::from_url() - New constructor that fetches config and sets up vended credentials
• VendedCredentialProvider trait - FileIO support for catalog-vended S3 credentials
• Expression parsing - SQL-like predicate parser for filter expressions
• Partition evaluation - Evaluates predicates against partition values
• Manifest reading - Enhanced manifest parsing with partition value extraction

Files Changed

Category	Files	Description
CLI	9 files	New CLI binary and command modules
Catalog	4 files	REST catalog with simplified URL+token auth
FileIO	2 files	Vended credentials support
Expressions	4 files	Predicate parsing and evaluation
Compaction	4 files	Table compaction orchestration
Scanning	2 files	Partition pruning integration
Docs	2 files	Implementation plans

Testing

• Validated against Cloudflare R2 Data Catalog
• Successfully connected and listed tables with simplified CLI
• All existing tests pass

Breaking Changes

• CLI arguments changed: removed --arn, --r2-account, --r2-bucket
• Now uses --catalog-url + --token for all catalog types

Test Plan

• cargo test passes
• cargo clippy passes (warnings only)
• CLI connects to real R2 catalog with --catalog-url + --token
• catalog info shows "Connected" status
• table list --namespace default returns tables
• table info default.logs shows schema