refactor parquet -> icebug-disk

docs/icebug-disk.md

@@ -0,0 +1,54 @@
+# Icebug-Disk Storage Format
+
+## Overview
+
+This is LadybugDB's implementation of [Icebug-Disk](https://github.com/Ladybug-Memory/icebug-format), a read-only graph storage format based on Parquet files. It is designed for efficient analytical queries on large graphs.
+
+## V1
+
+Implements Icebug-Disk v1.
+
+### Version
+
+Version is stored in each file's metadata footer as a key-value pair: `icebug_disk_version = v1`.
+
+### schema.cypher
+
+The graph schema is declared in a `schema.cypher`, which can be loaded using `lbug -i schema.cypher` to create tables in Ladybug.
+
+```cypher
+CREATE NODE TABLE city(id INT32, name STRING, population INT64, PRIMARY KEY(id)) WITH (storage = '<path-to-dir>', format = 'icebug-disk');
+CREATE NODE TABLE user(id INT32, name STRING, age INT64, PRIMARY KEY(id)) WITH (storage = '<path-to-dir>', format = 'icebug-disk');
+CREATE REL TABLE follows(FROM user TO user, since INT32) WITH (storage = '<path-to-dir>', format = 'icebug-disk');
+CREATE REL TABLE livesin(FROM user TO city) WITH (storage = '<path-to-dir>', format = 'icebug-disk');
+```
+
+File paths can be relative or absolute and are resolved as `<path-to-dir>/nodes_{tableName}.parquet` for node tables, and `<path-to-dir>/indices_{tableName}.parquet` and `<path-to-dir>/indptr_{tableName}.parquet` for relationship tables.
+
+Object-store URIs (e.g. `s3://bucket/path`, `https://host/path`) are supported as `storage` values and are passed through unchanged.
+
+If `storage` is omitted when `format = 'icebug-disk'` is set, files are resolved relative to the current working directory.
+
+Tables can also be created by manually running the above queries in the Ladybug CLI.
+
+If the directory is moved, the affected tables must be dropped and re-created with the updated path. This updates the file pointers in the catalog. A fresh db instance can also be used to run the `CREATE TABLE` queries with the new paths.
+
+Mixed tables are not supported — `CREATE REL TABLE` queries involving both `icebug-disk` and non-`icebug-disk` tables will throw a `BinderException`.
+
+Note: Icebug-disk tables are **immutable**: `ALTER TABLE` operations, Data insertions, updates, and deletions are not supported.
+
+### Node tables
+
+For each node table, there is a corresponding Parquet file named `nodes_{tableName}.parquet` containing a primary key column and one column per property as declared in the schema.
+
+### Indices
+
+Each relationship table has a corresponding `indices_{tableName}.parquet` file containing one row per edge. The first column is always `target` (the destination node offset), followed by zero or more edge property columns as declared in the schema.
+
+### Indptr
+
+Each relationship table has a corresponding `indptr_{tableName}.parquet` file containing the CSR row pointers. It has a single integer column with `N+1` entries, where `N` is the number of source nodes.
+
+## Convert from other formats
+
+You can convert from other graph formats (e.g. duckdb, parquet tables) to Icebug-Disk using the script at https://github.com/Ladybug-Memory/icebug-format

docs/morsel_parallelism.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-Morsel-driven parallelism is ladybug's approach to parallel query execution where work is divided into small, independently processable chunks called "morsels." This document explains how it works for native node tables versus columnar formats (Arrow, Parquet).
+Morsel-driven parallelism is ladybug's approach to parallel query execution where work is divided into small, independently processable chunks called "morsels." This document explains how it works for native node tables versus columnar formats (Arrow, Icebug Disk).
 
 ## Architecture
 

src/binder/bind/bind_ddl.cpp

@@ -10,7 +10,9 @@
 #include "binder/expression_visitor.h"
 #include "catalog/catalog.h"
 #include "catalog/catalog_entry/node_table_catalog_entry.h"
+#include "catalog/catalog_entry/rel_group_catalog_entry.h"
 #include "catalog/catalog_entry/sequence_catalog_entry.h"
+#include "common/constants.h"
 #include "common/enums/extend_direction_util.h"
 #include "common/exception/binder.h"
 #include "common/exception/message.h"
@@ -190,14 +192,23 @@ static ExtendDirection getStorageDirection(const case_insensitive_map_t<Value>&
     return DEFAULT_EXTEND_DIRECTION;
 }
 
+static std::string getStorageFormat(const case_insensitive_map_t<Value>& options) {
+    if (options.contains(TableOptionConstants::STORAGE_FORMAT_OPTION)) {
+        return options.at(TableOptionConstants::STORAGE_FORMAT_OPTION).toString();
+    }
+
+    return "";
+}
+
 BoundCreateTableInfo Binder::bindCreateNodeTableInfo(const CreateTableInfo* info) {
     auto propertyDefinitions = bindPropertyDefinitions(info->propertyDefinitions, info->tableName);
     auto& extraInfo = info->extraInfo->constCast<ExtraCreateNodeTableInfo>();
     validatePrimaryKey(extraInfo.pKName, propertyDefinitions);
     auto boundOptions = bindParsingOptions(extraInfo.options);
     auto storage = getStorage(boundOptions);
+    auto storageFormat = getStorageFormat(boundOptions);
     auto boundExtraInfo = std::make_unique<BoundExtraCreateNodeTableInfo>(extraInfo.pKName,
-        std::move(propertyDefinitions), std::move(storage));
+        std::move(propertyDefinitions), std::move(storage), std::move(storageFormat));
     return BoundCreateTableInfo(CatalogEntryType::NODE_TABLE_ENTRY, info->tableName,
         info->onConflict, std::move(boundExtraInfo), clientContext->useInternalCatalogEntry());
 }
@@ -220,15 +231,18 @@ BoundCreateTableInfo Binder::bindCreateRelTableGroupInfo(const CreateTableInfo*
     auto boundOptions = bindParsingOptions(extraInfo.options);
     auto storageDirection = getStorageDirection(boundOptions);
     auto storage = getStorage(boundOptions);
+    auto storageFormat = getStorageFormat(boundOptions);
     std::optional<function::TableFunction> scanFunction = std::nullopt;
     std::optional<std::unique_ptr<function::TableFuncBindData>> scanBindData = std::nullopt;
     std::string foreignDatabaseName;
     if (!storage.empty()) {
         auto dotPos = storage.find('.');
         // Check if storage is database.table format by verifying the attached database exists
+        // Handle special case where icebug-disk storage could contain a dot
         // Otherwise, treat as file path (e.g., "dataset/demo-db/icebug-disk/demo" or
         // "data.parquet")
-        if (dotPos != std::string::npos) {
+        if (!TableOptionConstants::isIceBugDiskFormat(storageFormat) &&
+            dotPos != std::string::npos) {
             std::string dbName = storage.substr(0, dotPos);
             std::string tableName = storage.substr(dotPos + 1);
             if (!dbName.empty()) {
@@ -319,6 +333,26 @@ BoundCreateTableInfo Binder::bindCreateRelTableGroupInfo(const CreateTableInfo*
             }
         }
 
+        bool isSrcIcebugDisk =
+            srcEntry->getType() == CatalogEntryType::NODE_TABLE_ENTRY ?
+                TableOptionConstants::isIceBugDiskFormat(
+                    srcEntry->ptrCast<NodeTableCatalogEntry>()->getStorageFormat()) :
+                false;
+        bool isDstIcebugDisk =
+            dstEntry->getType() == CatalogEntryType::NODE_TABLE_ENTRY ?
+                TableOptionConstants::isIceBugDiskFormat(
+                    dstEntry->ptrCast<NodeTableCatalogEntry>()->getStorageFormat()) :
+                false;
+        bool isRelIcebugDisk = TableOptionConstants::isIceBugDiskFormat(storageFormat);
+
+        // We don't allow mixing icebug-disk tables with non-icebug-disk tables
+        // We only allow icebug-disk rel tables to connect icebug-disk node tables
+        if ((!isRelIcebugDisk && (isSrcIcebugDisk || isDstIcebugDisk)) ||
+            (isRelIcebugDisk && (!isSrcIcebugDisk || !isDstIcebugDisk))) {
+            throw BinderException(
+                "Cannot mix icebug-disk tables with non-icebug-disk tables in CREATE REL TABLE.");
+        }
+
         // Use the actual shadow table IDs, not FOREIGN_TABLE_ID
         // The shadow tables allow the query planner to distinguish between different node tables
         auto srcTableID = srcEntry->getTableID();
@@ -333,8 +367,8 @@ BoundCreateTableInfo Binder::bindCreateRelTableGroupInfo(const CreateTableInfo*
     }
     auto boundExtraInfo = std::make_unique<BoundExtraCreateRelTableGroupInfo>(
         std::move(propertyDefinitions), srcMultiplicity, dstMultiplicity, storageDirection,
-        std::move(nodePairs), std::move(storage), std::move(scanFunction), std::move(scanBindData),
-        std::move(foreignDatabaseName));
+        std::move(nodePairs), std::move(storage), std::move(storageFormat), std::move(scanFunction),
+        std::move(scanBindData), std::move(foreignDatabaseName));
     return BoundCreateTableInfo(CatalogEntryType::REL_GROUP_ENTRY, info->tableName,
         info->onConflict, std::move(boundExtraInfo), clientContext->useInternalCatalogEntry());
 }
@@ -534,8 +568,36 @@ std::unique_ptr<BoundStatement> Binder::bindDrop(const Statement& statement) {
     return std::make_unique<BoundDrop>(drop.getDropInfo());
 }
 
+static void validateNotIceDiskTable(main::ClientContext* clientContext,
+    const std::string& tableName) {
+    auto catalog = Catalog::Get(*clientContext);
+    auto transaction = transaction::Transaction::Get(*clientContext);
+
+    if (!catalog->containsTable(transaction, tableName)) {
+        return;
+    }
+
+    auto tableEntry = catalog->getTableCatalogEntry(transaction, tableName);
+    std::string storageFormat;
+
+    if (tableEntry->getTableType() == common::TableType::NODE) {
+        storageFormat = tableEntry->ptrCast<NodeTableCatalogEntry>()->getStorageFormat();
+    } else if (tableEntry->getTableType() == common::TableType::REL) {
+        storageFormat = tableEntry->ptrCast<RelGroupCatalogEntry>()->getStorageFormat();
+    }
+
+    if (TableOptionConstants::isIceBugDiskFormat(storageFormat)) {
+        throw BinderException(
+            std::format("Cannot alter table {}: icebug-disk tables are immutable.", tableName));
+    }
+}
+
 std::unique_ptr<BoundStatement> Binder::bindAlter(const Statement& statement) {
     auto& alter = statement.constCast<Alter>();
+
+    // we don't support alter operations on icebug-disk tables
+    validateNotIceDiskTable(clientContext, alter.getInfo()->tableName);
+
     switch (alter.getInfo()->type) {
     case AlterType::RENAME: {
         return bindRenameTable(statement);

src/catalog/catalog.cpp

@@ -203,10 +203,11 @@ CatalogEntry* Catalog::createRelGroupEntry(Transaction* transaction,
     for (auto& nodePair : extraInfo->nodePairs) {
         relTableInfos.emplace_back(nodePair, tables->getNextOID());
     }
-    auto relGroupEntry = std::make_unique<RelGroupCatalogEntry>(info.tableName,
-        extraInfo->srcMultiplicity, extraInfo->dstMultiplicity, extraInfo->storageDirection,
-        std::move(relTableInfos), extraInfo->storage, extraInfo->scanFunction,
-        std::move(extraInfo->scanBindData), extraInfo->foreignDatabaseName);
+    auto relGroupEntry =
+        std::make_unique<RelGroupCatalogEntry>(info.tableName, extraInfo->srcMultiplicity,
+            extraInfo->dstMultiplicity, extraInfo->storageDirection, std::move(relTableInfos),
+            extraInfo->storage, extraInfo->storageFormat, extraInfo->scanFunction,
+            std::move(extraInfo->scanBindData), extraInfo->foreignDatabaseName);
     for (auto& definition : extraInfo->propertyDefinitions) {
         relGroupEntry->addProperty(definition);
     }
@@ -561,7 +562,7 @@ CatalogEntry* Catalog::createNodeTableEntry(Transaction* transaction,
     const BoundCreateTableInfo& info) {
     const auto extraInfo = info.extraInfo->constPtrCast<BoundExtraCreateNodeTableInfo>();
     auto entry = std::make_unique<NodeTableCatalogEntry>(info.tableName, extraInfo->primaryKeyName,
-        extraInfo->storage);
+        extraInfo->storage, extraInfo->storageFormat);
     for (auto& definition : extraInfo->propertyDefinitions) {
         entry->addProperty(definition);
     }

src/catalog/catalog_entry/node_table_catalog_entry.cpp

@@ -24,20 +24,26 @@ void NodeTableCatalogEntry::serialize(common::Serializer& serializer) const {
     serializer.write(primaryKeyName);
     serializer.writeDebuggingInfo("storage");
     serializer.write(storage);
+    serializer.writeDebuggingInfo("storageFormat");
+    serializer.write(storageFormat);
 }
 
 std::unique_ptr<NodeTableCatalogEntry> NodeTableCatalogEntry::deserialize(
     common::Deserializer& deserializer) {
     std::string debuggingInfo;
     std::string primaryKeyName;
     std::string storage;
+    std::string storageFormat;
     deserializer.validateDebuggingInfo(debuggingInfo, "primaryKeyName");
     deserializer.deserializeValue(primaryKeyName);
     deserializer.validateDebuggingInfo(debuggingInfo, "storage");
     deserializer.deserializeValue(storage);
+    deserializer.validateDebuggingInfo(debuggingInfo, "storageFormat");
+    deserializer.deserializeValue(storageFormat);
     auto nodeTableEntry = std::make_unique<NodeTableCatalogEntry>();
     nodeTableEntry->primaryKeyName = primaryKeyName;
     nodeTableEntry->storage = storage;
+    nodeTableEntry->storageFormat = storageFormat;
     return nodeTableEntry;
 }
 
@@ -66,6 +72,7 @@ std::unique_ptr<TableCatalogEntry> NodeTableCatalogEntry::copy() const {
     auto other = std::make_unique<NodeTableCatalogEntry>();
     other->primaryKeyName = primaryKeyName;
     other->storage = storage;
+    other->storageFormat = storageFormat;
     other->scanFunction = scanFunction;
     other->createBindDataFunc = createBindDataFunc;
     other->foreignDatabaseName = foreignDatabaseName;
@@ -76,7 +83,7 @@ std::unique_ptr<TableCatalogEntry> NodeTableCatalogEntry::copy() const {
 std::unique_ptr<BoundExtraCreateCatalogEntryInfo> NodeTableCatalogEntry::getBoundExtraCreateInfo(
     transaction::Transaction*) const {
     return std::make_unique<BoundExtraCreateNodeTableInfo>(primaryKeyName,
-        copyVector(getProperties()), storage);
+        copyVector(getProperties()), storage, storageFormat);
 }
 
 } // namespace catalog

src/catalog/catalog_entry/rel_group_catalog_entry.cpp

@@ -105,6 +105,8 @@ void RelGroupCatalogEntry::serialize(Serializer& serializer) const {
     }
     serializer.writeDebuggingInfo("relTableInfos");
     serializer.serializeVector(relTableInfos);
+    serializer.writeDebuggingInfo("storageFormat");
+    serializer.serializeValue(storageFormat);
 }
 
 std::unique_ptr<RelGroupCatalogEntry> RelGroupCatalogEntry::deserialize(
@@ -114,6 +116,7 @@ std::unique_ptr<RelGroupCatalogEntry> RelGroupCatalogEntry::deserialize(
     auto dstMultiplicity = RelMultiplicity::MANY;
     auto storageDirection = ExtendDirection::BOTH;
     std::string storage;
+    std::string storageFormat;
     std::vector<RelTableCatalogInfo> relTableInfos;
     deserializer.validateDebuggingInfo(debuggingInfo, "srcMultiplicity");
     deserializer.deserializeValue(srcMultiplicity);
@@ -132,11 +135,14 @@ std::unique_ptr<RelGroupCatalogEntry> RelGroupCatalogEntry::deserialize(
     }
     deserializer.validateDebuggingInfo(debuggingInfo, "relTableInfos");
     deserializer.deserializeVector(relTableInfos);
+    deserializer.validateDebuggingInfo(debuggingInfo, "storageFormat");
+    deserializer.deserializeValue(storageFormat);
     auto relGroupEntry = std::make_unique<RelGroupCatalogEntry>();
     relGroupEntry->srcMultiplicity = srcMultiplicity;
     relGroupEntry->dstMultiplicity = dstMultiplicity;
     relGroupEntry->storageDirection = storageDirection;
     relGroupEntry->storage = storage;
+    relGroupEntry->storageFormat = storageFormat;
     relGroupEntry->scanFunction = scanFunction;
     relGroupEntry->relTableInfos = relTableInfos;
     return relGroupEntry;
@@ -198,6 +204,7 @@ std::unique_ptr<TableCatalogEntry> RelGroupCatalogEntry::copy() const {
     other->dstMultiplicity = dstMultiplicity;
     other->storageDirection = storageDirection;
     other->storage = storage;
+    other->storageFormat = storageFormat;
     other->scanFunction = scanFunction;
     other->scanBindData = std::nullopt; // TODO: implement copy for bindData if needed
     other->foreignDatabaseName = foreignDatabaseName;
@@ -214,7 +221,7 @@ RelGroupCatalogEntry::getBoundExtraCreateInfo(transaction::Transaction*) const {
     }
     return std::make_unique<binder::BoundExtraCreateRelTableGroupInfo>(
         copyVector(propertyCollection.getDefinitions()), srcMultiplicity, dstMultiplicity,
-        storageDirection, std::move(nodePairs));
+        storageDirection, std::move(nodePairs), storage, storageFormat);
 }
 
 } // namespace catalog

src/graph/on_disk_graph.cpp

@@ -18,8 +18,8 @@
 #include "storage/local_storage/local_storage.h"
 #include "storage/storage_manager.h"
 #include "storage/storage_utils.h"
+#include "storage/table/ice_disk_rel_table.h"
 #include "storage/table/node_table.h"
-#include "storage/table/parquet_rel_table.h"
 #include "storage/table/rel_table.h"
 
 using namespace lbug::catalog;
@@ -129,8 +129,8 @@ OnDiskGraphNbrScanState::OnDiskGraphNbrScanState(ClientContext* context,
         }
 
         std::unique_ptr<RelTableScanState> scanState;
-        if (dynamic_cast<ParquetRelTable*>(table) != nullptr) {
-            scanState = std::make_unique<ParquetRelTableScanState>(*mm, srcNodeIDVector.get(),
+        if (dynamic_cast<IceDiskRelTable*>(table) != nullptr) {
+            scanState = std::make_unique<IceDiskRelTableScanState>(*mm, srcNodeIDVector.get(),
                 outVectors, state);
         } else {
             scanState = std::make_unique<RelTableScanState>(*MemoryManager::Get(*context),