From 3e382230f8f513ceffde13e374cc2de60fd20e79 Mon Sep 17 00:00:00 2001 From: Ally Heev Date: Wed, 25 Mar 2026 18:33:07 +0530 Subject: [PATCH 1/2] feat: add v1 spec --- README.md | 4 +- format/icebug-disk/architecture_overview.svg | 52 +++++++++++ format/icebug-disk/spec.md | 92 ++++++++++++++++++++ format/icebug-memory/spec.md | 91 +++++++++++++++++++ 4 files changed, 237 insertions(+), 2 deletions(-) create mode 100644 format/icebug-disk/architecture_overview.svg create mode 100644 format/icebug-disk/spec.md create mode 100644 format/icebug-memory/spec.md diff --git a/README.md b/README.md index 2f65499..12ad340 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ Icebug is a standardized graph format designed for efficient graph data interchange. It comes in two formats: -- **icebug-disk**: Parquet-based format for object storage -- **icebug-memory**: Apache Arrow-based format for in-memory processing +- **icebug-disk**: CSR adjacency list/join indices for disk storage +- **icebug-memory**: CSR adjacency list/join indices for in-memory processing This project provides tools to convert graph data from simple DuckDB databases or Parquet files containing `nodes_*` and `edges_*` tables, along with a `schema.cypher` file, into standardized graph formats for efficient processing. diff --git a/format/icebug-disk/architecture_overview.svg b/format/icebug-disk/architecture_overview.svg new file mode 100644 index 0000000..193c5da --- /dev/null +++ b/format/icebug-disk/architecture_overview.svg @@ -0,0 +1,52 @@ + + + + + + + + + + + + + Metadata File + + + Init Script + + + + + + Internal ID Mappings + + + + + Node Tables + + + + + Edge Ranges + + + + + Relationship Tables + + + + + + + diff --git a/format/icebug-disk/spec.md b/format/icebug-disk/spec.md new file mode 100644 index 0000000..50b7f0e --- /dev/null +++ b/format/icebug-disk/spec.md @@ -0,0 +1,92 @@ +# Icebug-Disk graph specification + +The Icebug-Disk graph format is designed to store graph data on disk efficiently, enabling high read performance for large graphs. + +## Versioning + +### Version 1 +Initial version of the format. Uses compressed sparse row-based (CSR) adjacency list/join indices for efficient graph traversals. + +## Architecture Diagram + +![Icebug-Disk Architecture Overview](architecture_overview.svg) + + +## Components + +### Metadata +A metadata file containing version information and the initialization script path. + +Example metadata file: +```json +{ + "version": "1", + "init_script": "path/to/schema.cypher" +} +``` + +### Initialization script +A script (typically containing node and relationship table creation rules) that can be executed by the query engine to create the graph in the database. This script is responsible for creating the graph structure, including node tables and relationship tables, and performing necessary validations. + +Example initialization script: +```cypher +CREATE NODE TABLE node_table_1 ( + id STRING PRIMARY KEY, + name STRING, + age INT +) WITH ( + format = 'parquet', + file_path = 'path/to/node_table_1.parquet' +); + +CREATE REL TABLE rel_table_1 ( + from_node_id STRING, + to_node_id STRING, + weight FLOAT +) WITH ( + format = 'parquet', + file_path = 'path/to/rel_table_1.parquet' +); +``` + +### Node tables +Node tables store the actual node data in columnar formats like Parquet, Lance, Vortex, etc. Ideally, one file per table should be sufficient for storing the node data. However, for very large graphs, multiple files can be used to store partitions of the node data. + +Example node table: + +| id | name | age | +|------|-------|-----| +| 1 | Alice | 30 | +| 2 | Bob | 25 | + +### Relationship tables +Relationship tables store the relationship data (i.e., `from_node_id`, `to_node_id`, and edge properties) in columnar format. Similar to node tables, there can be multiple files to store partitions of the relationship data for very large graphs. + +Example relationship table: + +| from | to | weight | +|------|-----|--------| +| 1 | 2 | 0.5 | +| 2 | 1 | 0.8 | + +### Internal ID mapping files +Each internal ID mapping file(per node table) stores the mapping between the original node IDs (`primary_key`) and the internal node offsets (`row_idx`) used in the graph. This is necessary for efficient storage and retrieval of nodes and relationships, as well as for performing graph traversals. + +Example internal ID mapping file: + +| node_id | node_offset | +|--------------------------------------------|-------------| +| 0020a216-0626-11ea-9f44-8c16456798f1 | 0 | +| 0020a216-0626-11ea-9f44-8c16456798f2 | 1 | + +### Edge range files +Each edge range file(per rel table) stores the edge ranges (in the relationship tables). These files are used to efficiently retrieve the edges connected to a specific node during graph traversals. + +Example edge range file: + +| edge_id | +|---------------------------------------------------------------| +| 0 -> start edge offset for node 1 | +| 1 -> end edge offset for node 1, start edge offset for node 2 | +... +| 2 -> end edge offset for node n | diff --git a/format/icebug-memory/spec.md b/format/icebug-memory/spec.md new file mode 100644 index 0000000..f6484d4 --- /dev/null +++ b/format/icebug-memory/spec.md @@ -0,0 +1,91 @@ +# Icebug-Memory graph specification +The Icebug-Memory graph format is designed to store graph data in-memory efficiently, enabling high read performance for large graphs. + +## Versioning + +### Version 1 +Initial version of the format. Uses compressed sparse row-based (CSR) adjacency list/join indices for efficient graph traversals. + +## Architecture Diagram + +![Icebug-Memory Architecture Overview](../icebug-disk/architecture_overview.svg) + + +## Components + +### Metadata +A metadata file containing version information and the initialization script path. + +Example metadata file: +```json +{ + "version": "1", + "init_script": "path/to/schema.cypher" +} +``` + +### Initialization script +A script (typically containing node and relationship table creation rules) that can be executed by the query engine to create the graph in the database. This script is responsible for creating the graph structure, including node tables and relationship tables, and performing necessary validations. + +Example initialization script: +```cypher +CREATE NODE TABLE node_table_1 ( + id STRING PRIMARY KEY, + name STRING, + age INT +) WITH ( + format = 'arrow', + arrow_id = '0' +); + +CREATE REL TABLE rel_table_1 ( + from_node_id STRING, + to_node_id STRING, + weight FLOAT +) WITH ( + format = 'arrow', + arrow_id = '1' +); +``` + +### Node tables +Node tables store the actual node data in columnar formats like arrow. + +Example node table: + +| id | name | age | +|------|-------|-----| +| 1 | Alice | 30 | +| 2 | Bob | 25 | + +### Relationship tables +Relationship tables store the relationship data (i.e., `from_node_id`, `to_node_id`, and edge properties) in columnar format. + +Example relationship table: + +| from | to | weight | +|------|-----|--------| +| 1 | 2 | 0.5 | +| 2 | 1 | 0.8 | + +### Internal ID mappings +Each Internal ID mapping(per node table) stores the mapping between the original node IDs (`primary_key`) and the internal node offsets (`row_idx`) used in the graph. This is necessary for efficient storage and retrieval of nodes and relationships, as well as for performing graph traversals. + +Example internal ID mapping: + +| node_id | node_offset | +|--------------------------------------------|-------------| +| 0020a216-0626-11ea-9f44-8c16456798f1 | 0 | +| 0020a216-0626-11ea-9f44-8c16456798f2 | 1 | + +### Edge range structures +Each edge range structure(per rel table) stores the edge ranges (in the relationship tables) for each node. These structures are used to efficiently retrieve the edges connected to a specific node during graph traversals. + +Example edge range: + +| edge_id | +|---------------------------------------------------------------| +| 0 -> start edge offset for node 1 | +| 1 -> end edge offset for node 1, start edge offset for node 2 | +... +| 2 -> end edge offset for node n | From 0864f94fcfb3d9626e91b49660805e3fdca20358 Mon Sep 17 00:00:00 2001 From: Ally Heev Date: Fri, 27 Mar 2026 10:08:30 +0530 Subject: [PATCH 2/2] remove lbug impl specific details --- format/icebug-disk/architecture_overview.svg | 56 +++++------- format/icebug-disk/spec.md | 88 ++++++------------- format/icebug-memory/spec.md | 91 +------------------- 3 files changed, 47 insertions(+), 188 deletions(-) diff --git a/format/icebug-disk/architecture_overview.svg b/format/icebug-disk/architecture_overview.svg index 193c5da..169232c 100644 --- a/format/icebug-disk/architecture_overview.svg +++ b/format/icebug-disk/architecture_overview.svg @@ -1,8 +1,5 @@ - - - - - - Metadata File - - - Init Script - - - - - - Internal ID Mappings - - - - - Node Tables - - - - - Edge Ranges - - - - - Relationship Tables - - - - - + + + + + Initialization Files + + + + + Node Tables + + + + + Indices Files + + + + + Indptr Files diff --git a/format/icebug-disk/spec.md b/format/icebug-disk/spec.md index 50b7f0e..6b2037d 100644 --- a/format/icebug-disk/spec.md +++ b/format/icebug-disk/spec.md @@ -5,88 +5,52 @@ The Icebug-Disk graph format is designed to store graph data on disk efficiently ## Versioning ### Version 1 + Initial version of the format. Uses compressed sparse row-based (CSR) adjacency list/join indices for efficient graph traversals. ## Architecture Diagram ![Icebug-Disk Architecture Overview](architecture_overview.svg) - ## Components -### Metadata -A metadata file containing version information and the initialization script path. - -Example metadata file: -```json -{ - "version": "1", - "init_script": "path/to/schema.cypher" -} -``` - -### Initialization script -A script (typically containing node and relationship table creation rules) that can be executed by the query engine to create the graph in the database. This script is responsible for creating the graph structure, including node tables and relationship tables, and performing necessary validations. - -Example initialization script: -```cypher -CREATE NODE TABLE node_table_1 ( - id STRING PRIMARY KEY, - name STRING, - age INT -) WITH ( - format = 'parquet', - file_path = 'path/to/node_table_1.parquet' -); - -CREATE REL TABLE rel_table_1 ( - from_node_id STRING, - to_node_id STRING, - weight FLOAT -) WITH ( - format = 'parquet', - file_path = 'path/to/rel_table_1.parquet' -); -``` +### Initialization files + +Initialization files could be format version, schema definitions, table creation statements etc. These files are used to initialize the graph ### Node tables + Node tables store the actual node data in columnar formats like Parquet, Lance, Vortex, etc. Ideally, one file per table should be sufficient for storing the node data. However, for very large graphs, multiple files can be used to store partitions of the node data. Example node table: -| id | name | age | -|------|-------|-----| -| 1 | Alice | 30 | -| 2 | Bob | 25 | +| id | name | age | +| -- | ----- | --- | +| 1 | Alice | 30 | +| 2 | Bob | 25 | + +### Indices files -### Relationship tables -Relationship tables store the relationship data (i.e., `from_node_id`, `to_node_id`, and edge properties) in columnar format. Similar to node tables, there can be multiple files to store partitions of the relationship data for very large graphs. +Each indices file[^1] is a table that stores relationship data (i.e., `from_node_id`, `to_node_id`, and edge properties) in columnar format. Similar to node tables, there can be multiple files to store partitions of the relationship data for very large graphs. -Example relationship table: +Example indices file: -| from | to | weight | -|------|-----|--------| -| 1 | 2 | 0.5 | -| 2 | 1 | 0.8 | +| from | to | weight | +| ---- | -- | ------ | +| 1 | 2 | 0.5 | +| 2 | 1 | 0.8 | -### Internal ID mapping files -Each internal ID mapping file(per node table) stores the mapping between the original node IDs (`primary_key`) and the internal node offsets (`row_idx`) used in the graph. This is necessary for efficient storage and retrieval of nodes and relationships, as well as for performing graph traversals. +### Indptr files -Example internal ID mapping file: +Each indptr file[^1] (per rel table) stores the offset range (in the indices file) for each node. These files are used to efficiently retrieve the edges connected to a specific node during graph traversals. -| node_id | node_offset | -|--------------------------------------------|-------------| -| 0020a216-0626-11ea-9f44-8c16456798f1 | 0 | -| 0020a216-0626-11ea-9f44-8c16456798f2 | 1 | +Example indptr file: -### Edge range files -Each edge range file(per rel table) stores the edge ranges (in the relationship tables). These files are used to efficiently retrieve the edges connected to a specific node during graph traversals. +| edge_id | +| ------- | +| 0 | +| 1 | +| 2 | -Example edge range file: +[^1]: https://www.usenix.org/system/files/login/articles/login_winter20_16_kelly.pdf -| edge_id | -|---------------------------------------------------------------| -| 0 -> start edge offset for node 1 | -| 1 -> end edge offset for node 1, start edge offset for node 2 | -... -| 2 -> end edge offset for node n | diff --git a/format/icebug-memory/spec.md b/format/icebug-memory/spec.md index f6484d4..0d8c400 100644 --- a/format/icebug-memory/spec.md +++ b/format/icebug-memory/spec.md @@ -1,91 +1,2 @@ # Icebug-Memory graph specification -The Icebug-Memory graph format is designed to store graph data in-memory efficiently, enabling high read performance for large graphs. - -## Versioning - -### Version 1 -Initial version of the format. Uses compressed sparse row-based (CSR) adjacency list/join indices for efficient graph traversals. - -## Architecture Diagram - -![Icebug-Memory Architecture Overview](../icebug-disk/architecture_overview.svg) - - -## Components - -### Metadata -A metadata file containing version information and the initialization script path. - -Example metadata file: -```json -{ - "version": "1", - "init_script": "path/to/schema.cypher" -} -``` - -### Initialization script -A script (typically containing node and relationship table creation rules) that can be executed by the query engine to create the graph in the database. This script is responsible for creating the graph structure, including node tables and relationship tables, and performing necessary validations. - -Example initialization script: -```cypher -CREATE NODE TABLE node_table_1 ( - id STRING PRIMARY KEY, - name STRING, - age INT -) WITH ( - format = 'arrow', - arrow_id = '0' -); - -CREATE REL TABLE rel_table_1 ( - from_node_id STRING, - to_node_id STRING, - weight FLOAT -) WITH ( - format = 'arrow', - arrow_id = '1' -); -``` - -### Node tables -Node tables store the actual node data in columnar formats like arrow. - -Example node table: - -| id | name | age | -|------|-------|-----| -| 1 | Alice | 30 | -| 2 | Bob | 25 | - -### Relationship tables -Relationship tables store the relationship data (i.e., `from_node_id`, `to_node_id`, and edge properties) in columnar format. - -Example relationship table: - -| from | to | weight | -|------|-----|--------| -| 1 | 2 | 0.5 | -| 2 | 1 | 0.8 | - -### Internal ID mappings -Each Internal ID mapping(per node table) stores the mapping between the original node IDs (`primary_key`) and the internal node offsets (`row_idx`) used in the graph. This is necessary for efficient storage and retrieval of nodes and relationships, as well as for performing graph traversals. - -Example internal ID mapping: - -| node_id | node_offset | -|--------------------------------------------|-------------| -| 0020a216-0626-11ea-9f44-8c16456798f1 | 0 | -| 0020a216-0626-11ea-9f44-8c16456798f2 | 1 | - -### Edge range structures -Each edge range structure(per rel table) stores the edge ranges (in the relationship tables) for each node. These structures are used to efficiently retrieve the edges connected to a specific node during graph traversals. - -Example edge range: - -| edge_id | -|---------------------------------------------------------------| -| 0 -> start edge offset for node 1 | -| 1 -> end edge offset for node 1, start edge offset for node 2 | -... -| 2 -> end edge offset for node n | +Similar to Icebug-Disk, but entirely in-memory