DenseVault

DenseVault is a zero-dependency, single-file immutable storage server designed for robust, high-performance archiving and live data access. It transforms any directory into a WORM (Write-Once-Read-Many) vault, accessible via standard WebDAV.

Built entirely on the Python Standard Library, it features Strided Content-Defined Chunking, Entropy-Adaptive Compression, Delta Encoding for versioned files, and O(log n) Random Access — making it suitable for both cold archival storage and running live AI workloads (like LLMs) directly from the vault.

---

Quickstart

DenseVault depends only on the Python Standard Library. No package installation is required.

1. Save the script as densevault.py.
2. Run the interactive wizard:

python3 densevault.py

3. Follow the prompts to create or open a vault and (optionally) set a password.
4. Connect to the WebDAV share from your OS:
   • Windows: Map Network Drive → http://localhost:8000
   • macOS: Finder → Connect to Server → http://localhost:8000
   • Linux: sudo mount -t davfs http://localhost:8000 /mnt/vault
5. Drag and drop files. They are chunked, deduplicated, intelligently compressed, and sealed immutably.

---

Key Features

1. Compute-Ready Archive

Unlike traditional backups that require full extraction before use, DenseVault supports O(log n) Random Access via HTTP Range requests.

• AI Inference: Run large AI models (GGUF, PyTorch) directly from the vault over WebDAV. No extraction needed.
• Mechanism: Pre-computed chunk_offsets in the asset manifest combined with bisect binary search allow the server to fetch only the specific chunk(s) covering a requested byte range.
• Verified: Successfully tested serving gemma-3-270m-it-Q4_K_M.gguf to llamafile directly from a vault over WebDAV.

Note: Delta-encoded assets require full reconstruction before slicing, so they do not support efficient Range reads. Use delta encoding only for versioned archival files that are always read in full (e.g. model checkpoints). Use normal storage for files served to live inference engines.

2. Delta Encoding for Versioned Files

For files that evolve incrementally (model checkpoints, dataset snapshots, binary releases), DenseVault can store only the difference from a previous version rather than the full file.

• How it works: A block-fingerprint algorithm scans the new file and emits COPY (reference existing base data) and INSERT (new literal bytes) operations. The operations stream is zlib-compressed and stored as a single chunk.
• Automatic fallback: If the delta is not smaller than DELTA_THRESHOLD (75%) of the estimated full chunked size, normal chunked storage is used instead — automatically, with no user intervention.
• Safe chaining: Delta chains are resolved recursively up to a depth of 8 (MAX_DELTA_DEPTH) to prevent runaway recursion from circular references.
• Wire format: Each delta blob is self-describing: it carries BLAKE2b-256 digests of both the base and target, CRC32 integrity protection, and the compressed operations stream.

To store a file as a delta via WebDAV:

curl -T model_v2.gguf \
     -H 'X-Base-File: /models/model_v1.gguf' \
     http://127.0.0.1:8000/models/model_v2.gguf

3. Strided Content-Defined Chunking

DenseVault uses a Gear-Hash rolling window CDC algorithm with a stride of 32 — scanning every 32nd byte rather than every byte to find content boundaries.

• ~30× faster chunking in pure Python vs. standard single-byte CDC, with comparable deduplication accuracy.
• Two-speed mask: A 14-bit mask is used for chunks below the 256 KB target size; a 19-bit mask above it. This keeps the average chunk size near the target while bounding the minimum at 64 KB and maximum at 1 MB.
• The gear table is seeded with splitmix64 for a uniform distribution of hash values across all 256 byte values.

4. Entropy-Adaptive Compression

The AdaptiveCompressor estimates Shannon entropy from a stratified sample (beginning, middle, and end of each chunk) to decide whether compression is worth attempting — avoiding wasted CPU cycles on incompressible data.

Entropy range	Action
≥ 7.5 (encrypted / media / already compressed)	Store raw — zero compression CPU.
< 2.0 (plain text / log files / zero-filled pages)	Compress with zlib level 1.
2.0 – 7.5 (mixed content)	Try zlib level 1; keep compressed only if ratio ≥ 1.05–1.10.

5. Enterprise-Grade Storage Engine

• Single-file container: All data, metadata, chunk indexes, and delta blobs live in one portable .vault (SQLite) file.
• ACID transactions: Crash-safe WAL (Write-Ahead Logging) mode with BLAKE2b-256 per-chunk integrity hashes.
• Connection pooling: 8 reusable SQLite connections handle concurrent WebDAV reads without serialising on a single lock.
• LRU caching: Path resolution and manifest lookups are cached with a 30-second TTL to reduce database round-trips on repeated reads.
• Integrity mapping: --check mode maps each corrupt chunk back to the specific filenames that reference it.

---

Architecture

Ingest Pipeline

Chunking, compression, and database writes run in a parallel three-stage pipeline:

main thread         compression workers (×4)       writer thread
───────────         ────────────────────────       ─────────────
read stream    →    hash + compress chunks    →    batch INSERT OR IGNORE
feed work_queue     push to write_queue            flush every 100 chunks

INSERT OR IGNORE on the chunks table provides automatic, transparent deduplication: identical chunks across different files share a single stored copy with zero extra logic.

Manifest V3

Every asset stores a manifest containing:

• chunks — ordered list of BLAKE2b-256 chunk hashes
• chunk_offsets — cumulative byte offsets for O(log n) range lookup
• chunk_sizes — individual chunk sizes
• root_hash — BLAKE2b-256 over the concatenated chunk hash list
• compression — per-asset compression statistics
• is_delta, base_asset_id, base_hash — populated for delta assets

Range request flow:

1. Client sends Range: bytes=5000-6000.
2. Server binary-searches chunk_offsets (bisect) to find the first relevant chunk.
3. Server fetches and decompresses only that chunk (and any immediately following ones that overlap the range).
4. Server slices and streams the exact requested bytes.

Storage Behaviour Reference

File type	Action	CPU cost	Storage cost
Media / encrypted	Entropy sampled → stored raw	Minimal	~0% reduction
Text / logs	Compressed zlib L1	Low	50–80% reduction
Exact duplicate	Manifest links to existing chunks	Near zero	0 bytes
Similar / versioned file	Shared chunks linked; unique chunks stored	Low	Proportional to diff
Versioned file (delta)	Stores only COPY+INSERT ops vs. base	Moderate (one-time)	Up to 90%+ reduction
AI model (GGUF)	Chunked, deduplicated, range-ready	Minimal	Variable

---

Usage & CLI Reference

Interactive Wizard

Running without arguments launches the setup wizard, which lists existing vaults, prompts for a port, and handles password setup:

python3 densevault.py

Direct CLI

python3 densevault.py [VAULT_FILE] [options]

Server options

Argument	Description	Default
VAULT_FILE	Path to the .vault file (created if absent).	Interactive
-p, --password	Vault password. Prefer the interactive prompt.	—
--host	Bind address.	127.0.0.1
--port	WebDAV server port.	8000
-v, --verbose	Enable debug-level logging.	Off

Maintenance flags (run and exit)

Argument	Description
--check	Full integrity check: verifies all chunk hashes and delta chains, maps errors to filenames.
--gc	Garbage collection: deletes orphaned chunks, then runs VACUUM to reclaim disk space.
--backup-meta FILE	Export all asset manifests to a TSV file (no chunk data).
--optimize-deltas	Scan the vault for delta optimisation opportunities and print suggestions.

Standalone delta tools (no vault needed)

Argument	Description
--delta-create BASE TARGET OUTPUT	Create a standalone delta file from two arbitrary binary files.
--delta-apply BASE DELTA OUTPUT	Reconstruct a target file by applying a delta to a base file.

Usage Examples

# Start a vault (non-interactive)
python3 densevault.py archive.vault --port 8080

# Integrity check
python3 densevault.py archive.vault --check

# Garbage collection
python3 densevault.py archive.vault --gc

# Find delta optimisation opportunities
python3 densevault.py archive.vault --optimize-deltas

# Export metadata catalogue (no chunk data)
python3 densevault.py archive.vault --backup-meta catalogue.tsv

# Standalone delta: create and apply
python3 densevault.py --delta-create model_v1.gguf model_v2.gguf patch.bin
python3 densevault.py --delta-apply  model_v1.gguf patch.bin model_v2_out.gguf

---

WebDAV Client Configuration

DenseVault speaks standard WebDAV and is compatible with any compliant client.

Windows

1. Open This PC → Map Network Drive.
2. Enter http://<server_ip>:8000.
3. Check Connect using different credentials if a password is set.

macOS

Finder → Go → Connect to Server → http://<server_ip>:8000

Linux

sudo mount -t davfs http://localhost:8000 /mnt/vault

Behaviour Reference

Action	Result
Upload	HTTP 201 Created. File is chunked, deduplicated, and sealed.
Upload (delta)	HTTP 201 Created. Delta stored if beneficial; normal chunking used as fallback.
Download	HTTP 200. Transparent decompression and delta reconstruction.
Range download	HTTP 206 Partial Content. O(log n) chunk lookup for non-delta assets.
Overwrite	HTTP 409 Conflict. Original preserved (WORM policy).
Delete	HTTP 403 Forbidden. WORM policy.

---

Configuration File

DenseVault optionally reads vault.json from the working directory at startup:

{
  "vault":    "archive.vault",
  "host":     "127.0.0.1",
  "port":     8000,
  "password": "changeme"
}

Security: Storing a password in vault.json is convenient but risky. Restrict file permissions or omit the password field and use the interactive prompt instead.

---

License

MIT — see the license header at the top of densevault.py.