Feature/172 ttl support

.github/workflows/benchmark.yml

@@ -0,0 +1,89 @@
+name: Performance Benchmarks
+
+# Run benchmarks on a schedule and for important changes
+on:
+  # Run weekly on Sunday at 2 AM UTC
+  schedule:
+    - cron: "0 2 * * 0"
+
+  # Allow manual triggering for on-demand performance testing
+  workflow_dispatch:
+
+  # Run on pushes to main branch that affect performance-critical code
+  push:
+    branches:
+      - main
+    paths:
+      - "d-engine-server/src/storage/**"
+      - "d-engine-server/benches/**"
+      - "d-engine-core/src/**"
+
+env:
+  CARGO_TERM_COLOR: always
+  RUST_BACKTRACE: full
+
+jobs:
+  benchmark:
+    name: Run Performance Benchmarks
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # Full history for performance comparison
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y protobuf-compiler
+
+      - name: Cache Rust dependencies
+        uses: Swatinem/rust-cache@v2
+        with:
+          # Cache key includes benchmark to avoid interference with test cache
+          key: benchmark-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}
+
+      - name: Run benchmarks
+        run: |
+          echo "========================================="
+          echo "Running d-engine Performance Benchmarks"
+          echo "========================================="
+          echo ""
+          echo "Performance Targets:"
+          echo "  • Without TTL: < 10ns overhead"
+          echo "  • With TTL passive check: < 50ns overhead"
+          echo "  • Piggyback cleanup: < 1ms"
+          echo ""
+
+          # Run benchmarks and save output
+          cargo bench --package d-engine-server 2>&1 | tee benchmark_output.txt
+
+          echo ""
+          echo "✅ Benchmarks completed successfully"
+
+      - name: Upload detailed results as artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-results
+          path: |
+            target/criterion/
+            benchmark_output.txt
+          retention-days: 30
+
+      - name: Performance summary
+        run: |
+          echo "========================================="
+          echo "Performance Benchmark Summary"
+          echo "========================================="
+          echo ""
+          echo "📊 Benchmark results uploaded as artifacts"
+          echo ""
+          echo "To view detailed results:"
+          echo "  1. Download the artifact"
+          echo "  2. Open target/criterion/report/index.html in your browser"
+          echo ""
+          echo "✅ Performance monitoring complete"

.github/workflows/ci.yml

@@ -66,6 +66,12 @@ jobs:
         run: |
           cargo llvm-cov nextest --all-features --workspace --lcov --output-path lcov.info --ignore-filename-regex "src/generated/.*|src/errors.rs"
 
+      - name: Check benchmarks compile
+        run: |
+          echo "Checking that benchmark code compiles correctly..."
+          cargo bench --no-run --package d-engine-server
+          echo "✅ Benchmarks compile successfully"
+
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v3
         with:

.github/workflows/commit-message-check.yml

@@ -16,16 +16,16 @@ jobs:
           BASE_SHA=${{ github.event.pull_request.base.sha }}
           HEAD_SHA=${{ github.event.pull_request.head.sha }}
 
-          ALLOWED_TYPES="feat|fix|refs|doc|perf|refactor|chore"
+          ALLOWED_TYPES="feat|fix|refs|doc|perf|refactor|style|test|chore|ci|revert"
           SCOPE_PATTERN="([a-z][a-z0-9-]*)"
           TICKET_PATTERN="#[0-9]+:"
           FULL_PATTERN="^($ALLOWED_TYPES)(\($SCOPE_PATTERN\))? $TICKET_PATTERN.+$"
 
           # Validate the format of all new submissions
           for commit in $(git rev-list $BASE_SHA..$HEAD_SHA); do
             msg=$(git show -s --format=%s $commit)
-            
-            # skip merge 
+
+            # skip merge
             if [[ "$msg" =~ ^Merge ]]; then
               continue
             fi

.github/workflows/dependency-audit.yml

@@ -12,6 +12,9 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+
       - name: Install protoc
         run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
 
@@ -26,7 +29,7 @@ jobs:
 
       - name: Audit dependencies
         run: |
-          cargo install cargo-deny
+          cargo install cargo-deny --locked
           cargo deny check all
 
       - name: Update dependencies

.gitignore

@@ -46,3 +46,4 @@ docker/monitoring/prometheus/data/
 lcov.info
 
 CONTEXT
+*.bak*

CHANGELOG.md

@@ -4,6 +4,69 @@ All notable changes to this project will be documented in this file.
 
 ---
 
+## [v0.2.0] - 2025-11-12 [✅ Released]
+
+### 🚀 Features
+
+- **TTL/Lease Support**: Implemented time-to-live (TTL) functionality with configurable cleanup strategies (piggyback, lazy, scheduled) for automatic key expiration
+- **Unified NodeBuilder API**: Simplified node startup with new `start_server()` method that combines `build()`, `start_rpc_server()`, and `ready()` into a single async call
+- **Improved State Machine Initialization**: Enhanced state machine lifecycle with `try_inject_lease()` and `post_start_init()` hooks for transparent lease configuration
+- **etcd-Compatible TTL Semantics**: TTL now uses absolute expiration time (compatible with etcd lease semantics) instead of relative TTL, ensuring correct behavior across restarts
+
+### 🔄 Breaking Changes
+
+- **WAL Format Change (File-based State Machine)**: ⚠️ **CRITICAL BREAKING CHANGE**
+  - WAL entries now store absolute expiration time (`expire_at_secs: u64`) instead of relative TTL (`ttl_secs: u32`)
+  - This enables crash-safe TTL semantics and etcd-compatible lease behavior
+  - **Migration Required**: See [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md) for upgrade instructions
+  - Existing WAL files from pre-v0.2.0 are **not compatible** and must be migrated
+
+- **NodeBuilder API**: `build().start_rpc_server().await.ready()` is now replaced with `.start_server().await`
+  - See [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md) for detailed migration instructions
+  - Old API is no longer supported
+
+### 📝 Documentation
+
+- Added comprehensive MIGRATION_GUIDE.md for API changes
+- Updated all README files with new `start_server()` API
+- Updated server guide documentation for custom implementations
+- Updated quick-start examples in overview documentation
+
+### 🐛 Fixes
+
+- Fixed clippy warning: empty line after doc comments in RocksDB state machine
+- Fixed duplicate trace logging in BufferedRaftLog initialization
+- Fixed log level filtering: RUST_LOG now correctly limits to DEBUG level (no more TRACE spam in tests)
+- Fixed Zed editor clippy warnings in benchmark code
+- Fixed unused imports in test utilities
+- Fixed crash-safety bug: snapshot restore now persists TTL metadata to RocksDB CF
+- Fixed WAL replay: expired entries are now correctly skipped during recovery
+
+### ⚡ Performance
+
+- **Benchmark Optimization**: Reduced TTL benchmark execution time by ~10x
+  - `worst_case_all_expired`: 213s → 20s (10.6x faster)
+  - `mixed_ttl_workload`: 200s → 20s (10x faster)
+  - `piggyback_high_frequency`: 200s → 20s (10x faster)
+  - Used `iter_batched` to separate setup from measurement
+  - Reduced sample size to 10 for tests with sleep operations
+
+### ✨ Quality
+
+- All benchmarks pass clippy without warnings
+- TTL benchmarks validate cleanup performance targets
+- State machine benchmarks validate scaling characteristics
+- Added tests for crash-safe WAL replay behavior
+- Added tests for TTL persistence across snapshot restore
+
+### 🔧 Internal Improvements
+
+- Refactored RocksDB options configuration into `configure_db_options()` helper (DRY)
+- Removed high-frequency trace logs from hot paths to reduce noise
+- Improved test output clarity by filtering log levels correctly
+
+---
+
 ## [v0.1.4] - 2025-10-12 [✅ Released]
 
 ### Features