Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
d8bd658
feat #43: When a Raft leader is elected, it should first send an empt…
JoshuaChi Apr 19, 2025
8fb91b3
Feature/59 benchmark (#72)
JoshuaChi Apr 20, 2025
2f97f6c
chore(ci) #73: prepare release for v0.1.2
JoshuaChi Apr 20, 2025
1756873
Merge branch 'main' into develop
JoshuaChi Apr 22, 2025
e53ba82
feat #79: Snapshot feature, Learner join cluster feature and develope…
JoshuaChi Sep 3, 2025
729e100
feat #79: optimize v0.1.3 CHANGELOG
JoshuaChi Sep 3, 2025
cebfdbb
Merge branch 'main' into develop
JoshuaChi Sep 3, 2025
0a057c8
fix #133: Example project: File/RocksDB storage engines can fail to s…
JoshuaChi Sep 7, 2025
bd44015
fix #133: fix fmt
JoshuaChi Sep 7, 2025
265c250
fix #133: fix fmt
JoshuaChi Sep 7, 2025
2e88aae
fix #133: fix fmt
JoshuaChi Sep 7, 2025
598f011
fix #133: update REAMDE
JoshuaChi Sep 7, 2025
68de4c7
feat #135: Enable tokio via three-nodes-cluster example
JoshuaChi Sep 17, 2025
3fe4cdc
feat #135: Enable tokio_metrics via three-nodes-cluster example
JoshuaChi Sep 17, 2025
f7911c1
feat #135: Enable tokio_metrics via three-nodes-cluster example
JoshuaChi Sep 18, 2025
4e30171
feat #138: v0.1.4 - target 100K throughput
JoshuaChi Oct 13, 2025
b03a606
fix #145: fix Bytes compatible bugs in examples folder
JoshuaChi Oct 14, 2025
f3f3430
fix #150: type issue
JoshuaChi Oct 14, 2025
32cc365
fix #151: fix mmap bug
JoshuaChi Oct 14, 2025
0713943
fix #152: Update README by introduce new feature from v0.1.4
JoshuaChi Oct 15, 2025
ff1e251
fix #153: fix code review feedback
JoshuaChi Oct 15, 2025
8659663
fix #154: code review feedback
JoshuaChi Oct 15, 2025
901008f
fix #155: fix code review feedback
JoshuaChi Oct 15, 2025
a3c314d
fix #156: fix code review feedback
JoshuaChi Oct 15, 2025
58f97dc
fix #157: fix code review feedback
JoshuaChi Oct 15, 2025
f145325
fix #158, #159: fix code review feedback
JoshuaChi Oct 16, 2025
7e0d96e
fix #164: fix code review bug
JoshuaChi Oct 16, 2025
b3e119e
fix #161: fix code review feedback
JoshuaChi Oct 16, 2025
9ea5116
fix #164: fix doc issues
JoshuaChi Oct 16, 2025
2169613
Merge branch 'main' into develop
JoshuaChi Oct 18, 2025
be93d34
feat #167: workspace structure (#175)
JoshuaChi Nov 5, 2025
d9636d1
feat #172: Feature ttl support (#176)
JoshuaChi Nov 12, 2025
2569af0
fix #172: format code to pass rustfmt checks
JoshuaChi Nov 12, 2025
e1143d0
fix #172: Replace tokio-tar with astral-tokio-tar to resolve CVE-2025…
JoshuaChi Nov 13, 2025
803400a
fix #172: Resolve Tokio runtime nesting issue in TTL benchmark
JoshuaChi Nov 13, 2025
69279ee
fix #172: Resolve clippy unused_unit warnings and improve Makefile be…
JoshuaChi Nov 13, 2025
56f1410
feat #174: Implement lock-free Watch mechanism for real-time key moni…
JoshuaChi Nov 25, 2025
6c90b57
feat #179: Implement single-node cluster support for dynamic expansio…
JoshuaChi Nov 26, 2025
97fab24
feat #182: Add EmbeddedEngine API with leader election notification &…
JoshuaChi Dec 1, 2025
ec736ab
doc #185: Add service discovery pattern guide for DNS/registry use cases
JoshuaChi Dec 3, 2025
c653330
doc #187: Restructure examples documentation to match folder structure
JoshuaChi Dec 3, 2025
635b3cd
doc #187: Fix broken benchmark documentation link in three-nodes-clus…
JoshuaChi Dec 3, 2025
5ed3544
docs #192: add comprehensive read consistency documentation
JoshuaChi Dec 4, 2025
e0b4142
feat(bench) #193: add --key-space parameter for hot-key testing
JoshuaChi Dec 5, 2025
9220e58
perf(watch) #194: Skip protobuf decoding when no active watchers exist
JoshuaChi Dec 5, 2025
0312010
docs(bench) #195: Update v0.2.0 benchmark report with accurate v0.1.4…
JoshuaChi Dec 5, 2025
208df8b
feat #196: Implement Watch feature with gRPC streaming and examples
JoshuaChi Dec 11, 2025
6d8ce99
docs #204: Prepare v0.2.0 release documentation
JoshuaChi Dec 11, 2025
2381550
fix #204: Suppress rustls-pemfile unmaintained warning in dependency …
JoshuaChi Dec 11, 2025
4163f9a
fix #204: Fix flaky test and update documentation for v0.2.0
JoshuaChi Dec 11, 2025
9123484
fix #205: Enable graceful shutdown during node startup phase
JoshuaChi Dec 12, 2025
0f94d9e
docs #204: Fix documentation links and optimize workspace doc generation
JoshuaChi Dec 12, 2025
52e1e74
doc #204: Simplify feature flags and add use-case guidance
JoshuaChi Dec 13, 2025
f7b0418
chore #204: Prepare v0.2.0 release with workspace cleanup
JoshuaChi Dec 13, 2025
793ab4c
style #204: Fix import ordering to comply with rustfmt stable
JoshuaChi Dec 13, 2025
faa62b3
fix #204: Address code review feedback for v0.2.0 release
JoshuaChi Dec 13, 2025
1b55a6f
perf #208: eliminate redundant async calls in leader write hot path
JoshuaChi Dec 13, 2025
86a1278
fix #204: update both fields in update_cluster_metadata to prevent st…
JoshuaChi Dec 13, 2025
bcf2239
fix #204: Increase election RPC timeout to fix flaky tests
JoshuaChi Dec 13, 2025
f601479
ci #204: Support docs as alias for doc in commit message validation
JoshuaChi Dec 14, 2025
b8eb206
docs #204: Prepare v0.2.0 for HN launch - simplify docs and set expec…
JoshuaChi Dec 14, 2025
8204c5c
docs #204: Fix documentation links and add integration mode quick sta…
JoshuaChi Dec 16, 2025
6e0c610
refactor(raft) #209: Architecture review - consolidate v0.2.0 improve…
JoshuaChi Dec 31, 2025
84c1539
doc #224: Fix rustdoc links and improve documentation generation
JoshuaChi Dec 31, 2025
40d7c47
doc #224: Fix markdown table formatting and typo in documentation
JoshuaChi Dec 31, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
28 changes: 28 additions & 0 deletions .claudeignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@

node_modules/
vendor/
.venv/
venv/


dist/
build/
out/
target/


.git/
.svn/


*.log
*.pdf
*.zip
*.tar.gz
*.mp4
*.mp3


.DS_Store
*.pyc
__pycache__/
8 changes: 8 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Nextest configuration for d-engine

[profile.default]

# Force serial execution for tests that use /tmp/db
[[profile.default.overrides]]
filter = "test(tmp_db) | test(config_path_env)"
threads-required = 'num-cpus'
83 changes: 83 additions & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: Performance Benchmarks

permissions:
contents: read

# Run benchmarks on a schedule and manual trigger only
on:
# Run weekly on Sunday at 2 AM UTC
schedule:
- cron: "0 2 * * 0"

# Allow manual triggering for on-demand performance testing
workflow_dispatch:

env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: full

jobs:
benchmark:
name: Run Performance Benchmarks
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history for performance comparison

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y protobuf-compiler

- name: Cache Rust dependencies
uses: Swatinem/rust-cache@v2
with:
# Cache key includes benchmark to avoid interference with test cache
key: benchmark-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}

- name: Run benchmarks
run: |
echo "========================================="
echo "Running d-engine Performance Benchmarks"
echo "========================================="
echo ""
echo "Performance Targets:"
echo " • Without TTL: < 10ns overhead"
echo " • With TTL passive check: < 50ns overhead"
echo " • Piggyback cleanup: < 1ms"
echo ""

# Run benchmarks and save output
cargo bench --package d-engine-server 2>&1 | tee benchmark_output.txt

echo ""
echo "✅ Benchmarks completed successfully"

- name: Upload detailed results as artifact
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: |
target/criterion/
benchmark_output.txt
retention-days: 30

- name: Performance summary
run: |
echo "========================================="
echo "Performance Benchmark Summary"
echo "========================================="
echo ""
echo "📊 Benchmark results uploaded as artifacts"
echo ""
echo "To view detailed results:"
echo " 1. Download the artifact"
echo " 2. Open target/criterion/report/index.html in your browser"
echo ""
echo "✅ Performance monitoring complete"
Comment thread Fixed
Comment thread
coderabbitai[bot] marked this conversation as resolved.
31 changes: 17 additions & 14 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
name: CI

permissions:
contents: read

on:
push:
pull_request:
Expand Down Expand Up @@ -38,32 +41,32 @@ jobs:
name: Test and Collect Coverage
runs-on: ubuntu-latest
needs: lint
permissions:
contents: read
env:
RUST_TOOLCHAIN: 1.88.0
CI: 1
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Cache dependencies
uses: actions/cache@v4
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@master
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-
toolchain: ${{ env.RUST_TOOLCHAIN }}
components: llvm-tools-preview

- name: Cache using Swatinem/rust-cache
uses: Swatinem/rust-cache@v2

- name: Install Protobuf compiler
run: sudo apt-get install -y protobuf-compiler

- name: Install cargo-llvm-cov
- name: Install cargo tools
run: |
cargo install cargo-llvm-cov --locked

- name: Install cargo-nextest
run: |
cargo install cargo-nextest --locked
cargo install cargo-nextest --version 0.9.114 --locked

- name: Run tests with nextest and generate coverage
run: |
Expand Down
11 changes: 8 additions & 3 deletions .github/workflows/commit-message-check.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
name: Validate Commit Messages

permissions:
contents: read

on:
pull_request:

jobs:
validate-commits:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v3
with:
Expand All @@ -16,16 +21,16 @@ jobs:
BASE_SHA=${{ github.event.pull_request.base.sha }}
HEAD_SHA=${{ github.event.pull_request.head.sha }}

ALLOWED_TYPES="feat|fix|refs|doc|perf|refactor|chore"
ALLOWED_TYPES="feat|fix|refs|doc|docs|perf|refactor|style|test|chore|ci|revert"
SCOPE_PATTERN="([a-z][a-z0-9-]*)"
TICKET_PATTERN="#[0-9]+:"
FULL_PATTERN="^($ALLOWED_TYPES)(\($SCOPE_PATTERN\))? $TICKET_PATTERN.+$"

# Validate the format of all new submissions
for commit in $(git rev-list $BASE_SHA..$HEAD_SHA); do
msg=$(git show -s --format=%s $commit)
# skip merge

# skip merge
if [[ "$msg" =~ ^Merge ]]; then
continue
fi
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/dependency-audit.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
name: Dependency Audit

permissions:
contents: read

on:
push:
branches: [main, develop]
Expand All @@ -12,6 +15,9 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

- name: Install protoc
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler

Expand All @@ -26,7 +32,7 @@ jobs:

- name: Audit dependencies
run: |
cargo install cargo-deny
cargo install cargo-deny --locked
cargo deny check all

- name: Update dependencies
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,6 @@ docker/monitoring/prometheus/data/
lcov.info

CONTEXT
*.bak*
.claudeignore
safe-split-commits.sh
87 changes: 87 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,93 @@ All notable changes to this project will be documented in this file.

---

## [v0.2.0] - 2025-12-11 [✅ Released]

### 🎯 Highlights for Developers

#### Workspace Structure - Modular Dependencies

**Problem**: v0.1.x pulled all dependencies even for client-only usage
**Solution**: Feature flags `client`/`server`/`full` - depend only on what you need

```toml
# Client-only (lightweight)
d-engine = { version = "0.2", features = ["client"] }

# Embedded server (full engine)
d-engine = { version = "0.2", features = ["server"] }
```

**Impact**: Faster builds, smaller binaries

#### TTL/Lease - Automatic Key Expiration

**Use Case**: Distributed locks, session management, temporary state
**API**: `client.put_with_ttl("session:123", data, Duration::from_secs(60))`
**Feature**: Crash-safe (survives restart via absolute expiration time)

#### Watch API - Real-Time Key Monitoring

**Use Case**: Config change notifications, service discovery
**Example**:

```rust,ignore
let mut watcher = client.watch("config/").await?;
while let Some(event) = watcher.next().await {
println!("Changed: {:?}", event);
}
```

**Performance**: Lock-free, <0.1ms notification latency

#### StandaloneServer - One-Line Deployment

**Use Case**: Independent server process (production deployment)
**API**: `run(shutdown_rx)` uses env config, `run_with(config_path, shutdown_rx)` uses explicit config
**Benefit**: Blocks until shutdown, no manual lifecycle management

#### EmbeddedEngine - In-Process Integration

**Use Case**: Embed d-engine in your Rust application
**API**: `start()` uses env config, `start_with(config_path)` uses explicit config, `start_custom(...)` for advanced usage
**Benefit**: Zero gRPC overhead via LocalKvClient (<0.1ms latency)

#### LocalKvClient - Zero-Overhead Embedded Access

**When**: Your app and d-engine in same process
**Benefit**: Skip gRPC serialization, direct memory access (<0.1ms)
**Example**: See `examples/service-discovery-embedded/`

---

### 📚 New Examples

- `examples/quick-start/` - 5-minute single-node setup
- `examples/single-node-expansion/` - Dynamic 1→3 node scaling
- `examples/service-discovery-embedded/` - LocalKvClient zero-overhead access
- `examples/service-discovery-standalone/` - Watch API pattern

---

### ⚠️ Breaking Changes

**File-based State Machine WAL Format**

- WAL now uses absolute expiration time (not relative TTL)
- **Action Required**: See [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md) if upgrading from v0.1.x
- **New users**: No action needed

---

### 🚀 Performance & Quality

- Watch API: Lock-free, tested with 1000+ concurrent watchers, <0.1ms notification latency
- TTL cleanup: Lazy (read-time check) + Background (scheduled task), <1% CPU overhead
- 1000+ new integration tests covering edge cases
- Zero clippy warnings across entire codebase

---

## [v0.1.4] - 2025-10-12 [✅ Released]

### Features
Expand Down
Loading
Loading