Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
d8bd658
feat #43: When a Raft leader is elected, it should first send an empt…
JoshuaChi Apr 19, 2025
8fb91b3
Feature/59 benchmark (#72)
JoshuaChi Apr 20, 2025
2f97f6c
chore(ci) #73: prepare release for v0.1.2
JoshuaChi Apr 20, 2025
1756873
Merge branch 'main' into develop
JoshuaChi Apr 22, 2025
e53ba82
feat #79: Snapshot feature, Learner join cluster feature and develope…
JoshuaChi Sep 3, 2025
729e100
feat #79: optimize v0.1.3 CHANGELOG
JoshuaChi Sep 3, 2025
cebfdbb
Merge branch 'main' into develop
JoshuaChi Sep 3, 2025
0a057c8
fix #133: Example project: File/RocksDB storage engines can fail to s…
JoshuaChi Sep 7, 2025
bd44015
fix #133: fix fmt
JoshuaChi Sep 7, 2025
265c250
fix #133: fix fmt
JoshuaChi Sep 7, 2025
2e88aae
fix #133: fix fmt
JoshuaChi Sep 7, 2025
598f011
fix #133: update REAMDE
JoshuaChi Sep 7, 2025
68de4c7
feat #135: Enable tokio via three-nodes-cluster example
JoshuaChi Sep 17, 2025
3fe4cdc
feat #135: Enable tokio_metrics via three-nodes-cluster example
JoshuaChi Sep 17, 2025
f7911c1
feat #135: Enable tokio_metrics via three-nodes-cluster example
JoshuaChi Sep 18, 2025
4e30171
feat #138: v0.1.4 - target 100K throughput
JoshuaChi Oct 13, 2025
b03a606
fix #145: fix Bytes compatible bugs in examples folder
JoshuaChi Oct 14, 2025
f3f3430
fix #150: type issue
JoshuaChi Oct 14, 2025
32cc365
fix #151: fix mmap bug
JoshuaChi Oct 14, 2025
0713943
fix #152: Update README by introduce new feature from v0.1.4
JoshuaChi Oct 15, 2025
ff1e251
fix #153: fix code review feedback
JoshuaChi Oct 15, 2025
8659663
fix #154: code review feedback
JoshuaChi Oct 15, 2025
901008f
fix #155: fix code review feedback
JoshuaChi Oct 15, 2025
a3c314d
fix #156: fix code review feedback
JoshuaChi Oct 15, 2025
58f97dc
fix #157: fix code review feedback
JoshuaChi Oct 15, 2025
f145325
fix #158, #159: fix code review feedback
JoshuaChi Oct 16, 2025
7e0d96e
fix #164: fix code review bug
JoshuaChi Oct 16, 2025
b3e119e
fix #161: fix code review feedback
JoshuaChi Oct 16, 2025
9ea5116
fix #164: fix doc issues
JoshuaChi Oct 16, 2025
2169613
Merge branch 'main' into develop
JoshuaChi Oct 18, 2025
be93d34
feat #167: workspace structure (#175)
JoshuaChi Nov 5, 2025
d9636d1
feat #172: Feature ttl support (#176)
JoshuaChi Nov 12, 2025
2569af0
fix #172: format code to pass rustfmt checks
JoshuaChi Nov 12, 2025
e1143d0
fix #172: Replace tokio-tar with astral-tokio-tar to resolve CVE-2025…
JoshuaChi Nov 13, 2025
803400a
fix #172: Resolve Tokio runtime nesting issue in TTL benchmark
JoshuaChi Nov 13, 2025
69279ee
fix #172: Resolve clippy unused_unit warnings and improve Makefile be…
JoshuaChi Nov 13, 2025
56f1410
feat #174: Implement lock-free Watch mechanism for real-time key moni…
JoshuaChi Nov 25, 2025
6c90b57
feat #179: Implement single-node cluster support for dynamic expansio…
JoshuaChi Nov 26, 2025
97fab24
feat #182: Add EmbeddedEngine API with leader election notification &…
JoshuaChi Dec 1, 2025
ec736ab
doc #185: Add service discovery pattern guide for DNS/registry use cases
JoshuaChi Dec 3, 2025
c653330
doc #187: Restructure examples documentation to match folder structure
JoshuaChi Dec 3, 2025
635b3cd
doc #187: Fix broken benchmark documentation link in three-nodes-clus…
JoshuaChi Dec 3, 2025
5ed3544
docs #192: add comprehensive read consistency documentation
JoshuaChi Dec 4, 2025
e0b4142
feat(bench) #193: add --key-space parameter for hot-key testing
JoshuaChi Dec 5, 2025
9220e58
perf(watch) #194: Skip protobuf decoding when no active watchers exist
JoshuaChi Dec 5, 2025
0312010
docs(bench) #195: Update v0.2.0 benchmark report with accurate v0.1.4…
JoshuaChi Dec 5, 2025
208df8b
feat #196: Implement Watch feature with gRPC streaming and examples
JoshuaChi Dec 11, 2025
6d8ce99
docs #204: Prepare v0.2.0 release documentation
JoshuaChi Dec 11, 2025
2381550
fix #204: Suppress rustls-pemfile unmaintained warning in dependency …
JoshuaChi Dec 11, 2025
4163f9a
fix #204: Fix flaky test and update documentation for v0.2.0
JoshuaChi Dec 11, 2025
9123484
fix #205: Enable graceful shutdown during node startup phase
JoshuaChi Dec 12, 2025
0f94d9e
docs #204: Fix documentation links and optimize workspace doc generation
JoshuaChi Dec 12, 2025
52e1e74
doc #204: Simplify feature flags and add use-case guidance
JoshuaChi Dec 13, 2025
f7b0418
chore #204: Prepare v0.2.0 release with workspace cleanup
JoshuaChi Dec 13, 2025
793ab4c
style #204: Fix import ordering to comply with rustfmt stable
JoshuaChi Dec 13, 2025
faa62b3
fix #204: Address code review feedback for v0.2.0 release
JoshuaChi Dec 13, 2025
1b55a6f
perf #208: eliminate redundant async calls in leader write hot path
JoshuaChi Dec 13, 2025
86a1278
fix #204: update both fields in update_cluster_metadata to prevent st…
JoshuaChi Dec 13, 2025
bcf2239
fix #204: Increase election RPC timeout to fix flaky tests
JoshuaChi Dec 13, 2025
f601479
ci #204: Support docs as alias for doc in commit message validation
JoshuaChi Dec 14, 2025
b8eb206
docs #204: Prepare v0.2.0 for HN launch - simplify docs and set expec…
JoshuaChi Dec 14, 2025
8204c5c
docs #204: Fix documentation links and add integration mode quick sta…
JoshuaChi Dec 16, 2025
6e0c610
refactor(raft) #209: Architecture review - consolidate v0.2.0 improve…
JoshuaChi Dec 31, 2025
84c1539
doc #224: Fix rustdoc links and improve documentation generation
JoshuaChi Dec 31, 2025
40d7c47
doc #224: Fix markdown table formatting and typo in documentation
JoshuaChi Dec 31, 2025
c4a6e1f
doc #225: Prepare multi-crate publishing documentation and configurat…
JoshuaChi Jan 1, 2026
ea370fe
chore #227: Merge branch 'main' into develop
JoshuaChi Jan 1, 2026
0db60ee
doc #227: fix working directory and version requirements
JoshuaChi Jan 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.

---

## [v0.2.0] - 2025-12-11 [✅ Released]
## [v0.2.0] - 2026-01-01 [✅ Released]

### 🎯 Highlights for Developers

Expand Down
4 changes: 0 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 5 additions & 6 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,12 @@ members = [
"d-engine-proto",
"d-engine-core",
"d-engine-client",
"d-engine-docs",
"d-engine-server"
]

exclude = [
"docs",
"examples/client_usage",
"examples/rocksdb-cluster",
"examples/sled-cluster",
"examples/three-nodes-cluster",
"examples/single-node-expansion",
Expand All @@ -32,10 +31,10 @@ repository = "https://github.com/deventlab/d-engine"
license = "MIT OR Apache-2.0"

[workspace.dependencies]
d-engine-proto = { path = "./d-engine-proto" }
d-engine-core = { path = "./d-engine-core" }
d-engine-client = { path = "./d-engine-client" }
d-engine-server = { path = "./d-engine-server" }
d-engine-proto = { path = "./d-engine-proto", version = "0.2.0" }
d-engine-core = { path = "./d-engine-core", version = "0.2.0" }
d-engine-client = { path = "./d-engine-client", version = "0.2.0" }
d-engine-server = { path = "./d-engine-server", version = "0.2.0" }

tokio = { version = "1", features = [
"macros",
Expand Down
12 changes: 6 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ RUST_LOG_LEVEL ?= d_engine_server=debug,d_engine_core=debug,d_engine_client=debu
RUST_BACKTRACE ?= 1

# Workspace member crates
WORKSPACE_MEMBERS := d-engine-proto d-engine-core d-engine-client d-engine-server d-engine-docs d-engine
WORKSPACE_MEMBERS := d-engine-proto d-engine-core d-engine-client d-engine-server d-engine

# Color codes for formatted output
RED := \033[0;31m
Expand Down Expand Up @@ -382,19 +382,19 @@ bench-gate: check-workspace
## docs Generate API documentation and open in browser (opens d-engine API hub)
docs: check-workspace
@echo "$(BLUE)Generating API documentation for public crates...$(NC)"
@$(CARGO) doc --package d-engine --package d-engine-docs --no-deps \
@$(CARGO) doc --package d-engine --no-deps \
--features "d-engine/client,d-engine/server,d-engine/rocksdb,d-engine/watch"
@echo "$(GREEN)✓ Documentation generated$(NC)"
@echo "$(CYAN)Opening documentation hub at: target/doc/d_engine_docs/index.html$(NC)"
@open "file://$$(pwd)/target/doc/d_engine_docs/index.html" 2>/dev/null || xdg-open "file://$$(pwd)/target/doc/d_engine_docs/index.html" 2>/dev/null || echo "Please open: target/doc/d_engine_docs/index.html"
@echo "$(CYAN)Opening documentation at: target/doc/d_engine/index.html$(NC)"
@open "file://$$(pwd)/target/doc/d_engine/index.html" 2>/dev/null || xdg-open "file://$$(pwd)/target/doc/d_engine/index.html" 2>/dev/null || true

## docs-all Generate documentation with all features enabled
docs-all: check-workspace
@echo "$(BLUE)Generating API documentation (all features)...$(NC)"
@$(CARGO) doc --workspace --all-features --no-deps
@echo "$(GREEN)✓ Documentation generated$(NC)"
@echo "$(CYAN)Opening documentation hub at: target/doc/d_engine_docs/index.html$(NC)"
@open "file://$$(pwd)/target/doc/d_engine_docs/index.html" 2>/dev/null || xdg-open "file://$$(pwd)/target/doc/d_engine_docs/index.html" 2>/dev/null || echo "Please open: target/doc/d_engine_docs/index.html"
@echo "$(CYAN)Opening documentation at: target/doc/d_engine/index.html$(NC)"
@open "file://$$(pwd)/target/doc/d_engine/index.html" 2>/dev/null || xdg-open "file://$$(pwd)/target/doc/d_engine/index.html" 2>/dev/null || true

## docs-check Check documentation without opening browser (CI-safe)
docs-check: check-workspace
Expand Down
20 changes: 0 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,26 +12,6 @@ with pluggable storage backends, built-in observability, and tokio runtime suppo

---

## 🚧 Project Status

d-engine is in **active development** (pre-1.0).

**For Production Use**:

- ✅ Core Raft consensus is stable (1000+ integration tests, Jepsen validated)
- ✅ API surface is stabilizing (v0.2.0 focuses on developer experience)
- ⚠️ Pre-1.0: Breaking changes may occur between minor versions (e.g., v0.2 → v0.3)
- 📋 [Semantic Versioning](https://semver.org/) will strictly apply after v1.0

**Compatibility Promise**:

- **Before v1.0**: Breaking changes documented in [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md)
- **After v1.0**: Breaking changes only in major versions (v1.x → v2.0)

**Current Focus**: API stabilization for v1.0 release (Q2 2026)

---

## Features

### New in v0.2.0 🎉
Expand Down
2 changes: 1 addition & 1 deletion benches/d-engine-bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Ensure you're using the correct d-engine version in your `Cargo.toml`:

```toml
[dependencies]
d-engine = "0.1.2" # Verify latest version
d-engine = "0.2.0" # Verify latest version
```

### 2. Setup Cluster
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
- ⚠️ **Read**: etcd 11.7x faster in linearizable reads (hardware difference)
- ✅ **Unique**: LeaseRead provides 6.9x performance vs linearizable with strong consistency

**vs v0.1.4**:
**vs v0.2.0**:
Comment thread
JoshuaChi marked this conversation as resolved.

- ✅ **All scenarios improved** 2.3% to 11% throughput
- ✅ **LeaseRead p99**: -44% latency (7.87ms vs 14.05ms) - biggest win
Expand Down
4 changes: 2 additions & 2 deletions d-engine-client/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ name = "d-engine-client"
version.workspace = true
edition.workspace = true
authors.workspace = true
description = "Client library for connecting to d-engine Raft consensus servers"
description = "Client library for interacting with d-engine Raft clusters via gRPC"
homepage.workspace = true
repository.workspace = true
license.workspace = true
keywords = ["raft", "consensus", "client", "async"]
categories = ["asynchronous", "network-programming"]
publish = false
publish = true

[package.metadata.docs.rs]
all-features = true
Expand Down
43 changes: 43 additions & 0 deletions d-engine-client/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# d-engine-client

[![Crates.io](https://img.shields.io/crates/v/d-engine-client.svg)](https://crates.io/crates/d-engine-client)
[![docs.rs](https://docs.rs/d-engine-client/badge.svg)](https://docs.rs/d-engine-client)

Client library for interacting with d-engine Raft clusters

---

## ⚠️ You Probably Don't Need This Crate

**Use [`d-engine`](https://crates.io/crates/d-engine) instead:**

```toml
[dependencies]
d-engine = { version = "0.2", features = ["client"] }
```

This provides the same API with simpler dependency management. The `d-engine-client` crate is automatically included when you enable the `client` feature.

---

## For Contributors

This crate exists for architectural reasons:

- **Clean boundaries** - Separates client logic from server implementation
- **Faster builds** - Workspace members can depend on client without pulling server deps
- **Testing** - Enables isolated client testing

If you're building an application, use `d-engine` with `features = ["client"]` instead.

---

## Quick Reference

See [`d-engine` README](https://github.com/deventlab/d-engine/blob/main/README.md) for usage examples and documentation.

---

## License

MIT or Apache-2.0
96 changes: 59 additions & 37 deletions d-engine-client/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,46 +1,68 @@
//! Client module for distributed consensus system
//! # d-engine-client
//!
//! Provides core components for interacting with the d_engine cluster:
//! - [`Client`] - Main entry point with cluster access
//! - [`ClientBuilder`] - Configurable client construction
//! - [`KvClient`] - Key-value store operations
//! - [`ClusterClient`] - Cluster management operations
//! - [`ConnectionPool`] - Underlying connection management
//! Client library for interacting with d-engine Raft clusters via gRPC
//!
//! ## ⚠️ You Probably Don't Need This Crate
//!
//! **Use [`d-engine`](https://crates.io/crates/d-engine) instead:**
//!
//! ```toml
//! [dependencies]
//! d-engine = { version = "0.2", features = ["client"] }
//! ```
//!
//! This provides the same API with simpler dependency management. The `d-engine-client` crate
//! is automatically included when you enable the `client` feature.
//!
//! ## For Contributors
//!
//! # Basic Usage
//! ```no_run
//! This crate exists for architectural reasons:
//! - Clean boundaries between client and server
//! - Faster builds during development
//! - Isolated client testing
//!
//! ## Quick Start
//!
//! ```rust,ignore
//! use d_engine_client::Client;
//! use d_engine_client::ClientBuilder;
//! use std::time::Duration;
//! use core::error::Error;
//!
//! #[tokio::main(flavor = "current_thread")]
//! async fn main(){
//! // Initialize client with automatic cluster discovery
//! let client = Client::builder(vec![
//! "http://node1:9081".into(),
//! "http://node2:9082".into()
//! ])
//! .connect_timeout(Duration::from_secs(3))
//! .request_timeout(Duration::from_secs(1))
//! .enable_compression(true)
//! .build()
//! .await
//! .unwrap();
//!
//! // Execute key-value operations
//! client.kv().put("user:1001", "Alice").await.unwrap();
//!
//! let value = client.kv().get("user:1001").await.unwrap();
//!
//! println!("User data: {:?}", value);
//!
//! // Perform cluster management
//! let members = client.cluster().list_members().await.unwrap();
//! println!("Cluster members: {:?}", members);
//!
//! #[tokio::main]
//! async fn main() -> Result<(), Box<dyn std::error::Error>> {
//! let client = Client::connect(vec!["http://localhost:50051"]).await?;
//!
//! // Write data
//! client.put(b"key".to_vec(), b"value".to_vec()).await?;
//!
//! // Read data
//! if let Some(value) = client.get(b"key".to_vec()).await? {
//! println!("Value: {:?}", value);
//! }
//!
//! Ok(())
//! }
//! ```
//!
//! ## Read Consistency
//!
//! Choose consistency level based on your needs:
//!
//! - `get_linearizable()` - Strong consistency (read from Leader)
//! - `get_eventual()` - Fast local reads (stale OK)
//! - `get_lease()` - Optimized with leader lease
//!
//! ## Features
//!
//! This crate provides:
//! - [`Client`] - Main entry point with cluster access
//! - [`ClientBuilder`] - Configurable client construction
//! - [`KvClient`] - Key-value store operations trait
//! - [`ClusterClient`] - Cluster management operations
//!
//! ## Documentation
//!
//! For comprehensive guides:
//! - [Read Consistency](https://docs.rs/d-engine/latest/d_engine/docs/client_guide/read_consistency/index.html)
//! - [Error Handling](https://docs.rs/d-engine/latest/d_engine/docs/client_guide/error_handling/index.html)

mod builder;
mod cluster;
Expand Down
4 changes: 2 additions & 2 deletions d-engine-core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ name = "d-engine-core"
version.workspace = true
edition.workspace = true
authors.workspace = true
description = "Core Raft consensus algorithm implementation and traits"
description = "Pure Raft consensus algorithm - for building custom Raft-based systems"
homepage.workspace = true
repository.workspace = true
license.workspace = true
keywords = ["raft", "consensus", "algorithm"]
categories = ["algorithms", "concurrency"]
publish = false
publish = true

[package.metadata.docs.rs]
all-features = true
Expand Down
69 changes: 69 additions & 0 deletions d-engine-core/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# d-engine-core

[![Crates.io](https://img.shields.io/crates/v/d-engine-core.svg)](https://crates.io/crates/d-engine-core)
[![docs.rs](https://docs.rs/d-engine-core/badge.svg)](https://docs.rs/d-engine-core)

Pure Raft consensus algorithm - for building custom Raft-based systems

---

## ⚠️ Internal Crate - Not Ready for Standalone Use

**Use [`d-engine`](https://crates.io/crates/d-engine) instead.**

This crate contains the pure Raft consensus algorithm used internally by d-engine. The API is unstable before v1.0.

```toml
# ❌ Don't use this directly
[dependencies]
d-engine-core = "0.2"

# ✅ Use this instead
[dependencies]
d-engine = "0.2"
```

---

## For Contributors

This crate provides the core Raft consensus algorithm:

- **Leader Election** - Automatic leader election with randomized timeouts
- **Log Replication** - Reliable log replication to followers
- **Membership Changes** - Dynamic cluster membership (add/remove nodes)
- **Snapshot Support** - Log compaction via snapshots

**Reference integration**: See how [d-engine-server](https://github.com/deventlab/d-engine/tree/main/d-engine-server) uses this crate.

**Key traits to understand**:

- `StorageEngine` - Persistent storage for Raft logs
- `StateMachine` - Application-specific state transitions
- `LogStore` - Log entry persistence
- `MetaStore` - Metadata persistence (term, voted_for)

See the [API documentation](https://docs.rs/d-engine-core) for detailed trait definitions.

---

## Future Vision

**Post-1.0 goal**: Become a standalone Raft library with stable API.

**Current status**: Internal to d-engine, API may change between minor versions.

---

## Documentation

For understanding d-engine internals:

- [Customize Storage Engine](https://docs.rs/d-engine/latest/d_engine/docs/server_guide/customize_storage_engine/index.html)
- [Customize State Machine](https://docs.rs/d-engine/latest/d_engine/docs/server_guide/customize_state_machine/index.html)

---

## License

MIT or Apache-2.0
Loading
Loading