feat: Implement DRepState bootstrapping - 264 #488

buddhisthead · 2025-12-12T17:13:47Z

Description

Implement DRepState bootstrapping (without history)

Move DRep types into their own file
Remove the history part for PR, we can add it back in a future commit

Related Issue(s)

How was this tested?

Modify your config file (omnibus.toml) to use the snapshot startup mode like this:

# ============================================================================
# Startup Configuration
# ============================================================================
[global.startup]
method = "snapshot"  # Options: "mithril" | "snapshot"
topic = "cardano.sequence.start"

Run
make run | egrep "DRepState|Received 400 DReps" to see that we're receiving the bootstrapping messages.

Checklist

My code builds and passes local tests
I added/updated tests for my changes, where applicable
I updated documentation (if applicable)
CI is green for this PR

Impact / Side effects

Moved all of the DRep types into their own file because adding more to "types.rs" seemed like generating more technical debt. The DRep types need to be shared across some modules and there was a circular dependency.

Reviewer notes / Areas to focus

Nothing special.

* Move DRep types into their own file * Remove the history part for PR, we can add it back in a future commit

Copilot

Pull request overview

This PR implements DRepState bootstrapping from CBOR snapshot files, enabling the system to initialize DRep state directly from snapshot data rather than replaying the entire chain. The implementation refactors DRep-related types into a dedicated module to resolve circular dependencies and adds the necessary message passing infrastructure to communicate bootstrap data between the snapshot parser and DRepState module.

Key changes:

Creates new common/src/drep.rs module consolidating all DRep types (DRepCredential, DRepRecord, Anchor, DRepChoice, etc.)
Implements DRepState bootstrap handler to receive and process DRep data from snapshots
Adds DRepBootstrapMessage and updates snapshot callback signatures to pass HashMap-based DRep data

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
processes/omnibus/omnibus.toml	Changes startup method from "mithril" to "snapshot" (appears to be test configuration that should be reverted)
modules/snapshot_bootstrapper/src/publisher.rs	Implements DRep bootstrap message publishing, updates callback to new signature, adds fields for tracking DRep count
modules/drep_state/src/state.rs	Adds bootstrap method to insert DRep records from snapshot data into state
modules/drep_state/src/drep_state.rs	Adds snapshot message subscription and handler to receive bootstrap messages
common/src/types.rs	Removes DRep types that were moved to dedicated module, updates imports
common/src/snapshot/streaming_snapshot.rs	Changes DRep parsing to return HashMap instead of Vec, updates internal types, converts from DRepInfo to DRepRecord
common/src/messages.rs	Adds DRepBootstrapMessage struct and DRepState variant to SnapshotStateMessage enum
common/src/lib.rs	Adds drep module and re-exports drep types
common/src/drep.rs	New file containing all DRep-related types: Anchor, DRepRecord, DRepChoice, DRepRegistration, DRepDeregistration, DRepUpdate, DRepVotingThresholds
common/examples/test_streaming_parser.rs	Updates example to work with new HashMap-based DRep callback signature

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-12T17:22:02Z

modules/drep_state/src/drep_state.rs

+                            SnapshotStateMessage::DRepState(drep_msg),
+                        )) => {
+                            info!("DRepState: Snapshot Bootstrap message received");
+                            state.bootstrap(drep_msg);


The bootstrapped state is modified but never committed to the state history. After calling state.bootstrap(drep_msg), you need to add history.lock().await.commit(drep_msg.epoch, state); to persist the changes. Without this, the bootstrapped DRep data will be lost and queries will return an empty state. This pattern is consistent with how other modules handle bootstrap messages (see accounts_state.rs line 122, epochs_state.rs line 103, and spo_state.rs line 759).

Suggested change

state.bootstrap(drep_msg);

state.bootstrap(drep_msg);

// Commit the bootstrapped state to history to persist changes

history.lock().await.commit(drep_msg.epoch, state);

This might actually be why the REST response is returning empty 🤔

Yep, definitely needed!

Copilot · 2025-12-12T17:22:03Z

common/src/drep.rs

+            minicbor::data::Type::String => d.str()?.to_string(),
+            _ => {
+                return Err(minicbor::decode::Error::message(
+                    "Expected bytes or string for URL",


The error message "Expected bytes or string for URL" is less specific than the original "Expected bytes or string for Anchor URL". Consider restoring the more specific error message to help developers better understand the context when this error occurs during CBOR decoding.

Suggested change

"Expected bytes or string for URL",

"Expected bytes or string for Anchor URL",

Copilot · 2025-12-12T17:22:03Z

modules/snapshot_bootstrapper/src/publisher.rs

+        tokio::spawn(async move {
+            if let Err(e) = context.publish(&snapshot_topic, message).await {
+                tracing::error!("Failed to publish DRepBootstrap message: {e}");
+            }
+        });


Using tokio::spawn here creates a race condition. The DRepBootstrap message may not be published before the snapshot completion signal, causing the DRepState module to miss the bootstrap data. This should use tokio::task::block_in_place with block_on instead, following the same pattern as on_accounts (lines 253-259) which has detailed comments explaining why blocking is necessary for message ordering guarantees.

I think Mr Copilot is right, unfortunately. It would be better if these callbacks were async themselves, not sure how much work that is!

Copilot · 2025-12-12T17:22:03Z

modules/drep_state/src/drep_state.rs

+                            SnapshotStateMessage::DRepState(drep_msg),
+                        )) => {
+                            info!("DRepState: Snapshot Bootstrap message received");
+                            state.bootstrap(drep_msg);


Consider adding a log message after bootstrap completes to confirm success and indicate how many DReps were loaded, following the pattern used in other modules (e.g., accounts_state.rs line 123: "Accounts state bootstrap complete"). This improves observability and helps with debugging bootstrap issues.

Suggested change

state.bootstrap(drep_msg);

state.bootstrap(drep_msg);

let drep_count = state.dreps.len();

info!("DRep state bootstrap complete: {} DReps loaded", drep_count);

Copilot · 2025-12-12T17:22:04Z

modules/drep_state/src/state.rs

+    pub fn bootstrap(&mut self, drep_msg: &DRepBootstrapMessage) {
+        for (cred, record) in &drep_msg.dreps {
+            self.dreps.insert(cred.clone(), record.clone());
+            // update historical state if enabled
+            /*
+            This will be needed once we want historical drep data from snapshots
+            if let Some(hist_map) = self.historical_dreps.as_mut() {
+                let cfg = self.config;
+                let entry = hist_map
+                    .entry(cred.clone())
+                    .or_insert_with(|| HistoricalDRepState::from_config(&cfg));
+                if let Some(info) = entry.info.as_mut() {
+                    info.deposit = record.deposit;
+                    info.expired = false;
+                    info.retired = false;
+                    info.active_epoch = None;
+                    info.last_active_epoch = 0; // unknown from snapshot
+                    info!(
+                        "Bootstrapped Historical DRepState: DRep {:?} => {:?}",
+                        cred, record
+                    );
+                }
+            }
+            */
+            // info!("Bootstrapped DRepState: DRep {:?} => {:?}", cred, record);
+        }
+    }


The new bootstrap method lacks test coverage. Consider adding unit tests to verify that DReps are correctly inserted from the bootstrap message, especially since this file already has comprehensive test coverage for other state operations (e.g., test_drep_process_one_certificate, test_drep_update_certificate). Tests should verify that the DRep records are properly stored and can be queried after bootstrap.

lowhung · 2025-12-14T17:29:55Z

When I try to query dreps with the following configuration, I get empty responses.

[module.drep-state]
# Enables /governance/dreps/{drep_id} endpoint (Requires store-delegators to be enabled)
store-info = true
# Enables /governance/dreps/{drep_id}/delegators endpoint
store-delegators = false
# Enables /governance/dreps/{drep_id}/metadata endpoint
store-metadata = false
# Enables /governance/dreps/{drep_id}/updates endpoint
store-updates = false
# Enables /governance/dreps/{drep_id}/votes endpoint
store-votes = false

[module.drdd-state]
store-drdd = true

With the following:

acropolis on  cet/drep-state-bootstrap-264 [$✘!]
➜ curl http://127.0.0.1:4340/governance/dreps
No current DRep state%

lowhung · 2025-12-12T18:06:01Z

common/examples/test_streaming_parser.rs

-            if let Some(anchor) = &drep.anchor {
+        for (i, (cred, record)) in dreps.iter().take(10).enumerate() {
+            let drep_id = match cred {
+                acropolis_common::StakeCredential::AddrKeyHash(hash) => {


There's a method on Credential that you can use instead of this match.

pub fn to_drep_bech32(&self) -> Result<String, anyhow::Error> {}

lowhung · 2025-12-12T18:06:40Z

common/examples/test_streaming_parser.rs

+                for (i, (cred, record)) in callbacks.sample_dreps.iter().enumerate() {
+                    let drep_id = match cred {
+                        acropolis_common::StakeCredential::AddrKeyHash(hash) => {
+                            format!("drep_{}", hex::encode(hash))


Same thing as above re: to_drep_bech32

lowhung · 2025-12-14T17:30:50Z

common/src/drep.rs

+    pub data_hash: Vec<u8>,
+}
+
+impl<'b, C> minicbor::Decode<'b, C> for Anchor {


Thanks for cleaning up types.rs 🙌🏻

lowhung · 2025-12-14T17:32:19Z

modules/drep_state/src/drep_state.rs

+                            SnapshotStateMessage::DRepState(drep_msg),
+                        )) => {
+                            info!("DRepState: Snapshot Bootstrap message received");
+                            state.bootstrap(drep_msg);


This might actually be why the REST response is returning empty 🤔

lowhung · 2025-12-14T17:59:28Z

I would have expected these warnings to have disappeared as a result of this bootstrap work. Perhaps we can track this down together?

2025-12-14T17:54:21.507136Z  WARN acropolis_common::stake_addresses: Delegated to unregistered DRep address AddrKeyHash(Hash<28>("55215f98aa7d9e289d215a55f62d258c6c3a71ab847b76de9ddbe661"))
2025-12-14T17:54:21.507161Z  WARN acropolis_common::stake_addresses: Delegated to unregistered DRep address AddrKeyHash(Hash<28>("86b4d0fb7fd85807003537d6c14f4053cfa75f2f9d4f670d23dec934"))
2025-12-14T17:54:21.507163Z  WARN acropolis_common::stake_addresses: Delegated to unregistered DRep address AddrKeyHash(Hash<28>("b6f4547ad049d7443ee5695761e1aa5e446cfccf72c7ed0d8ad8edfa"))
2025-12-14T17:54:21.507175Z  WARN acropolis_common::stake_addresses: Delegated to unregistered DRep address AddrKeyHash(Hash<28>("b504dbaab6de1a99cff4792682afd38c6c0aed226e563bc430530cff"))
2025-12-14T17:54:21.507140Z  WARN acropolis_common::stake_addresses: Delegated to unregistered DRep address AddrKeyHash(Hash<28>("ddb7316d7f4ff1d69084f2352e4613284381f17474fd133dfb4bb6ed"))
...

The accounts bootstrap does already include the dreps so there must be something amiss with the credential decoding.

sandtreader · 2025-12-15T10:45:30Z

common/src/types.rs

    pub deposit: Lovelace,
 }

-/// Anchor


+100 for splitting this out!

sandtreader · 2025-12-15T10:48:27Z

modules/drep_state/src/state.rs

+    pub fn bootstrap(&mut self, drep_msg: &DRepBootstrapMessage) {
+        for (cred, record) in &drep_msg.dreps {
+            self.dreps.insert(cred.clone(), record.clone());
+            // update historical state if enabled


We will never get all historical state I would think, but starting the history with what we just got seems right. Is there any reason not to enable this since it's gated on the historical_dreps being active anyway?

sandtreader

Noted the Copilot suggestion about committing state and publish race condition - maybe needs snapshot callbacks to be async?

sandtreader · 2025-12-15T10:50:37Z

modules/snapshot_bootstrapper/src/publisher.rs

+        tokio::spawn(async move {
+            if let Err(e) = context.publish(&snapshot_topic, message).await {
+                tracing::error!("Failed to publish DRepBootstrap message: {e}");
+            }
+        });


I think Mr Copilot is right, unfortunately. It would be better if these callbacks were async themselves, not sure how much work that is!

Implement DRepState bootstrapping

3352242

* Move DRep types into their own file * Remove the history part for PR, we can add it back in a future commit

Copilot AI review requested due to automatic review settings December 12, 2025 17:13

Copilot started reviewing on behalf of buddhisthead December 12, 2025 17:14 View session

buddhisthead added 2 commits December 12, 2025 09:15

Reorganize modules in lib.rs per check

c4dc150

Clippy

16d6e08

buddhisthead requested review from lowhung and sandtreader and removed request for Copilot December 12, 2025 17:17

Revert startup method to mithril for main branch

252dc2f

Copilot AI review requested due to automatic review settings December 12, 2025 17:21

Copilot AI reviewed Dec 12, 2025

View reviewed changes

Copilot started reviewing on behalf of buddhisthead December 12, 2025 17:22 View session

lowhung reviewed Dec 14, 2025

View reviewed changes

sandtreader reviewed Dec 15, 2025

View reviewed changes

common/src/types.rs

pub deposit: Lovelace,

}

/// Anchor

Copy link

Collaborator

sandtreader Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+100 for splitting this out!

sandtreader reviewed Dec 15, 2025

View reviewed changes

sandtreader requested changes Dec 15, 2025

View reviewed changes

	"Expected bytes or string for URL",
	"Expected bytes or string for Anchor URL",

feat: Implement DRepState bootstrapping - 264 #488

Are you sure you want to change the base?

feat: Implement DRepState bootstrapping - 264 #488

Conversation

buddhisthead commented Dec 12, 2025

Description

Related Issue(s)

How was this tested?

Checklist

Impact / Side effects

Reviewer notes / Areas to focus

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

lowhung commented Dec 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lowhung commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandtreader left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lowhung commented Dec 14, 2025 •

edited

Loading