Skip to content

fix: Learner promotion stuck due to voter count and missing role transition #212

Description

@JoshuaChi

Problem

When scaling a single-node cluster to 3 nodes, the third node (Learner) gets promoted to Voter by the Leader but fails to transition to Follower role, remaining stuck as Learner indefinitely.

Impact:

  • Learner cannot serve client read requests (returns "Learner can not process client read request")
  • Cluster appears healthy to Leader but is functionally degraded
  • Affects dynamic cluster scaling scenarios

Root Cause

Issue 1: Incorrect voter count calculation (d-engine-core/src/raft_role/leader_state.rs)

  • membership.voters() excludes the Leader itself
  • process_pending_promotions didn't account for Leader (+1)
  • Result: calculate_safe_batch_size(1, 1) → 0, blocking promotion

Issue 2: Missing role transition detection (d-engine-core/src/raft_role/learner_state.rs)

  • Learner received MembershipApplied event after promotion
  • Did not check if its own role changed to Voter
  • Failed to send BecomeFollower event to trigger transition

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    raft-membershipDynamic membership changes (node addition/removal from cluster)

    Type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions