feat: Implement Least Request Load Balancing Policy (gRFC A48) by emil10001 · Pull Request #2651 · grpc/grpc-rust

emil10001 · 2026-05-21T22:17:34Z

Implements the "all weights equal" Least Request Load Balancing policy in gRPC-Rust, in compliance with gRFC A48. The Least Request policy improves tail latencies in heterogeneous environments by tracking active request counts per endpoint and directing new requests to the backend with the lowest load.

Detailed Changes:

Core Load Balancing Policy (least_request.rs):
- Defined LeastRequestLoadBalancingConfig to parse and validate the choiceCount config parameter (default = 2, clamped from 2 to 10).
- Implemented LeastRequestBuilder registering policy name least_request_experimental.
- Implemented LeastRequestPolicy managing endpoint-level connections via ChildManager children delegating to pick_first.
- Maintained a persistent mapping of weak subchannel references to active request counters (subchannel_counters) so that outstanding request metrics survive picker updates and name re-resolutions.
- Implemented LeastRequestPicker utilizing a random sampling selection algorithm over choice_count subchannels.
Active Request Cancellation Safety:
- Identified and resolved a request counter leak bug where async task cancellations during dyn_invoke.await dropped the Pick closure without calling it.
- Implemented a custom, defusable ActiveRequestGuard using an AtomicBool inside LeastRequestPicker::pick. The guard guarantees that the active request count is decremented upon drop if the picker's on_complete callback is never invoked.
Channel & Service Config Integration:
- Registered the builder with the global LB registry in Channel::new inside channel.rs.
- Added CallbackRecvStream wrapping the stream in the channel's Invoke implementation to trigger on_complete callbacks when client streams are completed or dropped.
- Added LeastRequest variant to LbPolicyType enum in service_config.rs.
- Mapped LbPolicyType::LeastRequest configuration inside ResolverChannelController::update in channel.rs.
Test Additions & Verification:
- Added comprehensive unit tests in least_request.rs covering configuration parsing/clamping/validation, least request selection, tie-breaking, fewer subchannels than choice count, and cancellation drop-guard safety.
- Modified the InMemoryResolver in inmemory/mod.rs to dynamically set the LeastRequest load-balancing policy based on target URI path prefixes.
- Wrote a robust E2E integration test test_in_memory_least_request_load_balancing in inmemory/mod.rs verifying dynamic load balancing across multiple in-memory backends concurrently.

Motivation

Solution

Implements the "all weights equal" Least Request Load Balancing policy in gRPC-Rust, in compliance with gRFC A48. The Least Request policy improves tail latencies in heterogeneous environments by tracking active request counts per endpoint and directing new requests to the backend with the lowest load. Detailed Changes: 1. Core Load Balancing Policy (`least_request.rs`): - Defined `LeastRequestLoadBalancingConfig` to parse and validate the `choiceCount` config parameter (default = 2, clamped from 2 to 10). - Implemented `LeastRequestBuilder` registering policy name `least_request_experimental`. - Implemented `LeastRequestPolicy` managing endpoint-level connections via `ChildManager` children delegating to `pick_first`. - Maintained a persistent mapping of weak subchannel references to active request counters (`subchannel_counters`) so that outstanding request metrics survive picker updates and name re-resolutions. - Implemented `LeastRequestPicker` utilizing a random sampling selection algorithm over `choice_count` subchannels. 2. Active Request Cancellation Safety: - Identified and resolved a request counter leak bug where async task cancellations during `dyn_invoke.await` dropped the `Pick` closure without calling it. - Implemented a custom, defusable `ActiveRequestGuard` using an `AtomicBool` inside `LeastRequestPicker::pick`. The guard guarantees that the active request count is decremented upon drop if the picker's `on_complete` callback is never invoked. 3. Channel & Service Config Integration: - Registered the builder with the global LB registry in `Channel::new` inside `channel.rs`. - Added `CallbackRecvStream` wrapping the stream in the channel's `Invoke` implementation to trigger `on_complete` callbacks when client streams are completed or dropped. - Added `LeastRequest` variant to `LbPolicyType` enum in `service_config.rs`. - Mapped `LbPolicyType::LeastRequest` configuration inside `ResolverChannelController::update` in `channel.rs`. 4. Test Additions & Verification: - Added comprehensive unit tests in `least_request.rs` covering configuration parsing/clamping/validation, least request selection, tie-breaking, fewer subchannels than choice count, and cancellation drop-guard safety. - Modified the `InMemoryResolver` in `inmemory/mod.rs` to dynamically set the `LeastRequest` load-balancing policy based on target URI path prefixes. - Wrote a robust E2E integration test `test_in_memory_least_request_load_balancing` in `inmemory/mod.rs` verifying dynamic load balancing across multiple in-memory backends concurrently.

dfawley

Thanks for the PR! I haven't looked at the tests yet, but here's a first round of review comments.

dfawley · 2026-05-22T00:14:56Z

    #[default]
    PickFirst,
    RoundRobin,
+    LeastRequest,


This is a pre-defined list of LB policies limited to just PF and RR. But I think we can do this for now if we put a TODO to remove. cc @nathanielford for when he implements support for the newer field that lets you specify any policy.

I think I'd prefer to drop it now. It can be added back if it makes sense down the road.

dfawley · 2026-05-22T00:16:30Z

    fn build(&self, target: &Target, options: ResolverOptions) -> Box<dyn Resolver> {
        let path = target.path().strip_prefix('/').unwrap_or(target.path());
-        let ids: Vec<String> = path.split(',').map(|s| s.to_string()).collect();
+        let (lb_policy, rest) = if let Some(stripped) = path.strip_prefix("leastrequest/") {


The LB policy should get set by specifying a default service config, not through the in memory resolver. We don't have that yet, though, so this is fine for now since this module is unexported and unused except in tests.

Leaving this as a TODO comment.

dfawley · 2026-05-22T00:22:57Z

+    fn parse_config(
+        &self,
+        config: &ParsedJsonLbConfig,
+    ) -> Result<Option<<Self::LbPolicy as LbPolicy>::LbConfig>, String> {


I believe you can just name the type here if you want to avoid all the colons and "as"es.. Result<LeastRequestLoadBalancingConfig, String>?

dfawley · 2026-05-22T00:25:15Z

+        let choice_count = parsed.choice_count.min(10);
+        Ok(Some(LeastRequestLoadBalancingConfig { choice_count }))


Optional: A style choice. This could also be parsed.choice_count = ...min(10) and then return Ok(Some(parsed)). (parsed would need to be declared as mut)

dfawley · 2026-05-22T00:28:37Z

+pub(crate) struct LeastRequestPolicy {
+    child_manager: ChildManager<Endpoint>,
+    pick_first_builder: Arc<DynLbPolicyBuilder>,
+    choice_count: u32,


Consider storing an Option<Config> here instead of the flat config fields.

dfawley · 2026-05-22T22:47:49Z

+        PickResult::Pick(crate::client::load_balancing::Pick {
+            subchannel: selected.subchannel.clone(),
+            metadata: crate::metadata::MetadataMap::new(),


use please.

dfawley · 2026-05-22T22:59:08Z

+                        .state
+                        .picker
+                        .pick(&crate::core::RequestHeaders::default())


We should not be making picks for fake RPCs. If we can't map via the child itself, then we'll need another way to retrieve its active subchannel.

dfawley · 2026-05-22T23:00:01Z

+            // Clean up stale counters
+            self.subchannel_counters
+                .retain(|weak, _| weak.upgrade().is_some());


I think this would be better if it reused the known-active subchannels that we got above instead of keeping all the ones that happen to still upgrade.

dfawley · 2026-05-22T23:05:35Z

+        }
+        let aggregate_state = self.child_manager.aggregate_states();
+
+        if aggregate_state == ConnectivityState::Ready {


Optional: maybe let picker = if ... and send the update commonly between the if/else.

dfawley · 2026-05-22T23:07:35Z

+            let picker = self
+                .child_manager
+                .children()
+                .find(|cs| cs.state.connectivity_state == aggregate_state)
+                .map(|cs| cs.state.picker.clone())
+                .unwrap_or_else(|| {
+                    Arc::new(crate::client::load_balancing::QueuingPicker) as Arc<dyn Picker>
+                });


What picker is coming out of this? Is it just the first child whose state is the same as the aggregate state? We should round-robin over all of those children that match instead. You should ideally share the existing RoundRobin picker for this.

emil10001 · 2026-05-26T16:33:25Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new "least request" load balancing policy (least_request.rs) to the gRPC client, integrates it into the channel controller and service configuration, and adds comprehensive unit and integration tests. The code review feedback identifies several critical improvement opportunities: a bug where configuration updates (such as choice_count) are not propagated to the active picker because update_picker returns early when child states are unchanged; an issue where parent attributes are discarded when sharding resolver updates for child policies; and a recommendation to sample distinct subchannels (without replacement) to maximize the effectiveness of the "power of two choices" algorithm.

gemini-code-assist · 2026-05-26T16:35:05Z

+    fn update_picker(&mut self, channel_controller: &mut dyn ChannelController) {
+        if !self.child_manager.child_updated() {
+            return;
+        }


The update_picker method currently returns early if self.child_manager.child_updated() is false. However, if a resolver update only changes the choice_count configuration without changing any child states or endpoints, child_updated() will be false. This means the new choice_count will not be propagated to the active picker, and the policy will continue using the old choice_count.

To fix this, we should add a force parameter to update_picker and set it to true when called from resolver_update.

Suggested change

fn update_picker(&mut self, channel_controller: &mut dyn ChannelController) {

if !self.child_manager.child_updated() {

return;

}

fn update_picker(&mut self, channel_controller: &mut dyn ChannelController, force: bool) {

if !self.child_manager.child_updated() && !force {

return;

}

gemini-code-assist · 2026-05-26T16:35:06Z

+        let _ = self
+            .child_manager
+            .resolver_update(resolver_update, None, channel_controller);
+        self.update_picker(channel_controller);


Pass false to update_picker to match the new signature and avoid forcing an update when handling resolver errors.

Suggested change

self.update_picker(channel_controller);

self.update_picker(channel_controller, false);

gemini-code-assist · 2026-05-26T16:35:06Z

+            return Err(err.into());
+        }
+
+        self.update_picker(channel_controller);


Pass true to update_picker to force a picker update when a resolver update occurs, ensuring that any changes to choice_count are propagated even if child states are unchanged.

Suggested change

self.update_picker(channel_controller);

self.update_picker(channel_controller, true);

gemini-code-assist · 2026-05-26T16:35:06Z

+    ) {
+        self.child_manager
+            .subchannel_update(subchannel, state, channel_controller);
+        self.update_picker(channel_controller);


Pass false to update_picker to match the new signature.

Suggested change

self.update_picker(channel_controller);

self.update_picker(channel_controller, false);

gemini-code-assist · 2026-05-26T16:35:06Z

+
+    fn work(&mut self, channel_controller: &mut dyn ChannelController) {
+        self.child_manager.work(channel_controller);
+        self.update_picker(channel_controller);


Pass false to update_picker to match the new signature.

Suggested change

self.update_picker(channel_controller);

self.update_picker(channel_controller, false);

gemini-code-assist · 2026-05-26T16:35:06Z

+
+    fn exit_idle(&mut self, channel_controller: &mut dyn ChannelController) {
+        self.child_manager.exit_idle(channel_controller);
+        self.update_picker(channel_controller);


Pass false to update_picker to match the new signature.

Suggested change

self.update_picker(channel_controller);

self.update_picker(channel_controller, false);

gemini-code-assist · 2026-05-26T16:35:06Z

+            let update = ResolverUpdate {
+                attributes: crate::attributes::Attributes::default(),
+                endpoints: Ok(vec![e.clone()]),
+                service_config: update.service_config.clone(),
+                resolution_note: None,
+            };


When sharding the resolver update for child policies, the parent's update.attributes are discarded and replaced with Attributes::default(). This can cause important channel-level or security attributes resolved by the name resolver to be lost for the child policies.

We should propagate update.attributes.clone() to the child updates instead.

Suggested change

let update = ResolverUpdate {

attributes: crate::attributes::Attributes::default(),

endpoints: Ok(vec![e.clone()]),

service_config: update.service_config.clone(),

resolution_note: None,

};

let update = ResolverUpdate {

attributes: update.attributes.clone(),

endpoints: Ok(vec![e.clone()]),

service_config: update.service_config.clone(),

resolution_note: None,

};

gemini-code-assist · 2026-05-26T16:35:06Z

+        let sample_limit = self.choice_count.min(len);
+        let mut best_idx: Option<usize> = None;
+        let mut best_active_requests = usize::MAX;
+
+        for _ in 0..sample_limit {
+            let idx = if len == 1 {
+                0
+            } else {
+                rand::random_range(0..len)
+            };
+            let active_reqs = self.subchannels[idx]
+                .active_requests
+                .load(Ordering::Relaxed);
+            if best_idx.is_none() || active_reqs < best_active_requests {
+                best_idx = Some(idx);
+                best_active_requests = active_reqs;
+            }
+        }


The current random sampling algorithm uses sampling with replacement (rand::random_range(0..len) in a loop). When the number of subchannels is small (e.g., 2 or 3), there is a high probability of selecting the same subchannel multiple times, which reduces the effectiveness of the "power of two choices" algorithm.

We can improve this by tracking selected indices and ensuring we only sample distinct subchannels up to sample_limit.

let sample_limit = self.choice_count.min(len); let mut best_idx: Option<usize> = None; let mut best_active_requests = usize::MAX; let mut selected_indices = Vec::with_capacity(sample_limit); while selected_indices.len() < sample_limit { let idx = if len == 1 { 0 } else { rand::random_range(0..len) }; if !selected_indices.contains(&idx) { selected_indices.push(idx); let active_reqs = self.subchannels[idx] .active_requests .load(Ordering::Relaxed); if best_idx.is_none() || active_reqs < best_active_requests { best_idx = Some(idx); best_active_requests = active_reqs; } } }

emil10001 force-pushed the master branch from 24b4a7d to 87b8680 Compare May 21, 2026 22:50

emil10001 added 2 commits May 21, 2026 15:52

Merge branch 'grpc:master' into master

d4737a7

Merge branch 'master' into master

70d540d

dfawley reviewed May 22, 2026

View reviewed changes

gemini-code-assist Bot reviewed May 26, 2026

View reviewed changes

		let choice_count = parsed.choice_count.min(10);
		Ok(Some(LeastRequestLoadBalancingConfig { choice_count }))

	self.update_picker(channel_controller);
	self.update_picker(channel_controller, false);

	self.update_picker(channel_controller);
	self.update_picker(channel_controller, true);

Conversation

emil10001 commented May 21, 2026

Motivation

Solution

Uh oh!

dfawley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emil10001 commented May 26, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants