grpc: crate maintenance by dfawley · Pull Request #2654 · grpc/grpc-rust

dfawley · 2026-05-23T00:14:03Z

Rework the codegen crate:

protoc-gen-rust-grpc is now a real Rust crate that is able to build protoc (for now) and our plugin binary.
grpc-protobuf-build uses protoc-gen-rust-grpc to produce the binaries, instead of requiring the user to put them in their path.
Reworked the caching of the binaries a little. I thought we'd be able to use the standard Rust caching, but it didn't work for various reasons. So maybe this isn't much better than what was there before. LMK.
We now run cargo build for interop inside the CI's environment instead of in the run environment which uses MSYS and for some reason doesn't like the way we're doing the caching. (Maybe this is a good enough reason to go back to how the cache was working before?)

Add a grpc/README.md, and add more info to Cargo.tomls, remove publish=false to allow them to be published, etc.

arjan-bal · 2026-05-25T17:53:34Z

+    // If a prebuilt binary directory is specified but was empty, save the newly compiled
+    // binaries there so they can be cached and reused.


The cargo docs discourage writing artifacts outside of OUT_DIR. I suspect this may also lead to race conditions if there are multiple versions of the crate being built concurrently.

Is this being done only to speed up CI? I feel it's better to manually build the binaries using CMake once in CI instead of having a flag in the production build script to support this. Please let me know if you think otherwise.

Also, why does the regular Rust build cache not work in CI? Is it something that can be addressed by spending more time on the configuration, or is it a limitation in Swatinem/rust-cache@v2?

There were many issues with the Swatinem/rust-cache@v2 crate. First, I had to give it a shared cache location, and I had to set cache-workspace-crates. I ran into more problems, and ultimately I couldn't get it working, and also the cache files were very large. I think maybe we should revert back to the manual caching that it was doing before, and just change the way the compilation worked to use cargo instead of cmake from the commandline?

So it's easy enough to build this crate, then find the binary locations from the OUT_DIR (there's functions in lib.rs to tell us that). So we can find them and copy them out. But I think we have to be able to read them from the cache location and copy them back from there as part of build.rs -- if we're doing it outside of the build process, I don't know how we can determine where we should put them, and compiling the crate to call the functions it has would require building them. So I guess we can't completely go back to the way it was before. We could move the copying-out into it, though, though I suspect it's fine.

Also maybe we should publish our binary and download it / protoc instead of building by default?

Also maybe we should publish our binary and download it / protoc instead of building by default?

In the present state, only the first build takes a while to compile protobuf from source, subsequent builds are using cargo's cache. Maybe the first build times are not a huge problem? If it is, we could add such an option in the future.

It's not the time alone; it's also the C++/cmake dependencies. But regarding the time: because Rust puts the target directory local -- unlike Go -- if you're using grpc in 5 different projects (or if you use worktrees like I do, and don't manually configure things like I do so your target directory is shared), then you will have to rebuild it in every directory.

dfawley · 2026-05-27T00:00:41Z

PR is updated. Here is the current state:

protoc-gen-rust-grpc is a rust crate that always does cmake for protoc and our plugin except when running on docs.rs or in CS (via a different env var)
grpc-protobuf-build now has three ways of working:

it can be configured in the CodeGen itself where the binaries are, and it will use those first, then.....
if a "build-plugin" feature flag is set, it will dep on the protoc-gen-rust-grpc crate and use the binaries it makes
if a custom env var is set (GRPC_RUST_PROTOC_DIR), it will look there for both binaries.
lastly, it will scan your path for both binaries

The examples are updated to use grpc-protobuf-build now when running a grpc example. build.rs uses it instead of needing to externally run protoc. It doesn't use the build-plugin flag which means the user needs protoc and our plugin in their path (or in GRPC_RUST_PROTOC_DIR; or they can edit cargo.toml to make it add the build-plugin feature back in).
I added a workflow that detects a release of the plugin, which builds all 10 of the targets that protoc is built for, and attaches those binaries to the release. Those don't compile protoc, just our plugin. And I added support for a --version flag that will print the release version, which it either reads from Cargo.toml when building from the rust crate, or from the tag name when building for a release.

LMK what you think.

arjan-bal

Mostly looks good, some minor comments.

arjan-bal · 2026-05-27T07:40:24Z

+    fn find_in_path(binary_name: &str) -> Option<PathBuf> {
+        if let Some(paths) = std::env::var_os("PATH") {
+            for path in std::env::split_paths(&paths) {
+                let candidate = path.join(binary_name);
+                if candidate.is_file() {
+                    return Some(candidate);
+                }
+            }
+        }
+        None
+    }


This may fail when the fix exists but is not executable or when running on windows which has a PATHEXT environment variable and may check the cwd. Gemini mentioned that the which crate is the idiomatic way of finding binary paths. We could use it to simplify our code.

Instead of checking, we could also call the protoc or protoc-gen-rust-grpc binaries with the --version flag to check if the binaries are available.

Changed to use which and to execute the commands.

arjan-bal · 2026-05-27T07:41:50Z

+            let compiled_protoc = PathBuf::from(protoc_gen_rust_grpc::protoc());
+            let compiled_plugin = PathBuf::from(protoc_gen_rust_grpc::protoc_gen_rust_grpc());
+            if compiled_protoc.exists() && compiled_plugin.exists() {
+                // The files may not exist if a build setting instructed


Is this sentence incomplete?

Oops I forgot to finish my thought. Finished.

arjan-bal · 2026-05-27T09:05:50Z

+are compiled, and the functions `protoc` and `protoc_gen_rust_grpc` can be used
+to find their locations.
+
+## Building binaries manually


Just a thought that came up while reviewing the code: We could create a Rust binary crate that statically links to the C++ code (compiled via the cmake crate). The Rust binary would act as a thin wrapper, using FFI to forward command-line arguments to the C++ logic. This way, users can seamlessly build and install the protoc plugin using cargo install rather than invoking cmake manually.

Oh I like that idea! Except that we need two binaries for now. I talked to Em and he's open to the idea of publishing something similar for protoc itself.

(Sorry, to be clear; he was open to publishing something that can build protoc so we don't have to have something that builds both protoc and our plugin. I haven't talked to him about the FFI idea yet.)

arjan-bal · 2026-05-27T09:43:59Z

-              break
+          mkdir -p protoc-cache
+          # Find where cmake built the binaries.
+          find target/ -name "protoc" -o -name "protoc.exe" -o -name "protoc-gen-rust-grpc" -o -name "protoc-gen-rust-grpc.exe" | while read -r file; do


Since we're building protoc from source, we shouldn't need taiki-e/install-action@protoc, right?

Definitely not while building the plugin. I removed it from there. But the other ones might, because AIUI tonic requires protoc in your path. But maybe our CI doesn't except for the codegen job? I'll play around with this some more.

Aha! It's needed for the other jobs because the install rule includes standard proto include files -- not just the protoc binary.

arjan-bal · 2026-05-27T11:09:22Z


 [build-dependencies]
 tonic-prost-build = { path = "../tonic-prost-build" }
+grpc-protobuf-build = { path = "../grpc-protobuf-build", default-features = false }


Since the feature flags for the gRPC examples are enabled by default and the grpc-protobuf-build feature is disabled, running the Tonic examples now requires protoc and protoc-gen-rust-grpc to be installed and in the user's PATH. For instance, to run the Tonic routeguide server, I had to execute:

cargo run --bin routeguide-server --no-default-features --features=routeguide

Is this UX acceptable for Tonic users? If we want to improve this, we have a couple of options:

Use required-features: Disable the gRPC example features by default and add required-features` to each gRPC example binary. This ensures Cargo prints a clear error message instructing users to enable the exact features required.

Use a separate crate: Move the gRPC examples into a dedicated crate that has the necessary gRPC features enabled by default.

Thanks for pointing this out. I'd actually like even the grpc examples to be runnable without having proto in your path. So what I've done is added an env var (GRPC_RUST_REGENERATE_PROTO) to build.rs to cause it to regenerate the protobuf output. And if it's not set, then nothing is regenerated and nothing fails if the binaries aren't there.

arjan-bal · 2026-05-27T11:15:58Z

+    // If a prebuilt binary directory is specified but was empty, save the newly compiled
+    // binaries there so they can be cached and reused.


Also maybe we should publish our binary and download it / protoc instead of building by default?

In the present state, only the first build takes a while to compile protobuf from source, subsequent builds are using cargo's cache. Maybe the first build times are not a huge problem? If it is, we could add such an option in the future.

dfawley

Thanks for the review!

dfawley · 2026-05-27T14:49:57Z

+            let compiled_protoc = PathBuf::from(protoc_gen_rust_grpc::protoc());
+            let compiled_plugin = PathBuf::from(protoc_gen_rust_grpc::protoc_gen_rust_grpc());
+            if compiled_protoc.exists() && compiled_plugin.exists() {
+                // The files may not exist if a build setting instructed


Oops I forgot to finish my thought. Finished.

dfawley · 2026-05-27T15:13:31Z


 [build-dependencies]
 tonic-prost-build = { path = "../tonic-prost-build" }
+grpc-protobuf-build = { path = "../grpc-protobuf-build", default-features = false }


Thanks for pointing this out. I'd actually like even the grpc examples to be runnable without having proto in your path. So what I've done is added an env var (GRPC_RUST_REGENERATE_PROTO) to build.rs to cause it to regenerate the protobuf output. And if it's not set, then nothing is regenerated and nothing fails if the binaries aren't there.

dfawley · 2026-05-27T15:14:06Z

+    fn find_in_path(binary_name: &str) -> Option<PathBuf> {
+        if let Some(paths) = std::env::var_os("PATH") {
+            for path in std::env::split_paths(&paths) {
+                let candidate = path.join(binary_name);
+                if candidate.is_file() {
+                    return Some(candidate);
+                }
+            }
+        }
+        None
+    }


Changed to use which and to execute the commands.

dfawley · 2026-05-27T15:15:54Z

+are compiled, and the functions `protoc` and `protoc_gen_rust_grpc` can be used
+to find their locations.
+
+## Building binaries manually


Oh I like that idea! Except that we need two binaries for now. I talked to Em and he's open to the idea of publishing something similar for protoc itself.

dfawley · 2026-05-27T15:18:08Z

-              break
+          mkdir -p protoc-cache
+          # Find where cmake built the binaries.
+          find target/ -name "protoc" -o -name "protoc.exe" -o -name "protoc-gen-rust-grpc" -o -name "protoc-gen-rust-grpc.exe" | while read -r file; do


Definitely not while building the plugin. I removed it from there. But the other ones might, because AIUI tonic requires protoc in your path. But maybe our CI doesn't except for the codegen job? I'll play around with this some more.

…s) on a cache hit

LucioFranco

Overall LGTM have a few changes I think we should get in first

LucioFranco · 2026-05-27T16:38:31Z

+    println!("cargo:rerun-if-env-changed=GRPC_RUST_REGENERATE_PROTO");
+    let grpc_helloworld = env::var_os("CARGO_FEATURE_GRPC_HELLOWORLD").is_some();
+    let grpc_routeguide = env::var_os("CARGO_FEATURE_GRPC_ROUTEGUIDE").is_some();
+
+    if (grpc_helloworld || grpc_routeguide) && env::var_os("GRPC_RUST_REGENERATE_PROTO").is_some() {
+        let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap());
+
+        grpc_protobuf_build::CodeGen::new()
+            .output_dir(manifest_dir.join("src/grpc-helloworld/generated"))
+            .input("helloworld.proto")
+            .include(manifest_dir.join("proto/helloworld"))
+            .client_only()
+            .compile()
+            .unwrap();
+
+        grpc_protobuf_build::CodeGen::new()
+            .output_dir(manifest_dir.join("src/grpc-routeguide/generated"))
+            .input("route_guide.proto")
+            .include(manifest_dir.join("proto/routeguide"))
+            .client_only()
+            .compile()
+            .unwrap();
+    }


I think we should get some of these env var settings in a readme for the examples folder?

The CARGO_FEATURE env vars are cargo magic. I added something for the GRPC_RUST_REGENERATE_PROTO.

LucioFranco · 2026-05-27T16:41:40Z

+        #[cfg(feature = "build-plugin")]
+        {
+            let compiled_protoc = PathBuf::from(protoc_gen_rust_grpc::protoc());
+            let compiled_plugin = PathBuf::from(protoc_gen_rust_grpc::protoc_gen_rust_grpc());
+            if compiled_protoc.exists() && compiled_plugin.exists() {
+                // The files may not exist if a build setting instructed protoc-gen-rust-grpc to
+                // skip the C++ build (DOCS_RS / our CI setting).
+                return Ok((compiled_protoc, compiled_plugin));
+            }
+        }


we need to add a NO_VENDOR style env var (prob something like GRPC_RUST_NO_BUILD_PLUGIN) that even if build-plugin is enabled you can still disable it since feature flags are additive across the whole workspace if another crate enables it it may get enabled in your code. Can see the no vendor version here for openssl

There's already GRPC_RUST_SKIP_CPP_BUILD in protoc-gen-rust-grpc -- should we just give that a better name and document it?

Otherwise, because the cmake happens in build.rs, there is nothing we can do in this crate to stop it from happening.

LucioFranco · 2026-05-27T16:42:25Z

 *
 */

+//! Protobuf integration for the [`grpc` crate](grpc).


Suggested change

//! Protobuf integration for the [`grpc` crate](grpc).

//! Protobuf integration for the [`grpc`](grpc) crate.

LucioFranco · 2026-05-27T16:43:38Z


-
-(cd interop && cargo build --bins)
+# Skip building in github actions as the workflow already builds it (to


is this still needed? says to make sure cache works?

dfawley added this to the grpc-next milestone May 23, 2026

dfawley requested a review from arjan-bal May 23, 2026 00:14

dfawley assigned arjan-bal May 23, 2026

dfawley added the A-grpc-next label May 23, 2026

arjan-bal reviewed May 25, 2026

View reviewed changes

arjan-bal assigned dfawley May 25, 2026

dfawley force-pushed the alltogether branch 3 times, most recently from f53df38 to 9613c62 Compare May 26, 2026 22:28

dfawley added 2 commits May 26, 2026 15:43

Add readme and tidy up Cargo.tomls

945d424

rework codegen

a8d9400

dfawley force-pushed the alltogether branch 2 times, most recently from 40b30f9 to a356b0a Compare May 26, 2026 23:01

dfawley added 2 commits May 26, 2026 16:47

add protoc-gen-rust-grpc release workflow

2786a46

examples rework

625297b

dfawley force-pushed the alltogether branch from a356b0a to 625297b Compare May 26, 2026 23:52

arjan-bal mentioned this pull request May 27, 2026

Add grpc-rust documentation: language landing page, quickstart, and route guide (#1511) grpc/grpc.io#1517

Merged

arjan-bal reviewed May 27, 2026

View reviewed changes

review comments

c9d3d31

dfawley commented May 27, 2026

View reviewed changes

dfawley added 4 commits May 27, 2026 09:01

re-add protoc

1d38e50

fmt

9276224

remove manual trigger of binary upload

e7243f8

fix CI to skip rebuilding the plugin (and some other unnecessary step…

8383412

…s) on a cache hit

LucioFranco requested changes May 27, 2026

View reviewed changes

dfawley added 4 commits May 27, 2026 10:00

interop stop building plugin

edc5166

windows dlls?

22d9fa2

comments

bff69b2

windows!

cac8961

dfawley added 7 commits May 27, 2026 10:31

redundant

26226d2

windows? and udeps

899845e

:

3f051b0

actual udeps fix

8cc3853

back to skipping the build in CI interop script

9381eb1

combine ifs

cea737e

rename env var and document it

7119ac6

LucioFranco approved these changes May 27, 2026

View reviewed changes

missing sentence fragment

5e2bcbb

dfawley enabled auto-merge (squash) May 27, 2026 20:46

dfawley merged commit f7e89d1 into grpc:master May 27, 2026
22 checks passed

		// If a prebuilt binary directory is specified but was empty, save the newly compiled
		// binaries there so they can be cached and reused.

	//! Protobuf integration for the [`grpc` crate](grpc).
	//! Protobuf integration for the [`grpc`](grpc) crate.



		(cd interop && cargo build --bins)
		# Skip building in github actions as the workflow already builds it (to

Conversation

dfawley commented May 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dfawley commented May 27, 2026

Uh oh!

arjan-bal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dfawley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LucioFranco left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment