[sonic-grpc]: Add gNOI client and generated stubs to sonic-py-common by hdwhdw · Pull Request #28341 · sonic-net/sonic-buildimage

hdwhdw · 2026-07-09T22:01:25Z

Related: sonic-net/sonic-mgmt#26040
Supersedes: #27760

Why I did it

The PTF-free native gNOI test suite in sonic-mgmt needs an in-process Python gNOI client. The existing sonic-py-common distribution already reaches the SONiC host and runtime containers, making it the distribution point for the separate sonic_grpc import package.

How I did it

Added to src/sonic-py-common:

sonic_grpc.gnoi.GnoiClient, with System and File access over insecure TCP, unix://, or secure channels with caller-supplied credentials.
FakeGnoiServer test doubles for offline client tests.
Exact openconfig/gNOI v0.4.0 types, common, system, and file proto inputs, the Apache license, and immutable provenance.
A build_py hook that generates eight protobuf/gRPC modules with grpcio-tools==1.66.2.

Generation preserves upstream descriptor filenames while rewriting Python imports and message module identities to sonic_grpc.gnoi. The source distribution includes the proto inputs and client source. Python 3.9+ non-armhf wheels include the generated modules. Wheel installation does not run code generation; source and sdist builds generate the modules from the included inputs. Python 3.8 and armhf builds retain the existing sonic-py-common contents without sonic_grpc.

The distribution declares grpcio>=1.66.2 and protobuf>=5.29.6,<8 when it includes sonic_grpc.

Distribution

sonic_grpc ships through sonic-py-common's host and config-engine image propagation.
docker-sonic-mgmt installs the local Python 3 wheel under /opt/venv for native sonic-mgmt tests.
UDS access is deny-by-default: swss, syncd, and pmon receive no /var/run/gnmi mounts. Any in-box consumer must identify its owner and RPC use, document the threat model, and add only the required access in the consuming change.

The docker-sonic-mgmt test pipeline uses the PR target branch for sonic-mgmt and KVM test inputs, and the docker-mgmt source identity prevents a sonic-buildimage PR number from being interpreted as a sonic-mgmt PR.

Verification

41 focused tests pass on Python 3.13, including exact runtime-floor generation and configured arm-alias exclusion. The common 38-test subset passes on Python 3.9, 3.11, and 3.13.
A clean build with ambient grpcio-tools 1.51.1 generates modules with the pinned 1.66.2 tool.
An sdist-derived wheel contains all eight generated modules, and every generation input is included in the sonic-py-common content-SHA dependency set.
A protobuf 5.29.6-generated wheel imports with protobuf 7.35.1.
Python 3.8, armhf, and native/configured armv6l, armv7l, and armv8l builds exclude sonic_grpc; an armhf-built sdist retains its source and proto inputs.
Current-head Azure.sonic-buildimage run 1182341 passed all nine platform builds and all nine topology jobs.
Current-head Azure.docker-sonic-mgmt run 1182343 passed image build and seven of nine topology jobs. t1-lag-vpp repeatedly hit an existing VPP ARP setup failure, while t2 first hit a broad pre-existing Redis/LogAnalyzer failure and was canceled by Elastictest on retry after 652 tests passed. Its optional SBOM scan reports unchanged npm findings unrelated to this diff.
Current-head CodeQL, Semgrep, DCO, EasyCLA, file-protection, merge, and metadata checks pass.
Copilot reviewed all 28 current changed files with no findings and recommends human review.

Which release branch to backport

None.

Description for the changelog

Add a Python gNOI System and File client with reproducibly generated protobuf/gRPC modules to sonic-py-common for SONiC host, runtime-container, and sonic-mgmt consumers.

Dependencies

Consumed by [gnoi]: Add PTF-free native gNOI test suite (System.Time, File.Stat) sonic-mgmt#26040.
In-box UDS access policy: [doc]: Define in-box gNOI client access model SONiC#2482.

Tested branch

202605

Test result

202605: Local backport qualification passed on PR #29520 at 5a7511d3bffac9cba61c91e526da6c76a854cd74: 125 sonic-py-common tests per distribution (Bookworm, Trixie, Bullseye), runtime-floor wheel imports, and bounded management checks. Commands, environment, results, and limits. Full image/native ARM/device validation remains untested. This is backport evidence, not additional master validation.

202605 manual client-to-server smoke: Installed candidate 5a7511d3bffac9cba61c91e526da6c76a854cd74 passed three System.Time calls and a 38,912-byte File.Put with independent server filesystem verification. The real telemetry server was built locally from 202605 gitlink 06ec477f36e1f2ddf08ed71fd08eb0051b34c16b; this was not a full SONiC image. File.Get returned expected UNIMPLEMENTED. Disposable containers used plaintext loopback TCP with client authentication disabled. Physical devices, mTLS, host D-Bus, and reboot remain untested. The server logged a deferred double-close after successful Put. Full scope and observations; raw logs are retained locally.

Add a new standalone `sonic-grpc` Python distribution providing a native, in-process gNOI gRPC client for SONiC, and wire it into docker-sonic-mgmt. sonic_grpc.gnoi provides: - GnoiClient: a context-managed gRPC channel wrapper exposing gNOI service stubs as properties (client.system.*, client.file.*), over insecure TCP, unix:// UDS, or mTLS. Service-agnostic; new services plug in as properties. - Vendored flat gNOI proto stubs (System, File, types, common) regenerated against protobuf 4.x/5.x. No stub generation or gnoi.proto files needed at install time. - FakeGnoiServer test doubles for offline unit testing of gNOI-driven code. Packaged as its own minimal-dependency wheel (grpcio + protobuf only) so lightweight consumers - notably the docker-sonic-mgmt test image - can install it without pulling in a heavier package or its dependency tree. It installs anywhere, including: pip install "git+https://github.com/sonic-net/sonic-buildimage#subdirectory=src/sonic-grpc" Wired into docker-sonic-mgmt via rules/sonic-grpc.mk (SONIC_GRPC_PY3) and a uv-based install block in the Dockerfile (the image's /opt/venv has no pip, so the generic pip3 install_python_wheels macro is not used). grpcio is added to the image's package list as an explicit dependency. The gNOI client code originates from the framework in sonic-net#27760; this PR delivers it as a reusable standalone wheel that sonic-net#27760 and the sonic-mgmt gNOI test suite can both consume. Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

mssonicbld · 2026-07-09T22:01:33Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-09T22:01:56Z

Azure Pipelines successfully started running 1 pipeline(s).

The wheel-install block uses the copy_files macro, which must be imported at the top of the Dockerfile.j2 (as every other wheel-consuming Dockerfile does). Without the import the j2 template fails to render. Add the import. Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

mssonicbld · 2026-07-09T22:09:55Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-09T22:10:08Z

Azure Pipelines successfully started running 1 pipeline(s).

Every buildable component with GIT_CONTENT_SHA caching has a matching .dep file (included via Makefile.cache) that declares its DEP_FILES and cache mode. Add rules/sonic-grpc.dep for the new SONIC_GRPC_PY3 wheel, mirroring rules/sonic-py-common.dep, so the wheel participates in content-SHA build caching and rebuilds when its sources change. Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

mssonicbld · 2026-07-09T22:15:25Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-09T22:15:38Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR adds a new standalone Python wheel (sonic-grpc) under src/sonic-grpc/ that provides a minimal gNOI gRPC client framework (client wrapper + vendored gNOI proto stubs + test fakes) and wires that wheel into the docker-sonic-mgmt image build so consumers (notably sonic-mgmt’s native gNOI tests) can install it without pulling in a larger dependency tree.

Changes:

Introduces sonic_grpc.gnoi.GnoiClient plus a FakeGnoiServer test double, and vendors generated gNOI stubs (System/File/types/common) in the new sonic-grpc wheel.
Adds build rules for producing the new wheel and includes it in SONIC_PYTHON_WHEELS.
Updates docker-sonic-mgmt to include and install locally-built wheels via uv, and adds grpcio to the image’s Python dependencies.

Reviewed changes

Copilot reviewed 13 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/sonic-grpc/sonic_grpc/gnoi/types_pb2.py	Vendored generated protobuf stubs for gNOI types.
src/sonic-grpc/sonic_grpc/gnoi/types_pb2_grpc.py	Placeholder generated gRPC module for types (no services).
src/sonic-grpc/sonic_grpc/gnoi/common_pb2.py	Vendored generated protobuf stubs for gNOI common messages.
src/sonic-grpc/sonic_grpc/gnoi/common_pb2_grpc.py	Placeholder generated gRPC module for common (no services).
src/sonic-grpc/sonic_grpc/gnoi/system_pb2.py	Vendored generated protobuf stubs for gNOI System service messages.
src/sonic-grpc/sonic_grpc/gnoi/system_pb2_grpc.py	Vendored generated gRPC stubs for gNOI System service.
src/sonic-grpc/sonic_grpc/gnoi/file_pb2.py	Vendored generated protobuf stubs for gNOI File service messages.
src/sonic-grpc/sonic_grpc/gnoi/file_pb2_grpc.py	Vendored generated gRPC stubs for gNOI File service.
src/sonic-grpc/sonic_grpc/gnoi/client.py	Implements `GnoiClient` channel wrapper with per-service stub properties.
src/sonic-grpc/sonic_grpc/gnoi/testing.py	Implements `FakeGnoiServer` and `FakeSystemServicer` test doubles.
src/sonic-grpc/sonic_grpc/gnoi/init.py	Package entry point exporting `GnoiClient`.
src/sonic-grpc/sonic_grpc/init.py	Top-level `sonic_grpc` package docstring and future roadmap.
src/sonic-grpc/setup.py	Wheel metadata and runtime dependency declarations.
rules/sonic-grpc.mk	Adds wheel definition and registers it in `SONIC_PYTHON_WHEELS`.
rules/sonic-grpc.dep	Adds dependency tracking for caching/rebuild correctness.
rules/docker-sonic-mgmt.mk	Adds the `sonic-grpc` wheel to the docker-sonic-mgmt wheel set.
dockers/docker-sonic-mgmt/Dockerfile.j2	Installs locally-built wheels via `uv` and adds `grpcio` to dependencies.

Files not reviewed (4)

src/sonic-grpc/sonic_grpc/gnoi/common_pb2.py: Generated file
src/sonic-grpc/sonic_grpc/gnoi/file_pb2.py: Generated file
src/sonic-grpc/sonic_grpc/gnoi/system_pb2.py: Generated file
src/sonic-grpc/sonic_grpc/gnoi/types_pb2.py: Generated file

The SONiC bookworm/trixie wheel build rule runs `pytest` on each wheel's source unless <pkg>_TEST=n. sonic-grpc had no tests, so pytest collected 0 items and the wheel target failed. Add the gNOI client and FakeGnoiServer unit tests (31 tests, lifted from the framework in sonic-net#27760) under src/sonic-grpc/tests/, and declare a `testing` extra so the build's `pip install ".[testing]"` step is satisfied. Verified: `target/python-wheels/bookworm/sonic_grpc-1.0-py3-none-any.whl` now builds with `31 passed` in the test step. Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

mssonicbld · 2026-07-10T16:45:31Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-07-10T16:45:45Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Copilot reviewed 15 out of 20 changed files in this pull request and generated 2 comments.

Files not reviewed (4)

src/sonic-grpc/sonic_grpc/gnoi/common_pb2.py: Generated file
src/sonic-grpc/sonic_grpc/gnoi/file_pb2.py: Generated file
src/sonic-grpc/sonic_grpc/gnoi/system_pb2.py: Generated file
src/sonic-grpc/sonic_grpc/gnoi/types_pb2.py: Generated file

hdwhdw · 2026-07-14T18:20:30Z

@ronan-nexthop @ndas7 cannot tag you for review so FYI. Following a similar approach as #28339

@sneelam20 for review

ronan-nexthop · 2026-07-14T22:43:25Z

Interesting, is the a real use case for this on the box? If so the approach you take is good, althought I would have thought most of these gNOI operations would be handled off box? I appreciate gNOI RPC's don't change that frequently, in any case the regeneration story here isn't the best, appears file_pb2.py was generated by protobuf 5.29 tooling and grpcio-tools 1.70. system_pb2.py, types_pb2.py, common_pb2.py were generated by protobuf 4.25 tooling, and system_pb2_grpc.py by an older grpcio-tools. setup.py declares protobuf>=4.21 and unpinned grpcio. It would probably be good to have some form of regeneration script

hdwhdw · 2026-07-15T15:46:26Z

@ronan-nexthop Yes. There are Nvidia platform with multiple cards and a midplane network. Also Microsoft also have some orchestration that uses gnoi as a way to standardize on device operation.

This change is mostly for porting to sonic-mgmt instead of on the box container. (Although as a byproduct we also want it to be available on the box).

Agree with the generation. Probably can be built into the build system.

mssonicbld · 2026-09-17T21:16:04Z