nbes

An in-memory BES backend that forwards streams to multiple backends.

This project is a workaround for Bazel not supporting multiple --bes_backend args: bazelbuild/bazel#10908.

Features

• Run locally next to Bazel, optionally connecting over a unix domain socket
• Host as a service accepting many invocations
• Configure auth schemes per backend (e.g., remote headers, mTLS)
• Choose which backends to block on vs. upload asynchronously
• Supports TLS termination

Quick start

Download the latest nbes from the releases.

Start nbes and point it one or more BES backends. In this example, nbes points to an unauthenticated BuildBuddy endpoint.

nbes --listen 0.0.0.0:9000 --bes_backend=grpcs://remote.buildbuddy.io

Invoke Bazel with nbes as the backend.

bazel build //... --bes_backend=grpc://127.0.0.1:9000

Run nbes --help to see all options or see the usage documentation below.

Block vs. async

By default, nbes serializes the reponses from each backend into one response back to the client. A request succeeds if all backend requests succeed. If any backend returns a failure, so does nbes.

When async=true, the responses for that backend will be processed asynchronously. The client will not know whether the stream failed.

You can block on select backends and upload asynchronously to others.

Usage

Options can be specified as command-line args or in a configuration file. If both are specified, the cli args will take priority. Backend specifications will be combined.

CLI

Arg	Default	Description
--listen	0.0.0.0:9000	The socket address or unix domain socket path to listen on. E.g., 0.0.0.0:9000 or unix:/path/to/socket. When using a unix domain socket, invoke Bazel with a --bes_proxy arg and a --bes_backend arg, which is required but won't be used. E.g., bazel build //... --bes_proxy=unix:/path/to/socket --bes_backend=grpc://127.0.0.1.
--config	None	Path to a configuration file. See reference.
--bes_backend	[]	A BES backend to forward to. This argument may be repeated to declare multiple backends. In the simplest form, this is just a url, e.g. --bes_backend grpcs://foo.backend.org More options may be set for a backend using the comma-separated value form, e.g., --bes_backend name=buildbuddy,endpoint=grpcs://remote.buildbuddy.io See the full set of backend options. Support schemes include grpc and grpcs.
--server_tls_certificate	None	File path to the server PEM certificate to enable TLS.
--server_tls_key	None	File path to the server PEM private key for TLS.
--tls_certificate	[]	File path to a PEM certificate that is trusted to sign server certificates. Can be repeated for multiple certificates.
--concurrency_limit_per_connection	Unlimited	The number of concurrent inbound requests per connection. When used in combination with --load_shed_requests, requests will be rejected with a resource exhausted error instead of buffering when the concurrency limit is reached.
--load_shed_requests	false	Reject requests when the concurrency limit is reached. See --concurrency_limit_per_connection.
--max_concurrent_streams	Unlimited	Limit concurrent HTTP/2 streams per connection. Sets SETTINGS_MAX_CONCURRENT_STREAMS.
--max_connection_age	Unlimited	The maximum duration in seconds that a connection may exist.
--max_connection_age_grace	Unlimited	The maximum duration in seconds that a connection may continue to exist after a graceful shutdown period. This takes effect after the duration in --max_connection_age.

--bes_backend options

Specified as csv key-value pairs in a --bes_backend argument. E.g.,

nbes --bes_backend=name=foo,endpoint=grpcs://foo.backend.org,async=true

Name	Default	Description
name	Auto-generated if not provided	Name of the BES backend. Used to uniquely identify it, appears in logs.
endpoint	None	Endpoint of the backend in the form [SCHEME://]HOST[:PORT]. E.g., endpoint=grpcs://foo.backend.org
remote_header	[]	Remote header to send to the backend. May be repeated to declare multiple headers. E.g., remote_header=x-foobar-api-key=abcd1234
remote_header_file	[]	Remote header where the value is stored in a file. May be repeated to declare multiple headers. E.g., remote_header=x-foobar-api-key=/path/to/secret
async	false	Handle responses asynchronously instead of blocking on them to send back to the client. If the stream fails, the client won't be notified. Defaults to blocking behaviour (async=false).
tls_client_certificate	None	File path to a TLS PEM certificate used to identify the client to the backend. Use this when the backend requires mTLS authentication.
tls_client_key	None	File path to a TLS PEM private key used to identify the client to the backend. Use this when the backend requires mTLS authentication.
connect_timeout	Unlimited	Max duration in seconds to connect to the backend before timing out. Deafults to no timeout.
request_timeout	Unlimited	Max duration in seconds for requests to the backend before timing out. Defaults to no timeout.
request_buffer_size	500	The maximum number of requests that can be buffered waiting to send to this backend before adding back pressure on incoming requests. Increase this on slower backends that bottleneck other backends from receiving requests, at the cost of more memory usage. Defaults to 500.

Config file

Load options from a configuration file.

nbes -c config.yaml

Config reference

# Configuration for the nbes server
server:
  # Address to listen on, optionally a unix domain socket, e.g., unix:/path/to/socket
  listen: <SOCKET_ADDR>

  # TLS configuration. Set this to enable TLS (optional)
  tls:
    certificate: <PATH_TO_CERT>
    key: <PATH_TO_KEY>

  # Number of concurrent inbound requests per connection (optional)
  concurrency_limit_per_connection: <NUMBER>

  # Reject requests when the concurrency limit is reached (optional)
  load_shed_requests: <BOOL>

  # Limit concurrent HTTP/2 streams per connection (optional)
  max_concurrent_streams: <NUMBER>

  # The maximum duration in seconds that a connection may exist (optional)
  max_connection_age: <NUMBER>

  # The maximum duration in seconds that a connection may continue to exist after a graceful shutdown (optional)
  max_connection_age_grace: <NUMBER>

# BES backends to forward to
bes_backends:
  # Specify as an endpoint [SCHEME://]HOST[:PORT]
  - <URL>

  # ...or as an object to set additional properties
    # Identifier for the backend
  - name: <STRING> 

    # [SCHEME://]HOST[:PORT]
    endpoint: <URL>

    # Whether to handle responses asynchronously vs block to send back to the client (optional)
    async: <BOOL>

    # Remote headers to send to the backend (optional)
    remote_headers:
      <KEY>: <VALUE>

    # Remote headers where the value is stored in a file (optional)
    remote_header_files:
      <KEY>: <PATH>

    # TLS cert to use for mTLS auth (optional)
    tls_client_certificate: <PATH>

    # TLS key to use for mTLS auth (optional)
    tls_client_key: <PATH>

    # Max duration in seconds to connect to the backend before timing out (optional)
    connect_timeout: <NUMBER>

    # Max duration in seconds for requests to the backend before timing out (optional)
    request_timeout: <NUMBER>

    # The maximum number of requests that can be buffered waiting to send to this
    # backend before adding backpressure on incoming requests (optional)
    # Defaults to 500
    request_buffer_size: <NUMBER>

Deployment

The nbes service is released both as a binary under releases and as as an image. It can be deployed several ways.

systemd

Download the nbes binary from releases and create a config file.

Copy the following unit file to /etc/systemd/system/nbes.service substituting in the binary and config file paths.

[Unit]
Description=nbes service
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
ExecStart=/usr/local/bin/nbes -c /etc/nbes/config.yaml

[Install]
WantedBy=multi-user.target

Reload the systemd daemon.

sudo systemctl daemon-reload

Start the nbes service.

sudo systemctl start nbes.service

docker

Create a configuration file config.yaml. Then run:

docker container run -v ./config.yaml:/config.yaml -p 9000:9000 ghcr.io/kormide/nbes:latest

docker compose

compose.yaml

services:
  nbes:
    image: ghcr.io/kormide/nbes:latest
    ports:
      - 9000:9000
    volumes:
      - <PATH_TO_YOUR_CONFIG>:/config.yaml
    restart: always

Then run:

docker compose up -d

kubernetes

TODO

Compatibility

Note

Only BuildBuddy has been tested at this point because they are the only vendor with a public facing BES backend. If you have confirmed that another vendor works, please create a pull request to update this table. If you are a vendor and can grant me access for testing, I can add your backend.

BES Backend	Compatible
BuildBuddy	✅