diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1c9c3007..251d7118 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -183,8 +183,8 @@ jobs: - name: Verify protobufs up to date run: | make proto-gen - if [ -n "$(git status --porcelain -- pkg/server/proto)" ]; then - git status --short -- pkg/server/proto + if [ -n "$(git status --porcelain -- gen)" ]; then + git status --short -- gen exit 1 fi diff --git a/.github/workflows/codex-review.yml b/.github/workflows/codex-review.yml index 5150314b..4fe6dc47 100644 --- a/.github/workflows/codex-review.yml +++ b/.github/workflows/codex-review.yml @@ -85,7 +85,7 @@ jobs: STEP 1 — Enumerate. List every file changed by this PR, grouped by directory. Note (and skip review of) any generated files: anything under config/crd/, - config/rbac/role.yaml, pkg/server/proto/, any file named zz_generated*. These + config/rbac/role.yaml, gen/, any file named zz_generated*. These are regenerated, not hand-edited; findings on them are noise. STEP 2 — For each non-generated changed file, walk all four checks below. Even @@ -148,7 +148,7 @@ jobs: 4. Quality. New/changed behavior has tests (control-flow paths exercised, ideally over the wire); errors are wrapped not swallowed; code lands in the right package per the layout in CONTRIBUTING.md; generated code (config/crd, - zz_generated*, pkg/server/proto) is regenerated, not hand-edited. + zz_generated*, gen/) is regenerated, not hand-edited. ## Format constraints diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 46965b7f..17cfcd1a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -223,13 +223,13 @@ See the README's "Repository layout" for the full map. In short: | Controller / reconciler logic | `internal/controller/` | | Controller ↔ server HTTP wire type | `internal/controlplaneapi/` | | Pod-binding annotation / metadata contract | `internal/enginebinding/` | -| gRPC handlers, server wiring | `pkg/server/` | -| Cache-state index logic | `pkg/index/` | -| Mutable-slot rendering (the wedge) | `pkg/render/` | +| gRPC handlers, server wiring | `internal/server/` | +| Cache-state index logic | `internal/index/` | +| Planned reusable rendering API (reserved; not implemented) | `pkg/render/` | | Stable adapter extension contract | `pkg/adapters/{backend,runtime}/` | | Shipping adapter implementation / registration | `internal/adapters/builtin/` | -| Engine KV-event ingest implementation | `pkg/adapters/engine/` (pending the documented `internal/subscriber/` move) | -| Engine egress client (pre-tokenized request → engine; harness / benchmark, no binary owner) | `pkg/adapters/engineclient/` | +| Engine KV-event ingest implementation | `internal/subscriber/` | +| Engine egress client (pre-tokenized request → engine; harness / benchmark, no binary owner) | `pkg/engineclient/` | | The gRPC contract | `proto/` → then `make proto-gen` | Each package's `doc.go` (or package comment) states which binary owns it or why @@ -237,7 +237,7 @@ it is a supported external Go API. Follow [`docs/design/repository-boundaries.md`](docs/design/repository-boundaries.md) for dependency direction and the staged internal-package migration. -**Generated code** — `config/crd/`, `config/rbac/role.yaml`, `api/**/zz_generated*.go`, `pkg/server/proto/` — is committed but never hand-edited. Regenerate and commit it with the source change (`make pre-pr` verifies there's no drift). +**Generated code** — `config/crd/`, `config/rbac/role.yaml`, `api/**/zz_generated*.go`, `gen/` — is committed but never hand-edited. Regenerate and commit it with the source change (`make pre-pr` verifies there's no drift). **gRPC contract:** when you change `proto/`, update [`docs/design/grpc-contract.md`](docs/design/grpc-contract.md) in the same commit so the design doc stays accurate. The pre-commit hook blocks a commit that touches a `.proto` without touching that doc (override with `--no-verify` only if the change truly doesn't affect the contract). diff --git a/Makefile b/Makefile index 6c82f339..1a12b5d4 100644 --- a/Makefile +++ b/Makefile @@ -40,7 +40,7 @@ SBOM_TAG := $(subst /,_,$(TAG)) MINIMAL_IMAGE_DOCKERFILE ?= dockerfiles/Dockerfile MINIMAL_RUNTIME_BASE ?= gcr.io/distroless/static-debian13:nonroot@sha256:f7f8f729987ad0fdf6b05eeeae94b26e6a0f613bdf46feea7fc40f7bd72953e6 -version_pkg = $(MODULE)/pkg/version +version_pkg = $(MODULE)/internal/version LD_FLAGS += -X '$(version_pkg).GitVersion=$(TAG)' LD_FLAGS += -X '$(version_pkg).GitCommit=$(shell git rev-parse HEAD 2>/dev/null || echo unknown)' @@ -313,7 +313,7 @@ vulncheck: $(LOCALBIN) ## Scan dependencies + reachable code for known Go vulner COVER_MIN ?= 90 COVER_PROFILE ?= cover.out COVER_PROFILE_LOGIC ?= cover.logic.out -COVER_EXCLUDE := pkg/server/proto/|zz_generated|/cmd/|/hack/|pkg/testing/ +COVER_EXCLUDE := gen/|zz_generated|/cmd/|/hack/|internal/testutil/ .PHONY: cover cover: ## Run tests with coverage and print the per-function report (logic packages, cross-package counted). @@ -539,7 +539,7 @@ install-hooks: ## Install git hooks (vendor-neutral naming guard) via core.hooks .PHONY: verify-naming verify-naming: ## Fail if core-identity files reference OCI/Oracle (see CONTRIBUTING.md). @bad=$$(grep -rniEI '\boci\b|oci\.com|oraclecloud|\boracle\b' \ - api proto pkg/server/proto config/crd config/rbac config/default config/manager config/observability config/samples config/server config/webhook config/certmanager config/overlays docs/observability internal PROJECT go.mod 2>/dev/null || true); \ + api proto gen pkg config/crd config/rbac config/default config/manager config/observability config/samples config/server config/webhook config/certmanager config/overlays docs/observability internal PROJECT go.mod 2>/dev/null || true); \ if [ -n "$$bad" ]; then \ echo "✗ OCI/Oracle reference in core-identity files (banned per CONTRIBUTING.md):"; \ echo "$$bad" | sed 's/^/ /'; \ @@ -624,7 +624,7 @@ ci: verify-naming verify-no-internal-refs verify-dco test-dco reuse-lint verify- .PHONY: pre-pr pre-pr: ci ## Pre-PR gate: CI gate + generated-code drift check + sample admission check + review checklist. @$(MAKE) --no-print-directory manifests generate proto-gen >/dev/null - @gen='config/crd config/rbac/role.yaml config/webhook/manifests.yaml api/v1alpha1/zz_generated.deepcopy.go pkg/server/proto'; \ + @gen='config/crd config/rbac/role.yaml config/webhook/manifests.yaml api/v1alpha1/zz_generated.deepcopy.go gen'; \ if ! git diff --quiet -- $$gen; then \ echo "✗ generated-code drift — regenerate and commit these files:"; \ git --no-pager diff --name-only -- $$gen; \ diff --git a/README.md b/README.md index 0f855c6b..994a335c 100644 --- a/README.md +++ b/README.md @@ -91,17 +91,24 @@ the operator CLI and the CRDs. **`inferencecache-server`** (`cmd/server`) — gRPC policy server + cache-state index + metrics - `cmd/server/` — gRPC + HTTP server entrypoint -- `pkg/server/` — gRPC service (`LookupRoute`, `RenderTemplate`, …), health, metrics +- `internal/server/` — gRPC service (`LookupRoute`, `RenderTemplate`, …), health, metrics - `proto/` (+ generated stubs) — the gRPC contract -- `pkg/index/` — cache-state aggregator (`CacheIndex`) -- `pkg/render/` — mutable-slot prompt rendering engine (the wedge); importable library -- `pkg/adapters/engine/` — engine KV-event hook (feeds the index) +- `internal/index/` — cache-state aggregator (`CacheIndex`) +- `pkg/render/` — reserved path for a planned reusable renderer; no stable API yet + +**`kvevent-subscriber`** (`cmd/kvevent-subscriber`) — engine-side KV-event ingestion +- `cmd/kvevent-subscriber/` — composition and lifecycle entrypoint +- `internal/subscriber/` — engine KV-event hook (feeds the index) + +**Engine client library** — narrow pre-tokenized OpenAI-compatible completion client +- `pkg/engineclient/` — public `EngineClient` contract and `/v1/completions` implementation +- `internal/canary/` — repository-owned prefix-cache probe and live canary **`inferencecache`** (`cmd/inferencecache`) — operator CLI; `doctor` runs a read-only pre-flight diagnostic - `cmd/inferencecache/` — cobra entrypoint -- `pkg/cli/doctor/` — diagnostic checks + output formatters (see `docs/cli/doctor.md`) +- `internal/cli/doctor/` — diagnostic checks + output formatters (see `docs/cli/doctor.md`) -**Shared** — `pkg/version/`, `hack/`, `dockerfiles/`, `.githooks/` +**Shared** — `internal/version/`, `hack/`, `dockerfiles/`, `.githooks/` Private cross-binary HTTP DTOs live in `internal/controlplaneapi/`; pod-binding metadata shared by admission and reconcilers lives in `internal/enginebinding/`. diff --git a/REUSE.toml b/REUSE.toml index f78436f2..e6cf5235 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -15,8 +15,8 @@ path = [ "**/*.table", "**/*.xml", "LICENSE", - "pkg/adapters/engine/testdata/*.txt", - "pkg/cli/doctor/output/testdata/*.txt", + "internal/cli/doctor/output/testdata/*.txt", + "internal/subscriber/testdata/*.txt", ] precedence = "override" SPDX-FileCopyrightText = "2026 The inference-cache Authors" @@ -29,7 +29,7 @@ path = [ "config/crd/bases/*.yaml", "config/rbac/role.yaml", "config/webhook/manifests.yaml", - "pkg/server/proto/**/*.pb.go", + "gen/**/*.pb.go", ] precedence = "override" SPDX-FileCopyrightText = "2026 The inference-cache Authors" diff --git a/api/v1alpha1/cachepolicy_types.go b/api/v1alpha1/cachepolicy_types.go index dbdba0a0..afae0344 100644 --- a/api/v1alpha1/cachepolicy_types.go +++ b/api/v1alpha1/cachepolicy_types.go @@ -10,7 +10,7 @@ import ( ) // CachePolicyEvictionAlgorithm identifies an index entry-eviction algorithm. -// Each value has a corresponding implementation in pkg/index; the enum grows +// Each value has a corresponding implementation in internal/index; the enum grows // as new algorithms land. The choice is per-namespace: the controller flattens // it (lower-cased) into ResolvedPolicy.Eviction. The index reads it when the // entry cap is exceeded (to order victims) and, for LFU, on the lookup path (to diff --git a/api/v1alpha1/remaining_crds_types_test.go b/api/v1alpha1/remaining_crds_types_test.go index fc8ae385..57737cbf 100644 --- a/api/v1alpha1/remaining_crds_types_test.go +++ b/api/v1alpha1/remaining_crds_types_test.go @@ -26,7 +26,7 @@ func TestRemainingCRDSchemas(t *testing.T) { requireRequired(t, policySchema, "spec") policySpec := mustPath[map[string]any](t, policySchema, "properties", "spec") // Eviction selects the index cap-based eviction algorithm. Both values are - // implemented in pkg/index (LRU-by-lastSeen and LFU-by-access-count) and the + // implemented in internal/index (LRU-by-lastSeen and LFU-by-access-count) and the // controller propagates the choice (lower-cased) into ResolvedPolicy. evictionSchema := mustProperty(t, policySpec, "eviction") requireEnum(t, evictionSchema, []string{"LRU", "LFU"}) diff --git a/buf.yaml b/buf.yaml index 2affdba1..2f401d86 100644 --- a/buf.yaml +++ b/buf.yaml @@ -32,3 +32,13 @@ lint: # PREFIX_EVICTED, REPLICA_UPDATED, ALL_CLEARED) instead of a TYPE_ prefix; # the zero value is TYPE_UNSPECIFIED (ENUM_ZERO_VALUE_SUFFIX still enforced). - ENUM_VALUE_PREFIX +breaking: + use: + - FILE + ignore_only: + # The generated Go API intentionally moved out of the server implementation + # before the project was formally deployed. All other FILE compatibility + # rules remain enabled, while the repository boundary test pins this option + # to its new public gen/ path so a later import-path change still fails CI. + FILE_SAME_GO_PACKAGE: + - proto/inferencecache/v1alpha1/inferencecache.proto diff --git a/cmd/controller/main.go b/cmd/controller/main.go index 542d341a..42248774 100644 --- a/cmd/controller/main.go +++ b/cmd/controller/main.go @@ -24,10 +24,9 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" builtinadapters "github.com/cachebox-project/inference-cache/internal/adapters/builtin" "github.com/cachebox-project/inference-cache/internal/controller" + "github.com/cachebox-project/inference-cache/internal/version" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" cachewebhookv1alpha1 "github.com/cachebox-project/inference-cache/internal/webhook/v1alpha1" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" - "github.com/cachebox-project/inference-cache/pkg/version" ) const leaderLockName = "inference-cache-controller-leader-lock" @@ -70,7 +69,7 @@ func defaultOptions() options { cacheIndexRefreshEvery: controller.DefaultRefreshInterval, policyPushEvery: controller.DefaultPolicyPushInterval, subscriberImage: "", - policyServerGRPCAddress: adapterruntime.DefaultPolicyServerGRPCAddress, + policyServerGRPCAddress: "inference-cache-server.inference-cache-system.svc.cluster.local:9090", zapOpts: zap.Options{ TimeEncoder: zapcore.RFC3339TimeEncoder, }, @@ -133,10 +132,10 @@ func main() { // address are operator-supplied: pinning the image to a digest in // production and pointing the sidecar at the right Service DNS are // deployment concerns, not CR-level knobs. - adapterRegistries := builtinadapters.New( - adapterruntime.WithSubscriberImage(opts.subscriberImage), - adapterruntime.WithPolicyServerGRPCAddress(opts.policyServerGRPCAddress), - ) + adapterRegistries := builtinadapters.New(builtinadapters.Options{ + SubscriberImage: opts.subscriberImage, + PolicyServerGRPCAddress: opts.policyServerGRPCAddress, + }) adapterRegistry := adapterRegistries.Runtime // /probe wrapper for the CacheBackend reconciler's functional-probe gate. diff --git a/cmd/inferencecache/doctor.go b/cmd/inferencecache/doctor.go index 1bd775c2..c9de3561 100644 --- a/cmd/inferencecache/doctor.go +++ b/cmd/inferencecache/doctor.go @@ -27,8 +27,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor/checks" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor/output" + "github.com/cachebox-project/inference-cache/internal/cli/doctor/checks" + "github.com/cachebox-project/inference-cache/internal/cli/doctor/output" ) // Server-discovery defaults. The Service name and system namespace match the diff --git a/cmd/inferencecache/main.go b/cmd/inferencecache/main.go index 26592b03..777543e4 100644 --- a/cmd/inferencecache/main.go +++ b/cmd/inferencecache/main.go @@ -10,7 +10,7 @@ // The binary deliberately keeps its glue thin — flag parsing, Kubernetes/gRPC // client construction, and server-endpoint discovery live here, while the // diagnostic logic and output formatting live in the unit-tested -// github.com/cachebox-project/inference-cache/pkg/cli/doctor packages. +// internal/cli/doctor packages. package main import ( @@ -19,7 +19,7 @@ import ( "github.com/spf13/cobra" - "github.com/cachebox-project/inference-cache/pkg/version" + "github.com/cachebox-project/inference-cache/internal/version" ) func main() { diff --git a/cmd/kvevent-fake-engine/e2e_test.go b/cmd/kvevent-fake-engine/e2e_test.go index 2236db25..e73bbc3d 100644 --- a/cmd/kvevent-fake-engine/e2e_test.go +++ b/cmd/kvevent-fake-engine/e2e_test.go @@ -7,9 +7,9 @@ package main // GPU-free, per-PR end-to-end gate for the content-fingerprint routing path: // // fake engine (this package, real ZMQ PUB socket) -// → kvevent-subscriber pipeline (engine.Subscriber → engine.Reporter — +// → kvevent-subscriber pipeline (subscriber.Subscriber → subscriber.Reporter — // the same components cmd/kvevent-subscriber wires) -// → inference-cache server (pkg/server, real gRPC over loopback TCP) +// → inference-cache server (internal/server, real gRPC over loopback TCP) // → LookupRoute // // This is the regression lock for the all-NO_HINT bug: the engine's own KV @@ -42,10 +42,10 @@ import ( "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/server" + "github.com/cachebox-project/inference-cache/internal/subscriber" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - "github.com/cachebox-project/inference-cache/pkg/server" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" ) const ( @@ -155,20 +155,20 @@ func startSubscriberPipeline(t *testing.T, grpcAddr, endpoint, tenant string, lo } client := icpb.NewInferenceCacheClient(conn) - cfg := engine.Config{ + cfg := subscriber.Config{ ReplicaID: e2eReplica, ModelID: e2eModel, TenantID: tenant, HashScheme: e2eScheme, } - reporter := engine.NewReporter(client, cfg, - engine.WithWindow(10*time.Millisecond), - engine.WithLogger(logger)) - sub := engine.NewSubscriber(endpoint, e2eTopic, - engine.WithSubscriberLogger(logger), - engine.WithSubscriberBackoff(50*time.Millisecond)) - - out := make(chan *engine.EventBatch, 256) + reporter := subscriber.NewReporter(client, cfg, + subscriber.WithWindow(10*time.Millisecond), + subscriber.WithLogger(logger)) + sub := subscriber.NewSubscriber(endpoint, e2eTopic, + subscriber.WithSubscriberLogger(logger), + subscriber.WithSubscriberBackoff(50*time.Millisecond)) + + out := make(chan *subscriber.EventBatch, 256) subCtx, cancelSub := context.WithCancel(context.Background()) // Run only exits via context cancellation (it reconnects forever, fail-soft) diff --git a/cmd/kvevent-fake-engine/main_test.go b/cmd/kvevent-fake-engine/main_test.go index 9283ab67..89a5d930 100644 --- a/cmd/kvevent-fake-engine/main_test.go +++ b/cmd/kvevent-fake-engine/main_test.go @@ -8,7 +8,7 @@ import ( "bytes" "testing" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" + "github.com/cachebox-project/inference-cache/internal/subscriber" "github.com/cachebox-project/inference-cache/pkg/fingerprint" ) @@ -16,15 +16,15 @@ import ( // returns its BlockStored events. If the synthetic encoding drifts from what // the subscriber decodes, a smoke would assert against a key the subscriber // never produced (false green) — so every shape change must round-trip here. -func decodeStored(t *testing.T, payload []byte) []engine.BlockStored { +func decodeStored(t *testing.T, payload []byte) []subscriber.BlockStored { t.Helper() - batch, err := engine.DecodeEventBatch(payload) + batch, err := subscriber.DecodeEventBatch(payload) if err != nil { t.Fatalf("DecodeEventBatch: %v", err) } - out := make([]engine.BlockStored, 0, len(batch.Events)) + out := make([]subscriber.BlockStored, 0, len(batch.Events)) for i, ev := range batch.Events { - bs, ok := ev.(engine.BlockStored) + bs, ok := ev.(subscriber.BlockStored) if !ok { t.Fatalf("event %d = %T, want BlockStored", i, ev) } diff --git a/cmd/kvevent-subscriber/main.go b/cmd/kvevent-subscriber/main.go index 4bbbd595..8f5b4c40 100644 --- a/cmd/kvevent-subscriber/main.go +++ b/cmd/kvevent-subscriber/main.go @@ -33,8 +33,8 @@ import ( "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/subscriber" ) func main() { @@ -60,13 +60,13 @@ func main() { logger := slog.New(slog.NewJSONHandler(os.Stderr, nil)) - names, err := engine.ParseAdapterNames(*adapterNames) + names, err := subscriber.ParseAdapterNames(*adapterNames) if err != nil { logger.Error("invalid --lora-adapter-names", "value", *adapterNames, "err", err) os.Exit(2) } - cfg := engine.Config{ + cfg := subscriber.Config{ ReplicaID: *replica, ModelID: *model, TenantID: *tenant, @@ -77,9 +77,9 @@ func main() { logger.Error("invalid config", "err", err) os.Exit(2) } - tier := engine.CacheTier(*cacheTier) + tier := subscriber.CacheTier(*cacheTier) if !tier.IsValid() { - logger.Error("invalid --cache-tier", "value", *cacheTier, "valid", engine.ValidCacheTierNames()) + logger.Error("invalid --cache-tier", "value", *cacheTier, "valid", subscriber.ValidCacheTierNames()) os.Exit(2) } @@ -97,15 +97,15 @@ func main() { client := icpb.NewInferenceCacheClient(conn) - reporter := engine.NewReporter(client, cfg, - engine.WithWindow(*window), - engine.WithLogger(logger), - engine.WithIgnoreBlockRemoved(*ignoreBlockRemoved)) - sub := engine.NewSubscriber(*endpoint, *topic, engine.WithSubscriberLogger(logger)) + reporter := subscriber.NewReporter(client, cfg, + subscriber.WithWindow(*window), + subscriber.WithLogger(logger), + subscriber.WithIgnoreBlockRemoved(*ignoreBlockRemoved)) + sub := subscriber.NewSubscriber(*endpoint, *topic, subscriber.WithSubscriberLogger(logger)) - scraper := engine.NewMetricsScraper( + scraper := subscriber.NewMetricsScraper( &http.Client{Timeout: 5 * time.Second}, - engine.ScraperConfig{ + subscriber.ScraperConfig{ URL: *metricsURL, Tier: tier, ModelLabel: *engineModel, @@ -114,12 +114,12 @@ func main() { }, logger, ) - statsReporter := engine.NewStatsReporter(client, scraper, cfg, - engine.WithStatsInterval(*statsInterval), - engine.WithStatsLogger(logger), + statsReporter := subscriber.NewStatsReporter(client, scraper, cfg, + subscriber.WithStatsInterval(*statsInterval), + subscriber.WithStatsLogger(logger), ) - out := make(chan *engine.EventBatch, 256) + out := make(chan *subscriber.EventBatch, 256) // The reporter stops by draining a closed channel, not by signal — so on // shutdown the batches already buffered in `out` are flushed rather than diff --git a/cmd/server/main.go b/cmd/server/main.go index c0b3dcd1..d4099243 100644 --- a/cmd/server/main.go +++ b/cmd/server/main.go @@ -18,10 +18,10 @@ import ( "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" - "github.com/cachebox-project/inference-cache/pkg/server" - "github.com/cachebox-project/inference-cache/pkg/server/auth" + "github.com/cachebox-project/inference-cache/internal/server" + "github.com/cachebox-project/inference-cache/internal/server/auth" + "github.com/cachebox-project/inference-cache/internal/version" "github.com/cachebox-project/inference-cache/pkg/tokenize" - "github.com/cachebox-project/inference-cache/pkg/version" ) func main() { @@ -150,7 +150,7 @@ func main() { "snapshot_addr", cfg.SnapshotAddr, ) if err := server.ListenAndServe(ctx, cfg, opts...); err != nil { - // Terminal error — log once here. pkg/server.Serve does NOT log on + // Terminal error — log once here. internal/server.Serve does NOT log on // the errCh branch so we don't double-emit when a listener fails. slog.ErrorContext(ctx, "serve_error", "err", err) os.Exit(1) diff --git a/config/observability/alerting-rules.yaml b/config/observability/alerting-rules.yaml index ff7825bc..6edac3d1 100644 --- a/config/observability/alerting-rules.yaml +++ b/config/observability/alerting-rules.yaml @@ -82,7 +82,7 @@ groups: # have been present since vLLM ~0.18 (the release that added # LMCache-style offload support). This operator does not # currently scrape them - # (pkg/adapters/engine/metrics_scraper.go only reads + # (internal/subscriber/metrics_scraper.go only reads # vllm:prefix_cache_{hits,queries} for T1 plus # vllm:*_cache_usage_perc), so the alert binds directly to # vLLM's exposition. Operators should sanity-check the metric @@ -109,7 +109,7 @@ groups: # Python-prometheus-client convention (counter `foo` exposed as # `foo_total`) AND the unsuffixed form used by some non-Python # Prometheus clients. The repo's own scraper hedges the same - # way (pkg/adapters/engine/metrics_scraper.go `sumCounter`), so + # way (internal/subscriber/metrics_scraper.go `sumCounter`), so # the alert stays in lockstep with the scraper. # # SCRAPE-SCOPING REQUIREMENT: this expression matches EVERY diff --git a/config/server/server.yaml b/config/server/server.yaml index 67196fd8..8a4b69b6 100644 --- a/config/server/server.yaml +++ b/config/server/server.yaml @@ -184,7 +184,7 @@ spec: # the first eviction target under node memory pressure. Operators # running closer to the cap should bump both the request and the # limit in lockstep — and, if they need MORE than the cap, - # DefaultMaxEntries is a compile-time constant in pkg/index, so + # DefaultMaxEntries is a compile-time constant in internal/index, so # exceeding it currently requires building a custom server image. memory: 256Mi securityContext: diff --git a/docs/design/cachebackend-api.md b/docs/design/cachebackend-api.md index 71cb2745..9c578387 100644 --- a/docs/design/cachebackend-api.md +++ b/docs/design/cachebackend-api.md @@ -407,7 +407,7 @@ the pod, not the CacheBackend's current annotation), so flipping the annotation on a live backend takes effect as its pods roll. **Engine scope — vLLM only today.** The kernel-check init container is provided -by the runtime adapter via the optional `InitContainerProvider` interface, which +by the runtime adapter via the private internal `InitContainerProvider` capability, which only the vLLM+LMCache adapter implements. The SGLang+LMCache adapter does **not** implement it yet, so `inferencecache.io/lmcache-kernel-check` has no effect on SGLang engine pods and `EngineKernelsHealthy` is not published for them — even diff --git a/docs/design/crd-contract.md b/docs/design/crd-contract.md index 01c8e0f1..86af8108 100644 --- a/docs/design/crd-contract.md +++ b/docs/design/crd-contract.md @@ -79,7 +79,7 @@ So `CacheTenant` has `spec.quota.maxIndexEntries` and `status.indexEntries`, but Every new `v1alpha1` spec/status field ships in exactly **one of three states**: -1. **Wired at merge.** A runtime consumer in `pkg/server/`, `pkg/index/`, `pkg/adapters/`, or `internal/controller/` changes observable behavior on the field's value — verified by grep in the same PR that adds the field. +1. **Wired at merge.** A runtime consumer in `internal/server/`, `internal/index/`, `pkg/adapters/`, or `internal/controller/` changes observable behavior on the field's value — verified by grep in the same PR that adds the field. 2. **Intentionally declarative.** The field scaffolds a not-yet-built controller; the type godoc must say so explicitly *and* name the tracking work. This requirement binds **new** fields at merge. Three fields predate the invariant and belong in this state but do **not** yet fully satisfy the godoc bar — `CacheTenantCryptoSpec` (godoc'd only "reserved for future cryptographic isolation"), `PromptTemplate.slots`, and `PDTopology`'s prefill/decode pools — each is scaffolded ahead of its render / disaggregation controller but names no tracking effort in godoc. That is a known pre-existing gap, tracked as a follow-up; closing it (making each godoc name the work) brings them into compliance. The doc states the bar; these three are the outstanding exceptions, not compliant examples. 3. **Tracked for wiring.** A follow-up effort exists, the field's godoc names it, and the field comment says "inert until \". (Historical example: `CacheBackend.spec.storage.pvc.*` and `status.capacity` named the storage wire-up while they existed; both were later retired when the project decided durability is a backend choice rather than a generic volume knob — see `docs/design/lmcache-server-persistence.md`.) diff --git a/docs/design/grpc-contract.md b/docs/design/grpc-contract.md index d3c2746b..2cbc5685 100644 --- a/docs/design/grpc-contract.md +++ b/docs/design/grpc-contract.md @@ -13,11 +13,10 @@ This is the public API gateways and engines integrate against — the load-beari | proto file | `proto/inferencecache/v1alpha1/inferencecache.proto` | | package | `inferencecache.v1alpha1` | | service | `InferenceCache` | -| Go package | `github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1` | +| Go package | `github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1` | -The generated Go import currently remains under `pkg/server` for compatibility. -Its staged move to the neutral `gen/inferencecache/v1alpha1` path is documented -in [Repository boundaries](repository-boundaries.md#generated-protobuf-migration). +The generated Go import lives under the neutral `gen/inferencecache/v1alpha1` +path so clients do not import a server implementation package. Versioned `v1alpha1` → `v1beta1` → `v1`. No vendor tokens in the package or service (see CONTRIBUTING.md). @@ -94,7 +93,7 @@ What B4 originally landed (now partly superseded by B6, see below): fail-open st Still out of scope (later modules): template rendering (D-series), PD routing (Phase 2), and the event/metric **streams** `StreamCacheEvents` / `StreamMetrics` (M10). Java stubs are generated when the gateway client (E1) needs them. -**Update — B6 (cache index):** `LookupRoute`, `ReportCacheState`, `PublishEvent`, and `GetCacheState` are now backed by the in-memory `CacheIndex` (`pkg/index`): `ReportCacheState` ingests additive deltas; `PublishEvent` applies scheme-safe deltas only — `PREFIX_EVICTED` / `ALL_CLEARED` (removals) and `REPLICA_UPDATED` (replica liveness), while `PREFIX_ADDED` is a no-op (events carry no `hash_scheme`, so additions/refreshes come via `ReportCacheState`); `LookupRoute` returns ranked replicas (`PREFIX_MATCH` / `TENANT_HOT`), the `AFFINITY_HINT` stable-replica fallback when the prefix-match path downgrades to `StrategyNone` and `CachePolicy.spec.affinityRouting` is `Enabled` (the kubebuilder default — see "Affinity routing" below), a fail-open miss (`NO_HINT` — no match, no warm-tenant fallback, no usable affinity assignment, below the `CachePolicy.spec.minimumPrefixTokens` request-side gate under `affinityRouting: Disabled`, every candidate replica matched fewer tokens than the `CachePolicy.spec.minimumMatchedTokens` result-side floor — see "Matched-tokens floor" below — or the top per-replica score fell below the `CachePolicy.spec.routingFloorScore` post-score floor on the distinguishing-power-aware ranker — see [`lookuproute-ranking.md` §2.7](./lookuproute-ranking.md#27-the-replica-distinguishing-power-factor); with `affinityRouting: Enabled` the matched-tokens and routing-floor downgrades surface as `AFFINITY_HINT` instead, since the affinity fallback runs on the `StrategyNone` branch), a policy gate (`POLICY_REQUIRES_CHAIN` — `CachePolicy.spec.strategy.requireChain=true` and no valid wire block-hash chain, still fail-open), a deadline breach (`TIMEOUT` — `CachePolicy.spec.lookupTimeoutMs`, still fail-open), or one of the diagnostic codes (`UNKNOWN_TENANT` / `UNKNOWN_MODEL` / `UNKNOWN_HASH_SCHEME` — set-but-wrong contract key, see "Diagnostic reason codes" below — which keep precedence over `AFFINITY_HINT`); and `GetCacheState` returns the `(tenant, model)` aggregate. The lookup/index metrics (`inferencecache_index_entries`, `inferencecache_lookup_route_*`) are emitted on `/metrics`. `RenderTemplate`, `LookupPDRoute`, and the streams remain fail-open stubs. +**Update — B6 (cache index):** `LookupRoute`, `ReportCacheState`, `PublishEvent`, and `GetCacheState` are now backed by the in-memory `CacheIndex` (`internal/index`): `ReportCacheState` ingests additive deltas; `PublishEvent` applies scheme-safe deltas only — `PREFIX_EVICTED` / `ALL_CLEARED` (removals) and `REPLICA_UPDATED` (replica liveness), while `PREFIX_ADDED` is a no-op (events carry no `hash_scheme`, so additions/refreshes come via `ReportCacheState`); `LookupRoute` returns ranked replicas (`PREFIX_MATCH` / `TENANT_HOT`), the `AFFINITY_HINT` stable-replica fallback when the prefix-match path downgrades to `StrategyNone` and `CachePolicy.spec.affinityRouting` is `Enabled` (the kubebuilder default — see "Affinity routing" below), a fail-open miss (`NO_HINT` — no match, no warm-tenant fallback, no usable affinity assignment, below the `CachePolicy.spec.minimumPrefixTokens` request-side gate under `affinityRouting: Disabled`, every candidate replica matched fewer tokens than the `CachePolicy.spec.minimumMatchedTokens` result-side floor — see "Matched-tokens floor" below — or the top per-replica score fell below the `CachePolicy.spec.routingFloorScore` post-score floor on the distinguishing-power-aware ranker — see [`lookuproute-ranking.md` §2.7](./lookuproute-ranking.md#27-the-replica-distinguishing-power-factor); with `affinityRouting: Enabled` the matched-tokens and routing-floor downgrades surface as `AFFINITY_HINT` instead, since the affinity fallback runs on the `StrategyNone` branch), a policy gate (`POLICY_REQUIRES_CHAIN` — `CachePolicy.spec.strategy.requireChain=true` and no valid wire block-hash chain, still fail-open), a deadline breach (`TIMEOUT` — `CachePolicy.spec.lookupTimeoutMs`, still fail-open), or one of the diagnostic codes (`UNKNOWN_TENANT` / `UNKNOWN_MODEL` / `UNKNOWN_HASH_SCHEME` — set-but-wrong contract key, see "Diagnostic reason codes" below — which keep precedence over `AFFINITY_HINT`); and `GetCacheState` returns the `(tenant, model)` aggregate. The lookup/index metrics (`inferencecache_index_entries`, `inferencecache_lookup_route_*`) are emitted on `/metrics`. `RenderTemplate`, `LookupPDRoute`, and the streams remain fail-open stubs. #### Matched-tokens floor diff --git a/docs/design/grpc-tls.md b/docs/design/grpc-tls.md index 624f578d..6faa5ea4 100644 --- a/docs/design/grpc-tls.md +++ b/docs/design/grpc-tls.md @@ -1,6 +1,6 @@ # Design: gRPC TLS posture (policy server `:9090`) -Status: implemented (Phase 1) · Implements: B5 (gRPC TLS posture) · Relates: gRPC contract (`grpc-contract.md`), B5 server (`pkg/server`), E1 gateway client, default install (`config/default`) +Status: implemented (Phase 1) · Implements: B5 (gRPC TLS posture) · Relates: gRPC contract (`grpc-contract.md`), B5 server (`internal/server`), E1 gateway client, default install (`config/default`) ## Decision @@ -9,7 +9,7 @@ The **locked design decision** for the gRPC policy server is **one-sided Service TLS is **optional at the binary level**, controlled by flags (`--tls-cert-file` / `--tls-key-file`). The server *binary* fully supports TLS, but **`config/default` ships `:9090` plaintext, and TLS is an opt-in overlay** (`config/overlays/server-tls`). **Why opt-in, not on-by-default (yet).** Both gRPC clients of `:9090` are plaintext-only today: -- the in-cluster **`kvevent-subscriber` producer** (C1) dials `:9090` to call `ReportCacheState` with `insecure.NewCredentials()` (`cmd/kvevent-subscriber`), targeting the policy Service (`DefaultPolicyServerGRPCAddress` in `pkg/adapters/runtime/lmcache_shared.go`); and +- the in-cluster **`kvevent-subscriber` producer** (C1) dials `:9090` to call `ReportCacheState` with `insecure.NewCredentials()` (`cmd/kvevent-subscriber`), targeting the policy Service (the built-in default lives in `internal/adapters/builtin/runtime/subscriber.go`); and - the **external gateway client** (E1) isn't built yet. Flipping `config/default` to require TLS would break cache-state **ingestion** (the subscriber's handshake fails → no `ReportCacheState`). So this ticket **locks the decision and lands the full server-side mechanism** (flags, reloading cert, cert-manager Issuer/Certificate, posture metric, opt-in overlay, tests), and **defers the default flip** until both clients are TLS-aware — at which point enabling it is just making `config/overlays/server-tls` the default (and the subscriber needs the server CA distributed into engine-pod namespaces; see *Client trust anchor* below). Operators who want TLS now apply the overlay. @@ -38,7 +38,7 @@ cert-manager Issuer (self-signed) ──mints──▶ Certificate ──▶ - **In-process termination.** TLS is terminated by the server binary itself — no sidecar, Envoy, or Ingress. Keeps the "one binary, one Service" deployment shape. - **Reloading keypair.** The server serves the cert via a `tls.Config.GetCertificate` hook that re-reads `tls.crt`/`tls.key` when the file mtime advances, so a cert-manager-rotated Secret is picked up on the next handshake — no pod restart. A bad keypair still fails fast at startup (loaded once up front). -- **Client trust anchor + distribution.** With the self-signed Issuer, cert-manager publishes the issuing CA as `ca.crt` in the server's Secret. That Secret is mounted only into the server pod, so a client in another Deployment/namespace needs the CA distributed to it. **Caveat — the default `selfSigned` Issuer mints the serving cert as its own root, so `ca.crt` equals the leaf and rotates with it on renewal.** That makes a one-time static copy of `ca.crt` fragile: a client holding the old bundle fails after the leaf renews (cert-manager renews at ~2/3 of lifetime). It's fine for kind/dev (a short-lived cluster never reaches renewal), but for anything longer-lived a **stable** trust anchor is required — either (a) a real/org CA (clients already trust the root; the recommended production path), (b) a cert-manager CA-Issuer chain (a long-lived `isCA` CA Certificate → CA `Issuer` → rotating leaf, so `ca.crt` stays constant), or (c) dynamic propagation via [trust-manager](https://cert-manager.io/docs/trust/trust-manager/) redistributing the `Bundle` on every rotation. The concrete client-side trust wiring (and which of these an operator picks) is owned by the gateway client's connection/discovery doc (E1); this ticket fixes the server posture it must match and ships the dev-grade self-signed default. The install smoke proves the chain is real *at a point in time*: it pulls `ca.crt` from the serving Secret and runs `grpcurl -cacert -authority ` (so grpcurl verifies against the FQDN even though the port-forward terminates at `localhost`), asserting both that the cert verifies and that a wrong authority is rejected. The unit tests (`pkg/server/tls_test.go`) additionally verify the chain + DNS-name match in-process. +- **Client trust anchor + distribution.** With the self-signed Issuer, cert-manager publishes the issuing CA as `ca.crt` in the server's Secret. That Secret is mounted only into the server pod, so a client in another Deployment/namespace needs the CA distributed to it. **Caveat — the default `selfSigned` Issuer mints the serving cert as its own root, so `ca.crt` equals the leaf and rotates with it on renewal.** That makes a one-time static copy of `ca.crt` fragile: a client holding the old bundle fails after the leaf renews (cert-manager renews at ~2/3 of lifetime). It's fine for kind/dev (a short-lived cluster never reaches renewal), but for anything longer-lived a **stable** trust anchor is required — either (a) a real/org CA (clients already trust the root; the recommended production path), (b) a cert-manager CA-Issuer chain (a long-lived `isCA` CA Certificate → CA `Issuer` → rotating leaf, so `ca.crt` stays constant), or (c) dynamic propagation via [trust-manager](https://cert-manager.io/docs/trust/trust-manager/) redistributing the `Bundle` on every rotation. The concrete client-side trust wiring (and which of these an operator picks) is owned by the gateway client's connection/discovery doc (E1); this ticket fixes the server posture it must match and ships the dev-grade self-signed default. The install smoke proves the chain is real *at a point in time*: it pulls `ca.crt` from the serving Secret and runs `grpcurl -cacert -authority ` (so grpcurl verifies against the FQDN even though the port-forward terminates at `localhost`), asserting both that the cert verifies and that a wrong authority is rejected. The unit tests (`internal/server/tls_test.go`) additionally verify the chain + DNS-name match in-process. - **`grpc.health.v1` rides the same listener**, so a TLS server answers the health check over TLS for any client that dials it (gateway, grpcurl). The kubelet probe is a separate matter — see *kubelet probe compatibility* below. - **Plaintext fallback.** With both flags empty the server builds `grpc.NewServer()` with no credentials and serves plaintext — the default posture (`config/default`), until the opt-in overlay supplies the flags. diff --git a/docs/design/kvevent-subscriber-wiring.md b/docs/design/kvevent-subscriber-wiring.md index be1f1b1b..ba32a0da 100644 --- a/docs/design/kvevent-subscriber-wiring.md +++ b/docs/design/kvevent-subscriber-wiring.md @@ -44,7 +44,7 @@ Concretely: * `KVCacheRuntimeAdapter` gains `ObservationSidecar(cb, pod) (*corev1.Container, error)`. The vLLM/LMCache, vLLM/Mooncake, and SGLang/LMCache adapters return the `kvevent-subscriber` - container spec (via the shared `RenderSubscriberSidecar` — the KV-event stream is the + container spec (via their shared internal subscriber renderer — the KV-event stream is the engine's own ZMQ publisher, independent of the L2 store; each adapter pins its engine's `--hash-scheme` tag + ZMQ port); the reference adapter returns `(nil, nil)`. External ownership stays on the runtime/cache adapter and can attach observation. @@ -54,7 +54,7 @@ Concretely: fail open, matching the rest of the webhook. * **The vLLM/LMCache, vLLM/Mooncake, and SGLang/LMCache adapters return nil unless the controller's `--kvevent-subscriber-image` flag is set** (all go through the same shared - `RenderSubscriberSidecar`, so the opt-in behaviour is identical). An unconfigured image would put the sidecar + internal renderer, so the opt-in behaviour is identical). An unconfigured image would put the sidecar container into `ImagePullBackOff`, which keeps the engine pod from going Ready — the exact "cache becomes a serving dependency" failure mode the fail-open posture exists to prevent. Defaulting auto-attach off lets the controller install cleanly into any @@ -167,8 +167,8 @@ predates the T2 downgrade (earlier this path simply suppressed the eviction and left the entry stale at T1) and is kept for backward compatibility, but the signal it carries is unchanged. When set, a `BlockRemoved` becomes a T2 downgrade; when unset, it forwards `PREFIX_EVICTED`. `AllBlocksCleared` and `BlockStored` -flow normally in both modes. The shared `RenderSubscriberSidecar` helper -(`pkg/adapters/runtime/kvevent_subscriber.go`) — which the vLLM/LMCache, +flow normally in both modes. The shared internal subscriber renderer +(`internal/adapters/builtin/runtime/subscriber.go`) — which the vLLM/LMCache, vLLM/Mooncake, and SGLang/LMCache adapters all call — sets the flag **per integration mode**, because the L2 tier is present only in one of them: diff --git a/docs/design/lookuproute-ranking.md b/docs/design/lookuproute-ranking.md index 3402ab72..3ac477be 100644 --- a/docs/design/lookuproute-ranking.md +++ b/docs/design/lookuproute-ranking.md @@ -1019,11 +1019,11 @@ positive routing assertion. ## 9. Where the code lives -- Scoring + strategy orchestration: [`pkg/index/index.go`](../../pkg/index/index.go) +- Scoring + strategy orchestration: [`internal/index/index.go`](../../internal/index/index.go) — see `Lookup`, `lookupExact`, `lookupChain` (§2.5 chain walk), `LookupRoute`, `tenantHotCandidates`, `RankerConfig`. - Handler glue (proto ↔ index, `Strategy` → `reason_code`): - [`pkg/server/inferencecache_service.go`](../../pkg/server/inferencecache_service.go). + [`internal/server/inferencecache_service.go`](../../internal/server/inferencecache_service.go). - Tests covering each strategy and the baseline-preservation invariant: - [`pkg/index/index_test.go`](../../pkg/index/index_test.go) and - [`pkg/server/server_test.go`](../../pkg/server/server_test.go). + [`internal/index/index_test.go`](../../internal/index/index_test.go) and + [`internal/server/server_test.go`](../../internal/server/server_test.go). diff --git a/docs/design/policy-propagation.md b/docs/design/policy-propagation.md index 38925c06..ec46c50c 100644 --- a/docs/design/policy-propagation.md +++ b/docs/design/policy-propagation.md @@ -89,7 +89,7 @@ Stage values: Stage names: - `ingest` — Stage A. Verifies the in-process index ingest path accepts writes. NOTE: this stage writes via `index.Ingest` directly, NOT through the gRPC `ReportCacheState` handler the real subscriber uses (the handler drops messages with `tenant_id = inferencecache.io/probe` by design). A pass proves the index ingest path is healthy; a fail definitively means it's broken. Neither alone proves the wire subscriber path is healthy end-to-end. -- `routing` — Stage B. Verifies the in-process `index.LookupRoute` (the orchestrated ranking entrypoint that the gRPC handler delegates to) returns `PREFIX_MATCH` for the probe-synthesized hash against the just-ingested entry. NOTE: this stage calls `index.LookupRoute` directly, NOT the gRPC `inferenceCacheService.LookupRoute` handler. The handler short-circuits `tenant_id = inferencecache.io/probe` to `NO_HINT` by design (defense against external lookups against the reserved scope), so the probe cannot route through it. Handler-level concerns — policy gating (`minimumPrefixTokens`), `lookupTimeoutMs` deadline, proto→domain translation — are not covered by this stage and have their own unit tests under `pkg/server`. +- `routing` — Stage B. Verifies the in-process `index.LookupRoute` (the orchestrated ranking entrypoint that the gRPC handler delegates to) returns `PREFIX_MATCH` for the probe-synthesized hash against the just-ingested entry. NOTE: this stage calls `index.LookupRoute` directly, NOT the gRPC `inferenceCacheService.LookupRoute` handler. The handler short-circuits `tenant_id = inferencecache.io/probe` to `NO_HINT` by design (defense against external lookups against the reserved scope), so the probe cannot route through it. Handler-level concerns — policy gating (`minimumPrefixTokens`), `lookupTimeoutMs` deadline, proto→domain translation — are not covered by this stage and have their own unit tests under `internal/server`. - `t2` — Stage C. Verifies a tier-2 put/get round trip via the supplied `T2Prober` (LMCache backends; skipped otherwise). Status codes: @@ -420,15 +420,15 @@ namespace key `CachePolicy` uses — see the tenant-quota row below. | Field | Where it lands | |---|---| -| `evictionTTL` | `pkg/index` `TTLResolver` — per-tenant `freshness()` decay + `evictExpired()` cutoff. | -| `eviction` | `pkg/index` `EvictionResolver` — selects the per-namespace cap-based eviction algorithm. `lru` evicts oldest-by-`lastSeen`; `lfu` evicts the lowest per-entry access count, tie-broken on oldest `lastSeen`. The cap sweep (over `MaxEntries`) consults it to order victims. In `lfu` namespaces the lookup path also reads it to record which entries a *delivered* `LookupRoute` hint credits — the bump is lock-free and applied only when the response is actually returned (a `TIMEOUT`'d lookup credits nothing) and never changes a lookup result. The TTL sweep is algorithm-independent. Emitted as `inferencecache_index_evictions_total{algorithm,reason}`. | +| `evictionTTL` | `internal/index` `TTLResolver` — per-tenant `freshness()` decay + `evictExpired()` cutoff. | +| `eviction` | `internal/index` `EvictionResolver` — selects the per-namespace cap-based eviction algorithm. `lru` evicts oldest-by-`lastSeen`; `lfu` evicts the lowest per-entry access count, tie-broken on oldest `lastSeen`. The cap sweep (over `MaxEntries`) consults it to order victims. In `lfu` namespaces the lookup path also reads it to record which entries a *delivered* `LookupRoute` hint credits — the bump is lock-free and applied only when the response is actually returned (a `TIMEOUT`'d lookup credits nothing) and never changes a lookup result. The TTL sweep is algorithm-independent. Emitted as `inferencecache_index_evictions_total{algorithm,reason}`. | | `minimumPrefixTokens` | Request-side gate on `LookupRouteRequest`'s effective prefix token count: chain-bearing requests use `sum(block_token_counts)` and fall back to `prefix_token_count` only when the chain is empty (`effectivePrefixTokens` in the handler). A request shorter than the threshold never surfaces as `PREFIX_MATCH` **or `TENANT_HOT`**. With `affinityRouting: Disabled` the gate fires as a pre-lookup short-circuit straight to `NO_HINT` (cheap path: no index touch). With `affinityRouting: Enabled` (the default) the request goes through the full lookup so the index can classify `UNKNOWN_TENANT` / `UNKNOWN_MODEL` / `UNKNOWN_HASH_SCHEME` diagnostics before any fallback fires; if the index returns `StrategyPrefixMatch` **or `StrategyTenantHot`**, the handler downgrades it to `StrategyNone` as a result-side filter, and the affinity fallback then surfaces `AFFINITY_HINT` with a stable replica pick. Either path enforces the operator intent "tiny prompts don't surface PREFIX_MATCH or TENANT_HOT" (i.e. no positive cache-evidence hint); the difference is the cost (cheap short-circuit vs full lookup) and the fallback reason code. Gateway authors should size the gate against the chain budget they actually send. | | `minimumMatchedTokens` | Post-lookup floor on each replica's realized `matched_tokens`. The handler resolves the per-tenant floor via `PolicyStore.MinimumMatchedTokens`, which falls back to `DefaultMinimumMatchedTokens` (= 64) for tenants with no `CachePolicy`. Replicas whose `matched_tokens` falls below the floor are filtered from the scored result; if none survive, the response downgrades from `PREFIX_MATCH` to `StrategyNone`, which then surfaces as `reason_code: AFFINITY_HINT` with a stable single replica when `affinityRouting: Enabled` (the default) or as `reason_code: NO_HINT` with empty scores when `affinityRouting: Disabled`. The downgrade runs **before** the LFU `CreditHits` step so a non-delivered hint never bumps the per-entry access counter. See [`lookuproute-ranking.md`](./lookuproute-ranking.md). | | `routingFloorScore` | Post-score floor on the per-replica score from the distinguishing-power-aware ranker. The handler resolves the per-tenant floor via `PolicyStore.RoutingFloorScore`, which falls back to `DefaultRoutingFloorScore` (`0.1`) for tenants with no `CachePolicy`. When the top surviving replica's score falls below the floor, the response downgrades from `PREFIX_MATCH` to `StrategyNone`, which then surfaces as `AFFINITY_HINT` or `NO_HINT` per the `affinityRouting` toggle (same shape as the matched-tokens downgrade row above). Composes with `minimumMatchedTokens` — the matched-tokens floor runs first (per-replica filter), then this score floor checks the top survivor. Both downgrades run **before** the LFU `CreditHits` step so a non-delivered hint never bumps an LFU counter. See [`lookuproute-ranking.md`](./lookuproute-ranking.md). | | `lookupTimeoutMs` | `LookupRoute` derives a `context.WithTimeout`. A breach yields `reason_code: TIMEOUT` (still fail-open: empty scores). `TIMEOUT` keeps absolute precedence over `AFFINITY_HINT`. | | `affinityRouting` | Per-namespace toggle for the consistent-hash fallback on the `StrategyNone` branch. Resolved via `PolicyStore.AffinityRoutingEnabled`, which falls back to `DefaultAffinityRoutingEnabled` (`true`) for tenants with no `CachePolicy`. When enabled (default), `tryAffinityResponse` reads the index-known replica set for the request's `(tenant, model, hash_scheme)` engine domain from `servingByScope` (scheme-aware, mirroring the `TENANT_HOT` Pass 2 check), sorts by `replica_id` for cross-restart determinism, and modulos the SHA-256 of the length-prefixed `block_hashes` (fall-back to `prefix_hash`) against the sorted set — same prompt content → same replica every time. When disabled, the response stays on `NO_HINT`. Diagnostic codes (`UNKNOWN_TENANT` / `UNKNOWN_MODEL` / `UNKNOWN_HASH_SCHEME`) and `TIMEOUT` keep precedence over `AFFINITY_HINT`; affinity never preempts a real `PREFIX_MATCH` or `TENANT_HOT` that cleared the request-side gates (one exception: a tiny request below the per-namespace `minimumPrefixTokens` gate has its positive-hint result — including `TENANT_HOT` — downgraded to StrategyNone, so the affinity fallback can still fire on it; the operator intent "tiny prompts don’t surface a positive hint" outranks the TENANT_HOT-vs-affinity precedence). See [`grpc-contract.md` § "Affinity routing"](./grpc-contract.md). | | `strategy.enableChainMatching` / `strategy.requireChain` / `strategy.enableTenantHot` | Handler-side strategy gates. Chain matching disabled strips request block-hash fields before the index call; chain required rejects non-chain requests with `reason_code: POLICY_REQUIRES_CHAIN` before touching the index; tenant-hot disabled downgrades tenant-hot results to `NO_HINT`. The index remains policy-agnostic. | -| `CacheTenant.spec.quota.maxIndexEntries` | `pkg/index` `TenantQuotaResolver`. Pushed as a `ResolvedTenant{tenantID, maxIndexEntries, isolationMode}` slice alongside the policies. At ingest, if the tenant's distinct-prefix count exceeds the budget, the index evicts that tenant's oldest prefixes (Fairness) down to budget. Fail-open when no `CacheTenant` matches the ingest's `tenant_id`. | +| `CacheTenant.spec.quota.maxIndexEntries` | `internal/index` `TenantQuotaResolver`. Pushed as a `ResolvedTenant{tenantID, maxIndexEntries, isolationMode}` slice alongside the policies. At ingest, if the tenant's distinct-prefix count exceeds the budget, the index evicts that tenant's oldest prefixes (Fairness) down to budget. Fail-open when no `CacheTenant` matches the ingest's `tenant_id`. | The server is fail-open by construction on the hot path (no error returned to the gateway); a `CachePolicy`-level fail-open knob is not diff --git a/docs/design/repository-boundaries.md b/docs/design/repository-boundaries.md index ab9d2cc8..8df7db2e 100644 --- a/docs/design/repository-boundaries.md +++ b/docs/design/repository-boundaries.md @@ -21,9 +21,9 @@ The repository structure should make the following questions easy to answer: The default rule for new code is: -> Put code under the `internal/` component that owns it. Promote it to `pkg/` -> only after a concrete external consumer exists and the project is prepared to -> maintain its Go API compatibility. +> Put repository-private code under the `internal/` component that owns it. Use +> `pkg/` only for APIs intentionally exposed as extension contracts, SDKs, or +> reusable libraries. ## Top-level directory responsibilities @@ -70,8 +70,9 @@ The rules are: 6. `internal/controlplaneapi` owns private HTTP DTOs shared by the controller and server. Neither binary imports the other's implementation for wire types. -7. `internal/enginebinding` owns generic engine-pod metadata shared by admission - and controllers. Controllers do not import webhook packages. +7. `internal/enginebinding` owns private engine-pod metadata and coordination + contracts shared by built-in adapters, admission, and controllers. + Controllers do not import webhook or built-in adapter packages. 8. `pkg/adapters` contains the build-time extension contracts. Shipping implementations and registration belong under `internal/adapters`. 9. `internal/adapters/builtin.New` owns the complete registry composition shipped @@ -92,6 +93,15 @@ plugin system: - the repository does not load Go plugins or discover adapter implementations at runtime. +The seam is an implementation boundary, not a complete third-party integration +platform. An adapter can extend engine wiring within the API and validation +model the repository already exposes. Adding a new runtime, cache, or provider +identifier still requires a custom controller build plus the corresponding API, +CRD, admission, and, where applicable, reconciliation changes. Adding new +configuration semantics likewise remains a core repository or custom-fork +change; implementing `KVCacheRuntimeAdapter` alone does not bypass schema or +admission validation. + This build-time seam is the designated extension point, but its Go source contract is pre-stable. The Phase 0 audit found no real out-of-tree adapter consumer, so the structured-binding change intentionally did not retain a @@ -144,7 +154,7 @@ The baseline passes `go test ./...` and `git diff --check`. | `pkg/adapters/runtime` | Supported | Runtime extension interfaces and registry | Keep, narrow to contract-only code | | `internal/adapters/builtin/runtime` | Internal | Shipping vLLM/SGLang implementations and engine wire rendering | Keep | | `pkg/adapters/engine` | Internalize | Subscriber-side event ingest, metrics, and reporting | `internal/subscriber` | -| `pkg/adapters/engineclient` | Internalize by default | Canary/harness client with no current external consumer | `internal/engineclient` | +| `pkg/adapters/engineclient` | Supported, narrow and reclassify | Public inference-engine request client for gateways, benchmarks, and canaries | `pkg/engineclient` | | `pkg/fingerprint` | Supported | Language-neutral fingerprint contract used across integrations | Keep | | `pkg/tokenize` | Supported | Optional tokenizer boundary, including the tagged cgo implementation | Keep | | `pkg/index` | Internalize | Server-owned mutable cache-state implementation | `internal/index` | @@ -152,13 +162,15 @@ The baseline passes `go test ./...` and `git diff --check`. | `pkg/server/auth` | Internalize | Server-owned HTTP authentication | `internal/server/auth` | | `pkg/server/proto/...` | Migrate | Generated public gRPC API under a server-owned path | `gen/inferencecache/v1alpha1` | | `pkg/cli/doctor/...` | Internalize | `cmd/inferencecache` implementation | `internal/cli/doctor` | -| `pkg/render` | Remove placeholder | Empty server-owned placeholder with no implementation | Delete; create `internal/server/render` when implemented | +| `pkg/render` | Reserved public API | Planned reusable `RenderTemplate` library; no production implementation or stable Go API yet | Keep and clarify package status | | `pkg/testing` | Internalize | In-repository envtest helpers | `internal/testutil` | | `pkg/version` | Internalize | Repository binary build metadata | `internal/version` | Every completed migration must update package documentation and repository docs in the same commit. A remaining `pkg/` package must explicitly document its -external consumer or supported extension contract. +external consumer, supported extension/SDK contract, or approved reserved-public +status. A reserved package must state that no implementation or compatibility +guarantee exists yet and must not accumulate speculative placeholder APIs. ## Target source layout @@ -179,6 +191,8 @@ gen/ internal/ ├── adapters/ │ └── builtin/ +│ ├── options.go +│ ├── registry.go │ ├── runtime/ │ │ ├── vllm_lmcache.go │ │ ├── vllm_lmcache_wire.go @@ -200,8 +214,8 @@ internal/ │ ├── policy.go │ ├── probe.go │ └── snapshot.go +├── canary/ ├── enginebinding/ -├── engineclient/ ├── index/ ├── server/ │ └── auth/ @@ -216,7 +230,9 @@ pkg/ ├── adapters/ │ ├── backend/ │ └── runtime/ +├── engineclient/ ├── fingerprint/ +├── render/ └── tokenize/ proto/ @@ -290,21 +306,52 @@ Proposed commit: refactor(adapters): narrow public adapter contracts ``` -- [ ] Move shipping `Options`, subscriber image/server configuration, and - subscriber sidecar rendering from `pkg/adapters/runtime` into - `internal/adapters/builtin` or its runtime implementation package. -- [ ] Keep the runtime interfaces, registry, runtime identifiers, - supported-pair types, required structured-binding contract, and required - cross-component wire contracts public. -- [ ] Convert the concrete reference adapter into a Go example or test fixture +- [x] Replace the shipping functional options with a plain + `internal/adapters/builtin.Options` value containing `SubscriberImage` and + `PolicyServerGRPCAddress`; `cmd/controller` passes that value to + `internal/adapters/builtin.New` rather than configuring built-ins through the + public runtime API. +- [x] In `internal/adapters/builtin.New`, map the composition-level `Options` + explicitly to a narrower `internal/adapters/builtin/runtime.SubscriberConfig` + value accepted by each shipping runtime constructor. The runtime subpackage + must not import its parent `builtin` package. +- [x] Remove the shipping `Option`, `WithSubscriberImage`, and + `WithPolicyServerGRPCAddress` functional-option API. Move subscriber + image/server defaults, config normalization, and sidecar rendering to + `internal/adapters/builtin/runtime/subscriber.go`; preserve existing zero-value + behavior. +- [x] Keep only the core extension contract public under + `pkg/adapters/runtime`: `KVCacheRuntimeAdapter`, `RuntimeID`, `Registry`, + `SupportedPair`, `PairLister`, `ResolveRuntimeID`, and the required + structured-binding methods. The public contract does not promise that an + out-of-tree adapter participates in shipping subscriber or kernel-health + status behavior. +- [x] Move the private `InitContainerProvider` capability, + `SubscriberContainerName`, LMCache kernel-check container/annotation/mode/ + message/env contracts, mode validation, and shared engine-host-network helper + to `internal/enginebinding`. Built-in adapters, admission, and controllers + import that neutral private owner rather than one another's implementation. +- [x] Move built-in vLLM, SGLang, LMCache, and HiCache environment names, + defaults, endpoint rendering, and other implementation helpers to + `internal/adapters/builtin/runtime`. +- [x] Keep `Binding`, `Protocol`, `RenderedStorage`, `Provider`, and their + registries public under `pkg/adapters/backend`. Move provider-owned endpoint + and binding validation, including the existing external and LMCache endpoint + validators, from the runtime package to this backend contract package. +- [x] Convert the concrete reference adapter into a Go example or test fixture so it documents the extension contract without expanding the production API. -- [ ] Keep the LMCache kernel-check implementation in +- [x] Replace webhook and built-in tests that import the production reference + adapter with local test fixtures, and remove migration-only contract aliases. +- [x] Keep the LMCache kernel-check implementation in `internal/adapters/builtin/runtime/lmcachecheck.go`. -- [ ] Preserve registry selection, pod mutation, and admission behavior. -- [ ] Update documentation that still references the pre-refactor adapter paths. +- [x] Preserve registry selection, pod mutation, and admission behavior. +- [x] Update documentation that still references the pre-refactor adapter paths. This commit narrows source-level API exposure but must not redesign the adapter -contract. +contract. The public seam supports custom build-time implementations within the +existing API and validation model; a new runtime, cache, provider, or config +shape can still require API, CRD, webhook, and controller changes in the custom +fork. It is not a dynamic or schema-extensible plugin mechanism. ### A2. Move generated gRPC bindings @@ -314,14 +361,17 @@ Proposed commit: refactor(proto): move generated grpc API under gen ``` -- [ ] Change `go_package` to +- [x] Change `go_package` to `github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1`. -- [ ] Regenerate the Go protobuf and gRPC bindings under `gen/`. -- [ ] Update all server, subscriber, test, and tool imports in the same commit. -- [ ] Update generation and generated-drift checks to treat `gen/` as the only - Go output target. -- [ ] Remove `pkg/server/proto`. -- [ ] Preserve the protobuf package, service names, field numbers, and wire +- [x] Regenerate the Go protobuf and gRPC bindings under `gen/`. +- [x] Update all server, subscriber, test, and tool imports in the same commit. +- [x] Move the package documentation and generated-contract tests to the new + neutral owner. +- [x] Update Makefile generation, coverage exclusions, generated-drift checks, + CI generated-code checks, review-tool generated-path configuration, and + gRPC documentation to treat `gen/` as the only Go output target. +- [x] Remove `pkg/server/proto`. +- [x] Preserve the protobuf package, service names, field numbers, and wire behavior. Because inference-cache has not been formally deployed, the default plan is not @@ -337,12 +387,19 @@ Proposed commit: refactor(controlplane): extract snapshot wire contract ``` -- [ ] Add `internal/controlplaneapi/snapshot.go` for the `/snapshot` JSON DTOs. -- [ ] Keep mutable index domain types separate from the HTTP representation. -- [ ] Map index state to the HTTP DTO in the server boundary. -- [ ] Update the controller poller to import `internal/controlplaneapi`, not the +- [x] Add `internal/controlplaneapi/snapshot.go` with `Snapshot`, + `ReplicaSnapshot`, and `TenantSnapshot` as the owners of the `/snapshot` JSON + field names, optional-field behavior, and controller/server skew contract. +- [x] Keep mutable index domain types separate from the HTTP representation; + do not use aliases between index state and `controlplaneapi` DTOs. +- [x] Keep `Index.Snapshot()` returning an index-owned domain snapshot and add + an explicit field-by-field mapping to the HTTP DTO at the server boundary. +- [x] Update the controller poller to import `internal/controlplaneapi`, not the index implementation. -- [ ] Add JSON wire-shape tests before moving the index package. +- [x] Move exact-key, `omitempty`, and presence-bit/skew tests to + `internal/controlplaneapi`; keep aggregation, sorting, and accounting + invariants with the index implementation. +- [x] Add an endpoint-level mapping test before moving the index package. ### A4. Internalize the mutable cache index @@ -352,11 +409,11 @@ Proposed commit: refactor(index): internalize the cache index ``` -- [ ] Move `pkg/index` to `internal/index`. -- [ ] Update the server, tests, and `hack/index-sizing` imports. -- [ ] Preserve ingest, lookup, ranking, quota, TTL, eviction, and soft-state +- [x] Move `pkg/index` to `internal/index`. +- [x] Update the server, tests, and `hack/index-sizing` imports. +- [x] Preserve ingest, lookup, ranking, quota, TTL, eviction, and soft-state behavior. -- [ ] Do not split `index.go` in this commit. +- [x] Do not split `index.go` in this commit. ### A5. Internalize the server implementation @@ -366,15 +423,15 @@ Proposed commit: refactor(server): internalize the server implementation ``` -- [ ] Move `pkg/server` to `internal/server`. -- [ ] Move `pkg/server/auth` to `internal/server/auth` and retain it as a +- [x] Move `pkg/server` to `internal/server`. +- [x] Move `pkg/server/auth` to `internal/server/auth` and retain it as a cohesive security-focused subpackage. -- [ ] Update `cmd/server` and integration-test imports. -- [ ] Remove temporary `internal/controlplaneapi` type and constant aliases from +- [x] Update `cmd/server` and integration-test imports. +- [x] Remove temporary `internal/controlplaneapi` type and constant aliases from the server package after all callers use the neutral owner directly. -- [ ] Preserve HTTP routes, gRPC methods, metrics, TLS, authentication, and +- [x] Preserve HTTP routes, gRPC methods, metrics, TLS, authentication, and fail-open behavior. -- [ ] Do not split server implementation files in this commit. +- [x] Do not split server implementation files in this commit. ### A6. Internalize the KV-event subscriber implementation @@ -384,11 +441,11 @@ Proposed commit: refactor(subscriber): internalize kv event ingestion ``` -- [ ] Move `pkg/adapters/engine` to `internal/subscriber`. -- [ ] Keep `cmd/kvevent-subscriber` as a thin composition and lifecycle layer. -- [ ] Preserve ZMQ decoding, positional fingerprinting, metrics scraping, +- [x] Move `pkg/adapters/engine` to `internal/subscriber`. +- [x] Keep `cmd/kvevent-subscriber` as a thin composition and lifecycle layer. +- [x] Preserve ZMQ decoding, positional fingerprinting, metrics scraping, batching, reconnect, gRPC reporting, and fail-soft behavior. -- [ ] Keep tests and testdata beside the implementation. +- [x] Keep tests and testdata beside the implementation. ### A7. Internalize the doctor CLI implementation @@ -398,43 +455,58 @@ Proposed commit: refactor(cli): internalize doctor implementation ``` -- [ ] Move `pkg/cli/doctor` to `internal/cli/doctor`. -- [ ] Preserve the existing `checks` and `output` subpackages. -- [ ] Preserve CLI flags, finding codes, JSON field names, output formats, and +- [x] Move `pkg/cli/doctor` to `internal/cli/doctor`. +- [x] Preserve the existing `checks` and `output` subpackages. +- [x] Preserve CLI flags, finding codes, JSON field names, output formats, and exit-code behavior. The CLI output is a user-facing contract even though the Go package is private. -### A8. Internalize repository support packages +### A8. Reclassify repository support packages Proposed commit: ```text -refactor(repo): internalize repository support packages +refactor(repo): reclassify repository support packages ``` -- [ ] Move `pkg/testing` to `internal/testutil`. -- [ ] Move `pkg/version` to `internal/version`. -- [ ] Update Makefile `-ldflags` package paths. -- [ ] Delete the empty `pkg/render` placeholder. -- [ ] Create `internal/server/render` only when `RenderTemplate` receives a real - implementation. +- [x] Move `pkg/testing` to `internal/testutil`. +- [x] Move `pkg/version` to `internal/version`. +- [x] Update Makefile `-ldflags` package paths. +- [x] Keep `pkg/render` as the reserved public package for the planned reusable + `RenderTemplate` implementation. +- [x] Update `pkg/render` documentation to state that no production + implementation or stable Go API exists yet and that the current server does + not depend on it. +- [x] Do not add speculative render interfaces or placeholder behavior before + the implementation requirement is defined. -### A9. Reclassify the engine egress client +### A9. Narrow and reclassify the public engine client Proposed commit: ```text -refactor(engineclient): internalize the canary engine client +refactor(engineclient): establish public engine client boundary ``` -- [ ] Confirm that there is still no external SDK consumer. -- [ ] Move `pkg/adapters/engineclient` to `internal/engineclient` by default. -- [ ] Remove or explicitly isolate the unimplemented gRPC placeholder. -- [ ] Retain the OpenAI-compatible canary/harness behavior and tests. - -If a concrete external gateway consumer exists before this step, stop and -define the supported SDK contract instead of performing the move automatically. +- [x] Move `pkg/adapters/engineclient` to the neutral public path + `pkg/engineclient`; this is a Go source import-path change. +- [x] Keep the public package focused on `EngineClient`, `CompletionParams`, + `Completion`, `OpenAIClient`, `NewOpenAI`, and the supported pre-tokenized + OpenAI-compatible `/v1/completions` request/response mapping. +- [x] Move `PrefixCacheProbe`, Prometheus scraping helpers, and live canary tests + to `internal/canary` when they remain reference-stack infrastructure rather + than general engine-client behavior. +- [x] Delete the unimplemented gRPC client and `ErrNotImplemented` when it has no + remaining caller. Add a gRPC transport only with a concrete engine consumer + and validated protocol. +- [x] Preserve token-ID request encoding, zero-temperature semantics, response + size limits, status/error handling, and completion/usage parsing. +- [x] Document the current boundary explicitly: it does not yet promise + authentication, retries, endpoint discovery, load balancing, streaming, + tracing, or a complete OpenAI API SDK. +- [x] Update reference-stack scripts, package documentation, and tests for the + new public import path and internal canary location. ## Phase B: split large files without creating new package boundaries @@ -643,13 +715,16 @@ Additional checks by change type: | Test/reference-stack move | Affected Make target and GitHub workflow command paths | | End of each phase | `go test -race ./...` or the repository `make ci` gate as practical | -Add a lightweight repository-boundary verification before completing Phase A: +Keep repository-boundary verification lightweight before completing Phase A: + +- [x] Require every package under `pkg/` to have package documentation that + states its intended public role. +- [x] Reject imports of module `internal/` packages from non-test Go code under + `api/`, `gen/`, or `pkg/`. +- [x] Require generated public Go protobuf code to live only under `gen/`. -- [ ] Maintain an explicit allow-list of supported `pkg/` packages. -- [ ] Reject production imports from `pkg/` into `internal/`. -- [ ] Reject imports from public adapter contracts into built-in adapters. -- [ ] Reject controller imports of server or mutable-index implementations. -- [ ] Require generated public Go protobuf code to live only under `gen/`. +Do not add component-by-component import bans or a hard-coded `pkg/` allow-list +in this phase. Use normal code review for the finer-grained dependency rules. ## Review discipline diff --git a/docs/observability/alerts.md b/docs/observability/alerts.md index cdd04f5b..80f852ba 100644 --- a/docs/observability/alerts.md +++ b/docs/observability/alerts.md @@ -333,7 +333,7 @@ in the upstream v0.18 docs page). Our alert and triage queries accept both the unsuffixed and `_total` forms via `{__name__=~"...(_total)?"}`. This operator has no in-process scrape of those upstream metrics — its -own scraper (`pkg/adapters/engine/metrics_scraper.go`) only reads the T1 +own scraper (`internal/subscriber/metrics_scraper.go`) only reads the T1 `vllm:prefix_cache_{hits,queries}` plus `vllm:*_cache_usage_perc`. That means the alert binds directly to vLLM's exposition, and an upstream rename, deprecation, or version skew can silently make the alert inert @@ -450,7 +450,7 @@ curl -s localhost:8080/metrics | grep 'inferencecache_lookup_route_calls_total' # 4. Confirm the index has entries for the model and tenant. # The snapshot endpoint is gated by a SA bearer with the -# `inferencecache.io/controller` audience (see pkg/server/auth/audience.go). +# `inferencecache.io/controller` audience (see internal/server/auth/audience.go). # From a controller pod: # TOKEN=$(cat /var/run/secrets/inferencecache.io/controller-token/token) # Or generate a one-off via `kubectl create token` against the @@ -593,8 +593,8 @@ Service-endpoint probe and Ready gate cannot catch: | `stage` label | What `failed` means | |---|---| -| `ingest` | The probe wrote a synthetic prefix entry through the server's **in-process** `index.Ingest` path and the entry did not land. This pins the index ingest path; it does **NOT** exercise the gRPC `ReportCacheState` handler nor the `kvevent-subscriber` sidecar (subscriber wire bugs are invisible to Stage A by design — see the design doc and `pkg/server/probe.go` lead-in). A failure here means the index itself is dropping writes — a regression in `pkg/index` keying, scheme handling, or eviction. | -| `routing` | The probe wrote the entry, the index recorded it, but `LookupRoute` returned `NO_HINT` for the probe's hash. Likely an index-key-scheme mismatch (the probe's `hashScheme` is derived from `spec.runtime`; an empty scheme fails open and produces `NO_HINT` on lookup) or a lookup-filter regression in `pkg/server`. | +| `ingest` | The probe wrote a synthetic prefix entry through the server's **in-process** `index.Ingest` path and the entry did not land. This pins the index ingest path; it does **NOT** exercise the gRPC `ReportCacheState` handler nor the `kvevent-subscriber` sidecar (subscriber wire bugs are invisible to Stage A by design — see the design doc and `internal/server/probe.go` lead-in). A failure here means the index itself is dropping writes — a regression in `internal/index` keying, scheme handling, or eviction. | +| `routing` | The probe wrote the entry, the index recorded it, but `LookupRoute` returned `NO_HINT` for the probe's hash. Likely an index-key-scheme mismatch (the probe's `hashScheme` is derived from `spec.runtime`; an empty scheme fails open and produces `NO_HINT` on lookup) or a lookup-filter regression in `internal/server`. | | `t2` | (When a `T2Prober` is wired into the server.) The tier-2 put/get cycle against the configured external backend (LMCache today) failed. No `T2Prober` is wired in this revision, so this stage reports `skipped` on every install — an alert here only fires once a follow-up registers a real `T2Prober`. | The alert uses `increase(...{result="failed"}[5m]) >= 2 for: 5m` — a @@ -619,7 +619,7 @@ By `stage` label: - `ingest` — the in-process index ingest path is dropping writes. Check the server's `inferencecache_index_entries` gauge to see if the index - is accumulating entries at all; check server logs for `pkg/index` + is accumulating entries at all; check server logs for `internal/index` errors; verify `inferencecache_server_up == 1`. (A subscriber → server wire bug is **not** what causes this stage to fail — subscriber bugs show up as a missing-state pattern on real workload, not on this diff --git a/docs/operations/index-sizing.md b/docs/operations/index-sizing.md index ddbdb48e..eb29d921 100644 --- a/docs/operations/index-sizing.md +++ b/docs/operations/index-sizing.md @@ -17,7 +17,7 @@ and `CacheTenant.spec.quota.maxIndexEntries` for their workload. **Source of measurements.** All numbers below come from the in-tree sizing harness [`hack/index-sizing`](../../hack/index-sizing) on `go1.26.4` / `darwin/arm64`, ingested -through the real `pkg/index` code path. Re-run the harness on your own platform if you +through the real `internal/index` code path. Re-run the harness on your own platform if you need same-arch numbers — the harness is documented at the top of `main.go`. --- @@ -48,7 +48,7 @@ The global cap (`DefaultMaxEntries = 1,000,000`), the global TTL fallback, and t The "peak RSS" column is `Maxrss` from the harness run — the high-water mark the process ever reached, not current RSS. For a one-shot bulk ingest the peak ≈ steady-state + transient ingest allocations; in production the steady-state is somewhat lower. Treat the column as a **conservative pod-budget figure**: if you provision for the peak, the steady-state has headroom built in. -The default global entry cap of `DefaultMaxEntries = 1,000,000` (see [`pkg/index/index.go`](../../pkg/index/index.go)) +The default global entry cap of `DefaultMaxEntries = 1,000,000` (see [`internal/index/index.go`](../../internal/index/index.go)) is sized for a **1 GiB server pod**. Raise either both (memory + cap) or neither — the cap is not currently a server flag (see [Knobs](#knobs-the-operator-actually-has)), so reaching for a larger footprint needs a recompile today. @@ -61,7 +61,7 @@ so reaching for a larger footprint needs a recompile today. field `tenants[].indexEntries`, and the metric `inferencecache_index_entries` (with a `model` label, e.g. `inferencecache_index_entries{model="meta-llama/Llama-3"}`) all count **distinct prefix keys** — one per `(tenant, model, hash_scheme, adapter, prefix_hash)`, -regardless of how many replicas hold it. The internal `pkg/index.DefaultMaxEntries` cap, +regardless of how many replicas hold it. The internal `internal/index.DefaultMaxEntries` cap, by contrast, counts **total storage entries** — one per `(prefix_key, replica)` tuple. A single prefix held by R replicas is 1 "indexEntries" but R "storage entries". When the doc says "entries" below, the column header makes which unit explicit. @@ -103,7 +103,7 @@ hundreds of bytes per replica, vs. hundreds of bytes per prefix entry. **The prefix-hash byte width does NOT dominate.** Going from 32-byte hashes (LMCache / SHA-256-style) to 16-byte hashes shaved only ~16 B/entry on the heap. The in-tree vLLM -adapter ([`pkg/adapters/engine/events.go`](../../pkg/adapters/engine/events.go) — +adapter ([`internal/subscriber/events.go`](../../internal/subscriber/events.go) — `uint64BE`) normalizes integer block hashes to **8-byte big-endian** under the `vllm` hash_scheme, which would shave a few more bytes. The map machinery and `time.Time` are the bulk of the cost, not the hash bytes — narrower hashes don't materially change pod @@ -139,7 +139,7 @@ Replicas multiply E by R but only at ~50 B/extra-replica per shared entry. The index stores **one entry per block hash**, not per prompt. The vLLM KV-event subscriber maps each `BlockStored` event into one `PrefixEntry` per block hash (see -[`pkg/adapters/engine/mapper.go`](../../pkg/adapters/engine/mapper.go) — `StoredPrefixes`), +[`internal/subscriber/mapper.go`](../../internal/subscriber/mapper.go) — `StoredPrefixes`), each with a cumulative `token_count`. A 1000-token prompt at vLLM's 16-token block size produces ~63 block hashes (`ceil(1000/16)`), which becomes ~63 entries per replica. Plan for this expansion, not the single-blob shape, when sizing. @@ -178,7 +178,7 @@ working set down to ~1M, and the rest of the prefixes won't have hints. Three ch roughly 3×, to ~830K, comfortably under the cap. Pod stays at 1 GiB. Cost: prefixes re-used at the 15-minute mark go to miss instead of hit. Cheapest tuning option and usually the right one. -3. **Rebuild with a higher cap.** Bump `DefaultMaxEntries` in `pkg/index/index.go` to, +3. **Rebuild with a higher cap.** Bump `DefaultMaxEntries` in `internal/index/index.go` to, say, 3M. At ~500 B/entry that's ~1.4 GiB peak RSS — size the pod for at least 2 GiB to leave the 20 % headroom on top of the linear scaling. Most heavyweight option; reach for it when the workload can't tolerate the hit-rate loss from option 1 or @@ -192,7 +192,7 @@ tolerate lower hit rate, tighter TTL, or a custom build. ## TTL trade-offs `CachePolicy.spec.evictionTTL` is a per-namespace knob with a server-side default -of 30 minutes (`pkg/index.DefaultTTL`). The fallback fires whenever the resolver +of 30 minutes (`internal/index.DefaultTTL`). The fallback fires whenever the resolver returns ≤ 0 — i.e. **only a CachePolicy that explicitly sets `evictionTTL` overrides the default**. A namespace with no CachePolicy, or a CachePolicy that omits `evictionTTL`, both fall through to `DefaultTTL`. @@ -232,7 +232,7 @@ prefix warm, the engine recomputes. So the cost of "TTL too long" is wasted inde | Sweep interval | server compile-time constant | `DefaultSweepInterval = 1m` | How often the TTL pass runs. Higher = more lag, less CPU. | **State today.** The global cap, global TTL, and sweep interval are compile-time -constants in `pkg/index/index.go`; flipping any of them requires a server rebuild. The +constants in `internal/index/index.go`; flipping any of them requires a server rebuild. The per-namespace and per-tenant CRs above are the runtime-tunable surface. --- diff --git a/docs/quickstart.md b/docs/quickstart.md index 098127f7..0cf48bbc 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -191,8 +191,8 @@ stage-specific diagnostic. By `.reason`: | `.reason` | Stage | What it means | First-response | |---|---|---|---| -| `ProbeIngestFailed` | ingest | The server's in-process index `Ingest` path is dropping writes. Does **not** indicate a subscriber problem — that path isn't exercised by the probe (the gRPC `PublishEvent` / `ReportCacheState` subscriber surface is bypassed by design; see the stage description at the top of this section). | Read the `FunctionalProbeOK` condition's `.message` for the server's stage diagnostic; check `inferencecache_backend_probe_result_total{backend="/", stage="ingest", result="failed"}` for the trend. The server-side `inferencecache_index_entries` gauge is fine as background index-health context but cannot confirm whether the synthetic probe entry landed — probe entries live under the reserved `inferencecache.io/probe` tenant, which is excluded from the cap-accounting gauge by design. Inspect server logs for `pkg/index` errors. Confirm `inferencecache_server_up == 1`. | -| `ProbeRoutingFailed` | lookup | `LookupRoute` did not return a clean `PREFIX_MATCH` for the probe's reserved replica. Two failure modes share this reason and are disambiguated by the condition `.message`: (a) the lookup returned a non-`PREFIX_MATCH` index strategy — `NO_HINT` (the cleanly-missing case), `TENANT_HOT`, `UNKNOWN_TENANT`, `UNKNOWN_MODEL`, or `UNKNOWN_HASH_SCHEME` — likely an internal `hash_scheme` regression that dropped the probe's scheme on ingest (an empty scheme fails open and produces `NO_HINT`) or a lookup-filter regression in `pkg/index`/`pkg/server`; (b) the lookup did return `PREFIX_MATCH` but the probe's reserved replica isn't among the scored replicas — a probe-id-derivation or reserved-replica-collision regression. (Note: `TIMEOUT` is produced by the gRPC `LookupRoute` handler's deadline path and is NOT reachable here — the probe calls `index.LookupRoute` directly through an in-process seam.) The probe's `hashScheme` derives from `spec.runtime` for canonical resources (admission rejects unsupported runtime/cache pairs). | Read the `FunctionalProbeOK` condition's `.message` first — the server names which failure mode hit. Check `inferencecache_backend_probe_result_total{backend="/", stage="routing", result="failed"}` for the trend. (The server-side `inferencecache_lookup_route_calls_total` is NOT the right surface here — same reason: the probe bypasses the gRPC handler that emits that metric.) Inspect server logs for `pkg/index` lookup-path errors. | +| `ProbeIngestFailed` | ingest | The server's in-process index `Ingest` path is dropping writes. Does **not** indicate a subscriber problem — that path isn't exercised by the probe (the gRPC `PublishEvent` / `ReportCacheState` subscriber surface is bypassed by design; see the stage description at the top of this section). | Read the `FunctionalProbeOK` condition's `.message` for the server's stage diagnostic; check `inferencecache_backend_probe_result_total{backend="/", stage="ingest", result="failed"}` for the trend. The server-side `inferencecache_index_entries` gauge is fine as background index-health context but cannot confirm whether the synthetic probe entry landed — probe entries live under the reserved `inferencecache.io/probe` tenant, which is excluded from the cap-accounting gauge by design. Inspect server logs for `internal/index` errors. Confirm `inferencecache_server_up == 1`. | +| `ProbeRoutingFailed` | lookup | `LookupRoute` did not return a clean `PREFIX_MATCH` for the probe's reserved replica. Two failure modes share this reason and are disambiguated by the condition `.message`: (a) the lookup returned a non-`PREFIX_MATCH` index strategy — `NO_HINT` (the cleanly-missing case), `TENANT_HOT`, `UNKNOWN_TENANT`, `UNKNOWN_MODEL`, or `UNKNOWN_HASH_SCHEME` — likely an internal `hash_scheme` regression that dropped the probe's scheme on ingest (an empty scheme fails open and produces `NO_HINT`) or a lookup-filter regression in `internal/index`/`internal/server`; (b) the lookup did return `PREFIX_MATCH` but the probe's reserved replica isn't among the scored replicas — a probe-id-derivation or reserved-replica-collision regression. (Note: `TIMEOUT` is produced by the gRPC `LookupRoute` handler's deadline path and is NOT reachable here — the probe calls `index.LookupRoute` directly through an in-process seam.) The probe's `hashScheme` derives from `spec.runtime` for canonical resources (admission rejects unsupported runtime/cache pairs). | Read the `FunctionalProbeOK` condition's `.message` first — the server names which failure mode hit. Check `inferencecache_backend_probe_result_total{backend="/", stage="routing", result="failed"}` for the trend. (The server-side `inferencecache_lookup_route_calls_total` is NOT the right surface here — same reason: the probe bypasses the gRPC handler that emits that metric.) Inspect server logs for `internal/index` lookup-path errors. | | `ProbeT2Failed` | tier-2 | The tier-2 put/get cycle failed (LMCache, today). Only reachable when a `T2Prober` is wired into the server — none is registered in the current revision, so this condition does **not** appear on a clean install. | Will be applicable once a `T2Prober` ships; not actionable today. | ### `FunctionalProbeOK=Unknown` / `ProbeError` diff --git a/docs/reference-stack/VERSIONS.md b/docs/reference-stack/VERSIONS.md index 26ca9b2a..74942372 100644 --- a/docs/reference-stack/VERSIONS.md +++ b/docs/reference-stack/VERSIONS.md @@ -124,7 +124,7 @@ replaces that placeholder.) Concretely: satisfy (engine ↔ worker lmcache parity, and lmcache kernels ↔ the SGLang base image's CUDA runtime) are spelled out in the build steps above. - **What IS validated without a GPU:** SGLang's exact event wire is covered by the - Go `pkg/adapters/engine` SGLang test; the Python synthetic publisher covers only + Go `internal/subscriber` SGLang test; the Python synthetic publisher covers only the shared decode/redaction. See [`manifests/sglang-lmcache/README.md`](manifests/sglang-lmcache/README.md). diff --git a/docs/reference-stack/manifests/sglang-lmcache/README.md b/docs/reference-stack/manifests/sglang-lmcache/README.md index f6f24f43..5bac6935 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/README.md +++ b/docs/reference-stack/manifests/sglang-lmcache/README.md @@ -34,9 +34,9 @@ SGLang adopted vLLM's KV-event wire wholesale: `--kv-events-config` drives a ZMQ 1. **The event-decode path is engine-agnostic and already covered by tests.** The shipped `kvevent-subscriber` decodes SGLang's stream unchanged; the only difference is the `--hash-scheme=sglang` tag. The Go decoder is exercised against a synthetic - SGLang-shaped frame in `pkg/adapters/engine/sglang_wire_test.go`, and the + SGLang-shaped frame in `internal/subscriber/sglang_wire_test.go`, and the cross-engine isolation (`hash_scheme` keeps SGLang and vLLM prefixes disjoint) in - `pkg/index` (`TestNoCrossEngineFalseHitVLLMvsSGLang`). + `internal/index` (`TestNoCrossEngineFalseHitVLLMvsSGLang`). 2. **You can validate the wire off-GPU** (below): the Go test covers SGLang's exact wire shape; the Python synthetic tooling covers the shared decode/redaction logic. @@ -292,7 +292,7 @@ Two complementary off-GPU checks, with an important scope distinction: shape — and that the subscriber's decoder tolerates the trailing `attn_dp_rank` and tags reports `hash_scheme=sglang` — is asserted by the Go fixture the shipped subscriber actually uses (from the repo root): - `go test ./pkg/adapters/engine/ -run SGLang` + `go test ./internal/subscriber/ -run SGLang` (`TestDecodeSGLangEventBatch` + `TestReporterTagsSGLangScheme`). The Python synthetic path above does **not** cover the 3-tuple; rely on the Go test for the SGLang-specific envelope. diff --git a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml index 6e0b2dc2..3a4e999a 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml +++ b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml @@ -31,7 +31,7 @@ # >> You need an NVIDIA GPU. << SGLang loads weights on CUDA and the LMCache MP # worker moves KV over CUDA-IPC. There is no first-class CPU fallback here, so this # manifest is GPU-only. SGLang's exact KV-event wire (the 3-tuple EventBatch -# envelope) is covered off-GPU by `go test ./pkg/adapters/engine/ -run SGLang`; the +# envelope) is covered off-GPU by `go test ./internal/subscriber/ -run SGLang`; the # Python scripts/ tooling exercises the shared decode + token-redaction (it models # vLLM's 2-tuple envelope). See this directory's README. # diff --git a/docs/reference-stack/scripts/canary_dual_input_tokenization.sh b/docs/reference-stack/scripts/canary_dual_input_tokenization.sh index 50d90113..002667e7 100755 --- a/docs/reference-stack/scripts/canary_dual_input_tokenization.sh +++ b/docs/reference-stack/scripts/canary_dual_input_tokenization.sh @@ -12,7 +12,8 @@ # This isolates the ENGINE half of the guarantee. The other halves are covered # by their own tests: the tokenizer (pkg/tokenize, verified against a real HF # tokenizer) and the content fingerprint (pkg/fingerprint, golden tests). The -# Go test TestPrefixCacheCanaryLive drives pkg/adapters/engineclient: it sends a long +# Go test TestPrefixCacheCanaryLive under internal/canary drives pkg/engineclient: +# it sends a long # token-ID prompt to /v1/completions twice and asserts vLLM's # prefix_cache_hits_total rises on the warm (identical) request. # @@ -80,7 +81,7 @@ export IC_ENGINE_MODEL export IC_ENGINE_PROMPT_TOKENS="$PROMPT_TOKENS" log "running the by-construction canary (token-ID prompt prefix-cache hit)" -go test ./pkg/adapters/engineclient/ -run TestPrefixCacheCanaryLive -count=1 -v \ +go test ./internal/canary/ -run TestPrefixCacheCanaryLive -count=1 -v \ || fail "canary failed: the engine did not prefix-cache the token-ID prompt" log "PASS: engine prefix-cached the token-ID prompt — routing fingerprint matches by construction" diff --git a/docs/reference/metrics.md b/docs/reference/metrics.md index 4bf4453c..9e5104e8 100644 --- a/docs/reference/metrics.md +++ b/docs/reference/metrics.md @@ -15,7 +15,7 @@ silently. - **Namespace.** Every metric the cache plane owns is prefixed `inferencecache_*`, in both binaries. Server-binary metrics derive the prefix from the `metricNamespace` constant in - [`pkg/server/metrics.go`](../../pkg/server/metrics.go); controller- + [`internal/server/metrics.go`](../../internal/server/metrics.go); controller- binary metrics declare it inline on each `prometheus.NewXVec` declaration in `internal/controller/` (see `backendServerRestartCascadesTotal` for the pattern) — the two @@ -48,7 +48,7 @@ silently. |---|---|---|---| | `inferencecache_server_up` | *(none)* | `1` if the cache policy server is serving requests, `0` otherwise. | Server starts (→`1`) / shuts down (→`0`). Liveness signal. | | `inferencecache_server_grpc_tls_enabled` | *(none)* | `1` if the gRPC server (`:9090`) is terminating TLS, `0` if serving plaintext. | Set once at startup from `--tls-cert-file`/`--tls-key-file` (both set → `1`, both empty → `0`). Confirms the prod wire posture from Prometheus. See `docs/design/grpc-tls.md`. | -| `inferencecache_index_entries` | `model` | **Distinct prefix entries** the in-memory `CacheIndex` currently holds for that model, **excluding reserved-tenant (`inferencecache.io/probe`) entries**. One entry = one unique `(tenant, model, hash_scheme, adapter, prefix_hash)` tuple, regardless of how many replicas hold it. | Rises on new `(scheme, adapter, hash)` from `ReportCacheState`; falls on `AllBlocksCleared` / TTL eviction / max-entries cap. A `BlockRemoved` only drops the entry in **single-tier (non-L2) mode**, where the subscriber forwards it as `PREFIX_EVICTED`; in **L2/Offload mode** a `BlockRemoved` **retains** the entry — the subscriber re-reports it at tier T2 rather than deleting it, so the count holds (the entry ages out only via TTL). Idempotent re-reports and T1↔T2 tier changes on an existing `(scheme, adapter, hash)` do **not** move it. The probe's synthetic state IS in the index during a Run but is excluded from this gauge so a scrape that races Stage C cannot transiently surface a probe-tenant count on a real model bucket — see `WithReservedTenants` in `pkg/index/index.go`. | +| `inferencecache_index_entries` | `model` | **Distinct prefix entries** the in-memory `CacheIndex` currently holds for that model, **excluding reserved-tenant (`inferencecache.io/probe`) entries**. One entry = one unique `(tenant, model, hash_scheme, adapter, prefix_hash)` tuple, regardless of how many replicas hold it. | Rises on new `(scheme, adapter, hash)` from `ReportCacheState`; falls on `AllBlocksCleared` / TTL eviction / max-entries cap. A `BlockRemoved` only drops the entry in **single-tier (non-L2) mode**, where the subscriber forwards it as `PREFIX_EVICTED`; in **L2/Offload mode** a `BlockRemoved` **retains** the entry — the subscriber re-reports it at tier T2 rather than deleting it, so the count holds (the entry ages out only via TTL). Idempotent re-reports and T1↔T2 tier changes on an existing `(scheme, adapter, hash)` do **not** move it. The probe's synthetic state IS in the index during a Run but is excluded from this gauge so a scrape that races Stage C cannot transiently surface a probe-tenant count on a real model bucket — see `WithReservedTenants` in `internal/index/index.go`. | ### Counters @@ -71,7 +71,7 @@ silently. ## Controller metrics (`inferencecache_*`) — exposed today -Emitted by the `cmd/controller` binary, registered into the controller-runtime metrics registry (`sigs.k8s.io/controller-runtime/pkg/metrics`), and served at the manager's `--metrics-bind-address` (default `:8080` on the controller binary — separate process from the server binary's `:8080`). This is a deliberately separate registry from the server's `pkg/server/metrics.go` one; the two processes have disjoint scrape targets. +Emitted by the `cmd/controller` binary, registered into the controller-runtime metrics registry (`sigs.k8s.io/controller-runtime/pkg/metrics`), and served at the manager's `--metrics-bind-address` (default `:8080` on the controller binary — separate process from the server binary's `:8080`). This is a deliberately separate registry from the server's `internal/server/metrics.go` one; the two processes have disjoint scrape targets. ### Gauges @@ -106,29 +106,29 @@ with OTEL collectors) without bumping `v1alpha1`. ### Server binary (`cmd/server`) -- **Definitions:** [`pkg/server/metrics.go`](../../pkg/server/metrics.go) (the +- **Definitions:** [`internal/server/metrics.go`](../../internal/server/metrics.go) (the `serverMetrics` struct + `newServerMetrics`). - **`indexEntries` writers:** the index pushes via the `index.Metrics` - interface (`SetIndexEntries`); see [`pkg/index/`](../../pkg/index/). The + interface (`SetIndexEntries`); see [`internal/index/`](../../internal/index/). The snapshot is taken under `reportMu` so concurrent reporters can't publish a stale count. - **`lookupCalls` + `lookupLatency` writers:** the `LookupRoute` handler in - [`pkg/server/inferencecache_service.go`](../../pkg/server/inferencecache_service.go) + [`internal/server/inferencecache_service.go`](../../internal/server/inferencecache_service.go) calls `metrics.observeLookup(...)` exactly once per request. - **`tenantEvictions` writer:** the index calls `AddTenantEvictions(...)` via the `index.Metrics` interface after a quota-driven eviction at ingest; see - [`pkg/index/`](../../pkg/index/). One increment per evicted distinct prefix. + [`internal/index/`](../../internal/index/). One increment per evicted distinct prefix. - **`indexEvictions` writer:** the index calls `AddIndexEvictions(algorithm, reason, n)` via the `index.Metrics` interface after the cap sweep (`reason="cap"`, on ingest) - and the TTL sweep (`reason="ttl"`); see [`pkg/index/`](../../pkg/index/). The + and the TTL sweep (`reason="ttl"`); see [`internal/index/`](../../internal/index/). The per-algorithm tally is emitted after the index lock is released. - **`snapshotAuth` + `policyAuth` + `probeAuth` writers:** the TokenReview - middleware in [`pkg/server/auth/`](../../pkg/server/auth/) reports one + middleware in [`internal/server/auth/`](../../internal/server/auth/) reports one outcome per request via the `auth.ResultRecorder` interface. The recorders themselves are returned by `serverMetrics.SnapshotAuthRecorder()`, `serverMetrics.PolicyAuthRecorder()`, and `serverMetrics.ProbeAuthRecorder()` - (in `pkg/server/metrics.go`) and wired into the per-endpoint authenticators - in `pkg/server/server.go`. One increment per `/snapshot`, `/policy`, or + (in `internal/server/metrics.go`) and wired into the per-endpoint authenticators + in `internal/server/server.go`. One increment per `/snapshot`, `/policy`, or `/probe` request reaching the middleware, labeled by `result`. All three endpoints share the controller ServiceAccount identity profile but emit per-endpoint counters and enforce endpoint-specific audiences so a dashboard @@ -258,7 +258,7 @@ Two binaries each expose their own `/metrics` endpoint — separate processes, s registration pattern: - **Server binary (`cmd/server`)**: add a field to `serverMetrics` in - `pkg/server/metrics.go`, construct it in `newServerMetrics`, and + `internal/server/metrics.go`, construct it in `newServerMetrics`, and register it on the `prometheus.NewRegistry()` block. Add a typed writer method on `*serverMetrics` (e.g. `observeLookup`, `SetIndexEntries`) and call it from the relevant handler or index @@ -282,7 +282,7 @@ Two binaries each expose their own `/metrics` endpoint — separate processes, s Include labels, meaning, and what makes it move. If the metric is a histogram, document the bucket array and *why* those buckets. 4. **Wire test coverage.** Server-binary metrics: add an assertion in - `pkg/server/metrics_test.go`. Controller-binary metrics: add an + `internal/server/metrics_test.go`. Controller-binary metrics: add an assertion in a `_test.go` file alongside the reconciler that increments them (e.g. `cachebackend_server_restart_test.go` — see the `cascadeRestartsCount` helper for the pattern). In both cases verify diff --git a/docs/reference/reason-codes.md b/docs/reference/reason-codes.md index f9e17fd0..d06ff466 100644 --- a/docs/reference/reason-codes.md +++ b/docs/reference/reason-codes.md @@ -46,11 +46,11 @@ module — until then, treat `NO_HINT` as the only `LookupPDRoute` answer. **Constants in code:** `reasonPrefixMatch`, `reasonTenantHot`, `reasonNoHint`, `reasonTimeout`, `reasonPolicyRequiresChain`, `reasonAffinityHint`, `reasonUnknownTenant`, `reasonUnknownModel`, `reasonUnknownHashScheme` in -`pkg/server/inferencecache_service.go`. See also [`../design/lookuproute-diagnostics.md`](../design/lookuproute-diagnostics.md) for the design rule and gateway-SDK guidance. +`internal/server/inferencecache_service.go`. See also [`../design/lookuproute-diagnostics.md`](../design/lookuproute-diagnostics.md) for the design rule and gateway-SDK guidance. ### Ranking inputs beyond `matched_tokens × freshness` -The server-side ranker (`pkg/index`) is configurable via `RankerConfig` (in- +The server-side ranker (`internal/index`) is configurable via `RankerConfig` (in- binary knobs) and `CachePolicy.spec` (per-namespace knobs). Each `RankerConfig` knob defaults to a value that reduces the **pressure / SLO** layers to the baseline when its supporting signal is absent — so a deployment without replica @@ -86,7 +86,7 @@ the cardinality factor and both floors still run. | `TEMPLATE_NOT_FOUND` | spec'd, not emitted | The referenced `template_ref` doesn't exist. | Promoted when D5 (`RenderTemplate` handler) lands. | | `RENDER_ERROR` | spec'd, not emitted | Template was found but rendering failed (missing/typed-wrong variables, runtime DSL error). | Promoted with D5. | -**Constants in code:** `reasonOK` in `pkg/server/inferencecache_service.go`. +**Constants in code:** `reasonOK` in `internal/server/inferencecache_service.go`. --- @@ -135,7 +135,7 @@ See [metrics.md](metrics.md) for the full metric surface. [`proto/inferencecache/v1alpha1/inferencecache.proto`](../../proto/inferencecache/v1alpha1/inferencecache.proto) (already done for `TENANT_HOT`, `TIMEOUT`, `TEMPLATE_NOT_FOUND`, `RENDER_ERROR`). Run `make proto-gen` if the comment touched the schema. -2. **Add a constant** in `pkg/server/inferencecache_service.go` next to +2. **Add a constant** in `internal/server/inferencecache_service.go` next to `reasonPrefixMatch` / `reasonNoHint` / `reasonOK`. Keep the constant name `reason`. 3. **Emit it** from the handler at the relevant decision point. Keep handlers diff --git a/pkg/server/proto/inferencecache/v1alpha1/doc.go b/gen/inferencecache/v1alpha1/doc.go similarity index 58% rename from pkg/server/proto/inferencecache/v1alpha1/doc.go rename to gen/inferencecache/v1alpha1/doc.go index c1da31a7..556e74a6 100644 --- a/pkg/server/proto/inferencecache/v1alpha1/doc.go +++ b/gen/inferencecache/v1alpha1/doc.go @@ -3,6 +3,5 @@ // SPDX-License-Identifier: Apache-2.0 // Package v1alpha1 contains generated Go bindings for the public -// inferencecache.v1alpha1 protobuf API. The current import path is retained -// during the documented migration to gen/inferencecache/v1alpha1. +// inferencecache.v1alpha1 protobuf API. package inferencecachev1alpha1pb diff --git a/pkg/server/proto/inferencecache/v1alpha1/inferencecache.pb.go b/gen/inferencecache/v1alpha1/inferencecache.pb.go similarity index 99% rename from pkg/server/proto/inferencecache/v1alpha1/inferencecache.pb.go rename to gen/inferencecache/v1alpha1/inferencecache.pb.go index fbe0e975..29faa3d0 100644 --- a/pkg/server/proto/inferencecache/v1alpha1/inferencecache.pb.go +++ b/gen/inferencecache/v1alpha1/inferencecache.pb.go @@ -1941,14 +1941,13 @@ var file_inferencecache_v1alpha1_inferencecache_proto_rawDesc = string([]byte{ 0x65, 0x61, 0x6d, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x69, 0x6e, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x63, 0x61, 0x63, 0x68, 0x65, 0x2e, 0x76, 0x31, 0x61, 0x6c, 0x70, 0x68, 0x61, 0x31, 0x2e, 0x4d, 0x65, 0x74, 0x72, - 0x69, 0x63, 0x30, 0x01, 0x42, 0x6f, 0x5a, 0x6d, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, + 0x69, 0x63, 0x30, 0x01, 0x42, 0x62, 0x5a, 0x60, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x63, 0x61, 0x63, 0x68, 0x65, 0x62, 0x6f, 0x78, 0x2d, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x2f, 0x69, 0x6e, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x2d, 0x63, 0x61, - 0x63, 0x68, 0x65, 0x2f, 0x70, 0x6b, 0x67, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x2f, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x69, 0x6e, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x63, 0x61, - 0x63, 0x68, 0x65, 0x2f, 0x76, 0x31, 0x61, 0x6c, 0x70, 0x68, 0x61, 0x31, 0x3b, 0x69, 0x6e, 0x66, - 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x63, 0x61, 0x63, 0x68, 0x65, 0x76, 0x31, 0x61, 0x6c, 0x70, - 0x68, 0x61, 0x31, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x63, 0x68, 0x65, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x69, 0x6e, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, + 0x65, 0x63, 0x61, 0x63, 0x68, 0x65, 0x2f, 0x76, 0x31, 0x61, 0x6c, 0x70, 0x68, 0x61, 0x31, 0x3b, + 0x69, 0x6e, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x63, 0x61, 0x63, 0x68, 0x65, 0x76, 0x31, + 0x61, 0x6c, 0x70, 0x68, 0x61, 0x31, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, }) var ( diff --git a/pkg/server/proto/inferencecache/v1alpha1/inferencecache_grpc.pb.go b/gen/inferencecache/v1alpha1/inferencecache_grpc.pb.go similarity index 100% rename from pkg/server/proto/inferencecache/v1alpha1/inferencecache_grpc.pb.go rename to gen/inferencecache/v1alpha1/inferencecache_grpc.pb.go diff --git a/pkg/server/proto/inferencecache/v1alpha1/replica_stats_client_version_test.go b/gen/inferencecache/v1alpha1/replica_stats_client_version_test.go similarity index 97% rename from pkg/server/proto/inferencecache/v1alpha1/replica_stats_client_version_test.go rename to gen/inferencecache/v1alpha1/replica_stats_client_version_test.go index 3222dbc6..29fdb510 100644 --- a/pkg/server/proto/inferencecache/v1alpha1/replica_stats_client_version_test.go +++ b/gen/inferencecache/v1alpha1/replica_stats_client_version_test.go @@ -18,7 +18,7 @@ import ( "google.golang.org/protobuf/encoding/protowire" "google.golang.org/protobuf/proto" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) func TestReplicaStats_ClientVersion_RoundtripSet(t *testing.T) { diff --git a/hack/index-sizing/main.go b/hack/index-sizing/main.go index 8ae9a952..93f183c4 100644 --- a/hack/index-sizing/main.go +++ b/hack/index-sizing/main.go @@ -8,7 +8,7 @@ // prints heap + peak RSS so operators can pick CacheTenant.spec.quota.maxIndexEntries, // the per-namespace CachePolicy.spec.evictionTTL, and pod memory limits with // real numbers instead of a guess. The global server cap is the compile-time -// constant pkg/index.DefaultMaxEntries. +// constant internal/index.DefaultMaxEntries. // // Not a shipping binary; not built by `make build`. Run with: // @@ -25,7 +25,7 @@ import ( "syscall" "time" - "github.com/cachebox-project/inference-cache/pkg/index" + "github.com/cachebox-project/inference-cache/internal/index" ) // runPlan is the derived per-run shape: per-bucket key count, the actually- @@ -96,7 +96,7 @@ func planRun(keys, replicas, hashSize, tenants, models, batchSize int) (runPlan, func main() { keys := flag.Int("keys", 1_000_000, "distinct prefix keys to ingest") replicas := flag.Int("replicas", 1, "replicas reporting each prefix (entries = keys × replicas)") - hashSize := flag.Int("hash-bytes", 32, "prefix-hash bytes per entry. Conservative default representing LMCache-style SHA hashes; the in-tree vLLM adapter normalizes integer block hashes to 8 bytes big-endian (see pkg/adapters/engine/events.go uint64BE). Minimum 8 to guarantee uniqueness across the keys range.") + hashSize := flag.Int("hash-bytes", 32, "prefix-hash bytes per entry. Conservative default representing LMCache-style SHA hashes; the in-tree vLLM adapter normalizes integer block hashes to 8 bytes big-endian (see internal/subscriber/events.go uint64BE). Minimum 8 to guarantee uniqueness across the keys range.") tenants := flag.Int("tenants", 1, "distinct tenant IDs (keys are split evenly across tenants×models)") models := flag.Int("models", 1, "distinct model IDs") batchSize := flag.Int("batch", 1_000, "prefixes per Ingest call") diff --git a/hack/verify-samples/admission_test.go b/hack/verify-samples/admission_test.go index 55590fd4..2a033896 100644 --- a/hack/verify-samples/admission_test.go +++ b/hack/verify-samples/admission_test.go @@ -100,7 +100,7 @@ func TestVerifySamplesAdmissionEndToEnd(t *testing.T) { if err != nil { t.Fatalf("ctrl.NewManager: %v", err) } - registries := builtinadapters.New() + registries := builtinadapters.New(builtinadapters.Options{}) if err := cachewebhookv1alpha1.SetupCacheBackendWebhookWithManager(mgr, registries.Runtime); err != nil { t.Fatalf("register CacheBackend webhook: %v", err) } diff --git a/hack/verify-samples/main.go b/hack/verify-samples/main.go index 0090193e..056f896f 100644 --- a/hack/verify-samples/main.go +++ b/hack/verify-samples/main.go @@ -155,7 +155,7 @@ func run() error { // intentionally NOT registered: its MutatingWebhookConfiguration uses // failurePolicy=Ignore, so Pod creates (none in this suite anyway) // would just bypass it; CacheBackend is what we need to exercise. - registries := builtinadapters.New() + registries := builtinadapters.New(builtinadapters.Options{}) if err := cachewebhookv1alpha1.SetupCacheBackendWebhookWithManager(mgr, registries.Runtime); err != nil { return fmt.Errorf("register CacheBackend webhook: %w", err) } diff --git a/internal/adapters/builtin/boundaries_test.go b/internal/adapters/builtin/boundaries_test.go deleted file mode 100644 index fd3947e9..00000000 --- a/internal/adapters/builtin/boundaries_test.go +++ /dev/null @@ -1,125 +0,0 @@ -// SPDX-FileCopyrightText: 2026 The inference-cache Authors -// -// SPDX-License-Identifier: Apache-2.0 - -package builtin - -import ( - "go/parser" - "go/token" - "io/fs" - "path/filepath" - "runtime" - "sort" - "strconv" - "strings" - "testing" -) - -const modulePath = "github.com/cachebox-project/inference-cache" - -func TestProductionImportsRespectAdapterBoundaries(t *testing.T) { - t.Parallel() - - root := repositoryRoot(t) - scopes := []struct { - root string - banned []string - }{ - { - root: filepath.Join(root, "internal", "controller"), - banned: []string{ - modulePath + "/pkg/server", - modulePath + "/internal/webhook", - modulePath + "/internal/adapters/builtin", - }, - }, - { - root: filepath.Join(root, "internal", "webhook"), - banned: []string{modulePath + "/internal/adapters/builtin"}, - }, - } - for _, scope := range scopes { - err := filepath.WalkDir(scope.root, func(path string, entry fs.DirEntry, walkErr error) error { - if walkErr != nil { - return walkErr - } - if entry.IsDir() || filepath.Ext(path) != ".go" || strings.HasSuffix(path, "_test.go") { - return nil - } - file, err := parser.ParseFile(token.NewFileSet(), path, nil, parser.ImportsOnly) - if err != nil { - t.Fatalf("parse %s: %v", path, err) - } - for _, imported := range file.Imports { - pathValue, err := strconv.Unquote(imported.Path.Value) - if err != nil { - t.Fatalf("unquote import in %s: %v", path, err) - } - for _, prefix := range scope.banned { - if pathValue == prefix || strings.HasPrefix(pathValue, prefix+"/") { - t.Errorf("%s imports implementation package %q", filepath.Base(path), pathValue) - } - } - } - return nil - }) - if err != nil { - t.Fatalf("walk production files: %v", err) - } - } -} - -func TestPublicPackagesHaveDocumentation(t *testing.T) { - t.Parallel() - - root := repositoryRoot(t) - packageDirs := map[string]bool{} - err := filepath.WalkDir(filepath.Join(root, "pkg"), func(path string, entry fs.DirEntry, walkErr error) error { - if walkErr != nil { - return walkErr - } - if entry.IsDir() || filepath.Ext(path) != ".go" || strings.HasSuffix(path, "_test.go") { - return nil - } - file, err := parser.ParseFile(token.NewFileSet(), path, nil, parser.PackageClauseOnly|parser.ParseComments) - if err != nil { - return err - } - dir := filepath.Dir(path) - if _, seen := packageDirs[dir]; !seen { - packageDirs[dir] = false - } - if file.Doc != nil && strings.TrimSpace(file.Doc.Text()) != "" { - packageDirs[dir] = true - } - return nil - }) - if err != nil { - t.Fatalf("scan pkg packages: %v", err) - } - - var undocumented []string - for dir, documented := range packageDirs { - if !documented { - rel, err := filepath.Rel(root, dir) - if err != nil { - t.Fatalf("relative package path: %v", err) - } - undocumented = append(undocumented, rel) - } - } - sort.Strings(undocumented) - if len(undocumented) > 0 { - t.Fatalf("pkg packages without package documentation: %s", strings.Join(undocumented, ", ")) - } -} - -func repositoryRoot(t *testing.T) string { - t.Helper() - _, file, _, ok := runtime.Caller(0) - if !ok { - t.Fatal("locate repository root") - } - return filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", "..")) -} diff --git a/internal/adapters/builtin/registry.go b/internal/adapters/builtin/registry.go index e4fc742d..3e349ae7 100644 --- a/internal/adapters/builtin/registry.go +++ b/internal/adapters/builtin/registry.go @@ -18,13 +18,24 @@ type Registries struct { Storage *backendadapter.Registry } -// New constructs the complete built-in registries. Runtime options are applied +// Options configures the runtime adapters shipped by the controller binary. +// It belongs to the built-in composition rather than the public adapter seam. +type Options struct { + SubscriberImage string + PolicyServerGRPCAddress string +} + +// New constructs the complete built-in registries. Subscriber settings are applied // consistently to every adapter that injects the subscriber sidecar. -func New(opts ...adapterruntime.Option) Registries { +func New(opts Options) Registries { + subscriber := builtinruntime.SubscriberConfig{ + Image: opts.SubscriberImage, + PolicyServerGRPCAddress: opts.PolicyServerGRPCAddress, + } runtimeRegistry := adapterruntime.NewRegistry() - runtimeRegistry.Register(builtinruntime.NewVLLMLMCacheAdapter(opts...)) - runtimeRegistry.Register(builtinruntime.NewSGLangLMCacheAdapter(opts...)) - runtimeRegistry.Register(builtinruntime.NewSGLangHiCacheAdapter(opts...)) + runtimeRegistry.Register(builtinruntime.NewVLLMLMCacheAdapter(subscriber)) + runtimeRegistry.Register(builtinruntime.NewSGLangLMCacheAdapter(subscriber)) + runtimeRegistry.Register(builtinruntime.NewSGLangHiCacheAdapter(subscriber)) return Registries{ Runtime: runtimeRegistry, diff --git a/internal/adapters/builtin/registry_test.go b/internal/adapters/builtin/registry_test.go index f978d9b3..7e26bcb2 100644 --- a/internal/adapters/builtin/registry_test.go +++ b/internal/adapters/builtin/registry_test.go @@ -14,7 +14,7 @@ import ( func TestNewIncludesEveryShippingRuntimeAdapter(t *testing.T) { t.Parallel() - registry := New().Runtime + registry := New(Options{}).Runtime for _, tc := range []struct { name string runtime adapterruntime.RuntimeID @@ -40,7 +40,7 @@ func TestNewIncludesEveryShippingRuntimeAdapter(t *testing.T) { func TestNewIncludesShippingStorageProviders(t *testing.T) { t.Parallel() - registry := New().Storage + registry := New(Options{}).Storage for _, provider := range []cachev1alpha1.CacheBackendRemoteStorageProvider{ cachev1alpha1.CacheBackendRemoteStorageProviderRedis, cachev1alpha1.CacheBackendRemoteStorageProviderLMCacheServer, diff --git a/internal/adapters/builtin/runtime/contract_aliases_test.go b/internal/adapters/builtin/runtime/contract_aliases_test.go deleted file mode 100644 index 5cebaa7c..00000000 --- a/internal/adapters/builtin/runtime/contract_aliases_test.go +++ /dev/null @@ -1,43 +0,0 @@ -// SPDX-FileCopyrightText: 2026 The inference-cache Authors -// -// SPDX-License-Identifier: Apache-2.0 - -package runtime - -import adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" - -type ( - KVCacheRuntimeAdapter = adapterruntime.KVCacheRuntimeAdapter - InitContainerProvider = adapterruntime.InitContainerProvider - Options = adapterruntime.Options - Option = adapterruntime.Option - RuntimeID = adapterruntime.RuntimeID - SupportedPair = adapterruntime.SupportedPair - SubscriberSidecarParams = adapterruntime.SubscriberSidecarParams -) - -const ( - RuntimeVLLM = adapterruntime.RuntimeVLLM - RuntimeSGLang = adapterruntime.RuntimeSGLang - RuntimeReference = adapterruntime.RuntimeReference - LMCacheKernelCheckContainerName = adapterruntime.LMCacheKernelCheckContainerName - AnnotationLMCacheKernelCheck = adapterruntime.AnnotationLMCacheKernelCheck - KernelCheckModeAuto = adapterruntime.KernelCheckModeAuto - KernelCheckModeReportOnly = adapterruntime.KernelCheckModeReportOnly - KernelCheckModeStrict = adapterruntime.KernelCheckModeStrict - KernelCheckModeOff = adapterruntime.KernelCheckModeOff - KernelCheckMsgOK = adapterruntime.KernelCheckMsgOK - KernelCheckMsgFailPrefix = adapterruntime.KernelCheckMsgFailPrefix - EnvKernelCheckStrict = adapterruntime.EnvKernelCheckStrict - DefaultSubscriberImage = adapterruntime.DefaultSubscriberImage - DefaultPolicyServerGRPCAddress = adapterruntime.DefaultPolicyServerGRPCAddress - SubscriberContainerName = adapterruntime.SubscriberContainerName -) - -var ( - RenderSubscriberSidecar = adapterruntime.RenderSubscriberSidecar - WithSubscriberImage = adapterruntime.WithSubscriberImage - WithPolicyServerGRPCAddress = adapterruntime.WithPolicyServerGRPCAddress - NewRegistry = adapterruntime.NewRegistry - NewReferenceAdapter = adapterruntime.NewReferenceAdapter -) diff --git a/internal/adapters/builtin/runtime/doc.go b/internal/adapters/builtin/runtime/doc.go index 0e2d04e7..59b7ab6b 100644 --- a/internal/adapters/builtin/runtime/doc.go +++ b/internal/adapters/builtin/runtime/doc.go @@ -16,7 +16,6 @@ // prefix bytes). The engine-side LMCache *launch* surface differs from vLLM // (--enable-lmcache + LMCACHE_USE_EXPERIMENTAL rather than // --kv-transfer-config). Managed cache-server rendering belongs to -// internal/adapters/builtin/storage; subscriber-sidecar rendering remains shared -// in pkg/adapters/runtime/kvevent_subscriber.go, with common defaults in -// pkg/adapters/runtime/lmcache_shared.go. +// internal/adapters/builtin/storage; subscriber-sidecar rendering and its +// deployment defaults are private implementation details in this package. package runtime diff --git a/internal/adapters/builtin/runtime/lmcachecheck.go b/internal/adapters/builtin/runtime/lmcachecheck.go index bf50ea32..66366893 100644 --- a/internal/adapters/builtin/runtime/lmcachecheck.go +++ b/internal/adapters/builtin/runtime/lmcachecheck.go @@ -9,7 +9,7 @@ import ( "k8s.io/apimachinery/pkg/api/resource" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" + "github.com/cachebox-project/inference-cache/internal/enginebinding" ) // gpuResourceName is the extended resource an engine container requests when @@ -69,17 +69,17 @@ except BaseException as e: // arrive — the fallback is a defense-in-depth default, not the typo guard. func resolveKernelCheckMode(cache *cachev1alpha1.CacheBackend) string { if cache == nil { - return adapterruntime.KernelCheckModeAuto + return enginebinding.KernelCheckModeAuto } - switch cache.Annotations[adapterruntime.AnnotationLMCacheKernelCheck] { - case adapterruntime.KernelCheckModeReportOnly: - return adapterruntime.KernelCheckModeReportOnly - case adapterruntime.KernelCheckModeStrict: - return adapterruntime.KernelCheckModeStrict - case adapterruntime.KernelCheckModeOff: - return adapterruntime.KernelCheckModeOff + switch cache.Annotations[enginebinding.AnnotationLMCacheKernelCheck] { + case enginebinding.KernelCheckModeReportOnly: + return enginebinding.KernelCheckModeReportOnly + case enginebinding.KernelCheckModeStrict: + return enginebinding.KernelCheckModeStrict + case enginebinding.KernelCheckModeOff: + return enginebinding.KernelCheckModeOff default: - return adapterruntime.KernelCheckModeAuto + return enginebinding.KernelCheckModeAuto } } @@ -154,31 +154,31 @@ func (vllmLMCacheAdapter) KernelCheckInitContainer(cache *cachev1alpha1.CacheBac return nil, nil } mode := resolveKernelCheckMode(cache) - if mode == adapterruntime.KernelCheckModeOff { + if mode == enginebinding.KernelCheckModeOff { return nil, nil } engine := engineContainerForKernelCheck(pod) if engine == nil || engine.Image == "" { return nil, nil } - if mode == adapterruntime.KernelCheckModeAuto && !requestsGPU(engine) { + if mode == enginebinding.KernelCheckModeAuto && !requestsGPU(engine) { return nil, nil } strictValue := "0" - if mode == adapterruntime.KernelCheckModeStrict { + if mode == enginebinding.KernelCheckModeStrict { strictValue = "1" } env := make([]corev1.EnvVar, 0, len(engine.Env)+1) for _, entry := range engine.Env { - if entry.Name != adapterruntime.EnvKernelCheckStrict { + if entry.Name != enginebinding.EnvKernelCheckStrict { env = append(env, entry) } } - env = append(env, corev1.EnvVar{Name: adapterruntime.EnvKernelCheckStrict, Value: strictValue}) + env = append(env, corev1.EnvVar{Name: enginebinding.EnvKernelCheckStrict, Value: strictValue}) return &corev1.Container{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, Image: engine.Image, ImagePullPolicy: engine.ImagePullPolicy, SecurityContext: engine.SecurityContext.DeepCopy(), diff --git a/internal/adapters/builtin/runtime/lmcachecheck_script_test.go b/internal/adapters/builtin/runtime/lmcachecheck_script_test.go index 1301a059..4ba754e9 100644 --- a/internal/adapters/builtin/runtime/lmcachecheck_script_test.go +++ b/internal/adapters/builtin/runtime/lmcachecheck_script_test.go @@ -10,6 +10,8 @@ import ( "path/filepath" "strings" "testing" + + "github.com/cachebox-project/inference-cache/internal/enginebinding" ) // runScript runs kernelCheckScript under python3 with PYTHONPATH=pkgParent and @@ -39,14 +41,14 @@ func runScript(t *testing.T, pkgParent string, strict bool) (string, int) { for _, kv := range os.Environ() { if strings.HasPrefix(kv, "PYTHONPATH=") || strings.HasPrefix(kv, "KERNEL_CHECK_MSG=") || - strings.HasPrefix(kv, EnvKernelCheckStrict+"=") { + strings.HasPrefix(kv, enginebinding.EnvKernelCheckStrict+"=") { continue } env = append(env, kv) } env = append(env, "PYTHONPATH="+pkgParent, "KERNEL_CHECK_MSG="+msg) if strict { - env = append(env, EnvKernelCheckStrict+"=1") + env = append(env, enginebinding.EnvKernelCheckStrict+"=1") } cmd.Env = env err = cmd.Run() @@ -81,7 +83,7 @@ func TestKernelCheckScriptNoNativeSoReportOnlyExitsZero(t *testing.T) { if code != 0 { t.Errorf("report-only exit code = %d, want 0", code) } - if !strings.HasPrefix(strings.TrimSpace(msg), KernelCheckMsgFailPrefix) { + if !strings.HasPrefix(strings.TrimSpace(msg), enginebinding.KernelCheckMsgFailPrefix) { t.Errorf("message = %q, want FAIL: prefix", msg) } if !strings.Contains(msg, "no native c_ops") { @@ -94,7 +96,7 @@ func TestKernelCheckScriptNoNativeSoStrictExitsOne(t *testing.T) { if code != 1 { t.Errorf("strict exit code = %d, want 1", code) } - if !strings.HasPrefix(strings.TrimSpace(msg), KernelCheckMsgFailPrefix) { + if !strings.HasPrefix(strings.TrimSpace(msg), enginebinding.KernelCheckMsgFailPrefix) { t.Errorf("message = %q, want FAIL: prefix", msg) } } @@ -160,7 +162,7 @@ func TestKernelCheckScriptHealthyExtensionReportsOK(t *testing.T) { if code != 0 { t.Errorf("exit = %d, want 0", code) } - if strings.TrimSpace(msg) != KernelCheckMsgOK { - t.Errorf("message = %q, want %q (a loadable native c_ops must report OK, not a false FAIL)", msg, KernelCheckMsgOK) + if strings.TrimSpace(msg) != enginebinding.KernelCheckMsgOK { + t.Errorf("message = %q, want %q (a loadable native c_ops must report OK, not a false FAIL)", msg, enginebinding.KernelCheckMsgOK) } } diff --git a/internal/adapters/builtin/runtime/lmcachecheck_test.go b/internal/adapters/builtin/runtime/lmcachecheck_test.go index c7691e4a..83b64458 100644 --- a/internal/adapters/builtin/runtime/lmcachecheck_test.go +++ b/internal/adapters/builtin/runtime/lmcachecheck_test.go @@ -13,6 +13,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/enginebinding" ) func gpuEnginePod(image string) *corev1.Pod { @@ -33,13 +34,13 @@ func cbWithKernelCheck(mode string) *cachev1alpha1.CacheBackend { Spec: cachev1alpha1.CacheBackendSpec{Type: cachev1alpha1.CacheBackendTypeLMCache}, } if mode != "" { - cb.Annotations = map[string]string{AnnotationLMCacheKernelCheck: mode} + cb.Annotations = map[string]string{enginebinding.AnnotationLMCacheKernelCheck: mode} } return cb } func TestKernelCheckAutoInjectsOnGPUPod(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) c, err := a.KernelCheckInitContainer(cbWithKernelCheck(""), gpuEnginePod("vllm/img:cu129")) if err != nil { t.Fatalf("unexpected error: %v", err) @@ -47,8 +48,8 @@ func TestKernelCheckAutoInjectsOnGPUPod(t *testing.T) { if c == nil { t.Fatal("expected an init container for a GPU LMCache engine pod under auto mode") } - if c.Name != LMCacheKernelCheckContainerName { - t.Errorf("name = %q, want %q", c.Name, LMCacheKernelCheckContainerName) + if c.Name != enginebinding.LMCacheKernelCheckContainerName { + t.Errorf("name = %q, want %q", c.Name, enginebinding.LMCacheKernelCheckContainerName) } if c.Image != "vllm/img:cu129" { t.Errorf("image = %q, want engine image", c.Image) @@ -77,9 +78,9 @@ func TestKernelCheckAutoInjectsOnGPUPod(t *testing.T) { } func TestKernelCheckCommandIdenticalAcrossModesEnvDiffers(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) - ro, _ := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeReportOnly), gpuEnginePod("img")) - st, _ := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeStrict), gpuEnginePod("img")) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) + ro, _ := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeReportOnly), gpuEnginePod("img")) + st, _ := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeStrict), gpuEnginePod("img")) if ro == nil || st == nil { t.Fatal("expected init containers for both modes") } @@ -105,7 +106,7 @@ func hasStrictEnv(c *corev1.Container) bool { return false } for _, e := range c.Env { - if e.Name == EnvKernelCheckStrict && e.Value == "1" { + if e.Name == enginebinding.EnvKernelCheckStrict && e.Value == "1" { return true } } @@ -118,7 +119,7 @@ func strictEnvValue(c *corev1.Container) (string, int) { return v, n } for _, e := range c.Env { - if e.Name == EnvKernelCheckStrict { + if e.Name == enginebinding.EnvKernelCheckStrict { v = e.Value n++ } @@ -127,26 +128,26 @@ func strictEnvValue(c *corev1.Container) (string, int) { } func TestKernelCheckStripsInheritedStrictEnv(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) pod := gpuEnginePod("img") // The engine container carries a stray KERNEL_CHECK_STRICT=1. It must NOT // leak into the report-only check (which would turn it fail-closed) and must // not appear twice. - pod.Spec.Containers[0].Env = []corev1.EnvVar{{Name: EnvKernelCheckStrict, Value: "1"}} + pod.Spec.Containers[0].Env = []corev1.EnvVar{{Name: enginebinding.EnvKernelCheckStrict, Value: "1"}} - ro, _ := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeReportOnly), pod) + ro, _ := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeReportOnly), pod) if v, n := strictEnvValue(ro); v != "0" || n != 1 { t.Errorf("report-only KERNEL_CHECK_STRICT = %q x%d, want \"0\" x1 (inherited value stripped + overridden)", v, n) } - st, _ := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeStrict), pod) + st, _ := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeStrict), pod) if v, n := strictEnvValue(st); v != "1" || n != 1 { t.Errorf("strict KERNEL_CHECK_STRICT = %q x%d, want \"1\" x1", v, n) } } func TestKernelCheckAutoSkipsCPUPod(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) cpuPod := &corev1.Pod{Spec: corev1.PodSpec{Containers: []corev1.Container{{ Name: EngineContainerName, Image: "vllm/cpu", }}}} @@ -160,36 +161,36 @@ func TestKernelCheckAutoSkipsCPUPod(t *testing.T) { } func TestKernelCheckOffSkipsEvenGPU(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) - c, _ := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeOff), gpuEnginePod("img")) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) + c, _ := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeOff), gpuEnginePod("img")) if c != nil { t.Fatal("off mode must never inject") } } func TestKernelCheckReportOnlyInjectsOnCPU(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) cpuPod := &corev1.Pod{Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: EngineContainerName, Image: "img"}}}} - c, _ := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeReportOnly), cpuPod) + c, _ := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeReportOnly), cpuPod) if c == nil { t.Fatal("report-only must inject regardless of GPU request") } for _, e := range c.Env { - if e.Name == EnvKernelCheckStrict && e.Value == "1" { + if e.Name == enginebinding.EnvKernelCheckStrict && e.Value == "1" { t.Error("report-only must not set STRICT=1") } } } func TestKernelCheckStrictSetsStrictEnv(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) - c, _ := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeStrict), gpuEnginePod("img")) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) + c, _ := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeStrict), gpuEnginePod("img")) if c == nil { t.Fatal("strict must inject") } got := "" for _, e := range c.Env { - if e.Name == EnvKernelCheckStrict { + if e.Name == enginebinding.EnvKernelCheckStrict { got = e.Value } } @@ -199,12 +200,12 @@ func TestKernelCheckStrictSetsStrictEnv(t *testing.T) { } func TestKernelCheckMultiContainerNoEngineNameSkips(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) pod := &corev1.Pod{Spec: corev1.PodSpec{Containers: []corev1.Container{ {Name: "foo", Image: "a", Resources: corev1.ResourceRequirements{Limits: corev1.ResourceList{gpuResourceName: resource.MustParse("1")}}}, {Name: "bar", Image: "b"}, }}} - c, err := a.KernelCheckInitContainer(cbWithKernelCheck(KernelCheckModeStrict), pod) + c, err := a.KernelCheckInitContainer(cbWithKernelCheck(enginebinding.KernelCheckModeStrict), pod) if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -214,7 +215,7 @@ func TestKernelCheckMultiContainerNoEngineNameSkips(t *testing.T) { } func TestKernelCheckCopiesEngineEnvironment(t *testing.T) { - a := NewVLLMLMCacheAdapter().(InitContainerProvider) + a := NewVLLMLMCacheAdapter(SubscriberConfig{}).(enginebinding.InitContainerProvider) nonRoot := true engineSC := &corev1.SecurityContext{RunAsNonRoot: &nonRoot} pod := gpuEnginePod("img") diff --git a/internal/adapters/builtin/runtime/sglang_hicache.go b/internal/adapters/builtin/runtime/sglang_hicache.go index 71ab545e..f68756c1 100644 --- a/internal/adapters/builtin/runtime/sglang_hicache.go +++ b/internal/adapters/builtin/runtime/sglang_hicache.go @@ -27,21 +27,13 @@ const ( ) type sglangHiCacheAdapter struct { - subscriberImage string - policyServerGRPCAddress string + subscriber SubscriberConfig } // NewSGLangHiCacheAdapter returns the endpoint-free adapter for SGLang's native // host-memory hierarchical cache. -func NewSGLangHiCacheAdapter(opts ...runtimeadapter.Option) runtimeadapter.KVCacheRuntimeAdapter { - var cfg runtimeadapter.Options - for _, option := range opts { - option(&cfg) - } - return sglangHiCacheAdapter{ - subscriberImage: cfg.SubscriberImage, - policyServerGRPCAddress: cfg.PolicyServerGRPCAddress, - } +func NewSGLangHiCacheAdapter(subscriber SubscriberConfig) runtimeadapter.KVCacheRuntimeAdapter { + return sglangHiCacheAdapter{subscriber: subscriber} } func (sglangHiCacheAdapter) Supports(runtime runtimeadapter.RuntimeID, cache *cachev1alpha1.CacheBackend) bool { @@ -184,9 +176,8 @@ func (sglangHiCacheAdapter) InjectRouterConfig(*corev1.PodSpec, *backendadapter. } func (a sglangHiCacheAdapter) ObservationSidecar(cache *cachev1alpha1.CacheBackend, pod *corev1.Pod) (*corev1.Container, error) { - return runtimeadapter.RenderSubscriberSidecar(runtimeadapter.SubscriberSidecarParams{ - Image: a.subscriberImage, - ServerAddr: a.policyServerGRPCAddress, + return renderSubscriberSidecar(subscriberSidecarParams{ + Config: a.subscriber, Cache: cache, Pod: pod, HashScheme: sglangSubscriberHashScheme, diff --git a/internal/adapters/builtin/runtime/sglang_hicache_test.go b/internal/adapters/builtin/runtime/sglang_hicache_test.go index f22953a2..082dc6b7 100644 --- a/internal/adapters/builtin/runtime/sglang_hicache_test.go +++ b/internal/adapters/builtin/runtime/sglang_hicache_test.go @@ -36,7 +36,7 @@ func newHiCacheBackend(spec *cachev1alpha1.SGLangHiCacheSpec) *cachev1alpha1.Cac } func TestHiCacheAdapterContract(t *testing.T) { - adapter := NewSGLangHiCacheAdapter() + adapter := NewSGLangHiCacheAdapter(SubscriberConfig{}) cache := newHiCacheBackend(&cachev1alpha1.SGLangHiCacheSpec{Ratio: "2"}) if !adapter.Supports(runtimeadapter.RuntimeSGLang, cache) { @@ -80,7 +80,7 @@ func TestHiCacheInjectsOnlyRequestedFlags(t *testing.T) { } beforeNonArgs := pod.DeepCopy() - if err := NewSGLangHiCacheAdapter().InjectEngineConfig(pod, nil, cache); err != nil { + if err := NewSGLangHiCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, nil, cache); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } args := pod.Containers[0].Args @@ -110,7 +110,7 @@ func TestHiCacheInjectsOnlyRequestedFlags(t *testing.T) { func TestHiCacheOptionalFieldsStayOmitted(t *testing.T) { cache := newHiCacheBackend(&cachev1alpha1.SGLangHiCacheSpec{Ratio: "1.5"}) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: "only"}}} - if err := NewSGLangHiCacheAdapter().InjectEngineConfig(pod, nil, cache); err != nil { + if err := NewSGLangHiCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, nil, cache); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } if got, ok := testArgValue(pod.Containers[0].Args, SGLangHiCacheRatioArg); !ok || got != "1.5" { @@ -143,7 +143,7 @@ func TestHiCacheMatchingArgsArePreserved(t *testing.T) { Name: SGLangEngineContainerName, Args: append([]string(nil), originalArgs...), }}} - if err := NewSGLangHiCacheAdapter().InjectEngineConfig(pod, nil, cache); err != nil { + if err := NewSGLangHiCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, nil, cache); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } if !reflect.DeepEqual(pod.Containers[0].Args, originalArgs) { @@ -182,7 +182,7 @@ func TestHiCacheConflictsFailAtomically(t *testing.T) { Volumes: []corev1.Volume{{Name: "keep"}}, } before := pod.DeepCopy() - if err := NewSGLangHiCacheAdapter().InjectEngineConfig(pod, nil, base); err == nil { + if err := NewSGLangHiCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, nil, base); err == nil { t.Fatal("InjectEngineConfig returned no error") } if !reflect.DeepEqual(pod, before) { @@ -213,7 +213,7 @@ func TestHiCacheOmittedOptionalArgsFailAtomicallyWhenMalformedOrDuplicated(t *te Env: []corev1.EnvVar{{Name: "KEEP", Value: "yes"}}, }}} before := pod.DeepCopy() - if err := NewSGLangHiCacheAdapter().InjectEngineConfig(pod, nil, cache); err == nil { + if err := NewSGLangHiCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, nil, cache); err == nil { t.Fatal("InjectEngineConfig returned no error") } if !reflect.DeepEqual(pod, before) { @@ -274,7 +274,7 @@ func TestHiCacheRejectsInvalidBackendAtAdapterBoundary(t *testing.T) { cache := newHiCacheBackend(&cachev1alpha1.SGLangHiCacheSpec{Ratio: "2"}) tc.mutate(cache) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: "sglang"}}} - if err := NewSGLangHiCacheAdapter().InjectEngineConfig(pod, nil, cache); err == nil { + if err := NewSGLangHiCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, nil, cache); err == nil { t.Fatal("InjectEngineConfig returned no error") } if len(pod.Containers[0].Args) != 0 { @@ -290,14 +290,14 @@ func TestHiCacheMultiContainerRequiresSGLangName(t *testing.T) { {Name: "engine"}, {Name: "metrics"}, }} - if err := NewSGLangHiCacheAdapter().InjectEngineConfig(pod, nil, cache); err == nil || + if err := NewSGLangHiCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, nil, cache); err == nil || !strings.Contains(err.Error(), `none is named "sglang"`) { t.Fatalf("InjectEngineConfig error = %v, want missing sglang container", err) } } func TestHiCacheReservedArgs(t *testing.T) { - got := NewSGLangHiCacheAdapter().ReservedArgs() + got := NewSGLangHiCacheAdapter(SubscriberConfig{}).ReservedArgs() want := []string{ SGLangEnableHiCacheArg, SGLangHiCacheSizeArg, @@ -309,7 +309,7 @@ func TestHiCacheReservedArgs(t *testing.T) { if !reflect.DeepEqual(got, want) { t.Fatalf("ReservedArgs = %v, want %v", got, want) } - if got := NewSGLangHiCacheAdapter().ReservedEnv(); len(got) != 0 { + if got := NewSGLangHiCacheAdapter(SubscriberConfig{}).ReservedEnv(); len(got) != 0 { t.Fatalf("ReservedEnv = %v, want empty", got) } } @@ -318,8 +318,7 @@ func TestHiCacheObservationSidecarReusesSGLangRenderer(t *testing.T) { cache := newHiCacheBackend(&cachev1alpha1.SGLangHiCacheSpec{Ratio: "2"}) cache.Spec.Observation = &cachev1alpha1.CacheBackendObservationSpec{ModelID: "model-a"} adapter := NewSGLangHiCacheAdapter( - runtimeadapter.WithSubscriberImage("subscriber:test"), - runtimeadapter.WithPolicyServerGRPCAddress("policy:50051"), + SubscriberConfig{Image: "subscriber:test", PolicyServerGRPCAddress: "policy:50051"}, ) sidecar, err := adapter.ObservationSidecar(cache, &corev1.Pod{}) if err != nil { diff --git a/internal/adapters/builtin/runtime/sglang_lmcache.go b/internal/adapters/builtin/runtime/sglang_lmcache.go index d2a9ce82..97ad31f9 100644 --- a/internal/adapters/builtin/runtime/sglang_lmcache.go +++ b/internal/adapters/builtin/runtime/sglang_lmcache.go @@ -51,30 +51,12 @@ const ( // GPU-validated end-to-end; full design: docs/design/sglang-lmcache-mp-mode.md. The // kvevent-subscriber sidecar rendering is still shared engine-agnostically. type sglangLMCacheAdapter struct { - // subscriberImage is the image the kvevent-subscriber sidecar runs. - // Empty (the default) disables sidecar auto-attach — ObservationSidecar - // returns nil — so an unconfigured controller install doesn't push engine - // pods into ImagePullBackOff on a nonexistent default image. - subscriberImage string - // policyServerGRPCAddress overrides the default in-cluster Service DNS the - // sidecar dials to ReportCacheState. Empty falls back to - // [runtimeadapter.DefaultPolicyServerGRPCAddress]. - policyServerGRPCAddress string + subscriber SubscriberConfig } -// NewSGLangLMCacheAdapter returns the runtime adapter for the (sglang, LMCache) pair. The -// optional [runtimeadapter.Option] helpers let the controller pin the -// subscriber sidecar's image + policy-server target — the same options -// the built-in composition applies uniformly to every shipping adapter. -func NewSGLangLMCacheAdapter(opts ...runtimeadapter.Option) runtimeadapter.KVCacheRuntimeAdapter { - var cfg runtimeadapter.Options - for _, o := range opts { - o(&cfg) - } - return sglangLMCacheAdapter{ - subscriberImage: cfg.SubscriberImage, - policyServerGRPCAddress: cfg.PolicyServerGRPCAddress, - } +// NewSGLangLMCacheAdapter returns the runtime adapter for the (sglang, LMCache) pair. +func NewSGLangLMCacheAdapter(subscriber SubscriberConfig) runtimeadapter.KVCacheRuntimeAdapter { + return sglangLMCacheAdapter{subscriber: subscriber} } // Supports matches SGLang engines against an LMCache CacheBackend. Every other @@ -131,7 +113,7 @@ func (sglangLMCacheAdapter) InjectRouterConfig(pod *corev1.PodSpec, binding *bac // ObservationSidecar returns the kvevent-subscriber container the Pod webhook // appends to an SGLang engine pod so its KV-cache events flow to the policy -// server. It delegates to the shared [runtimeadapter.RenderSubscriberSidecar], +// server. It delegates to the shared internal subscriber renderer, // pinning the SGLang-specific knobs: --hash-scheme=sglang (so the index keeps // SGLang prefixes disjoint from vLLM's) and SGLang's ZMQ PUB port. The // eviction-forwarding policy (--ignore-block-removed) is mode-dependent and @@ -140,9 +122,8 @@ func (sglangLMCacheAdapter) InjectRouterConfig(pod *corev1.PodSpec, binding *bac // binary decodes SGLang's KV-event stream unchanged because SGLang emits the // same msgspec BlockStored/BlockRemoved/AllBlocksCleared wire vLLM does. func (a sglangLMCacheAdapter) ObservationSidecar(cache *cachev1alpha1.CacheBackend, pod *corev1.Pod) (*corev1.Container, error) { - return runtimeadapter.RenderSubscriberSidecar(runtimeadapter.SubscriberSidecarParams{ - Image: a.subscriberImage, - ServerAddr: a.policyServerGRPCAddress, + return renderSubscriberSidecar(subscriberSidecarParams{ + Config: a.subscriber, Cache: cache, Pod: pod, HashScheme: sglangSubscriberHashScheme, diff --git a/internal/adapters/builtin/runtime/sglang_lmcache_test.go b/internal/adapters/builtin/runtime/sglang_lmcache_test.go index 4d17cf07..c6e6c428 100644 --- a/internal/adapters/builtin/runtime/sglang_lmcache_test.go +++ b/internal/adapters/builtin/runtime/sglang_lmcache_test.go @@ -18,6 +18,7 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" provideradapter "github.com/cachebox-project/inference-cache/internal/adapters/builtin/storage" + "github.com/cachebox-project/inference-cache/internal/enginebinding" backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" runtimeadapter "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) @@ -95,7 +96,7 @@ func resolveRedisServer(_ runtimeadapter.KVCacheRuntimeAdapter, cb *cachev1alpha } func TestSGLangSupports(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cases := []struct { name string runtime runtimeadapter.RuntimeID @@ -117,7 +118,7 @@ func TestSGLangSupports(t *testing.T) { } func TestSGLangSupportedPairs(t *testing.T) { - a := NewSGLangLMCacheAdapter().(interface { + a := NewSGLangLMCacheAdapter(SubscriberConfig{}).(interface { SupportedPairs() []runtimeadapter.SupportedPair }) got := a.SupportedPairs() @@ -128,7 +129,7 @@ func TestSGLangSupportedPairs(t *testing.T) { } func TestSGLangResolveCacheServer(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) pod, svc, err := resolveRedisServer(a, newSGLangBackend(nil)) if err != nil { t.Fatalf("ResolveCacheServer: %v", err) @@ -149,7 +150,7 @@ func TestSGLangResolveCacheServer(t *testing.T) { } func TestSGLangResolveCacheServerImageOverride(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(map[string]string{"redisImage": "registry.example.com/redis:pinned"}) pod, _, err := resolveRedisServer(a, cb) if err != nil { @@ -161,7 +162,7 @@ func TestSGLangResolveCacheServerImageOverride(t *testing.T) { } func TestSGLangResolveCacheServerNilCache(t *testing.T) { - if _, _, err := resolveRedisServer(NewSGLangLMCacheAdapter(), nil); err == nil { + if _, _, err := resolveRedisServer(NewSGLangLMCacheAdapter(SubscriberConfig{}), nil); err == nil { t.Fatalf("ResolveCacheServer(nil) returned no error") } } @@ -182,7 +183,7 @@ func TestSGLangCanonicalHostOnlyBindingDoesNotSelectRedis(t *testing.T) { }, } pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: "sglang", Image: "sglang:test"}}} - adapter := NewSGLangLMCacheAdapter() + adapter := NewSGLangLMCacheAdapter(SubscriberConfig{}) if !adapter.SupportsBinding(nil) { t.Fatal("SGLang LMCache adapter rejected host-only binding") } @@ -203,7 +204,7 @@ func TestSGLangCanonicalHostOnlyBindingDoesNotSelectRedis(t *testing.T) { } func TestSGLangInjectEngineConfig(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(nil) pod := &corev1.PodSpec{ Containers: []corev1.Container{ @@ -303,7 +304,7 @@ func TestSGLangInjectEngineConfig(t *testing.T) { } func TestSGLangInjectEngineConfigSingleContainerPodAcceptsAnyName(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: "engine", Image: "img"}}} if err := a.InjectEngineConfig(pod, respBinding("cache.ns1.svc:6379"), cb); err != nil { @@ -315,7 +316,7 @@ func TestSGLangInjectEngineConfigSingleContainerPodAcceptsAnyName(t *testing.T) } func TestSGLangInjectEngineConfigMultiContainerWithoutSGLangNameErrors(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{ {Name: "engine"}, @@ -333,7 +334,7 @@ func TestSGLangInjectEngineConfigMultiContainerWithoutSGLangNameErrors(t *testin } func TestSGLangInjectEngineConfigIdempotent(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: SGLangEngineContainerName, Image: "img"}}} if err := a.InjectEngineConfig(pod, respBinding("first.svc:6379"), cb); err != nil { @@ -374,7 +375,7 @@ func TestSGLangInjectEngineConfigIdempotent(t *testing.T) { } func TestSGLangInjectEngineConfigConfigOverrides(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(map[string]string{ "chunkSize": "512", "l1SizeGB": "8", @@ -408,7 +409,7 @@ func TestSGLangInjectEngineConfigReusesExistingDevShm(t *testing.T) { // SECOND mount at the same mountPath makes the Pod invalid (the API server // rejects duplicate mountPaths), so injection must REUSE the engine's volume for // the worker rather than adding its own. - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) pod := &corev1.PodSpec{ Containers: []corev1.Container{{ Name: SGLangEngineContainerName, @@ -460,7 +461,7 @@ func TestSGLangInjectEngineConfigRejectsConfigPathCollision(t *testing.T) { Image: "sglang:test", VolumeMounts: []corev1.VolumeMount{{Name: "operator-cfg", MountPath: "/etc/lmcache"}}, }}} - err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) + err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) if err == nil { t.Fatalf("want an error when the engine already mounts the adapter-owned config path") } @@ -516,7 +517,7 @@ func TestSGLangInjectEngineConfigRejectsForeignReservedNames(t *testing.T) { for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { before := tc.pod.DeepCopy() - err := NewSGLangLMCacheAdapter().InjectEngineConfig(tc.pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) + err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(tc.pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) if err == nil { t.Fatalf("want an error when %s", tc.name) } @@ -541,7 +542,7 @@ func TestSGLangInjectEngineConfigReinjectionConvergesOnCurrentRender(t *testing. // status.endpoint here). Value-equality against a fresh render would misread this // as foreign; the second injection must instead converge the worker on the new // endpoint rather than reject it, duplicate it, or leave the stale one. - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: SGLangEngineContainerName, Image: "img"}}} if err := a.InjectEngineConfig(pod, respBinding("first.svc:6379"), newSGLangBackend(nil)); err != nil { t.Fatalf("first InjectEngineConfig: %v", err) @@ -602,7 +603,7 @@ func TestSGLangInjectEngineConfigRejectsUnwritableDevShm(t *testing.T) { } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - err := NewSGLangLMCacheAdapter().InjectEngineConfig(tc.pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) + err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(tc.pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) if err == nil { t.Fatalf("want an error when the engine's /dev/shm is not writable scratch (%s)", tc.name) } @@ -628,7 +629,7 @@ func TestSGLangInjectEngineConfigReusesWritableNonEmptyDirDevShm(t *testing.T) { VolumeSource: corev1.VolumeSource{NFS: &corev1.NFSVolumeSource{Server: "s", Path: "/p", ReadOnly: false}}, }}, } - if err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { + if err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { t.Fatalf("InjectEngineConfig rejected a writable /dev/shm: %v", err) } w := findInitContainer(pod.InitContainers, "lmcache-mp-worker") @@ -655,7 +656,7 @@ func TestSGLangInjectEngineConfigWorkerSeesTheGPU(t *testing.T) { // the env is not dropped as dead weight — the failure it prevents is a wedged // engine, not a cache miss. pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: SGLangEngineContainerName, Image: "sglang:test"}}} - if err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { + if err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } w := findInitContainer(pod.InitContainers, "lmcache-mp-worker") @@ -679,7 +680,7 @@ func TestSGLangInjectEngineConfigWorkerRestrictedSecurityContext(t *testing.T) { // fail-open). And it must add NO capabilities (an added cap is itself a Restricted // violation; IPC_LOCK is not needed — GPU access is via device files, not caps). pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: SGLangEngineContainerName, Image: "sglang:test"}}} - if err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { + if err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } w := findInitContainer(pod.InitContainers, "lmcache-mp-worker") @@ -726,7 +727,7 @@ func TestSGLangInjectEngineConfigWorkerMirrorsEngineUserIdentity(t *testing.T) { RunAsNonRoot: &nonRoot, RunAsUser: &uid, RunAsGroup: &gid, }, }}} - if err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { + if err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } w := findInitContainer(pod.InitContainers, "lmcache-mp-worker") @@ -743,7 +744,7 @@ func TestSGLangInjectEngineConfigWorkerMirrorsEngineUserIdentity(t *testing.T) { // And it does NOT force a read-only rootfs or a fixed UID when the engine sets // none — that would risk breaking the vendor image's writes / CUDA-IPC. pod2 := &corev1.PodSpec{Containers: []corev1.Container{{Name: SGLangEngineContainerName, Image: "sglang:test"}}} - _ = NewSGLangLMCacheAdapter().InjectEngineConfig(pod2, respBinding("r.svc:6379"), newSGLangBackend(nil)) + _ = NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod2, respBinding("r.svc:6379"), newSGLangBackend(nil)) w2 := findInitContainer(pod2.InitContainers, "lmcache-mp-worker") if w2.SecurityContext.RunAsUser != nil { t.Errorf("runAsUser forced to %v when engine set none — must inherit from the pod, not override the image", w2.SecurityContext.RunAsUser) @@ -768,7 +769,7 @@ func TestSGLangInjectEngineConfigMirrorsDevShmSubPath(t *testing.T) { VolumeSource: corev1.VolumeSource{EmptyDir: &corev1.EmptyDirVolumeSource{Medium: corev1.StorageMediumMemory}}, }}, } - if err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { + if err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), newSGLangBackend(nil)); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } w := findInitContainer(pod.InitContainers, "lmcache-mp-worker") @@ -891,7 +892,7 @@ func TestSGLangInjectEngineConfigRejectsUnshareableDevShm(t *testing.T) { } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - err := NewSGLangLMCacheAdapter().InjectEngineConfig(tc.pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) + err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(tc.pod, respBinding("r.svc:6379"), newSGLangBackend(nil)) if err == nil { t.Fatalf("want an error when the engine's /dev/shm is unshareable (%s)", tc.name) } @@ -910,7 +911,7 @@ func TestSGLangInjectEngineConfigWorkerHasMemoryBudget(t *testing.T) { Name: SGLangEngineContainerName, Image: "sglang:test", }}} cb := newSGLangBackend(map[string]string{"l1SizeGB": "8"}) - if err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), cb); err != nil { + if err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), cb); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } w := findInitContainer(pod.InitContainers, "lmcache-mp-worker") @@ -958,7 +959,7 @@ func TestSGLangInjectEngineConfigSanitizesNumericConfig(t *testing.T) { t.Run(tc.key+"="+tc.bad, func(t *testing.T) { cb := newSGLangBackend(map[string]string{tc.key: tc.bad}) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: SGLangEngineContainerName, Image: "img"}}} - if err := NewSGLangLMCacheAdapter().InjectEngineConfig(pod, respBinding("r.svc:6379"), cb); err != nil { + if err := NewSGLangLMCacheAdapter(SubscriberConfig{}).InjectEngineConfig(pod, respBinding("r.svc:6379"), cb); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } joined := strings.Join(findInitContainer(pod.InitContainers, "lmcache-mp-worker").Args, " ") @@ -973,7 +974,7 @@ func TestSGLangInjectEngineConfigSanitizesNumericConfig(t *testing.T) { } func TestSGLangInjectEngineConfigFailOpen(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) trueVal, falseVal := true, false cases := []struct { name string @@ -1000,7 +1001,7 @@ func TestSGLangInjectEngineConfigFailOpen(t *testing.T) { } func TestSGLangInjectEngineConfigBadInput(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(nil) good := &corev1.PodSpec{Containers: []corev1.Container{{Name: SGLangEngineContainerName}}} cases := []struct { @@ -1029,7 +1030,7 @@ func TestSGLangInjectEngineConfigBadInput(t *testing.T) { } func TestSGLangInjectRouterConfigIsNoop(t *testing.T) { - a := NewSGLangLMCacheAdapter() + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) cb := newSGLangBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: "router", Env: []corev1.EnvVar{{Name: "EXISTING", Value: "x"}}}}} if err := a.InjectRouterConfig(pod, respBinding("x.svc:65432"), cb); err != nil { @@ -1046,7 +1047,7 @@ func TestSGLangInjectRouterConfigIsNoop(t *testing.T) { } func TestSGLangObservationSidecarShape(t *testing.T) { - a := NewSGLangLMCacheAdapter(runtimeadapter.WithSubscriberImage(runtimeadapter.DefaultSubscriberImage)) + a := NewSGLangLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newSGLangBackend(map[string]string{"model": "Qwen/Qwen2.5-0.5B-Instruct"}) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "sglang-a", Namespace: "engines"}} @@ -1057,15 +1058,15 @@ func TestSGLangObservationSidecarShape(t *testing.T) { if c == nil { t.Fatalf("ObservationSidecar returned nil for sglang+LMCache with a model + image set") } - if c.Name != runtimeadapter.SubscriberContainerName { - t.Fatalf("container name = %q, want %q", c.Name, runtimeadapter.SubscriberContainerName) + if c.Name != enginebinding.SubscriberContainerName { + t.Fatalf("container name = %q, want %q", c.Name, enginebinding.SubscriberContainerName) } if !envHasFieldRef(c.Env, "POD_NAME", "metadata.name") || !envHasFieldRef(c.Env, "POD_NAMESPACE", "metadata.namespace") { t.Fatalf("downward-API env missing: %v", c.Env) } wantArgs := []string{ "--engine-endpoint=tcp://127.0.0.1:5557", - "--server=" + runtimeadapter.DefaultPolicyServerGRPCAddress, + "--server=" + DefaultPolicyServerGRPCAddress, "--replica-id=$(POD_NAME)", "--tenant-id=$(POD_NAMESPACE)", "--model-id=Qwen/Qwen2.5-0.5B-Instruct", @@ -1092,7 +1093,7 @@ func TestSGLangObservationSidecarArgsParseAgainstSubscriberFlagSet(t *testing.T) // startup. Parse the rendered args through a FlagSet mirroring the binary's // event-path flag surface and assert they parse cleanly. Keep in sync with // cmd/kvevent-subscriber/main.go. - a := NewSGLangLMCacheAdapter(runtimeadapter.WithSubscriberImage(runtimeadapter.DefaultSubscriberImage)) + a := NewSGLangLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newSGLangBackend(map[string]string{"model": "Qwen/Qwen2.5-0.5B-Instruct"}) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "sglang-a", Namespace: "engines"}} c, err := a.ObservationSidecar(cb, pod) @@ -1123,10 +1124,10 @@ func TestSGLangObservationSidecarArgsParseAgainstSubscriberFlagSet(t *testing.T) } func TestSGLangObservationSidecarHonoursOptions(t *testing.T) { - a := NewSGLangLMCacheAdapter( - runtimeadapter.WithSubscriberImage("registry.example.com/subscriber:pinned"), - runtimeadapter.WithPolicyServerGRPCAddress("ic-server.custom-ns.svc.cluster.local:9090"), - ) + a := NewSGLangLMCacheAdapter(SubscriberConfig{ + Image: "registry.example.com/subscriber:pinned", + PolicyServerGRPCAddress: "ic-server.custom-ns.svc.cluster.local:9090", + }) cb := newSGLangBackend(map[string]string{"model": "MyOrg/MyModel"}) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "sglang-z", Namespace: "engines"}} c, err := a.ObservationSidecar(cb, pod) @@ -1142,7 +1143,7 @@ func TestSGLangObservationSidecarHonoursOptions(t *testing.T) { } func TestSGLangObservationSidecarSkipsWithoutModel(t *testing.T) { - a := NewSGLangLMCacheAdapter(runtimeadapter.WithSubscriberImage(runtimeadapter.DefaultSubscriberImage)) + a := NewSGLangLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newSGLangBackend(nil) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "sglang-a"}} c, err := a.ObservationSidecar(cb, pod) @@ -1155,7 +1156,7 @@ func TestSGLangObservationSidecarSkipsWithoutModel(t *testing.T) { } func TestSGLangObservationSidecarSkipsWithoutImage(t *testing.T) { - a := NewSGLangLMCacheAdapter() // no image configured → auto-attach opt-out + a := NewSGLangLMCacheAdapter(SubscriberConfig{}) // no image configured → auto-attach opt-out cb := newSGLangBackend(map[string]string{"model": "MyOrg/MyModel"}) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "sglang-a"}} c, err := a.ObservationSidecar(cb, pod) @@ -1168,7 +1169,7 @@ func TestSGLangObservationSidecarSkipsWithoutImage(t *testing.T) { } func TestSGLangObservationSidecarBadInput(t *testing.T) { - a := NewSGLangLMCacheAdapter(runtimeadapter.WithSubscriberImage(runtimeadapter.DefaultSubscriberImage)) + a := NewSGLangLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newSGLangBackend(map[string]string{"model": "m"}) cases := []struct { name string @@ -1188,7 +1189,7 @@ func TestSGLangObservationSidecarBadInput(t *testing.T) { } func TestSGLangReservedArgs(t *testing.T) { - got := NewSGLangLMCacheAdapter().ReservedArgs() + got := NewSGLangLMCacheAdapter(SubscriberConfig{}).ReservedArgs() want := []string{SGLangEnableLMCacheArg, SGLangConfigFileArg} if len(got) != len(want) { t.Fatalf("ReservedArgs = %v, want %v", got, want) @@ -1201,7 +1202,7 @@ func TestSGLangReservedArgs(t *testing.T) { } func TestSGLangReservedEnv(t *testing.T) { - got := NewSGLangLMCacheAdapter().ReservedEnv() + got := NewSGLangLMCacheAdapter(SubscriberConfig{}).ReservedEnv() want := []string{ EnvLMCacheUseExperimental, EnvInferenceCacheFailOpen, @@ -1232,7 +1233,7 @@ func TestSGLangReservedEnv(t *testing.T) { } func TestSGLangEngineContainerName(t *testing.T) { - if got := NewSGLangLMCacheAdapter().EngineContainerName(); got != SGLangEngineContainerName { + if got := NewSGLangLMCacheAdapter(SubscriberConfig{}).EngineContainerName(); got != SGLangEngineContainerName { t.Fatalf("EngineContainerName = %q, want %q", got, SGLangEngineContainerName) } } diff --git a/internal/adapters/builtin/runtime/subscriber.go b/internal/adapters/builtin/runtime/subscriber.go new file mode 100644 index 00000000..9d77d243 --- /dev/null +++ b/internal/adapters/builtin/runtime/subscriber.go @@ -0,0 +1,108 @@ +// SPDX-FileCopyrightText: 2026 The inference-cache Authors +// +// SPDX-License-Identifier: Apache-2.0 + +package runtime + +import ( + "fmt" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + + cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/enginebinding" +) + +const ( + DefaultSubscriberImage = "ghcr.io/cachebox-project/inference-cache-subscriber:dev" + + DefaultPolicyServerGRPCAddress = "inference-cache-server.inference-cache-system.svc.cluster.local:9090" +) + +// SubscriberConfig contains the shipping subscriber settings shared by the +// built-in runtime adapters. Its zero value preserves the existing behavior: +// no sidecar image is attached, and an empty server address selects the +// in-cluster default when a sidecar is rendered. +type SubscriberConfig struct { + Image string + PolicyServerGRPCAddress string +} + +type subscriberSidecarParams struct { + Config SubscriberConfig + Cache *cachev1alpha1.CacheBackend + Pod *corev1.Pod + HashScheme string + EngineZMQPortStr string +} + +func renderSubscriberSidecar(p subscriberSidecarParams) (*corev1.Container, error) { + if p.Cache == nil { + return nil, fmt.Errorf("observation sidecar: cache is nil") + } + if p.Pod == nil { + return nil, fmt.Errorf("observation sidecar: pod is nil") + } + if p.Config.Image == "" { + return nil, nil + } + modelID := p.Cache.Spec.EffectiveObservationModelID() + if modelID == "" { + return nil, nil + } + serverAddr := p.Config.PolicyServerGRPCAddress + if serverAddr == "" { + serverAddr = DefaultPolicyServerGRPCAddress + } + + args := []string{ + "--engine-endpoint=tcp://127.0.0.1:" + p.EngineZMQPortStr, + "--server=" + serverAddr, + "--replica-id=$(POD_NAME)", + "--tenant-id=$(POD_NAMESPACE)", + "--model-id=" + modelID, + "--hash-scheme=" + p.HashScheme, + } + if !p.Cache.Spec.IsEventsOnly() { + args = append(args, "--ignore-block-removed=true") + } + + nonRoot := true + noPrivEsc := false + readOnlyRoot := true + uid := int64(65532) + return &corev1.Container{ + Name: enginebinding.SubscriberContainerName, + Image: p.Config.Image, + ImagePullPolicy: corev1.PullIfNotPresent, + Env: []corev1.EnvVar{ + { + Name: "POD_NAME", + ValueFrom: &corev1.EnvVarSource{FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.name"}}, + }, + { + Name: "POD_NAMESPACE", + ValueFrom: &corev1.EnvVarSource{FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.namespace"}}, + }, + }, + Args: args, + Resources: corev1.ResourceRequirements{ + Requests: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("10m"), + corev1.ResourceMemory: resource.MustParse("64Mi"), + }, + Limits: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("200m"), + corev1.ResourceMemory: resource.MustParse("128Mi"), + }, + }, + SecurityContext: &corev1.SecurityContext{ + RunAsNonRoot: &nonRoot, + RunAsUser: &uid, + AllowPrivilegeEscalation: &noPrivEsc, + ReadOnlyRootFilesystem: &readOnlyRoot, + Capabilities: &corev1.Capabilities{Drop: []corev1.Capability{"ALL"}}, + }, + }, nil +} diff --git a/internal/adapters/builtin/runtime/vllm_lmcache.go b/internal/adapters/builtin/runtime/vllm_lmcache.go index 1cf0a20c..9281c2ec 100644 --- a/internal/adapters/builtin/runtime/vllm_lmcache.go +++ b/internal/adapters/builtin/runtime/vllm_lmcache.go @@ -10,6 +10,7 @@ import ( corev1 "k8s.io/api/core/v1" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/enginebinding" backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) @@ -43,31 +44,13 @@ const ( // the observation sidecar but uses its own MP engine wire and a Redis provider // binding rather than the standalone lmcache-server. type vllmLMCacheAdapter struct { - // subscriberImage is the image the kvevent-subscriber sidecar runs. - // Empty (the default) disables sidecar auto-attach — ObservationSidecar - // returns nil — so an unconfigured controller install doesn't push - // engine pods into ImagePullBackOff on a nonexistent default image. - subscriberImage string - // policyServerGRPCAddress overrides the default in-cluster Service DNS - // the sidecar dials to ReportCacheState. Empty falls back to - // [DefaultPolicyServerGRPCAddress]. - policyServerGRPCAddress string + subscriber SubscriberConfig } // NewVLLMLMCacheAdapter returns the adapter that wires vLLM engine pods to an -// LMCache CacheBackend. The optional [adapterruntime.Option] helpers let the controller pin -// the subscriber sidecar's image + policy-server target; the no-arg form -// reproduces the package defaults and keeps tests + the nil-Registry -// fallback paths working. -func NewVLLMLMCacheAdapter(opts ...adapterruntime.Option) adapterruntime.KVCacheRuntimeAdapter { - var cfg adapterruntime.Options - for _, o := range opts { - o(&cfg) - } - return vllmLMCacheAdapter{ - subscriberImage: cfg.SubscriberImage, - policyServerGRPCAddress: cfg.PolicyServerGRPCAddress, - } +// LMCache CacheBackend. +func NewVLLMLMCacheAdapter(subscriber SubscriberConfig) adapterruntime.KVCacheRuntimeAdapter { + return vllmLMCacheAdapter{subscriber: subscriber} } // Supports matches vLLM runtimes against an LMCache CacheBackend. Any other @@ -170,18 +153,12 @@ func (vllmLMCacheAdapter) InjectEngineConfig(pod *corev1.PodSpec, binding *backe } func injectMooncakeEngineHostNetwork(pod *corev1.PodSpec, cache *cachev1alpha1.CacheBackend) { - if EngineHostNetworkRequested(cache) { + if enginebinding.EngineHostNetworkRequested(cache) { pod.HostNetwork = true pod.DNSPolicy = corev1.DNSClusterFirstWithHostNet } } -// EngineHostNetworkRequested reports whether the operator opted engine pods -// using a Mooncake remote binding into host networking. -func EngineHostNetworkRequested(cache *cachev1alpha1.CacheBackend) bool { - return adapterruntime.EngineHostNetworkRequested(cache) -} - // InjectRouterConfig is a no-op for LMCache: the LMCache topology has no // router component the controller needs to wire. Returning nil keeps the // interface contract satisfied so a Registry caller can blindly invoke both @@ -199,7 +176,7 @@ func (vllmLMCacheAdapter) InjectRouterConfig(pod *corev1.PodSpec, binding *backe // ObservationSidecar returns the kvevent-subscriber container the Pod webhook // appends to a vLLM engine pod so its KV-cache events flow to the policy -// server. It delegates to the shared [adapterruntime.RenderSubscriberSidecar], pinning the +// server. It delegates to the shared internal subscriber renderer, pinning the // vLLM-specific knobs: --hash-scheme=vllm and the vLLM ZMQ PUB port. The // eviction-forwarding policy (--ignore-block-removed) is mode-dependent and // computed by the shared builder (suppressed in Offload where the L2 tier @@ -207,9 +184,8 @@ func (vllmLMCacheAdapter) InjectRouterConfig(pod *corev1.PodSpec, binding *backe // subscriber shape is identical for every vLLM-engine L2 backend (LMCache, // Mooncake) because the KV-event stream comes from vLLM itself, not the L2 store. func (a vllmLMCacheAdapter) ObservationSidecar(cache *cachev1alpha1.CacheBackend, pod *corev1.Pod) (*corev1.Container, error) { - return adapterruntime.RenderSubscriberSidecar(adapterruntime.SubscriberSidecarParams{ - Image: a.subscriberImage, - ServerAddr: a.policyServerGRPCAddress, + return renderSubscriberSidecar(subscriberSidecarParams{ + Config: a.subscriber, Cache: cache, Pod: pod, HashScheme: vllmSubscriberHashScheme, @@ -228,12 +204,3 @@ var ( kvTransferConfig = KVTransferConfig upsertArgPair = UpsertArgPair ) - -// ValidateExternalEndpoint is the shared canonical endpoint seam used by -// admission, reconciliation, and pod injection. It validates an -// operator-supplied endpoint against the selected remote provider's wire -// protocol. Bare host:port is portable across providers; explicit schemes are -// accepted only when the provider's engine wire consumes them. -func ValidateExternalEndpoint(provider cachev1alpha1.CacheBackendRemoteStorageProvider, endpoint string) error { - return adapterruntime.ValidateExternalEndpoint(provider, endpoint) -} diff --git a/internal/adapters/builtin/runtime/vllm_lmcache_test.go b/internal/adapters/builtin/runtime/vllm_lmcache_test.go index 53d171e7..84509562 100644 --- a/internal/adapters/builtin/runtime/vllm_lmcache_test.go +++ b/internal/adapters/builtin/runtime/vllm_lmcache_test.go @@ -18,7 +18,9 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" provideradapter "github.com/cachebox-project/inference-cache/internal/adapters/builtin/storage" + "github.com/cachebox-project/inference-cache/internal/enginebinding" backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" + runtimeadapter "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) func newLMCacheBackend(cfg map[string]string) *cachev1alpha1.CacheBackend { @@ -72,13 +74,13 @@ func lmCacheBinding(endpoint string) *backendadapter.Binding { // resolveLMCacheServer keeps the provider-rendering assertions independent // from the runtime adapter now that provider lifecycle is a separate seam. -func resolveLMCacheServer(_ KVCacheRuntimeAdapter, cb *cachev1alpha1.CacheBackend) (*corev1.PodSpec, *corev1.Service, error) { +func resolveLMCacheServer(_ runtimeadapter.KVCacheRuntimeAdapter, cb *cachev1alpha1.CacheBackend) (*corev1.PodSpec, *corev1.Service, error) { return provideradapter.ResolveLMCacheServer(cb) } // resolvePod unwraps the provider renderer for tests that only assert on the // rendered pod, failing on error or a nil result. -func resolvePod(t *testing.T, a KVCacheRuntimeAdapter, cb *cachev1alpha1.CacheBackend) *corev1.PodSpec { +func resolvePod(t *testing.T, a runtimeadapter.KVCacheRuntimeAdapter, cb *cachev1alpha1.CacheBackend) *corev1.PodSpec { t.Helper() pod, _, err := resolveLMCacheServer(a, cb) if err != nil { @@ -91,19 +93,19 @@ func resolvePod(t *testing.T, a KVCacheRuntimeAdapter, cb *cachev1alpha1.CacheBa } func TestVLLMLMCacheSupports(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cases := []struct { name string - runtime RuntimeID + runtime runtimeadapter.RuntimeID cache *cachev1alpha1.CacheBackend want bool }{ - {"vllm+lmcache", RuntimeVLLM, newLMCacheBackend(nil), true}, - {"vllm+unsupported", RuntimeVLLM, newCacheBackend(cachev1alpha1.CacheBackendType("unsupported"), "vllm"), false}, - {"sglang+lmcache", RuntimeSGLang, newLMCacheBackend(nil), false}, - {"reference+lmcache", RuntimeReference, newLMCacheBackend(nil), false}, - {"nil cache", RuntimeVLLM, nil, false}, + {"vllm+lmcache", runtimeadapter.RuntimeVLLM, newLMCacheBackend(nil), true}, + {"vllm+unsupported", runtimeadapter.RuntimeVLLM, newCacheBackend(cachev1alpha1.CacheBackendType("unsupported"), "vllm"), false}, + {"sglang+lmcache", runtimeadapter.RuntimeSGLang, newLMCacheBackend(nil), false}, + {"reference+lmcache", runtimeadapter.RuntimeID("reference"), newLMCacheBackend(nil), false}, + {"nil cache", runtimeadapter.RuntimeVLLM, nil, false}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -115,7 +117,7 @@ func TestVLLMLMCacheSupports(t *testing.T) { } func TestVLLMLMCacheResolveCacheServer(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) pod, svc, err := resolveLMCacheServer(a, cb) @@ -153,7 +155,7 @@ func TestVLLMLMCacheResolveCacheServer(t *testing.T) { } // Service spec: adapter fills Type + Ports only — ObjectMeta and Selector - // are the reconciler's responsibility (see KVCacheRuntimeAdapter docs). + // are the reconciler's responsibility (see runtimeadapter.KVCacheRuntimeAdapter docs). if svc.Spec.Type != corev1.ServiceTypeClusterIP { t.Fatalf("svc.Spec.Type = %q, want ClusterIP", svc.Spec.Type) } @@ -174,7 +176,7 @@ func TestVLLMLMCacheResolveCacheServer(t *testing.T) { // an overlay pod behind a virtual ClusterIP. hostNetwork is reserved for backends // whose data plane genuinely cannot work without it. func TestVLLMLMCacheResolveCacheServerStaysPodNetworkAndVirtualIP(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) pod, svc, err := resolveLMCacheServer(a, newLMCacheBackend(nil)) if err != nil { t.Fatalf("ResolveCacheServer: %v", err) @@ -193,7 +195,7 @@ func TestVLLMLMCacheResolveCacheServerHasReadinessProbe(t *testing.T) { // the server is actually serving — making status optimistic. The // adapter must render a TCP probe targeting the named lmcache port so // Ready waits on the real accept loop. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) pod, _, err := resolveLMCacheServer(a, newLMCacheBackend(nil)) if err != nil { t.Fatalf("ResolveCacheServer: %v", err) @@ -213,7 +215,7 @@ func TestVLLMLMCacheResolveCacheServerHasReadinessProbe(t *testing.T) { func TestVLLMLMCacheResolveCacheServerBoundsRawNilResources(t *testing.T) { // The renderer keeps the 4Gi/8Gi safety bounds even when an object bypasses // the mutating webhook and reaches the raw-struct path with nil resources. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) pod, _, err := resolveLMCacheServer(a, newLMCacheBackend(nil)) if err != nil { t.Fatalf("ResolveCacheServer: %v", err) @@ -234,7 +236,7 @@ func TestVLLMLMCacheResolveCacheServerHasCPURequestWhenAutoscaled(t *testing.T) // a CPU request on the lmcache-server container when spec.autoscaling // is set. A completely omitted resource block receives the bounded memory // fallback; an explicitly supplied limits-only block remains limits-only. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.Autoscaling = &cachev1alpha1.CacheBackendAutoscalingSpec{MaxReplicas: 3} pod, _, err := resolveLMCacheServer(a, cb) @@ -259,7 +261,7 @@ func TestVLLMLMCacheResolveCacheServerAutoscalingPreservesLimitsOnlyResources(t // memory was absent under autoscaling, which silently overrode // the operator's "limit-only" intent — that is the gap this // test pins shut. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.Autoscaling = &cachev1alpha1.CacheBackendAutoscalingSpec{MaxReplicas: 3} cb.Spec.RemoteStorage.LMCacheServer.Resources = &corev1.ResourceRequirements{ @@ -289,7 +291,7 @@ func TestVLLMLMCacheResolveCacheServerHonorsProviderResources(t *testing.T) { // test) — the CRD-schema default supplies memory limits to every // CacheBackend so the cache-server pod is bounded by the cgroup limit // rather than OOM-killed by the kubelet under T2 load. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.RemoteStorage.LMCacheServer.Resources = &corev1.ResourceRequirements{ Requests: corev1.ResourceList{ @@ -320,7 +322,7 @@ func TestVLLMLMCacheResolveCacheServerProviderResourcesNotMutated(t *testing.T) // place — controllers reconcile against an informer-cached object, // and a write through the pointer would propagate back to every // subsequent reader on the same shared cache. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.Autoscaling = &cachev1alpha1.CacheBackendAutoscalingSpec{MaxReplicas: 3} cb.Spec.RemoteStorage.LMCacheServer.Resources = &corev1.ResourceRequirements{ @@ -343,7 +345,7 @@ func TestVLLMLMCacheResolveCacheServerEmptyProviderResourcesIsRespected(t *testi // silently re-introduces limits the operator deliberately omitted. // (No autoscaling here either: the autoscaling-fallback test pins // the orthogonal HPA-CPU behavior.) - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.RemoteStorage.LMCacheServer.Resources = &corev1.ResourceRequirements{} pod := resolvePod(t, a, cb) @@ -362,7 +364,7 @@ func TestVLLMLMCacheResolveCacheServerAutoscalingFillsMissingCPU(t *testing.T) { // — otherwise the operator's memory-only spec.remoteStorage.lmCacheServer.resources silently // breaks the HPA metric path. The adapter MUST NOT overwrite a // CPU request the operator did supply. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.Autoscaling = &cachev1alpha1.CacheBackendAutoscalingSpec{MaxReplicas: 3} cb.Spec.RemoteStorage.LMCacheServer.Resources = &corev1.ResourceRequirements{ @@ -397,7 +399,7 @@ func TestVLLMLMCacheResolveCacheServerAutoscalingReplacesZeroCPU(t *testing.T) { // fallback at render time. A POSITIVE operator-supplied value // still survives — that case is pinned by // TestVLLMLMCacheResolveCacheServerAutoscalingRespectsOperatorCPU. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.Autoscaling = &cachev1alpha1.CacheBackendAutoscalingSpec{MaxReplicas: 3} cb.Spec.RemoteStorage.LMCacheServer.Resources = &corev1.ResourceRequirements{ @@ -415,7 +417,7 @@ func TestVLLMLMCacheResolveCacheServerAutoscalingRespectsOperatorCPU(t *testing. // fallback MUST NOT overwrite it — the operator's value is // authoritative for HPA-utilization math, and a silent overwrite // would surprise users tuning the denominator. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) cb.Spec.Autoscaling = &cachev1alpha1.CacheBackendAutoscalingSpec{MaxReplicas: 3} cb.Spec.RemoteStorage.LMCacheServer.Resources = &corev1.ResourceRequirements{ @@ -431,7 +433,7 @@ func TestVLLMLMCacheResolveCacheServerAutoscalingRespectsOperatorCPU(t *testing. } func TestVLLMLMCacheResolveCacheServerImageOverride(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(map[string]string{"serverImage": "registry.example.com/lmcache:pinned"}) pod, _, err := resolveLMCacheServer(a, cb) @@ -444,7 +446,7 @@ func TestVLLMLMCacheResolveCacheServerImageOverride(t *testing.T) { } func TestVLLMLMCacheResolveCacheServerCommandOverride(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(map[string]string{ "serverCommand": "python3 -m lmcache.v1.multiprocess.server --cpu-buffer-size 60", }) @@ -469,14 +471,14 @@ func TestVLLMLMCacheResolveCacheServerCommandOverride(t *testing.T) { } func TestVLLMLMCacheResolveCacheServerNilCache(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) if _, _, err := resolveLMCacheServer(a, nil); err == nil { t.Fatalf("ResolveCacheServer(nil) returned no error") } } func TestVLLMLMCacheInjectEngineConfig(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) pod := &corev1.PodSpec{ Containers: []corev1.Container{ @@ -548,7 +550,7 @@ func TestVLLMLMCacheInjectEngineConfig(t *testing.T) { func TestVLLMLMCacheInjectEngineConfigSingleContainerPodAcceptsAnyName(t *testing.T) { // A pod with exactly one container is accepted as the engine even when // the container is not named "vllm" — there's no sidecar to crash. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: "engine"}}} @@ -564,7 +566,7 @@ func TestVLLMLMCacheInjectEngineConfigMultiContainerWithoutVLLMNameErrors(t *tes // A multi-container pod with no container named "vllm" must be // rejected: blindly mutating every container would inject vLLM-only // flags onto sidecars and crash them. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{ {Name: "engine", Env: []corev1.EnvVar{{Name: "EXISTING", Value: "x"}}}, @@ -584,7 +586,7 @@ func TestVLLMLMCacheInjectEngineConfigMultiContainerWithoutVLLMNameErrors(t *tes } func TestVLLMLMCacheInjectEngineConfigIdempotent(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: EngineContainerName}}} @@ -627,7 +629,7 @@ func TestVLLMLMCacheInjectEngineConfigIdempotent(t *testing.T) { } func TestVLLMLMCacheInjectEngineConfigFailOpen(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) trueVal, falseVal := true, false cases := []struct { name string @@ -656,7 +658,7 @@ func TestVLLMLMCacheInjectEngineConfigFailOpen(t *testing.T) { } func TestVLLMLMCacheInjectEngineConfigRoleMapping(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cases := []struct { role cachev1alpha1.CacheBackendIntegrationRole wantKVRole string @@ -686,7 +688,7 @@ func TestVLLMLMCacheInjectEngineConfigRoleMapping(t *testing.T) { } func TestVLLMLMCacheInjectEngineConfigTypedOverrides(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(map[string]string{ "chunkSize": "512", "remoteSerde": "cachegen", @@ -726,7 +728,7 @@ func TestVLLMLMCacheHostOnlyEngineConfigUsesTypedConfig(t *testing.T) { {Name: "KEEP_ME", Value: "preserved"}, }, }}} - adapter := NewVLLMLMCacheAdapter() + adapter := NewVLLMLMCacheAdapter(SubscriberConfig{}) if err := adapter.InjectEngineConfig(pod, nil, cb); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } @@ -769,7 +771,7 @@ func TestVLLMLMCacheCanonicalMooncakeBindingHonorsEngineHostNetwork(t *testing.T Protocol: backendadapter.ProtocolMooncakeStore, Endpoint: "mooncake.engines.svc.cluster.local:50051", } - adapter := NewVLLMLMCacheAdapter() + adapter := NewVLLMLMCacheAdapter(SubscriberConfig{}) if err := adapter.InjectEngineConfig(pod, binding, cb); err != nil { t.Fatalf("InjectEngineConfig: %v", err) } @@ -792,7 +794,7 @@ func TestVLLMLMCacheCanonicalMooncakeBindingHonorsEngineHostNetwork(t *testing.T } func TestVLLMLMCacheInjectEngineConfigPassesThroughLMScheme(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: EngineContainerName}}} // A caller that already prefixed lm:// must not produce lm://lm://. @@ -809,7 +811,7 @@ func TestVLLMLMCacheInjectEngineConfigPassesThroughLMScheme(t *testing.T) { } func TestVLLMLMCacheInjectEngineConfigBadInput(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) good := &corev1.PodSpec{Containers: []corev1.Container{{Name: EngineContainerName}}} cases := []struct { @@ -831,7 +833,7 @@ func TestVLLMLMCacheInjectEngineConfigBadInput(t *testing.T) { } func TestVLLMLMCacheInjectRouterConfigIsNoop(t *testing.T) { - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) pod := &corev1.PodSpec{Containers: []corev1.Container{{Name: "router", Env: []corev1.EnvVar{{Name: "EXISTING", Value: "x"}}}}} if err := a.InjectRouterConfig(pod, lmCacheBinding("x.svc:65432"), cb); err != nil { @@ -845,11 +847,11 @@ func TestVLLMLMCacheInjectRouterConfigIsNoop(t *testing.T) { } func TestVLLMLMCacheInjectRouterConfigTrulyNoopsOnBadInput(t *testing.T) { - // The KVCacheRuntimeAdapter contract says backends without a router + // The runtimeadapter.KVCacheRuntimeAdapter contract says backends without a router // component should return nil without touching pod. The LMCache adapter // must honour that even for nil/empty inputs so callers can blindly // invoke InjectRouterConfig on every adapter without branching. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(nil) good := &corev1.PodSpec{Containers: []corev1.Container{{Name: "router"}}} cases := []struct { @@ -943,7 +945,7 @@ func TestValidateExternalEndpointProviderSchemes(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - err := ValidateExternalEndpoint(tt.provider, tt.endpoint) + err := backendadapter.ValidateExternalEndpoint(tt.provider, tt.endpoint) if tt.wantErr && err == nil { t.Fatalf("ValidateExternalEndpoint(%s, %q) succeeded, want error", tt.provider, tt.endpoint) } @@ -964,12 +966,12 @@ func TestVLLMLMCacheEngineContainerName(t *testing.T) { } func TestRegistryResolvesVLLMLMCache(t *testing.T) { - r := NewRegistry() - r.Register(NewVLLMLMCacheAdapter()) + r := runtimeadapter.NewRegistry() + r.Register(NewVLLMLMCacheAdapter(SubscriberConfig{})) if r.Len() == 0 { t.Fatalf("registry has no adapters") } - got, err := r.Select(RuntimeVLLM, newLMCacheBackend(nil)) + got, err := r.Select(runtimeadapter.RuntimeVLLM, newLMCacheBackend(nil)) if err != nil { t.Fatalf("Select(vllm, LMCache): %v", err) } @@ -1016,10 +1018,10 @@ func TestUpsertArgPairAppendsAndReplaces(t *testing.T) { func TestVLLMLMCacheObservationSidecarShape(t *testing.T) { // Auto-attach is opt-in: the operator passes the subscriber image via - // the controller flag. WithSubscriberImage here mirrors the production + // the controller flag. SubscriberConfig here mirrors the production // wiring. Without it ObservationSidecar would return nil (see // TestVLLMLMCacheObservationSidecarSkipsWithoutImage). - a := NewVLLMLMCacheAdapter(WithSubscriberImage(DefaultSubscriberImage)) + a := NewVLLMLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newLMCacheBackend(map[string]string{"model": "Qwen/Qwen2.5-0.5B-Instruct"}) pod := &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "engine-a", Namespace: "engines"}, @@ -1032,8 +1034,8 @@ func TestVLLMLMCacheObservationSidecarShape(t *testing.T) { if c == nil { t.Fatalf("ObservationSidecar returned nil for vLLM+LMCache with a model + image set") } - if c.Name != SubscriberContainerName { - t.Fatalf("container name = %q, want %q", c.Name, SubscriberContainerName) + if c.Name != enginebinding.SubscriberContainerName { + t.Fatalf("container name = %q, want %q", c.Name, enginebinding.SubscriberContainerName) } if c.Image != DefaultSubscriberImage { t.Fatalf("container image = %q, want %q", c.Image, DefaultSubscriberImage) @@ -1080,7 +1082,7 @@ func TestVLLMLMCacheInjectEngineConfigEventsOnlyIsNoOp(t *testing.T) { // must be left untouched so a hybrid-attention model's KV-cache manager is // not disabled — and it requires no endpoint (nothing dials a cache server), // so a nil binding must NOT error the way the managed path does. - a := NewVLLMLMCacheAdapter() + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) cb := newLMCacheBackend(map[string]string{"model": "Qwen/Qwen3.6-27B"}) cb.Spec.Integration = &cachev1alpha1.CacheBackendIntegrationSpec{ Mode: cachev1alpha1.CacheBackendIntegrationModeEventsOnly, @@ -1107,7 +1109,7 @@ func TestVLLMLMCacheObservationSidecarEventsOnlyForwardsEvictions(t *testing.T) // suppression flag must be ABSENT (the binary defaults it to false). Pinning // its absence here keeps a future edit from silently re-suppressing // evictions in events-only and stranding stale routing hints. - a := NewVLLMLMCacheAdapter(WithSubscriberImage(DefaultSubscriberImage)) + a := NewVLLMLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newLMCacheBackend(map[string]string{"model": "Qwen/Qwen3.6-27B"}) cb.Spec.Integration = &cachev1alpha1.CacheBackendIntegrationSpec{ Mode: cachev1alpha1.CacheBackendIntegrationModeEventsOnly, @@ -1137,10 +1139,10 @@ func TestVLLMLMCacheObservationSidecarEventsOnlyForwardsEvictions(t *testing.T) } func TestVLLMLMCacheObservationSidecarHonoursOptions(t *testing.T) { - a := NewVLLMLMCacheAdapter( - WithSubscriberImage("registry.example.com/subscriber:pinned"), - WithPolicyServerGRPCAddress("ic-server.custom-ns.svc.cluster.local:9090"), - ) + a := NewVLLMLMCacheAdapter(SubscriberConfig{ + Image: "registry.example.com/subscriber:pinned", + PolicyServerGRPCAddress: "ic-server.custom-ns.svc.cluster.local:9090", + }) cb := newLMCacheBackend(map[string]string{"model": "MyOrg/MyModel"}) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "engine-z", Namespace: "engines"}} @@ -1161,7 +1163,7 @@ func TestVLLMLMCacheObservationSidecarSkipsWithoutModel(t *testing.T) { // subscriber binary would refuse to start (model-id is a required // flag), so the adapter returns (nil, nil) to skip the append. The next // admission picks up the sidecar once the operator sets the field. - a := NewVLLMLMCacheAdapter(WithSubscriberImage(DefaultSubscriberImage)) + a := NewVLLMLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newLMCacheBackend(nil) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "engine-a"}} @@ -1179,8 +1181,8 @@ func TestVLLMLMCacheObservationSidecarSkipsWithoutImage(t *testing.T) { // --kvevent-subscriber-image is unset, the adapter returns no sidecar // at all — even when observation.modelID is set — so an operator that // hasn't yet shipped a subscriber image can't end up with engine pods - // stuck in ImagePullBackOff. Opt-in by passing WithSubscriberImage. - a := NewVLLMLMCacheAdapter() // no image configured + // stuck in ImagePullBackOff. Opt-in by setting SubscriberConfig.Image. + a := NewVLLMLMCacheAdapter(SubscriberConfig{}) // no image configured cb := newLMCacheBackend(map[string]string{"model": "MyOrg/MyModel"}) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "engine-a"}} @@ -1202,7 +1204,7 @@ func TestVLLMLMCacheObservationSidecarArgsParseAgainstSubscriberFlagSet(t *testi // parse cleanly. Keep the flag set in sync with // cmd/kvevent-subscriber/main.go — adding a flag to the sidecar's args // before the binary learns it is what this guard exists to catch. - a := NewVLLMLMCacheAdapter(WithSubscriberImage(DefaultSubscriberImage)) + a := NewVLLMLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newLMCacheBackend(map[string]string{"model": "Qwen/Qwen2.5-0.5B-Instruct"}) pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "engine-a", Namespace: "engines"}} @@ -1239,7 +1241,7 @@ func TestVLLMLMCacheObservationSidecarArgsParseAgainstSubscriberFlagSet(t *testi } func TestVLLMLMCacheObservationSidecarBadInput(t *testing.T) { - a := NewVLLMLMCacheAdapter(WithSubscriberImage(DefaultSubscriberImage)) + a := NewVLLMLMCacheAdapter(SubscriberConfig{Image: DefaultSubscriberImage}) cb := newLMCacheBackend(map[string]string{"model": "m"}) cases := []struct { name string @@ -1271,19 +1273,6 @@ func vllmEnvHasFieldRef(env []corev1.EnvVar, name, path string) bool { return false } -func TestReferenceAdapterObservationSidecarIsNil(t *testing.T) { - a := NewReferenceAdapter() - cb := newCacheBackend(cachev1alpha1.CacheBackendTypeLMCache, "") - pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "ref-pod"}} - c, err := a.ObservationSidecar(cb, pod) - if err != nil { - t.Fatalf("ObservationSidecar: %v", err) - } - if c != nil { - t.Fatalf("reference adapter ObservationSidecar must return nil, got %+v", c) - } -} - func vllmContainsArg(args []string, want string) bool { for _, a := range args { if a == want { diff --git a/internal/adapters/builtin/runtime/vllm_lmcache_wire.go b/internal/adapters/builtin/runtime/vllm_lmcache_wire.go index 993a66d8..a79ec832 100644 --- a/internal/adapters/builtin/runtime/vllm_lmcache_wire.go +++ b/internal/adapters/builtin/runtime/vllm_lmcache_wire.go @@ -25,20 +25,19 @@ import ( corev1 "k8s.io/api/core/v1" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) // Engine env var names. The cache plane's contract with the engine: an // engine pod that carries these variables (plus the --kv-transfer-config // arg below) is wired to an LMCache-compatible cache. const ( - EnvLMCacheRemoteURL = adapterruntime.EnvLMCacheRemoteURL - EnvLMCacheRemoteSerde = adapterruntime.EnvLMCacheRemoteSerde - EnvLMCacheChunkSize = adapterruntime.EnvLMCacheChunkSize - EnvLMCacheLocalCPU = adapterruntime.EnvLMCacheLocalCPU - EnvLMCacheMaxLocalCPU = adapterruntime.EnvLMCacheMaxLocalCPU - EnvVLLMUseV1 = adapterruntime.EnvVLLMUseV1 - EnvInferenceCacheFailOpen = adapterruntime.EnvInferenceCacheFailOpen + EnvLMCacheRemoteURL = "LMCACHE_REMOTE_URL" + EnvLMCacheRemoteSerde = "LMCACHE_REMOTE_SERDE" + EnvLMCacheChunkSize = "LMCACHE_CHUNK_SIZE" + EnvLMCacheLocalCPU = "LMCACHE_LOCAL_CPU" + EnvLMCacheMaxLocalCPU = "LMCACHE_MAX_LOCAL_CPU_SIZE" + EnvVLLMUseV1 = "VLLM_USE_V1" + EnvInferenceCacheFailOpen = "INFERENCECACHE_FAIL_OPEN" // EnvPythonHashSeed pins Python's hash seed so the NONE_HASH that seeds // vLLM's prefix-cache block-hash chain is deterministic across the // scheduler and the TP worker processes. Under TP>1 those are separate @@ -47,7 +46,7 @@ const ( // stored hashes — LMCache reload silently 0-hits and the engine fully // recomputes with no crash and no error. A correctness invariant, not a // tunable. - EnvPythonHashSeed = adapterruntime.EnvPythonHashSeed + EnvPythonHashSeed = "PYTHONHASHSEED" ) // EngineContainerName is the conventional name of the vLLM container in an @@ -55,7 +54,7 @@ const ( // pod is treated as the engine; a multi-container pod is rejected — silently // mutating every container would inject vLLM-only flags onto sidecars and // crash them. -const EngineContainerName = adapterruntime.EngineContainerName +const EngineContainerName = "vllm" // Defaults the engine env carries when the operator does not override them // through typed LMCache config. The CPU-safe @@ -456,10 +455,6 @@ func ConfigOr(cfg map[string]string, key, fallback string) string { // stored values so the engine pod admits unwired rather than crashing). // Centralising the rule here means a future tightening only needs to // touch one place to ripple to all three layers. -func ValidateLMCacheEndpoint(s string) error { - return adapterruntime.ValidateLMCacheEndpoint(s) -} - // splitLMCacheHostPort parses a host:port string into its host and port // halves with bracket-aware IPv6 handling. Returns (host, port, hasPort) // so callers can tell apart `cache` (no port → hasPort=false) from diff --git a/internal/adapters/builtin/runtime/vllm_lmcache_wire_test.go b/internal/adapters/builtin/runtime/vllm_lmcache_wire_test.go index 5c9fb638..4e0f54b8 100644 --- a/internal/adapters/builtin/runtime/vllm_lmcache_wire_test.go +++ b/internal/adapters/builtin/runtime/vllm_lmcache_wire_test.go @@ -12,6 +12,7 @@ import ( "k8s.io/apimachinery/pkg/api/resource" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" ) // lookupInjectedEnv returns the value of the env var named name on the engine @@ -335,7 +336,7 @@ func TestValidateLMCacheEndpoint(t *testing.T) { } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - err := ValidateLMCacheEndpoint(tc.input) + err := backendadapter.ValidateLMCacheEndpoint(tc.input) if tc.wantErr { if err == nil { t.Fatalf("ValidateLMCacheEndpoint(%q) = nil, want error containing %q", tc.input, tc.wantMatch) diff --git a/internal/boundarytest/repository_test.go b/internal/boundarytest/repository_test.go new file mode 100644 index 00000000..6ee4da48 --- /dev/null +++ b/internal/boundarytest/repository_test.go @@ -0,0 +1,175 @@ +// SPDX-FileCopyrightText: 2026 The inference-cache Authors +// +// SPDX-License-Identifier: Apache-2.0 + +package boundarytest + +import ( + "go/parser" + "go/token" + "io/fs" + "os" + "path/filepath" + "runtime" + "sort" + "strconv" + "strings" + "testing" +) + +const modulePath = "github.com/cachebox-project/inference-cache" + +func TestPublicPackagesHaveDocumentation(t *testing.T) { + t.Parallel() + + root := repositoryRoot(t) + packageDirs := map[string]bool{} + err := filepath.WalkDir(filepath.Join(root, "pkg"), func(path string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() || filepath.Ext(path) != ".go" || strings.HasSuffix(path, "_test.go") { + return nil + } + file, err := parser.ParseFile(token.NewFileSet(), path, nil, parser.PackageClauseOnly|parser.ParseComments) + if err != nil { + return err + } + dir := filepath.Dir(path) + if _, seen := packageDirs[dir]; !seen { + packageDirs[dir] = false + } + if file.Doc != nil && strings.TrimSpace(file.Doc.Text()) != "" { + packageDirs[dir] = true + } + return nil + }) + if err != nil { + t.Fatalf("scan public packages: %v", err) + } + + var undocumented []string + for dir, documented := range packageDirs { + if documented { + continue + } + rel, err := filepath.Rel(root, dir) + if err != nil { + t.Fatalf("relative package path: %v", err) + } + undocumented = append(undocumented, rel) + } + sort.Strings(undocumented) + if len(undocumented) > 0 { + t.Fatalf("public packages without package documentation: %s", strings.Join(undocumented, ", ")) + } +} + +func TestPublicProductionCodeDoesNotImportInternalPackages(t *testing.T) { + t.Parallel() + + root := repositoryRoot(t) + for _, topLevel := range []string{"api", "gen", "pkg"} { + topLevel := topLevel + t.Run(topLevel, func(t *testing.T) { + t.Parallel() + err := filepath.WalkDir(filepath.Join(root, topLevel), func(path string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() || filepath.Ext(path) != ".go" || strings.HasSuffix(path, "_test.go") { + return nil + } + file, err := parser.ParseFile(token.NewFileSet(), path, nil, parser.ImportsOnly) + if err != nil { + return err + } + for _, imported := range file.Imports { + importPath, err := strconv.Unquote(imported.Path.Value) + if err != nil { + return err + } + if strings.HasPrefix(importPath, modulePath+"/internal/") { + rel, relErr := filepath.Rel(root, path) + if relErr != nil { + return relErr + } + t.Errorf("%s imports private package %q", rel, importPath) + } + } + return nil + }) + if err != nil { + t.Fatalf("scan %s production code: %v", topLevel, err) + } + }) + } +} + +func TestGeneratedProtobufCodeLivesUnderGen(t *testing.T) { + t.Parallel() + + root := repositoryRoot(t) + err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() { + switch entry.Name() { + case ".git", "bin": + return filepath.SkipDir + default: + return nil + } + } + if !strings.HasSuffix(entry.Name(), ".pb.go") { + return nil + } + rel, err := filepath.Rel(root, path) + if err != nil { + return err + } + if !strings.HasPrefix(filepath.ToSlash(rel), "gen/") { + t.Errorf("generated protobuf Go file must live under gen/: %s", rel) + } + return nil + }) + if err != nil { + t.Fatalf("scan generated protobuf files: %v", err) + } +} + +func TestProtobufGoPackageIsPinnedToGen(t *testing.T) { + t.Parallel() + + root := repositoryRoot(t) + protoPath := filepath.Join(root, "proto", "inferencecache", "v1alpha1", "inferencecache.proto") + contents, err := os.ReadFile(protoPath) + if err != nil { + t.Fatalf("read protobuf contract: %v", err) + } + want := `option go_package = "` + modulePath + `/gen/inferencecache/v1alpha1;inferencecachev1alpha1pb";` + var got string + for _, line := range strings.Split(string(contents), "\n") { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, "option go_package") { + continue + } + if got != "" { + t.Fatalf("protobuf contract contains multiple go_package options: %q and %q", got, line) + } + got = line + } + if got != want { + t.Fatalf("protobuf go_package must remain pinned to the public gen path: got %q, want %q", got, want) + } +} + +func repositoryRoot(t *testing.T) string { + t.Helper() + _, file, _, ok := runtime.Caller(0) + if !ok { + t.Fatal("locate repository root") + } + return filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..")) +} diff --git a/pkg/adapters/engineclient/canary.go b/internal/canary/canary.go similarity index 89% rename from pkg/adapters/engineclient/canary.go rename to internal/canary/canary.go index 74b61fd4..9a089857 100644 --- a/pkg/adapters/engineclient/canary.go +++ b/internal/canary/canary.go @@ -2,7 +2,10 @@ // // SPDX-License-Identifier: Apache-2.0 -package engineclient +// Package canary provides repository-owned live and test probes for inference +// engines. It is operational infrastructure, not part of the public engine +// client API. +package canary import ( "bufio" @@ -13,6 +16,8 @@ import ( "net/http" "strconv" "strings" + + "github.com/cachebox-project/inference-cache/pkg/engineclient" ) // PrefixCacheProbe is the by-construction canary: it sends the SAME token-ID @@ -24,10 +29,10 @@ import ( // guarantee (the tokenizer half is proven by pkg/tokenize, the fingerprint half // by pkg/fingerprint). type PrefixCacheProbe struct { - Client EngineClient // sends the token-ID prompt (typically NewOpenAI) - HTTP *http.Client // scrapes /metrics; defaults to http.DefaultClient - EngineURL string // base URL, e.g. http://host:8000 - MetricsURL string // optional; defaults to EngineURL + "/metrics" + Client engineclient.EngineClient // sends the token-ID prompt (typically NewOpenAI) + HTTP *http.Client // scrapes /metrics; defaults to http.DefaultClient + EngineURL string // base URL, e.g. http://host:8000 + MetricsURL string // optional; defaults to EngineURL + "/metrics" Model string // HitsMetric / QueriesMetric name the vLLM prefix-cache counters to read. // Empty defaults to the standard names below; override when a vLLM build @@ -45,15 +50,15 @@ const ( type ProbeResult struct { HitsDelta int // vllm:prefix_cache_hits_total change across the warm request QueriesDelta int // vllm:prefix_cache_queries_total change across the warm request - Cold Completion - Warm Completion + Cold engineclient.Completion + Warm engineclient.Completion } // Run fires the cold request (populating the cache), then measures the // prefix-cache counters immediately before and after an identical warm request, // so HitsDelta reflects only the warm request. A HitsDelta > 0 is the success // signal. -func (p *PrefixCacheProbe) Run(ctx context.Context, tokens []uint32, params CompletionParams) (ProbeResult, error) { +func (p *PrefixCacheProbe) Run(ctx context.Context, tokens []uint32, params engineclient.CompletionParams) (ProbeResult, error) { if p.Client == nil { return ProbeResult{}, errors.New("canary: PrefixCacheProbe.Client is nil") } diff --git a/pkg/adapters/engineclient/canary_test.go b/internal/canary/canary_test.go similarity index 95% rename from pkg/adapters/engineclient/canary_test.go rename to internal/canary/canary_test.go index ba0fd42d..7871e9df 100644 --- a/pkg/adapters/engineclient/canary_test.go +++ b/internal/canary/canary_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engineclient +package canary import ( "context" @@ -15,6 +15,8 @@ import ( "strings" "sync" "testing" + + "github.com/cachebox-project/inference-cache/pkg/engineclient" ) // mockVLLM mimics just enough of a vLLM OpenAI server for the canary: it serves @@ -70,12 +72,12 @@ func TestPrefixCacheProbeDetectsWarmHit(t *testing.T) { defer srv.Close() probe := &PrefixCacheProbe{ - Client: NewOpenAI(nil), + Client: engineclient.NewOpenAI(nil), EngineURL: srv.URL, Model: "m", } tokens := tokenRange(0, 64) - res, err := probe.Run(context.Background(), tokens, CompletionParams{MaxTokens: 1}) + res, err := probe.Run(context.Background(), tokens, engineclient.CompletionParams{MaxTokens: 1}) if err != nil { t.Fatalf("probe: %v", err) } @@ -203,14 +205,14 @@ func TestPrefixCacheCanaryLive(t *testing.T) { } probe := &PrefixCacheProbe{ - Client: NewOpenAI(nil), + Client: engineclient.NewOpenAI(nil), EngineURL: engineURL, Model: model, // Allow metric-name overrides for vLLM builds that rename the counters. HitsMetric: os.Getenv("IC_ENGINE_HITS_METRIC"), QueriesMetric: os.Getenv("IC_ENGINE_QUERIES_METRIC"), } - res, err := probe.Run(context.Background(), tokens, CompletionParams{MaxTokens: 1, Temperature: 0}) + res, err := probe.Run(context.Background(), tokens, engineclient.CompletionParams{MaxTokens: 1, Temperature: 0}) if err != nil { t.Fatalf("live canary: %v", err) } diff --git a/pkg/cli/doctor/checks/cachebackend.go b/internal/cli/doctor/checks/cachebackend.go similarity index 99% rename from pkg/cli/doctor/checks/cachebackend.go rename to internal/cli/doctor/checks/cachebackend.go index 0d227d69..e51b7ca5 100644 --- a/pkg/cli/doctor/checks/cachebackend.go +++ b/internal/cli/doctor/checks/cachebackend.go @@ -14,7 +14,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) const checkCacheBackendHealth = "CacheBackendHealth" diff --git a/pkg/cli/doctor/checks/checks.go b/internal/cli/doctor/checks/checks.go similarity index 99% rename from pkg/cli/doctor/checks/checks.go rename to internal/cli/doctor/checks/checks.go index d3ce5445..bd9eb7db 100644 --- a/pkg/cli/doctor/checks/checks.go +++ b/internal/cli/doctor/checks/checks.go @@ -34,7 +34,7 @@ import ( "google.golang.org/grpc" healthpb "google.golang.org/grpc/health/grpc_health_v1" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) // DefaultStaleWindow is how long a CacheBackend may go without a fresh KV event diff --git a/pkg/cli/doctor/checks/checks_test.go b/internal/cli/doctor/checks/checks_test.go similarity index 99% rename from pkg/cli/doctor/checks/checks_test.go rename to internal/cli/doctor/checks/checks_test.go index caf83469..1814e180 100644 --- a/pkg/cli/doctor/checks/checks_test.go +++ b/internal/cli/doctor/checks/checks_test.go @@ -29,7 +29,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) // --- shared helpers --------------------------------------------------------- diff --git a/pkg/cli/doctor/checks/endpoints.go b/internal/cli/doctor/checks/endpoints.go similarity index 99% rename from pkg/cli/doctor/checks/endpoints.go rename to internal/cli/doctor/checks/endpoints.go index 461a105c..365fd6d1 100644 --- a/pkg/cli/doctor/checks/endpoints.go +++ b/internal/cli/doctor/checks/endpoints.go @@ -13,7 +13,7 @@ import ( healthpb "google.golang.org/grpc/health/grpc_health_v1" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) const ( diff --git a/pkg/cli/doctor/checks/podaudit.go b/internal/cli/doctor/checks/podaudit.go similarity index 99% rename from pkg/cli/doctor/checks/podaudit.go rename to internal/cli/doctor/checks/podaudit.go index 5fb6d397..732137a7 100644 --- a/pkg/cli/doctor/checks/podaudit.go +++ b/internal/cli/doctor/checks/podaudit.go @@ -15,7 +15,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) const ( diff --git a/pkg/cli/doctor/checks/tenant_policy.go b/internal/cli/doctor/checks/tenant_policy.go similarity index 98% rename from pkg/cli/doctor/checks/tenant_policy.go rename to internal/cli/doctor/checks/tenant_policy.go index cf346a1a..b9322dfb 100644 --- a/pkg/cli/doctor/checks/tenant_policy.go +++ b/internal/cli/doctor/checks/tenant_policy.go @@ -13,7 +13,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) const ( diff --git a/pkg/cli/doctor/finding.go b/internal/cli/doctor/finding.go similarity index 100% rename from pkg/cli/doctor/finding.go rename to internal/cli/doctor/finding.go diff --git a/pkg/cli/doctor/finding_test.go b/internal/cli/doctor/finding_test.go similarity index 100% rename from pkg/cli/doctor/finding_test.go rename to internal/cli/doctor/finding_test.go diff --git a/pkg/cli/doctor/output/human.go b/internal/cli/doctor/output/human.go similarity index 97% rename from pkg/cli/doctor/output/human.go rename to internal/cli/doctor/output/human.go index cd8cb897..59f1c695 100644 --- a/pkg/cli/doctor/output/human.go +++ b/internal/cli/doctor/output/human.go @@ -8,7 +8,7 @@ import ( "fmt" "io" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) // ANSI SGR codes for the human format. Kept minimal: a color per severity plus diff --git a/pkg/cli/doctor/output/json.go b/internal/cli/doctor/output/json.go similarity index 96% rename from pkg/cli/doctor/output/json.go rename to internal/cli/doctor/output/json.go index 772a9650..28895c85 100644 --- a/pkg/cli/doctor/output/json.go +++ b/internal/cli/doctor/output/json.go @@ -8,7 +8,7 @@ import ( "encoding/json" "io" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) // jsonReport is the stable schema emitted by `--output=json`. It is a documented diff --git a/pkg/cli/doctor/output/output.go b/internal/cli/doctor/output/output.go similarity index 96% rename from pkg/cli/doctor/output/output.go rename to internal/cli/doctor/output/output.go index c21076c0..7db2e136 100644 --- a/pkg/cli/doctor/output/output.go +++ b/internal/cli/doctor/output/output.go @@ -17,7 +17,7 @@ import ( "fmt" "io" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) // Format identifies an output rendering. diff --git a/pkg/cli/doctor/output/output_test.go b/internal/cli/doctor/output/output_test.go similarity index 96% rename from pkg/cli/doctor/output/output_test.go rename to internal/cli/doctor/output/output_test.go index 470d551f..f40a646f 100644 --- a/pkg/cli/doctor/output/output_test.go +++ b/internal/cli/doctor/output/output_test.go @@ -12,7 +12,7 @@ import ( "strings" "testing" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) var update = flag.Bool("update", false, "regenerate golden files") @@ -42,7 +42,7 @@ func assertGolden(t *testing.T, name string, got []byte) { } want, err := os.ReadFile(path) if err != nil { - t.Fatalf("read golden %s: %v (run `go test ./pkg/cli/doctor/output -update`)", path, err) + t.Fatalf("read golden %s: %v (run `go test ./internal/cli/doctor/output -update`)", path, err) } if !bytes.Equal(got, want) { t.Errorf("output mismatch for %s\n--- got ---\n%s\n--- want ---\n%s", name, got, want) diff --git a/pkg/cli/doctor/output/table.go b/internal/cli/doctor/output/table.go similarity index 94% rename from pkg/cli/doctor/output/table.go rename to internal/cli/doctor/output/table.go index 17666429..cc659ac8 100644 --- a/pkg/cli/doctor/output/table.go +++ b/internal/cli/doctor/output/table.go @@ -10,7 +10,7 @@ import ( "strings" "text/tabwriter" - "github.com/cachebox-project/inference-cache/pkg/cli/doctor" + "github.com/cachebox-project/inference-cache/internal/cli/doctor" ) // renderTable writes one tab-aligned row per finding under a STATUS/CODE/CHECK/ diff --git a/pkg/cli/doctor/output/testdata/human.txt b/internal/cli/doctor/output/testdata/human.txt similarity index 100% rename from pkg/cli/doctor/output/testdata/human.txt rename to internal/cli/doctor/output/testdata/human.txt diff --git a/pkg/cli/doctor/output/testdata/human_color.txt b/internal/cli/doctor/output/testdata/human_color.txt similarity index 100% rename from pkg/cli/doctor/output/testdata/human_color.txt rename to internal/cli/doctor/output/testdata/human_color.txt diff --git a/pkg/cli/doctor/output/testdata/report.json b/internal/cli/doctor/output/testdata/report.json similarity index 100% rename from pkg/cli/doctor/output/testdata/report.json rename to internal/cli/doctor/output/testdata/report.json diff --git a/pkg/cli/doctor/output/testdata/report.table b/internal/cli/doctor/output/testdata/report.table similarity index 100% rename from pkg/cli/doctor/output/testdata/report.table rename to internal/cli/doctor/output/testdata/report.table diff --git a/internal/controller/cachebackend_controller.go b/internal/controller/cachebackend_controller.go index 3098fda8..2ef46dfa 100644 --- a/internal/controller/cachebackend_controller.go +++ b/internal/controller/cachebackend_controller.go @@ -617,7 +617,7 @@ func (r *CacheBackendReconciler) reconcileExternal(ctx context.Context, backend // operator running kubectl describe sees the same // shape complaint they would get on a fresh kubectl // apply. - if err := adapterruntime.ValidateExternalEndpoint(storage.Provider, endpoint); err != nil { + if err := backendadapter.ValidateExternalEndpoint(storage.Provider, endpoint); err != nil { readyReason = conditionReasonExternalEndpointInvalid readyMsg = "spec.remoteStorage." + err.Error() break diff --git a/internal/controller/cachebackend_controller_test.go b/internal/controller/cachebackend_controller_test.go index 13312dd5..280042ed 100644 --- a/internal/controller/cachebackend_controller_test.go +++ b/internal/controller/cachebackend_controller_test.go @@ -32,6 +32,7 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" builtinadapters "github.com/cachebox-project/inference-cache/internal/adapters/builtin" builtinruntime "github.com/cachebox-project/inference-cache/internal/adapters/builtin/runtime" + "github.com/cachebox-project/inference-cache/internal/enginebinding" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" @@ -81,7 +82,7 @@ func configureTestRegistries(r *CacheBackendReconciler) { if r.Registry != nil && r.BackendRegistry != nil { return } - registries := builtinadapters.New() + registries := builtinadapters.New(builtinadapters.Options{}) if r.Registry == nil { r.Registry = registries.Runtime } @@ -314,7 +315,7 @@ func TestReconcileCanonicalHostOnlyCacheReportsEngineDiagnostics(t *testing.T) { cb.Spec.EngineSelector = &cachev1alpha1.CacheBackendEngineSelector{ MatchLabels: map[string]string{"app": "engine"}, } - pod := strictPodWithKernelStatus(termed(1, adapterruntime.KernelCheckMsgFailPrefix+" lmcache c_ops failed")) + pod := strictPodWithKernelStatus(termed(1, enginebinding.KernelCheckMsgFailPrefix+" lmcache c_ops failed")) pod.ObjectMeta = metav1.ObjectMeta{ Name: "engine", Namespace: cb.Namespace, @@ -1293,7 +1294,7 @@ func TestReconcileEventsOnlyAdapterRejectingHostOnlyBindingIsUnmanaged(t *testin } r := newReconciler(scheme, cb) r.Registry = adapterruntime.NewRegistry() - r.Registry.Register(remoteOnlyRuntimeAdapter{KVCacheRuntimeAdapter: builtinruntime.NewVLLMLMCacheAdapter()}) + r.Registry.Register(remoteOnlyRuntimeAdapter{KVCacheRuntimeAdapter: builtinruntime.NewVLLMLMCacheAdapter(builtinruntime.SubscriberConfig{})}) reconcile(t, r, "cache", "ns1") @@ -1329,7 +1330,7 @@ func TestReconcileEventsOnlyTakesPrecedenceOverExternal(t *testing.T) { } r := newReconciler(scheme, cb) r.Registry = adapterruntime.NewRegistry() - r.Registry.Register(builtinruntime.NewVLLMLMCacheAdapter()) + r.Registry.Register(builtinruntime.NewVLLMLMCacheAdapter(builtinruntime.SubscriberConfig{})) reconcile(t, r, "cache", "ns1") diff --git a/internal/controller/cachebackend_engine_compat.go b/internal/controller/cachebackend_engine_compat.go index 9f8d67c0..cce57068 100644 --- a/internal/controller/cachebackend_engine_compat.go +++ b/internal/controller/cachebackend_engine_compat.go @@ -142,7 +142,7 @@ func engineContainerStatus(pod *corev1.Pod, engineContainerName string) *corev1. var engine *corev1.ContainerStatus nonSidecar := 0 for i := range pod.Status.ContainerStatuses { - if pod.Status.ContainerStatuses[i].Name == adapterruntime.SubscriberContainerName { + if pod.Status.ContainerStatuses[i].Name == enginebinding.SubscriberContainerName { continue } engine = &pod.Status.ContainerStatuses[i] diff --git a/internal/controller/cachebackend_engine_compat_test.go b/internal/controller/cachebackend_engine_compat_test.go index 3ecf0f67..31ebaab5 100644 --- a/internal/controller/cachebackend_engine_compat_test.go +++ b/internal/controller/cachebackend_engine_compat_test.go @@ -14,8 +14,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + "github.com/cachebox-project/inference-cache/internal/enginebinding" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) const testBackendUID = "be-uid-123" @@ -54,7 +54,7 @@ func TestDetectEngineConnectorCrashLoop(t *testing.T) { scheme := newScheme(t) const ns, name = "ns1", "cache" injectedBy := ns + "/" + name - sidecar := adapterruntime.SubscriberContainerName + sidecar := enginebinding.SubscriberContainerName clbo := func(n string) ctrState { return ctrState{n, crashLoopBackOffReason} } run := func(n string) ctrState { return ctrState{n, ""} } diff --git a/internal/controller/cachebackend_kernelcheck.go b/internal/controller/cachebackend_kernelcheck.go index 25f2f38f..94dfccfe 100644 --- a/internal/controller/cachebackend_kernelcheck.go +++ b/internal/controller/cachebackend_kernelcheck.go @@ -18,7 +18,6 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" "github.com/cachebox-project/inference-cache/internal/enginebinding" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) // EngineKernelsHealthy gate: surfaces the engine-side native CUDA-kernel @@ -112,7 +111,7 @@ func evaluateEngineKernelHealth( if cond.Status == metav1.ConditionFalse && strictFail { v.downgradeReady = true v.readyReason = reasonEngineKernelDegraded - v.readyMessage = "lmcache CUDA kernels failed to load on one or more engine pods; in strict mode those pods stay in Init holding their GPU reservation without serving — fix the engine image's lmcache/CUDA alignment or set " + adapterruntime.AnnotationLMCacheKernelCheck + "=report-only" + v.readyMessage = "lmcache CUDA kernels failed to load on one or more engine pods; in strict mode those pods stay in Init holding their GPU reservation without serving — fix the engine image's lmcache/CUDA alignment or set " + enginebinding.AnnotationLMCacheKernelCheck + "=report-only" } return v } @@ -153,7 +152,7 @@ func aggregateKernelHealth(backend *cachev1alpha1.CacheBackend, pods []corev1.Po } msg := strings.TrimSpace(term.Message) switch { - case strings.HasPrefix(msg, adapterruntime.KernelCheckMsgFailPrefix): + case strings.HasPrefix(msg, enginebinding.KernelCheckMsgFailPrefix): nFail++ if failMsg == "" { failMsg = msg @@ -161,7 +160,7 @@ func aggregateKernelHealth(backend *cachev1alpha1.CacheBackend, pods []corev1.Po if kernelCheckAdmittedStrict(&pods[i]) { strictFail = true } - case msg == adapterruntime.KernelCheckMsgOK && term.ExitCode == 0: + case msg == enginebinding.KernelCheckMsgOK && term.ExitCode == 0: nOK++ default: // Terminated without our OK/FAIL: contract: python3 missing (exit @@ -218,7 +217,7 @@ func aggregateKernelHealth(backend *cachev1alpha1.CacheBackend, pods []corev1.Po // has been observed yet. func kernelCheckInSpec(pod *corev1.Pod) bool { for i := range pod.Spec.InitContainers { - if pod.Spec.InitContainers[i].Name == adapterruntime.LMCacheKernelCheckContainerName { + if pod.Spec.InitContainers[i].Name == enginebinding.LMCacheKernelCheckContainerName { return true } } @@ -232,11 +231,11 @@ func kernelCheckInSpec(pod *corev1.Pod) bool { func kernelCheckAdmittedStrict(pod *corev1.Pod) bool { for i := range pod.Spec.InitContainers { c := &pod.Spec.InitContainers[i] - if c.Name != adapterruntime.LMCacheKernelCheckContainerName { + if c.Name != enginebinding.LMCacheKernelCheckContainerName { continue } for _, e := range c.Env { - if e.Name == adapterruntime.EnvKernelCheckStrict && e.Value == "1" { + if e.Name == enginebinding.EnvKernelCheckStrict && e.Value == "1" { return true } } @@ -248,7 +247,7 @@ func kernelCheckAdmittedStrict(pod *corev1.Pod) bool { // on pod, or nil if absent. func findKernelCheckStatus(pod *corev1.Pod) *corev1.ContainerStatus { for i := range pod.Status.InitContainerStatuses { - if pod.Status.InitContainerStatuses[i].Name == adapterruntime.LMCacheKernelCheckContainerName { + if pod.Status.InitContainerStatuses[i].Name == enginebinding.LMCacheKernelCheckContainerName { return &pod.Status.InitContainerStatuses[i] } } diff --git a/internal/controller/cachebackend_kernelcheck_integration_test.go b/internal/controller/cachebackend_kernelcheck_integration_test.go index 58b7025e..e41dd9a2 100644 --- a/internal/controller/cachebackend_kernelcheck_integration_test.go +++ b/internal/controller/cachebackend_kernelcheck_integration_test.go @@ -16,8 +16,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/enginebinding" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) // TestIntegrationEngineKernelHealthGate exercises the EngineKernelsHealthy @@ -51,7 +51,7 @@ func TestIntegrationEngineKernelHealthGate(t *testing.T) { // strict Ready downgrade. var initEnv []corev1.EnvVar if strict { - initEnv = []corev1.EnvVar{{Name: adapterruntime.EnvKernelCheckStrict, Value: "1"}} + initEnv = []corev1.EnvVar{{Name: enginebinding.EnvKernelCheckStrict, Value: "1"}} } pod := &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ @@ -67,7 +67,7 @@ func TestIntegrationEngineKernelHealthGate(t *testing.T) { }, Spec: corev1.PodSpec{ InitContainers: []corev1.Container{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, Image: "registry.example.com/lmcache-kernel-check:test", Env: initEnv, }}, @@ -89,11 +89,11 @@ func TestIntegrationEngineKernelHealthGate(t *testing.T) { } before := livePod.DeepCopy() livePod.Status.InitContainerStatuses = []corev1.ContainerStatus{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, State: corev1.ContainerState{ Terminated: &corev1.ContainerStateTerminated{ ExitCode: 1, - Message: adapterruntime.KernelCheckMsgFailPrefix + " lmcache c_ops CUDA kernel version mismatch", + Message: enginebinding.KernelCheckMsgFailPrefix + " lmcache c_ops CUDA kernel version mismatch", }, }, }} @@ -170,7 +170,7 @@ func TestIntegrationEngineKernelHealthGate(t *testing.T) { // Use a backend WITH the strict annotation: kernel mismatch must // downgrade Ready to False. - cb := kernelCheckBackend("cache", ns, adapterruntime.KernelCheckModeStrict) + cb := kernelCheckBackend("cache", ns, enginebinding.KernelCheckModeStrict) if err := k8s.Create(ctx, cb); err != nil { t.Fatalf("create: %v", err) } @@ -245,7 +245,7 @@ func kernelCheckBackend(name, ns, kernelCheckMode string) *cachev1alpha1.CacheBa if cb.Annotations == nil { cb.Annotations = map[string]string{} } - cb.Annotations[adapterruntime.AnnotationLMCacheKernelCheck] = kernelCheckMode + cb.Annotations[enginebinding.AnnotationLMCacheKernelCheck] = kernelCheckMode } return cb } diff --git a/internal/controller/cachebackend_kernelcheck_test.go b/internal/controller/cachebackend_kernelcheck_test.go index c9ea9bde..0ad33d8f 100644 --- a/internal/controller/cachebackend_kernelcheck_test.go +++ b/internal/controller/cachebackend_kernelcheck_test.go @@ -15,17 +15,17 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/enginebinding" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) func podWithKernelStatus(state corev1.ContainerState) corev1.Pod { return corev1.Pod{ Spec: corev1.PodSpec{InitContainers: []corev1.Container{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, }}}, Status: corev1.PodStatus{InitContainerStatuses: []corev1.ContainerStatus{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, State: state, }}}, } @@ -35,7 +35,7 @@ func podWithKernelStatus(state corev1.ContainerState) corev1.Pod { // observed status yet — a just-created / unscheduled pod. func specOnlyKernelPod() corev1.Pod { return corev1.Pod{Spec: corev1.PodSpec{InitContainers: []corev1.Container{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, }}}} } @@ -45,8 +45,8 @@ func specOnlyKernelPod() corev1.Pod { func strictPodWithKernelStatus(state corev1.ContainerState) corev1.Pod { p := podWithKernelStatus(state) p.Spec.InitContainers = []corev1.Container{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, - Env: []corev1.EnvVar{{Name: adapterruntime.EnvKernelCheckStrict, Value: "1"}}, + Name: enginebinding.LMCacheKernelCheckContainerName, + Env: []corev1.EnvVar{{Name: enginebinding.EnvKernelCheckStrict, Value: "1"}}, }} return p } @@ -76,9 +76,9 @@ func TestAggregateKernelHealth(t *testing.T) { podWithKernelStatus(termed(0, "FAIL: ImportError: libcudart.so.13")), }, metav1.ConditionFalse, reasonKernelLoadFailed, true}, {"strict crashloop fail via lastState", []corev1.Pod{{ - Spec: corev1.PodSpec{InitContainers: []corev1.Container{{Name: adapterruntime.LMCacheKernelCheckContainerName}}}, + Spec: corev1.PodSpec{InitContainers: []corev1.Container{{Name: enginebinding.LMCacheKernelCheckContainerName}}}, Status: corev1.PodStatus{InitContainerStatuses: []corev1.ContainerStatus{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, State: corev1.ContainerState{Waiting: &corev1.ContainerStateWaiting{Reason: "CrashLoopBackOff"}}, LastTerminationState: termed(1, "FAIL: ImportError: libcudart.so.13"), }}}, @@ -163,7 +163,7 @@ func TestEvaluateEngineKernelHealthAnnotationStrictButReportOnlyPodDoesNotDowngr // the pod was already running. Pod truth wins: do NOT downgrade Ready (that // pod is actually serving), though the condition still surfaces False. cb := &cachev1alpha1.CacheBackend{ObjectMeta: metav1.ObjectMeta{Name: "cb", Namespace: "ns", - Annotations: map[string]string{adapterruntime.AnnotationLMCacheKernelCheck: adapterruntime.KernelCheckModeStrict}}} + Annotations: map[string]string{enginebinding.AnnotationLMCacheKernelCheck: enginebinding.KernelCheckModeStrict}}} up := kvReadiness{readyStatus: metav1.ConditionTrue} v := evaluateEngineKernelHealth(cb, up, []corev1.Pod{podWithKernelStatus(termed(0, "FAIL: ImportError: libcudart.so.13"))}, true) if v.downgradeReady { diff --git a/internal/controller/cachebackend_probe_test.go b/internal/controller/cachebackend_probe_test.go index 6bff20c6..f3da7f38 100644 --- a/internal/controller/cachebackend_probe_test.go +++ b/internal/controller/cachebackend_probe_test.go @@ -677,7 +677,7 @@ func TestProbeResultMetricEmitsPerStage(t *testing.T) { // metricCounter reads the inferencecache_backend_probe_result_total counter for // one label combination by collecting the metric directly. Avoids pulling // in the prometheus/testutil dependency just for one assertion (the -// pkg/server tests use the same trick). +// internal/server tests use the same trick). func metricCounter(t *testing.T, backend, stage, result string) float64 { t.Helper() m, err := probeResultMetric.GetMetricWithLabelValues(backend, stage, result) diff --git a/internal/controller/cachebackend_server_restart.go b/internal/controller/cachebackend_server_restart.go index 448cfbda..0fcfe90e 100644 --- a/internal/controller/cachebackend_server_restart.go +++ b/internal/controller/cachebackend_server_restart.go @@ -124,7 +124,7 @@ const cascadeRestartReasonServerInstanceChanged = "server_instance_changed" // Partitioned by namespaced CacheBackend identity and a short reason // code. Registered into the controller-runtime metrics registry on // package init so it appears on the manager's /metrics endpoint (no -// per-Service registry — see pkg/server/metrics.go for the +// per-Service registry — see internal/server/metrics.go for the // other-direction posture). Safe to mutate concurrently; tests // reset its inner state via resetBackendServerRestartCascadesTotalForTest. var backendServerRestartCascadesTotal = prometheus.NewCounterVec( diff --git a/internal/controller/cacheindex_authed_integration_test.go b/internal/controller/cacheindex_authed_integration_test.go index 579826db..b8b189aa 100644 --- a/internal/controller/cacheindex_authed_integration_test.go +++ b/internal/controller/cacheindex_authed_integration_test.go @@ -21,8 +21,8 @@ import ( "k8s.io/client-go/kubernetes" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/index" - "github.com/cachebox-project/inference-cache/pkg/server/auth" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/server/auth" ) // TestIntegrationCacheIndexPollerAgainstAuthedSnapshot drives the full @@ -31,13 +31,13 @@ import ( // CacheIndexPoller.refresh // -> bearerToken() reads the SA token from a tmpfile (kubelet-shape) // -> fetchSnapshot() sends Authorization: Bearer -// -> in-process httptest server wrapped in pkg/server/auth.Middleware +// -> in-process httptest server wrapped in internal/server/auth.Middleware // -> Authenticator calls TokenReview against the envtest apiserver // -> apiserver validates the token it minted via TokenRequest -// -> handler returns a synthetic index.Snapshot +// -> handler returns a synthetic controlplaneapi.Snapshot // -> poller decodes it and writes CacheIndex.status against envtest // -// pkg/server/auth/integration_test.go already covers the middleware in +// internal/server/auth/integration_test.go already covers the middleware in // isolation with raw http requests. This test stitches the production // CLIENT code (the poller) onto the same backend, which is the surface // the bearer-token rollout actually changes for downstream callers. @@ -90,9 +90,9 @@ func TestIntegrationCacheIndexPollerAgainstAuthedSnapshot(t *testing.T) { // (the controller treats prefix-only replicas with no stats reported as // hidden from the cluster-wide CacheIndex.status surface — see the // per-backend CacheBackend.status.indexParticipation path for those). - served := index.Snapshot{ + served := controlplaneapi.Snapshot{ TotalPrefixes: 7, - Replicas: []index.ReplicaSnapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "r1", CacheMemoryBytes: 200, HitRate: 0.75, LastUpdate: time.Now()}, }, } @@ -191,7 +191,7 @@ func TestIntegrationCacheIndexPollerAgainstAuthedSnapshot(t *testing.T) { // rejects it under TokenReview.Audiences=[controller], so the middleware // returns 401 even though the SA identity would otherwise be admitted. // This is the over-the-wire complement to the in-process middleware - // envtest in pkg/server/auth and pins the same audience-binding contract + // envtest in internal/server/auth and pins the same audience-binding contract // against the controller's actual poller code path. Fail-soft expectation // matches the wrong-SA / no-token branches above. wrongAudienceTokenFile := mintTokenFileWithAudience(controllerSA, "https://kubernetes.default.svc") diff --git a/internal/controller/cacheindex_controller.go b/internal/controller/cacheindex_controller.go index ccf43bc4..3ddd29bd 100644 --- a/internal/controller/cacheindex_controller.go +++ b/internal/controller/cacheindex_controller.go @@ -29,8 +29,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/log" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" "github.com/cachebox-project/inference-cache/internal/enginebinding" - "github.com/cachebox-project/inference-cache/pkg/index" ) // Defaults for the CacheIndex status poller. @@ -196,7 +196,7 @@ func (p *CacheIndexPoller) refresh(ctx context.Context) error { // from churning resourceVersions (the same discipline as the CacheIndex write // and the CacheBackend status writers). A patch failure for one tenant does not // abort the others. -func (p *CacheIndexPoller) reconcileTenantStatuses(ctx context.Context, snap index.Snapshot) error { +func (p *CacheIndexPoller) reconcileTenantStatuses(ctx context.Context, snap controlplaneapi.Snapshot) error { var tenants cachev1alpha1.CacheTenantList if err := p.Client.List(ctx, &tenants); err != nil { if apierrors.IsNotFound(err) { @@ -205,7 +205,7 @@ func (p *CacheIndexPoller) reconcileTenantStatuses(ctx context.Context, snap ind return fmt.Errorf("list CacheTenants: %w", err) } - observedByID := make(map[string]index.TenantSnapshot, len(snap.Tenants)) + observedByID := make(map[string]controlplaneapi.TenantSnapshot, len(snap.Tenants)) for _, t := range snap.Tenants { observedByID[t.TenantID] = t } @@ -226,7 +226,7 @@ func (p *CacheIndexPoller) reconcileTenantStatuses(ctx context.Context, snap ind // reflects 0 rather than staying nil. obs, ok := observedByID[ct.Spec.TenantID] if !ok { - obs = index.TenantSnapshot{TenantID: ct.Spec.TenantID} + obs = controlplaneapi.TenantSnapshot{TenantID: ct.Spec.TenantID} } // Any CR whose tenantID is owned by a DIFFERENT CacheTenant is a shadowed // duplicate — whether or not it declares a quota of its own. A no-quota @@ -282,7 +282,7 @@ func (p *CacheIndexPoller) reconcileTenantStatuses(ctx context.Context, snap ind // arbitrary, and a fabricated 0 would mislead operators, so backend hit-rate // aggregation is deliberately left to a follow-up; the presence bit added by // the pointer-harmonize change is consumed only by CacheIndex.status. -func (p *CacheIndexPoller) refreshCacheBackendParticipation(ctx context.Context, snap index.Snapshot) error { +func (p *CacheIndexPoller) refreshCacheBackendParticipation(ctx context.Context, snap controlplaneapi.Snapshot) error { var backends cachev1alpha1.CacheBackendList if err := p.Client.List(ctx, &backends); err != nil { return fmt.Errorf("list CacheBackends: %w", err) @@ -590,31 +590,31 @@ func (p *CacheIndexPoller) bearerToken() (string, error) { // fetchSnapshot GETs and decodes the server's /snapshot JSON. When token is // non-empty it is sent as an Authorization: Bearer header so the server's // auth middleware can validate it via TokenReview. -func fetchSnapshot(ctx context.Context, hc *http.Client, url, token string) (index.Snapshot, error) { +func fetchSnapshot(ctx context.Context, hc *http.Client, url, token string) (controlplaneapi.Snapshot, error) { req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { - return index.Snapshot{}, fmt.Errorf("build snapshot request %q: %w", url, err) + return controlplaneapi.Snapshot{}, fmt.Errorf("build snapshot request %q: %w", url, err) } if token != "" { req.Header.Set("Authorization", "Bearer "+token) } resp, err := hc.Do(req) if err != nil { - return index.Snapshot{}, fmt.Errorf("scrape snapshot %s: %w", url, err) + return controlplaneapi.Snapshot{}, fmt.Errorf("scrape snapshot %s: %w", url, err) } defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { - return index.Snapshot{}, fmt.Errorf("snapshot %s: unexpected status %d", url, resp.StatusCode) + return controlplaneapi.Snapshot{}, fmt.Errorf("snapshot %s: unexpected status %d", url, resp.StatusCode) } - var snap index.Snapshot + var snap controlplaneapi.Snapshot if err := json.NewDecoder(resp.Body).Decode(&snap); err != nil { - return index.Snapshot{}, fmt.Errorf("decode snapshot: %w", err) + return controlplaneapi.Snapshot{}, fmt.Errorf("decode snapshot: %w", err) } return snap, nil } // buildCacheIndexStatus converts an index snapshot into CacheIndex status. -func buildCacheIndexStatus(snap index.Snapshot, serverURL string, now time.Time) cachev1alpha1.CacheIndexStatus { +func buildCacheIndexStatus(snap controlplaneapi.Snapshot, serverURL string, now time.Time) cachev1alpha1.CacheIndexStatus { st := cachev1alpha1.CacheIndexStatus{ Prefixes: cachev1alpha1.PrefixStatus{ Summary: cachev1alpha1.PrefixSummary{Total: int64(snap.TotalPrefixes), Hot: int64(snap.HotPrefixes)}, @@ -633,7 +633,7 @@ func buildCacheIndexStatus(snap index.Snapshot, serverURL string, now time.Time) // collide on `id` — pick the lexicographically-later tenant // deterministically so the chosen row is stable across ticks. // The `tenant` field on each row keeps the source identifiable. - byID := make(map[string]index.ReplicaSnapshot, len(snap.Replicas)) + byID := make(map[string]controlplaneapi.ReplicaSnapshot, len(snap.Replicas)) for _, r := range snap.Replicas { if r.LastUpdate.IsZero() { continue @@ -790,7 +790,7 @@ func effectiveTenantOwners(items []cachev1alpha1.CacheTenant) map[string]types.N // owns the same spec.tenantID and is the one actually enforced. Such a duplicate // must NOT report its own budget as effective: it goes Ready=False/Duplicate and // QuotaExceeded=False/NotEffective so the operator sees it is being ignored. -func buildCacheTenantStatus(ct *cachev1alpha1.CacheTenant, obs index.TenantSnapshot, shadowedBy *types.NamespacedName) cachev1alpha1.CacheTenantStatus { +func buildCacheTenantStatus(ct *cachev1alpha1.CacheTenant, obs controlplaneapi.TenantSnapshot, shadowedBy *types.NamespacedName) cachev1alpha1.CacheTenantStatus { st := cachev1alpha1.CacheTenantStatus{ObservedGeneration: ct.Generation} // Seed from existing conditions so meta.SetStatusCondition keeps each // condition's LastTransitionTime stable when its Status doesn't flip. diff --git a/internal/controller/cacheindex_controller_test.go b/internal/controller/cacheindex_controller_test.go index cdf9b7ed..f0cb5aee 100644 --- a/internal/controller/cacheindex_controller_test.go +++ b/internal/controller/cacheindex_controller_test.go @@ -28,19 +28,19 @@ import ( "github.com/prometheus/client_golang/prometheus/testutil" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" - "github.com/cachebox-project/inference-cache/pkg/index" ) func TestBuildCacheIndexStatus(t *testing.T) { now := time.Unix(1_700_000_000, 0) - snap := index.Snapshot{ + snap := controlplaneapi.Snapshot{ TotalPrefixes: 5, HotPrefixes: 0, - Replicas: []index.ReplicaSnapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "r1", Tenant: "ns-a", CacheMemoryBytes: 100, HitRate: 0.8, Pressure: 0.5, LastUpdate: now, StatsReported: true}, }, - Tenants: []index.TenantSnapshot{ + Tenants: []controlplaneapi.TenantSnapshot{ // MemoryUsed is non-zero here on purpose: it simulates an older / // skewed server still reporting the deprecated, double-counted // per-tenant memory. The controller must DISCARD it (hard-zero), @@ -101,8 +101,8 @@ func derefStr(p *string) string { // wins deterministically (preserves listMapKey=id uniqueness). func TestBuildCacheIndexStatusFiltersPrefixOnlyAndPicksWinner(t *testing.T) { now := time.Unix(1_700_000_000, 0) - snap := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + snap := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "ns-a", CacheMemoryBytes: 100, LastUpdate: now, StatsReported: true}, {ReplicaID: "vllm-0", Tenant: "ns-b", CacheMemoryBytes: 200, LastUpdate: now, StatsReported: true}, {ReplicaID: "prefix-only", Tenant: "ns-a", PrefixCount: 5}, @@ -135,14 +135,14 @@ func TestBuildCacheIndexStatusFiltersPrefixOnlyAndPicksWinner(t *testing.T) { // — that is what this test pins. func TestBuildCacheIndexStatusHitRateNilWhenUnreported(t *testing.T) { now := time.Unix(1_700_000_000, 0) - snap := index.Snapshot{ + snap := controlplaneapi.Snapshot{ TotalPrefixes: 4, - Replicas: []index.ReplicaSnapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ // Stats-bearing replica reporting a real 0% hit rate: HitRate is // present as "0", NOT nil — an observed zero, not an absence. {ReplicaID: "r-reported", Tenant: "ns-a", HitRate: 0, LastUpdate: now, StatsReported: true}, }, - Tenants: []index.TenantSnapshot{ + Tenants: []controlplaneapi.TenantSnapshot{ // Tenant with index entries but no reported stats: HitRate nil, // IndexEntries present (a real observed 0-vs-N count). {TenantID: "t-unreported", IndexEntries: 4, HitRate: 0, HitRateReported: false}, @@ -191,12 +191,12 @@ func TestBuildCacheIndexStatusHitRateNilWhenUnreported(t *testing.T) { func TestBuildCacheIndexStatusSkewFallbackPreservesOldServerHitRate(t *testing.T) { now := time.Unix(1_700_000_000, 0) // Simulate an old server: presence bits unset, but real reported values. - snap := index.Snapshot{ + snap := controlplaneapi.Snapshot{ TotalPrefixes: 3, - Replicas: []index.ReplicaSnapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "r-old", Tenant: "ns-a", CacheMemoryBytes: 100, HitRate: 0.66, LastUpdate: now, StatsReported: false}, }, - Tenants: []index.TenantSnapshot{ + Tenants: []controlplaneapi.TenantSnapshot{ {TenantID: "t-old", IndexEntries: 3, HitRate: 0.66, HitRateReported: false}, }, } @@ -214,7 +214,7 @@ func TestBuildCacheIndexStatusSkewFallbackPreservesOldServerHitRate(t *testing.T func TestEmptyIndexStatusRendersZeroSummary(t *testing.T) { // An empty index must still render prefixes.summary.{total,hot}=0 explicitly // (not omit them), matching the contract shape. - st := buildCacheIndexStatus(index.Snapshot{}, "http://server/snapshot", time.Unix(1, 0)) + st := buildCacheIndexStatus(controlplaneapi.Snapshot{}, "http://server/snapshot", time.Unix(1, 0)) b, err := json.Marshal(st) if err != nil { t.Fatalf("marshal: %v", err) @@ -250,7 +250,7 @@ func TestStatusEqualIgnoresTimestamps(t *testing.T) { } func TestFetchSnapshot(t *testing.T) { - want := index.Snapshot{TotalPrefixes: 7, Replicas: []index.ReplicaSnapshot{{ReplicaID: "r1"}}} + want := controlplaneapi.Snapshot{TotalPrefixes: 7, Replicas: []controlplaneapi.ReplicaSnapshot{{ReplicaID: "r1"}}} srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(want) })) @@ -284,7 +284,7 @@ func TestFetchSnapshotSendsBearerToken(t *testing.T) { var gotAuth string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotAuth = r.Header.Get("Authorization") - _ = json.NewEncoder(w).Encode(index.Snapshot{TotalPrefixes: 1}) + _ = json.NewEncoder(w).Encode(controlplaneapi.Snapshot{TotalPrefixes: 1}) })) defer srv.Close() @@ -370,7 +370,7 @@ func TestRefreshCreatesThenUpdatesOnlyOnChange(t *testing.T) { Build() var mu sync.Mutex - served := index.Snapshot{TotalPrefixes: 3, Replicas: []index.ReplicaSnapshot{{ReplicaID: "r1", CacheMemoryBytes: 100, HitRate: 0.8, LastUpdate: time.Unix(1_700_000_000, 0)}}} + served := controlplaneapi.Snapshot{TotalPrefixes: 3, Replicas: []controlplaneapi.ReplicaSnapshot{{ReplicaID: "r1", CacheMemoryBytes: 100, HitRate: 0.8, LastUpdate: time.Unix(1_700_000_000, 0)}}} srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { mu.Lock() defer mu.Unlock() @@ -408,7 +408,7 @@ func TestRefreshCreatesThenUpdatesOnlyOnChange(t *testing.T) { // Change the served snapshot → status updates. mu.Lock() - served = index.Snapshot{TotalPrefixes: 9, Replicas: []index.ReplicaSnapshot{{ReplicaID: "r1", CacheMemoryBytes: 500, HitRate: 0.9, LastUpdate: time.Unix(1_700_000_100, 0)}}} + served = controlplaneapi.Snapshot{TotalPrefixes: 9, Replicas: []controlplaneapi.ReplicaSnapshot{{ReplicaID: "r1", CacheMemoryBytes: 500, HitRate: 0.9, LastUpdate: time.Unix(1_700_000_100, 0)}}} mu.Unlock() if err := p.refresh(ctx); err != nil { t.Fatalf("third refresh: %v", err) @@ -425,7 +425,7 @@ func TestRefreshCreatesThenUpdatesOnlyOnChange(t *testing.T) { // buildPollerWithFixtures spins up a fake client + httptest server and returns // a poller wired to both. CacheBackends and engine pods are pre-loaded; the // served Snapshot is read under the supplied mutex. -func buildPollerWithFixtures(t *testing.T, backends []*cachev1alpha1.CacheBackend, enginePods []*corev1.Pod, served *index.Snapshot, mu *sync.Mutex) (*CacheIndexPoller, client.Client, *httptest.Server) { +func buildPollerWithFixtures(t *testing.T, backends []*cachev1alpha1.CacheBackend, enginePods []*corev1.Pod, served *controlplaneapi.Snapshot, mu *sync.Mutex) (*CacheIndexPoller, client.Client, *httptest.Server) { t.Helper() scheme := runtime.NewScheme() if err := cachev1alpha1.AddToScheme(scheme); err != nil { @@ -514,9 +514,9 @@ func TestRefreshProjectsParticipationPerBackend(t *testing.T) { t3 := time.Unix(1_700_000_100, 0).UTC() var mu sync.Mutex - served := index.Snapshot{ + served := controlplaneapi.Snapshot{ TotalPrefixes: 6, - Replicas: []index.ReplicaSnapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 2, LastEventAt: t1}, {ReplicaID: "vllm-a-1", Tenant: "default", PrefixCount: 3, LastEventAt: t2}, {ReplicaID: "vllm-b-0", Tenant: "default", PrefixCount: 1, LastEventAt: t3}, @@ -565,8 +565,8 @@ func TestRefreshNoEventsForBackendPublishesZeroParticipation(t *testing.T) { podA := enginePod("vllm-a-0", "default", map[string]string{"app": "vllm-a"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 1, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -605,8 +605,8 @@ func TestRefreshClearsStaleParticipationOnReplicaDrain(t *testing.T) { podA := enginePod("vllm-a-0", "default", map[string]string{"app": "vllm-a"}) var mu sync.Mutex tEvent := time.Unix(1_700_000_000, 0).UTC() - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 5, LastEventAt: tEvent}, }, } @@ -627,7 +627,7 @@ func TestRefreshClearsStaleParticipationOnReplicaDrain(t *testing.T) { // Drain: a successful scrape with zero matching replicas. mu.Lock() - served = index.Snapshot{Replicas: nil} + served = controlplaneapi.Snapshot{Replicas: nil} mu.Unlock() if err := p.refresh(ctx); err != nil { t.Fatalf("second refresh: %v", err) @@ -651,8 +651,8 @@ func TestRefreshSameNameDifferentNamespaceAttributesByLabel(t *testing.T) { podNS1 := enginePod("vllm-0", "ns-1", map[string]string{"app": "vllm"}) podNS2 := enginePod("vllm-0", "ns-2", map[string]string{"app": "vllm"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "ns-1", PrefixCount: 2, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vllm-0", Tenant: "ns-2", PrefixCount: 5, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, @@ -685,8 +685,8 @@ func TestRefreshDeletedEnginePodSkipsAttribution(t *testing.T) { podA0 := enginePod("vllm-a-0", "default", map[string]string{"app": "vllm-a"}) // vllm-a-1 reported in snapshot but no corresponding pod fixture. var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 2, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vllm-a-1", Tenant: "default", PrefixCount: 99, LastEventAt: time.Unix(1_700_000_999, 0).UTC()}, }, @@ -717,8 +717,8 @@ func TestRefreshBackendWithNoEngineSelectorSkipped(t *testing.T) { cbA := cbFixture("backend-a", "default", map[string]string{"app": "vllm-a"}) podA := enginePod("vllm-a-0", "default", map[string]string{"app": "vllm-a"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 2, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -748,8 +748,8 @@ func TestRefreshNoChurnOnIdenticalSnapshot(t *testing.T) { cbA := cbFixture("backend-a", "default", map[string]string{"app": "vllm-a"}) podA := enginePod("vllm-a-0", "default", map[string]string{"app": "vllm-a"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 4, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -786,8 +786,8 @@ func TestRefreshHitRateStaysNil(t *testing.T) { podA0 := enginePod("vllm-a-0", "default", map[string]string{"app": "vllm-a"}) podA1 := enginePod("vllm-a-1", "default", map[string]string{"app": "vllm-a"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 2, HitRate: 0.75, StatsReported: true, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vllm-a-1", Tenant: "default", PrefixCount: 3, HitRate: 0.85, StatsReported: true, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, @@ -824,8 +824,8 @@ func TestRefreshT2HitRatePresence(t *testing.T) { podB := enginePod("vllm-b-0", "default", map[string]string{"app": "vllm-b"}) podC := enginePod("vllm-c-0", "default", map[string]string{"app": "vllm-c"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-h-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 750, T2QueryTokens: 1000, LastUpdate: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vllm-b-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 0, T2QueryTokens: 500, LastUpdate: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vllm-c-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 0, T2QueryTokens: 0, LastUpdate: time.Unix(1_700_000_000, 0).UTC()}, @@ -873,7 +873,7 @@ func TestRefreshT2HitRateGauge(t *testing.T) { podB := enginePod("vb-0", "default", map[string]string{"app": "vb"}) podC := enginePod("vc-0", "default", map[string]string{"app": "vc"}) var mu sync.Mutex - served := index.Snapshot{Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vh-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 750, T2QueryTokens: 1000, LastUpdate: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vb-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 0, T2QueryTokens: 500, LastUpdate: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vc-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 0, T2QueryTokens: 0, LastUpdate: time.Unix(1_700_000_000, 0).UTC()}, @@ -915,7 +915,7 @@ inferencecache_backend_t2_query_tokens_total{backend="default/t2-h"} 0 // rate is cumulative, idleness alone would NOT prune it — drop-out does. Its // series must be pruned, not left at a stale 0. mu.Lock() - served.Replicas = []index.ReplicaSnapshot{ + served.Replicas = []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vh-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 1200, T2QueryTokens: 1500, LastUpdate: time.Unix(1_700_000_100, 0).UTC()}, } mu.Unlock() @@ -1050,7 +1050,7 @@ func TestRefreshT2HitRateCumulativeAfterRegression(t *testing.T) { cb := cbFixture("t2-r", "default", map[string]string{"app": "vr"}) pod := enginePod("vr-0", "default", map[string]string{"app": "vr"}) var mu sync.Mutex - served := index.Snapshot{Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vr-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 750, T2QueryTokens: 1000, LastUpdate: time.Unix(1_700_000_000, 0).UTC()}, }} p, _, srv := buildPollerWithFixtures(t, @@ -1062,7 +1062,7 @@ func TestRefreshT2HitRateCumulativeAfterRegression(t *testing.T) { // Healthy: 750/1000 = 0.75. Now queries climb (1000 -> 5000) while hits stay // flat at 750 — the tier stopped serving reloads. mu.Lock() - served.Replicas = []index.ReplicaSnapshot{ + served.Replicas = []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vr-0", Tenant: "default", PrefixCount: 1, T2HitTokens: 750, T2QueryTokens: 5000, LastUpdate: time.Unix(1_700_000_100, 0).UTC()}, } mu.Unlock() @@ -1087,8 +1087,8 @@ func TestRefreshScrapeFailureDoesNotClearParticipation(t *testing.T) { cbA := cbFixture("backend-a", "default", map[string]string{"app": "vllm-a"}) podA := enginePod("vllm-a-0", "default", map[string]string{"app": "vllm-a"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: "default", PrefixCount: 7, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1128,8 +1128,8 @@ func TestRefreshOverlappingSelectorsFirstNameWins(t *testing.T) { cbBeta := cbFixture("beta", "default", map[string]string{"app": "vllm", "model": "llama"}) podMatch := enginePod("vllm-0", "default", map[string]string{"app": "vllm", "model": "llama"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "default", PrefixCount: 4, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1164,8 +1164,8 @@ func TestRefreshAnnotationOwnedBackendWithNoSelector(t *testing.T) { cbOther := cbFixture("other", "default", map[string]string{"app": "vllm"}) pod := enginePodInjectedBy("vllm-0", "default", "default", "owner", map[string]string{"app": "vllm"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "default", PrefixCount: 3, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1198,8 +1198,8 @@ func TestRefreshAnnotationOverridesSelectorMatch(t *testing.T) { cbBeta := cbFixture("beta", "default", map[string]string{"app": "vllm"}) podMatch := enginePodInjectedBy("vllm-0", "default", "default", "beta", map[string]string{"app": "vllm"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "default", PrefixCount: 6, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1243,8 +1243,8 @@ func TestRefreshPodLookupErrorPreservesPriorStatus(t *testing.T) { // First refresh: clean client, publishes a positive participation. var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "default", PrefixCount: 9, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1309,8 +1309,8 @@ func TestRefreshUsesRealisticSidecarIdentityShape(t *testing.T) { // Pod name shaped like a real Deployment-managed ReplicaSet pod. pod := enginePod("vllm-7d9c8b6f4-abcd", "default", map[string]string{"app": "vllm"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-7d9c8b6f4-abcd", Tenant: "default", PrefixCount: 12, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1342,8 +1342,8 @@ func TestRefreshAnnotationPointsAtMissingBackend(t *testing.T) { other := cbFixture("other", "default", map[string]string{"app": "vllm"}) pod := enginePodInjectedBy("vllm-0", "default", "default", "gone", map[string]string{"app": "vllm"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "default", PrefixCount: 3, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1374,8 +1374,8 @@ func TestRefreshAnnotationInWrongNamespaceFallsBack(t *testing.T) { // Pod in ns-pod, annotation points at ns-other/foreign — cross-namespace. pod := enginePodInjectedBy("vllm-0", "ns-pod", "ns-other", "foreign", map[string]string{"app": "vllm"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "ns-pod", PrefixCount: 4, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1411,8 +1411,8 @@ func TestRefreshSelectorClearedAfterPublishingDrains(t *testing.T) { cb := cbFixture("backend", "default", map[string]string{"app": "vllm"}) pod := enginePod("vllm-0", "default", map[string]string{"app": "vllm"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: "default", PrefixCount: 8, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -1526,7 +1526,7 @@ func TestReconcileTenantStatusesShadowedDuplicate(t *testing.T) { ctx := context.Background() // 3 distinct prefixes for "shared": under the effective budget (5). - snap := index.Snapshot{Tenants: []index.TenantSnapshot{{TenantID: "shared", IndexEntries: 3}}} + snap := controlplaneapi.Snapshot{Tenants: []controlplaneapi.TenantSnapshot{{TenantID: "shared", IndexEntries: 3}}} if err := p.reconcileTenantStatuses(ctx, snap); err != nil { t.Fatalf("reconcile: %v", err) } @@ -1594,7 +1594,7 @@ func TestReconcileTenantStatusesProjectsAndFlapsQuota(t *testing.T) { } // Observed under budget: Ready=True, QuotaExceeded=False, indexEntries=2. - under := index.Snapshot{Tenants: []index.TenantSnapshot{{TenantID: "team-vision", IndexEntries: 2}}} + under := controlplaneapi.Snapshot{Tenants: []controlplaneapi.TenantSnapshot{{TenantID: "team-vision", IndexEntries: 2}}} if err := p.reconcileTenantStatuses(ctx, under); err != nil { t.Fatalf("reconcile (under): %v", err) } @@ -1610,7 +1610,7 @@ func TestReconcileTenantStatusesProjectsAndFlapsQuota(t *testing.T) { } // Observed over budget: QuotaExceeded flaps True (OverEntryBudget). - over := index.Snapshot{Tenants: []index.TenantSnapshot{{TenantID: "team-vision", IndexEntries: 5}}} + over := controlplaneapi.Snapshot{Tenants: []controlplaneapi.TenantSnapshot{{TenantID: "team-vision", IndexEntries: 5}}} if err := p.reconcileTenantStatuses(ctx, over); err != nil { t.Fatalf("reconcile (over): %v", err) } @@ -1651,7 +1651,7 @@ func TestReconcileTenantStatusesAbsentTenantObservedAsZero(t *testing.T) { // A successful scrape with no row for team-quiet means it currently holds // zero prefixes — an observed 0, not "unknown". - if err := p.reconcileTenantStatuses(ctx, index.Snapshot{}); err != nil { + if err := p.reconcileTenantStatuses(ctx, controlplaneapi.Snapshot{}); err != nil { t.Fatalf("reconcile: %v", err) } var got cachev1alpha1.CacheTenant diff --git a/internal/controller/cacheindex_integration_test.go b/internal/controller/cacheindex_integration_test.go index e5d982ba..32316be8 100644 --- a/internal/controller/cacheindex_integration_test.go +++ b/internal/controller/cacheindex_integration_test.go @@ -18,7 +18,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/index" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" ) // TestIntegrationCacheIndexPoller exercises the CacheIndex poller against a @@ -36,11 +36,11 @@ func TestIntegrationCacheIndexPoller(t *testing.T) { t.Run("StatusSurfaceOnCreate", func(t *testing.T) { lastUpdate := time.Unix(1_700_000_000, 0).UTC() - snap := index.Snapshot{ + snap := controlplaneapi.Snapshot{ TotalPrefixes: 7, // HotPrefixes intentionally left at 0: the controller should render // the deferred access-counting surface explicitly as hot: 0. - Replicas: []index.ReplicaSnapshot{{ + Replicas: []controlplaneapi.ReplicaSnapshot{{ ReplicaID: "vllm-0", Tenant: "tenant-a", CacheMemoryBytes: 2048, @@ -49,7 +49,7 @@ func TestIntegrationCacheIndexPoller(t *testing.T) { LastUpdate: lastUpdate, StatsReported: true, }}, - Tenants: []index.TenantSnapshot{{ + Tenants: []controlplaneapi.TenantSnapshot{{ TenantID: "tenant-a", IndexEntries: 7, HitRate: 0.75, @@ -143,9 +143,9 @@ func TestIntegrationCacheIndexPoller(t *testing.T) { t.Run("SnapshotPollWritesOnlyOnChange", func(t *testing.T) { var mu sync.Mutex - served := index.Snapshot{ + served := controlplaneapi.Snapshot{ TotalPrefixes: 3, - Replicas: []index.ReplicaSnapshot{{ + Replicas: []controlplaneapi.ReplicaSnapshot{{ ReplicaID: "vllm-0", Tenant: "tenant-a", CacheMemoryBytes: 100, @@ -153,7 +153,7 @@ func TestIntegrationCacheIndexPoller(t *testing.T) { LastUpdate: time.Unix(1_700_000_000, 0).UTC(), StatsReported: true, }}, - Tenants: []index.TenantSnapshot{{TenantID: "tenant-a", IndexEntries: 3, HitRate: 0.8, HitRateReported: true}}, + Tenants: []controlplaneapi.TenantSnapshot{{TenantID: "tenant-a", IndexEntries: 3, HitRate: 0.8, HitRateReported: true}}, } var requests int srv := newSnapshotServer(t, &served, &snapshotServerHooks{ @@ -194,9 +194,9 @@ func TestIntegrationCacheIndexPoller(t *testing.T) { if requests != 2 { t.Fatalf("snapshot requests after two refreshes = %d, want 2", requests) } - served = index.Snapshot{ + served = controlplaneapi.Snapshot{ TotalPrefixes: 9, - Replicas: []index.ReplicaSnapshot{{ + Replicas: []controlplaneapi.ReplicaSnapshot{{ ReplicaID: "vllm-0", Tenant: "tenant-a", CacheMemoryBytes: 500, @@ -204,7 +204,7 @@ func TestIntegrationCacheIndexPoller(t *testing.T) { LastUpdate: time.Unix(1_700_000_100, 0).UTC(), StatsReported: true, }}, - Tenants: []index.TenantSnapshot{{TenantID: "tenant-a", IndexEntries: 9, HitRate: 0.9, HitRateReported: true}}, + Tenants: []controlplaneapi.TenantSnapshot{{TenantID: "tenant-a", IndexEntries: 9, HitRate: 0.9, HitRateReported: true}}, } }() @@ -267,7 +267,7 @@ func TestIntegrationCacheIndexPoller(t *testing.T) { t.Fatalf("spec after pruning = %#v, want omitted or legacy empty object", spec) } - snap := index.Snapshot{TotalPrefixes: 1, Tenants: []index.TenantSnapshot{{TenantID: "tenant-a", IndexEntries: 1}}} + snap := controlplaneapi.Snapshot{TotalPrefixes: 1, Tenants: []controlplaneapi.TenantSnapshot{{TenantID: "tenant-a", IndexEntries: 1}}} srv := newSnapshotServer(t, &snap, nil) defer srv.Close() poller := &CacheIndexPoller{ @@ -318,7 +318,7 @@ type snapshotServerHooks struct { OnRequest func() } -func newSnapshotServer(t *testing.T, served *index.Snapshot, hooks *snapshotServerHooks) *httptest.Server { +func newSnapshotServer(t *testing.T, served *controlplaneapi.Snapshot, hooks *snapshotServerHooks) *httptest.Server { t.Helper() return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/snapshot" { diff --git a/internal/controller/cachepolicy_affinity_routing_integration_test.go b/internal/controller/cachepolicy_affinity_routing_integration_test.go index 61aaeabb..36658409 100644 --- a/internal/controller/cachepolicy_affinity_routing_integration_test.go +++ b/internal/controller/cachepolicy_affinity_routing_integration_test.go @@ -14,7 +14,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" ) // TestIntegrationCachePolicyAffinityRouting exercises the full @@ -37,7 +38,7 @@ import ( // - A namespace with NO CachePolicy reports DefaultAffinityRoutingEnabled // (the server-wide default fires for unconfigured tenants). // -// Complements the pkg/server unit tests by exercising the real +// Complements the internal/server unit tests by exercising the real // apiserver-side kubebuilder defaulting AND the controller→server // propagation path together. func TestIntegrationCachePolicyAffinityRouting(t *testing.T) { @@ -88,9 +89,9 @@ func TestIntegrationCachePolicyAffinityRouting(t *testing.T) { if got := store.AffinityRoutingEnabled(nsDisabled); got != false { t.Fatalf("explicit-Disabled namespace = %v, want false (operator opt-out)", got) } - if got := store.AffinityRoutingEnabled(nsUnconfigured); got != cacheserver.DefaultAffinityRoutingEnabled { + if got := store.AffinityRoutingEnabled(nsUnconfigured); got != controlplaneapi.DefaultAffinityRoutingEnabled { t.Fatalf("unconfigured namespace = %v, want DefaultAffinityRoutingEnabled (%v) — server-wide fallback failed", - got, cacheserver.DefaultAffinityRoutingEnabled) + got, controlplaneapi.DefaultAffinityRoutingEnabled) } // Belt-and-braces: read the omitted-field CR back from the apiserver and diff --git a/internal/controller/cachepolicy_authed_integration_test.go b/internal/controller/cachepolicy_authed_integration_test.go index 157c7aaf..aa9634a3 100644 --- a/internal/controller/cachepolicy_authed_integration_test.go +++ b/internal/controller/cachepolicy_authed_integration_test.go @@ -19,8 +19,8 @@ import ( "k8s.io/client-go/kubernetes" ctrl "sigs.k8s.io/controller-runtime" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" - "github.com/cachebox-project/inference-cache/pkg/server/auth" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" + "github.com/cachebox-project/inference-cache/internal/server/auth" ) // TestIntegrationCachePolicyPushAgainstAuthedEndpoint is the mirror of @@ -31,12 +31,12 @@ import ( // ControlPlaneReconciler.pushSnapshot // -> bearerToken() reads the SA token from a tmpfile (kubelet-shape) // -> POST carries Authorization: Bearer -// -> in-process httptest server wrapped in pkg/server/auth.Middleware +// -> in-process httptest server wrapped in internal/server/auth.Middleware // -> Authenticator calls TokenReview against the envtest apiserver // -> apiserver validates the token it minted via TokenRequest // -> handler accepts and returns 204 // -// The auth middleware unit tests in pkg/server/auth already cover the +// The auth middleware unit tests in internal/server/auth already cover the // middleware in isolation; this test pins the production CLIENT code (the // reconciler) onto the same backend, since that's the surface this hardening // changes for downstream callers. diff --git a/internal/controller/cachepolicy_eviction_integration_test.go b/internal/controller/cachepolicy_eviction_integration_test.go index ef651beb..309c7a29 100644 --- a/internal/controller/cachepolicy_eviction_integration_test.go +++ b/internal/controller/cachepolicy_eviction_integration_test.go @@ -14,14 +14,14 @@ import ( ctrl "sigs.k8s.io/controller-runtime" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/index" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" + "github.com/cachebox-project/inference-cache/internal/index" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" ) // TestIntegrationCachePolicyEvictionAlgorithm exercises the full // CachePolicy.spec.eviction loop against a real apiserver: the reconciler // flattens the CRD enum (lower-cased) into the PolicyStore, and an index wired -// with that store as its EvictionResolver (exactly as pkg/server.New does) +// with that store as its EvictionResolver (exactly as internal/server.New does) // picks LFU vs LRU victims accordingly when the entry cap is exceeded. // // One index per namespace/algorithm, each holding only its own tenant's diff --git a/internal/controller/cachepolicy_matched_tokens_floor_integration_test.go b/internal/controller/cachepolicy_matched_tokens_floor_integration_test.go index 54904adf..ae7c4a5a 100644 --- a/internal/controller/cachepolicy_matched_tokens_floor_integration_test.go +++ b/internal/controller/cachepolicy_matched_tokens_floor_integration_test.go @@ -14,7 +14,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" ) // TestIntegrationCachePolicyMinimumMatchedTokensFloor pins the wiring @@ -36,7 +37,7 @@ import ( // - A namespace with NO CachePolicy reports DefaultMinimumMatchedTokens // (the server-wide fallback fires for unconfigured tenants). // -// This test complements the pkg/server unit tests by exercising the real +// This test complements the internal/server unit tests by exercising the real // apiserver-side kubebuilder defaulting AND the controller→server propagation // path together — the C2 reconcile hot-loop class of bug (envtest exposed it // in the cap-eviction work and motivates this entire integration tier). @@ -97,18 +98,18 @@ func TestIntegrationCachePolicyMinimumMatchedTokensFloor(t *testing.T) { if got := store.MinimumMatchedTokens(nsExplicit); got != 256 { t.Fatalf("explicit-floor namespace = %d, want 256 (policy override)", got) } - if got := store.MinimumMatchedTokens(nsOmitted); got != cacheserver.DefaultMinimumMatchedTokens { + if got := store.MinimumMatchedTokens(nsOmitted); got != controlplaneapi.DefaultMinimumMatchedTokens { t.Fatalf("omitted-field namespace = %d, want DefaultMinimumMatchedTokens (%d) — "+ "kubebuilder default did not fill in 64 at apiserver admission, OR the controller "+ "didn't flatten the apiserver-defaulted value", - got, cacheserver.DefaultMinimumMatchedTokens) + got, controlplaneapi.DefaultMinimumMatchedTokens) } if got := store.MinimumMatchedTokens(nsDisabled); got != 0 { t.Fatalf("disabled-floor namespace = %d, want 0 (explicit opt-out)", got) } - if got := store.MinimumMatchedTokens(nsUnconfigured); got != cacheserver.DefaultMinimumMatchedTokens { + if got := store.MinimumMatchedTokens(nsUnconfigured); got != controlplaneapi.DefaultMinimumMatchedTokens { t.Fatalf("unconfigured namespace = %d, want DefaultMinimumMatchedTokens (%d) — server-wide fallback failed", - got, cacheserver.DefaultMinimumMatchedTokens) + got, controlplaneapi.DefaultMinimumMatchedTokens) } // Belt-and-braces: read the omitted-field CR back from the apiserver and @@ -125,8 +126,8 @@ func TestIntegrationCachePolicyMinimumMatchedTokensFloor(t *testing.T) { t.Fatalf("bare CachePolicy(%s).spec.minimumMatchedTokens is nil after apiserver round-trip — "+ "the +kubebuilder:default=64 marker did not materialize on the stored object", nsOmitted) } - if got := *readback.Spec.MinimumMatchedTokens; got != cacheserver.DefaultMinimumMatchedTokens { + if got := *readback.Spec.MinimumMatchedTokens; got != controlplaneapi.DefaultMinimumMatchedTokens { t.Fatalf("bare CachePolicy(%s).spec.minimumMatchedTokens = %d after apiserver round-trip, want %d "+ - "(the kubebuilder default)", nsOmitted, got, cacheserver.DefaultMinimumMatchedTokens) + "(the kubebuilder default)", nsOmitted, got, controlplaneapi.DefaultMinimumMatchedTokens) } } diff --git a/internal/controller/cachepolicy_routing_floor_integration_test.go b/internal/controller/cachepolicy_routing_floor_integration_test.go index 2d7cb893..6ba89d5c 100644 --- a/internal/controller/cachepolicy_routing_floor_integration_test.go +++ b/internal/controller/cachepolicy_routing_floor_integration_test.go @@ -14,7 +14,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" ) // TestIntegrationCachePolicyRoutingFloorScore exercises the full @@ -39,7 +40,7 @@ import ( // - A namespace with NO CachePolicy reports DefaultRoutingFloorScore // (the server-wide safety floor fires for unconfigured tenants). // -// Complements the pkg/server unit tests by exercising the real +// Complements the internal/server unit tests by exercising the real // apiserver-side kubebuilder defaulting AND the controller→server // propagation path together — the C2 reconcile hot-loop class of bug // (envtest exposed it in the cap-eviction work). @@ -101,18 +102,18 @@ func TestIntegrationCachePolicyRoutingFloorScore(t *testing.T) { if got := store.RoutingFloorScore(nsExplicit); !approx(got, 5.0) { t.Fatalf("explicit-floor namespace = %v, want 5.0 (policy override)", got) } - if got := store.RoutingFloorScore(nsOmitted); !approx(got, cacheserver.DefaultRoutingFloorScore) { + if got := store.RoutingFloorScore(nsOmitted); !approx(got, controlplaneapi.DefaultRoutingFloorScore) { t.Fatalf("omitted-field namespace = %v, want DefaultRoutingFloorScore (%v) — "+ "kubebuilder default did not fill in 0.1 at apiserver admission, OR the controller "+ "didn't flatten the apiserver-defaulted value", - got, cacheserver.DefaultRoutingFloorScore) + got, controlplaneapi.DefaultRoutingFloorScore) } if got := store.RoutingFloorScore(nsDisabled); got != 0 { t.Fatalf("disabled-floor namespace = %v, want 0 (explicit opt-out)", got) } - if got := store.RoutingFloorScore(nsUnconfigured); !approx(got, cacheserver.DefaultRoutingFloorScore) { + if got := store.RoutingFloorScore(nsUnconfigured); !approx(got, controlplaneapi.DefaultRoutingFloorScore) { t.Fatalf("unconfigured namespace = %v, want DefaultRoutingFloorScore (%v) — server-wide fallback failed", - got, cacheserver.DefaultRoutingFloorScore) + got, controlplaneapi.DefaultRoutingFloorScore) } // Belt-and-braces: read the omitted-field CR back from the apiserver and diff --git a/internal/controller/cachepolicy_strategy_integration_test.go b/internal/controller/cachepolicy_strategy_integration_test.go index ac4dd9c2..aec2a4c5 100644 --- a/internal/controller/cachepolicy_strategy_integration_test.go +++ b/internal/controller/cachepolicy_strategy_integration_test.go @@ -13,7 +13,7 @@ import ( ctrl "sigs.k8s.io/controller-runtime" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" ) func TestIntegrationCachePolicyStrategyGates(t *testing.T) { diff --git a/internal/controller/contract_coverage_sweep_test.go b/internal/controller/contract_coverage_sweep_test.go index 8a6a9ac5..054352ec 100644 --- a/internal/controller/contract_coverage_sweep_test.go +++ b/internal/controller/contract_coverage_sweep_test.go @@ -33,9 +33,8 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" builtinadapters "github.com/cachebox-project/inference-cache/internal/adapters/builtin" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" - "github.com/cachebox-project/inference-cache/pkg/index" ) // TestWebhookPollerSelectorFallbackAgreement is the load-bearing @@ -85,8 +84,8 @@ func TestWebhookPollerSelectorFallbackAgreement(t *testing.T) { const podName = "engine-a" pollerPod := enginePod(podName, ns, labels) // no injected-by annotation var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: podName, Tenant: ns, PrefixCount: 7, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -132,8 +131,8 @@ func TestRefreshNotFoundAndSuccessSameTickSameNamespace(t *testing.T) { // so the Get returns NotFound. livePod := enginePod("vllm-live-0", ns, map[string]string{"app": "vllm-live"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-live-0", Tenant: ns, PrefixCount: 5, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, {ReplicaID: "vllm-stale-0", Tenant: ns, PrefixCount: 99, LastEventAt: time.Unix(1_700_000_500, 0).UTC()}, }, @@ -178,8 +177,8 @@ func TestRefreshSelectorIsStrictSubsetOfPodLabels(t *testing.T) { "role": "engine", }) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: ns, PrefixCount: 3, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -218,8 +217,8 @@ func TestRefreshAnnotationNameIsPrefixOfAnotherBackend(t *testing.T) { map[string]string{"app": "vllm-longer"}) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-0", Tenant: ns, PrefixCount: 11, LastEventAt: time.Unix(1_700_000_000, 0).UTC()}, }, } @@ -265,8 +264,8 @@ func TestRefreshSamePodNameAcrossTenantsIsFailSoft(t *testing.T) { var mu sync.Mutex tsA := time.Unix(1_700_000_000, 0).UTC() tsB := time.Unix(1_700_000_500, 0).UTC() - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: podName, Tenant: nsA, PrefixCount: 4, LastEventAt: tsA}, {ReplicaID: podName, Tenant: nsB, PrefixCount: 9, LastEventAt: tsB}, }, @@ -350,14 +349,14 @@ func runPodWebhookAndCaptureInjectedBy(t *testing.T, namespace string, t.Fatalf("cachev1alpha1.AddToScheme: %v", err) } c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(cb1, cb2).Build() - registries := builtinadapters.New() + registries := builtinadapters.New(builtinadapters.Options{}) h := &podwebhook.EngineInjector{Reader: c, Registry: registries.Runtime, Log: logr.Discard()} pod := &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "engine-a", Labels: podLabels}, Spec: corev1.PodSpec{ Containers: []corev1.Container{{ - Name: adapterruntime.EngineContainerName, + Name: "vllm", Image: "vllm/vllm-openai-cpu:latest", Args: []string{"--model", "qwen"}, }}, diff --git a/internal/controller/controlplane_controller_test.go b/internal/controller/controlplane_controller_test.go index 152f7e02..4a0ff184 100644 --- a/internal/controller/controlplane_controller_test.go +++ b/internal/controller/controlplane_controller_test.go @@ -23,14 +23,15 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" ) // pushRecorder records every PolicySnapshot received over HTTP so the test // can assert on the body the controller would send to the real server. type pushRecorder struct { mu sync.Mutex - snapshots []cacheserver.PolicySnapshot + snapshots []controlplaneapi.PolicySnapshot method string authz string // last Authorization header observed } @@ -43,7 +44,7 @@ func (p *pushRecorder) handler() http.HandlerFunc { return } _ = r.Body.Close() - var snap cacheserver.PolicySnapshot + var snap controlplaneapi.PolicySnapshot if err := json.Unmarshal(body, &snap); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return @@ -63,11 +64,11 @@ func (p *pushRecorder) lastAuthz() string { return p.authz } -func (p *pushRecorder) latest() (cacheserver.PolicySnapshot, bool) { +func (p *pushRecorder) latest() (controlplaneapi.PolicySnapshot, bool) { p.mu.Lock() defer p.mu.Unlock() if len(p.snapshots) == 0 { - return cacheserver.PolicySnapshot{}, false + return controlplaneapi.PolicySnapshot{}, false } return p.snapshots[len(p.snapshots)-1], true } @@ -136,8 +137,8 @@ func TestPushSnapshotFlattensPoliciesAndTenants(t *testing.T) { if !ok { t.Fatal("expected a push") } - if snap.Version != cacheserver.PolicyPropagationVersion { - t.Fatalf("version = %d, want %d", snap.Version, cacheserver.PolicyPropagationVersion) + if snap.Version != controlplaneapi.PolicyPropagationVersion { + t.Fatalf("version = %d, want %d", snap.Version, controlplaneapi.PolicyPropagationVersion) } if len(snap.Policies) != 1 || snap.Policies[0].Namespace != "team-a" { t.Fatalf("policies = %+v, want one for team-a", snap.Policies) @@ -199,8 +200,8 @@ func TestPushSnapshotIncludesAllPolicies(t *testing.T) { if !ok { t.Fatal("expected at least one push") } - if snap.Version != cacheserver.PolicyPropagationVersion { - t.Fatalf("snapshot version = %d, want %d", snap.Version, cacheserver.PolicyPropagationVersion) + if snap.Version != controlplaneapi.PolicyPropagationVersion { + t.Fatalf("snapshot version = %d, want %d", snap.Version, controlplaneapi.PolicyPropagationVersion) } if rec.method != http.MethodPost { t.Fatalf("HTTP method = %q, want POST", rec.method) @@ -610,11 +611,11 @@ func TestResolveOnePolicyRoutingFloorScoreFallbackOnInvalidInput(t *testing.T) { // controller must still fall back to the safety default rather than 0. overflowing := "999999999999999999999999999999999999999999999999" cases := []tc{ - {name: "overflowing literal", spec: overflowing, want: cacheserver.DefaultRoutingFloorScore}, - {name: "malformed", spec: "not-a-number", want: cacheserver.DefaultRoutingFloorScore}, + {name: "overflowing literal", spec: overflowing, want: controlplaneapi.DefaultRoutingFloorScore}, + {name: "malformed", spec: "not-a-number", want: controlplaneapi.DefaultRoutingFloorScore}, // A negative leak — bypasses the CRD pattern but flows through to // the parser. Must fall back, not be honored. - {name: "negative", spec: "-1.5", want: cacheserver.DefaultRoutingFloorScore}, + {name: "negative", spec: "-1.5", want: controlplaneapi.DefaultRoutingFloorScore}, // Sanity: a well-formed value parses cleanly. {name: "valid", spec: "2.5", want: 2.5}, {name: "explicit zero opt-out", spec: "0", want: 0}, diff --git a/internal/controller/integration_test.go b/internal/controller/integration_test.go index 425469ea..ca0f19cf 100644 --- a/internal/controller/integration_test.go +++ b/internal/controller/integration_test.go @@ -38,8 +38,8 @@ import ( metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" podwebhook "github.com/cachebox-project/inference-cache/internal/webhook/pod" - "github.com/cachebox-project/inference-cache/pkg/index" ) // These tests run the reconciler against a real kube-apiserver (envtest), so they @@ -1491,8 +1491,8 @@ func TestIntegrationCacheIndexPollerProjectsParticipation(t *testing.T) { tEvent := time.Now().Add(-30 * time.Second).UTC().Truncate(time.Second) var mu sync.Mutex - served := index.Snapshot{ - Replicas: []index.ReplicaSnapshot{ + served := controlplaneapi.Snapshot{ + Replicas: []controlplaneapi.ReplicaSnapshot{ {ReplicaID: "vllm-a-0", Tenant: ns, PrefixCount: 4, LastEventAt: tEvent}, {ReplicaID: "vllm-b-0", Tenant: ns, PrefixCount: 1, LastEventAt: tEvent}, }, @@ -1536,9 +1536,9 @@ func TestIntegrationCacheIndexAcceptsUntenantedTenantRow(t *testing.T) { k8s, _, _ := startEnv(t) ctx := context.Background() - served := index.Snapshot{ + served := controlplaneapi.Snapshot{ TotalPrefixes: 5, - Tenants: []index.TenantSnapshot{ + Tenants: []controlplaneapi.TenantSnapshot{ {TenantID: "", IndexEntries: 2}, // untenanted bucket {TenantID: "team", IndexEntries: 3}, }, diff --git a/internal/controller/tenant_quota_integration_test.go b/internal/controller/tenant_quota_integration_test.go index 55b2ab1c..28159fe3 100644 --- a/internal/controller/tenant_quota_integration_test.go +++ b/internal/controller/tenant_quota_integration_test.go @@ -6,21 +6,19 @@ package controller import ( "context" - "encoding/json" "fmt" - "net/http" "net/http/httptest" "testing" "time" - "github.com/go-logr/logr" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" ctrl "sigs.k8s.io/controller-runtime" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - "github.com/cachebox-project/inference-cache/pkg/index" - cacheserver "github.com/cachebox-project/inference-cache/pkg/server" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" + cacheserver "github.com/cachebox-project/inference-cache/internal/server" ) // TestIntegrationCacheTenantQuota exercises the full CacheTenant quota loop @@ -28,10 +26,10 @@ import ( // APIs: // // - a real PolicyStore + index.Index (the index's quota resolver IS the store, -// exactly as pkg/server.New wires it); -// - the real /policy push handler and a /snapshot handler over the index; -// - the real ControlPlaneReconciler (CRD → push) and CacheIndexPoller (snapshot -// → CacheTenant.status). +// exactly as internal/server.New wires it); +// - the real /policy push handler; +// - the real ControlPlaneReconciler (CRD → push) and CacheIndexPoller's +// snapshot-DTO projection into CacheTenant.status. // // That covers what the fake client can't: real CRD validation/defaulting and // real Status().Patch semantics on CacheTenant. @@ -45,14 +43,8 @@ func TestIntegrationCacheTenantQuota(t *testing.T) { policySrv := httptest.NewServer(cacheserver.NewPolicyHTTPHandler(store)) defer policySrv.Close() - snapSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "application/json") - _ = json.NewEncoder(w).Encode(idx.Snapshot()) - })) - defer snapSrv.Close() - reconciler := &ControlPlaneReconciler{Client: k8s, ServerPolicyURL: policySrv.URL, HTTPClient: policySrv.Client()} - poller := &CacheIndexPoller{Client: k8s, SnapshotURL: snapSrv.URL, HTTPClient: snapSrv.Client(), Log: logr.Discard()} + poller := &CacheIndexPoller{Client: k8s} push := func() { t.Helper() if _, err := reconciler.Reconcile(ctx, ctrl.Request{}); err != nil { @@ -61,7 +53,17 @@ func TestIntegrationCacheTenantQuota(t *testing.T) { } scrape := func() { t.Helper() - if err := poller.reconcileTenantStatuses(ctx, idx.Snapshot()); err != nil { + domain := idx.Snapshot() + snap := controlplaneapi.Snapshot{TotalPrefixes: domain.TotalPrefixes, HotPrefixes: domain.HotPrefixes} + for _, tenant := range domain.Tenants { + snap.Tenants = append(snap.Tenants, controlplaneapi.TenantSnapshot{ + TenantID: tenant.TenantID, + IndexEntries: tenant.IndexEntries, + HitRate: tenant.HitRate, + HitRateReported: tenant.HitRateReported, + }) + } + if err := poller.reconcileTenantStatuses(ctx, snap); err != nil { t.Fatalf("scrape: %v", err) } } diff --git a/internal/controlplaneapi/doc.go b/internal/controlplaneapi/doc.go index 8db45f6b..15494c75 100644 --- a/internal/controlplaneapi/doc.go +++ b/internal/controlplaneapi/doc.go @@ -6,6 +6,6 @@ // the inference-cache controller and server binaries. // // These types are not a supported external Go API. Their JSON representation -// is the compatibility boundary for the controller-to-server /policy and -// /probe endpoints. +// is the compatibility boundary for the controller-to-server /snapshot, +// /policy, and /probe endpoints. package controlplaneapi diff --git a/internal/controlplaneapi/snapshot.go b/internal/controlplaneapi/snapshot.go new file mode 100644 index 00000000..9ae2e566 --- /dev/null +++ b/internal/controlplaneapi/snapshot.go @@ -0,0 +1,46 @@ +// SPDX-FileCopyrightText: 2026 The inference-cache Authors +// +// SPDX-License-Identifier: Apache-2.0 + +package controlplaneapi + +import "time" + +// Snapshot is the controller-facing representation returned by GET /snapshot. +// Its JSON tags are a private compatibility contract between independently +// rolled controller and server binaries. +type Snapshot struct { + Replicas []ReplicaSnapshot `json:"replicas"` + Tenants []TenantSnapshot `json:"tenants"` + TotalPrefixes int `json:"totalPrefixes"` + HotPrefixes int `json:"hotPrefixes"` +} + +// ReplicaSnapshot is the /snapshot representation of one replica's latest +// aggregate state. Presence bits preserve the distinction between an observed +// zero and a value that an older or not-yet-reporting producer omitted. +type ReplicaSnapshot struct { + ReplicaID string `json:"replicaId"` + Tenant string `json:"tenant,omitempty"` + CacheMemoryBytes int64 `json:"cacheMemoryBytes"` + HitRate float32 `json:"hitRate"` + Pressure float32 `json:"pressure"` + LastUpdate time.Time `json:"lastUpdate"` + PrefixCount int `json:"prefixCount"` + LastEventAt time.Time `json:"lastEventAt,omitempty"` + StatsReported bool `json:"statsReported,omitempty"` + T2HitTokens int64 `json:"t2HitTokens,omitempty"` + T2QueryTokens int64 `json:"t2QueryTokens,omitempty"` +} + +// TenantSnapshot is the /snapshot representation of one tenant's aggregate +// footprint. MemoryUsed is deprecated and intentionally remains a required +// zero-valued JSON key for controller/server skew compatibility. +type TenantSnapshot struct { + TenantID string `json:"tenantId"` + IndexEntries int64 `json:"indexEntries"` + HitRate float32 `json:"hitRate"` + HitRateReported bool `json:"hitRateReported,omitempty"` + // Deprecated: always 0; read ReplicaSnapshot.CacheMemoryBytes instead. + MemoryUsed int64 `json:"memoryUsed"` +} diff --git a/pkg/index/snapshot_wire_contract_test.go b/internal/controlplaneapi/snapshot_test.go similarity index 81% rename from pkg/index/snapshot_wire_contract_test.go rename to internal/controlplaneapi/snapshot_test.go index f6db3a26..c821d687 100644 --- a/pkg/index/snapshot_wire_contract_test.go +++ b/internal/controlplaneapi/snapshot_test.go @@ -2,9 +2,9 @@ // // SPDX-License-Identifier: Apache-2.0 -package index +package controlplaneapi -// Frozen wire-shape contract for pkg/index.Snapshot — the JSON the policy +// Frozen wire-shape contract for Snapshot — the JSON the policy // server publishes at /snapshot and the controller decodes in // CacheIndexPoller. A silent rename of any JSON tag (e.g. // json:"replicaId" → json:"replica_id") would still pass a round-trip test @@ -161,6 +161,35 @@ func TestSnapshotJSONOptionalTagWireShape(t *testing.T) { } } +// TestSnapshotPresenceBitsSupportRollingSkew pins both directions of the +// presence-bit contract: false values are omitted for old consumers, and a +// new controller decoding an old-server payload sees the zero-value false +// signal while retaining the legacy measurements used by its skew fallback. +func TestSnapshotPresenceBitsSupportRollingSkew(t *testing.T) { + newBody, err := json.Marshal(Snapshot{ + Replicas: []ReplicaSnapshot{{ReplicaID: "r1", HitRate: 0.5}}, + Tenants: []TenantSnapshot{{TenantID: "t1", HitRate: 0.5}}, + }) + if err != nil { + t.Fatalf("marshal new snapshot: %v", err) + } + if strings.Contains(string(newBody), `"statsReported"`) || strings.Contains(string(newBody), `"hitRateReported"`) { + t.Fatalf("false presence bits must be omitted for old consumers: %s", newBody) + } + + oldBody := []byte(`{"replicas":[{"replicaId":"r-old","cacheMemoryBytes":100,"hitRate":0.66,"pressure":0,"lastUpdate":"2023-11-14T22:13:20Z","prefixCount":3,"lastEventAt":"0001-01-01T00:00:00Z"}],"tenants":[{"tenantId":"t-old","indexEntries":3,"hitRate":0.66,"memoryUsed":0}],"totalPrefixes":3,"hotPrefixes":0}`) + var decoded Snapshot + if err := json.Unmarshal(oldBody, &decoded); err != nil { + t.Fatalf("decode old-server snapshot: %v", err) + } + if len(decoded.Replicas) != 1 || decoded.Replicas[0].StatsReported || decoded.Replicas[0].HitRate != 0.66 { + t.Fatalf("old replica skew decode = %+v", decoded.Replicas) + } + if len(decoded.Tenants) != 1 || decoded.Tenants[0].HitRateReported || decoded.Tenants[0].HitRate != 0.66 { + t.Fatalf("old tenant skew decode = %+v", decoded.Tenants) + } +} + // assertExactKeys verifies the set of top-level JSON keys in `got` is // exactly `want`, no more no less. Order doesn't matter; missing or // extra keys both fail. diff --git a/pkg/adapters/runtime/kernelcheck.go b/internal/enginebinding/runtime.go similarity index 58% rename from pkg/adapters/runtime/kernelcheck.go rename to internal/enginebinding/runtime.go index 0c17764f..e8899a76 100644 --- a/pkg/adapters/runtime/kernelcheck.go +++ b/internal/enginebinding/runtime.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package runtime +package enginebinding import ( corev1 "k8s.io/api/core/v1" @@ -10,9 +10,12 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" ) -// Kernel-check wire contract shared by the injecting built-in adapter and the -// controller that reads its annotation and termination message. +// Private wire contracts shared by built-in adapters, admission, and +// controllers. They are implementation details of the controller binary, not +// part of the public build-time adapter interface. const ( + SubscriberContainerName = "kvevent-subscriber" + LMCacheKernelCheckContainerName = "lmcache-kernel-check" AnnotationLMCacheKernelCheck = "inferencecache.io/lmcache-kernel-check" @@ -26,9 +29,9 @@ const ( EnvKernelCheckStrict = "KERNEL_CHECK_STRICT" ) -// InitContainerProvider is the optional capability implemented by an adapter -// that renders an engine-pod init container. Returning nil means no check is -// required for the given cache and pod. +// InitContainerProvider is the private capability implemented by a built-in +// adapter that renders an engine-pod init container. Returning nil means no +// check is required for the given cache and pod. type InitContainerProvider interface { KernelCheckInitContainer(cache *cachev1alpha1.CacheBackend, pod *corev1.Pod) (*corev1.Container, error) } @@ -43,3 +46,9 @@ func IsValidKernelCheckMode(s string) bool { return false } } + +// EngineHostNetworkRequested reports whether the operator opted an engine pod +// using a Mooncake remote binding into host networking. +func EngineHostNetworkRequested(cache *cachev1alpha1.CacheBackend) bool { + return cache != nil && cache.Spec.Integration != nil && cache.Spec.Integration.EngineHostNetwork +} diff --git a/pkg/index/adapter_partition_test.go b/internal/index/adapter_partition_test.go similarity index 100% rename from pkg/index/adapter_partition_test.go rename to internal/index/adapter_partition_test.go diff --git a/pkg/index/affinity_test.go b/internal/index/affinity_test.go similarity index 100% rename from pkg/index/affinity_test.go rename to internal/index/affinity_test.go diff --git a/pkg/index/aggregate_test.go b/internal/index/aggregate_test.go similarity index 100% rename from pkg/index/aggregate_test.go rename to internal/index/aggregate_test.go diff --git a/pkg/index/diagnostics_test.go b/internal/index/diagnostics_test.go similarity index 100% rename from pkg/index/diagnostics_test.go rename to internal/index/diagnostics_test.go diff --git a/pkg/index/distinguishing_power_lookup_test.go b/internal/index/distinguishing_power_lookup_test.go similarity index 99% rename from pkg/index/distinguishing_power_lookup_test.go rename to internal/index/distinguishing_power_lookup_test.go index 75cbbef4..fb1e3e8d 100644 --- a/pkg/index/distinguishing_power_lookup_test.go +++ b/internal/index/distinguishing_power_lookup_test.go @@ -31,7 +31,7 @@ func startTestIndex(t *testing.T) *Index { // TestLookupExactZeroDistinguishingWhenAllReplicasHoldPrefix is the headline // case: three replicas all hold a 16-token chat-template prefix. The // distinguishing-power factor must collapse to 0, so every Score is 0 — -// the service-layer post-score floor (covered in pkg/server) then downgrades +// the service-layer post-score floor (covered in internal/server) then downgrades // the response to NO_HINT. The index itself stays policy-unaware: it still // returns the matched replicas; the floor decides whether they ship. func TestLookupExactZeroDistinguishingWhenAllReplicasHoldPrefix(t *testing.T) { diff --git a/pkg/index/distinguishing_power_test.go b/internal/index/distinguishing_power_test.go similarity index 100% rename from pkg/index/distinguishing_power_test.go rename to internal/index/distinguishing_power_test.go diff --git a/internal/index/doc.go b/internal/index/doc.go new file mode 100644 index 00000000..8748d3b4 --- /dev/null +++ b/internal/index/doc.go @@ -0,0 +1,12 @@ +// SPDX-FileCopyrightText: 2026 The inference-cache Authors +// +// SPDX-License-Identifier: Apache-2.0 + +// Package index implements the inference-cache server's mutable cache-state +// index, populated from engine KV events and queried by LookupRoute. +// +// The index engine (the in-memory store, ingestion, eviction, ranking) runs only +// in the server binary. Snapshot types are index-owned domain values; the server +// maps them explicitly to the controller-facing HTTP contract in +// internal/controlplaneapi. +package index diff --git a/pkg/index/index.go b/internal/index/index.go similarity index 98% rename from pkg/index/index.go rename to internal/index/index.go index f707d725..2e23dc7e 100644 --- a/pkg/index/index.go +++ b/internal/index/index.go @@ -1003,7 +1003,7 @@ func (i *Index) lookupWithHits(req LookupRequest) ([]ReplicaScore, map[string][] // PREFIX_MATCH path when the top score falls below the per-namespace // routingFloorScore OR when every replica's matched_tokens falls below // the per-namespace minimumMatchedTokens floor — see -// pkg/server/inferencecache_service.go buildLookupResponse. The downgrade +// internal/server/inferencecache_service.go buildLookupResponse. The downgrade // lands on StrategyNone, which surfaces as AFFINITY_HINT under // default-enabled affinity with a usable seed + serving replica or as // NO_HINT under affinityRouting: Disabled. Old gateway clients that only @@ -1607,7 +1607,7 @@ func (i *Index) tenantHotCandidates(req LookupRequest) []ReplicaScore { // modulo trade-off); a Phase-2 follow-up may replace the modulo with // Rendezvous / HRW without altering this method's signature. // -// Wired by pkg/server/inferencecache_service.buildLookupResponse on the +// Wired by internal/server/inferencecache_service.buildLookupResponse on the // StrategyNone branch when CachePolicy.spec.affinityRouting is Enabled // (the kubebuilder default). The index is policy-unaware; the toggle // lives entirely in the server. @@ -1760,17 +1760,17 @@ func (i *Index) aggregateLocked() Aggregate { return agg } -// Snapshot is a point-in-time, cluster-wide view of the index for the -// CacheIndex status surface (consumed by the controller). Metadata only. +// Snapshot is a point-in-time, cluster-wide domain view of the index. The +// server maps it to the controller-facing /snapshot DTO. Metadata only. // // TotalPrefixes is the number of distinct prefix keys (a prefix held by // multiple replicas counts once), and it equals the sum of // tenants[].indexEntries — see Aggregate. type Snapshot struct { - Replicas []ReplicaSnapshot `json:"replicas"` - Tenants []TenantSnapshot `json:"tenants"` - TotalPrefixes int `json:"totalPrefixes"` - HotPrefixes int `json:"hotPrefixes"` // always 0: intentionally unwired. The per-entry LFU access counter exists but governs cap eviction only; it is not aggregated into a cluster-wide "hot prefix" count. + Replicas []ReplicaSnapshot + Tenants []TenantSnapshot + TotalPrefixes int + HotPrefixes int // always 0: intentionally unwired. The per-entry LFU access counter exists but governs cap eviction only; it is not aggregated into a cluster-wide "hot prefix" count. } // ReplicaSnapshot is the latest reported state for one replica, cluster-wide. @@ -1787,14 +1787,14 @@ type Snapshot struct { // consumer scope a pod lookup. Empty when the replica is only known through // older code paths that did not carry tenant context. type ReplicaSnapshot struct { - ReplicaID string `json:"replicaId"` - Tenant string `json:"tenant,omitempty"` - CacheMemoryBytes int64 `json:"cacheMemoryBytes"` - HitRate float32 `json:"hitRate"` - Pressure float32 `json:"pressure"` - LastUpdate time.Time `json:"lastUpdate"` - PrefixCount int `json:"prefixCount"` - LastEventAt time.Time `json:"lastEventAt,omitempty"` + ReplicaID string + Tenant string + CacheMemoryBytes int64 + HitRate float32 + Pressure float32 + LastUpdate time.Time + PrefixCount int + LastEventAt time.Time // StatsReported is true once the replica's stats reporter has emitted at // least one stats payload (the replica appears in the stats map). It is the // presence bit that lets a consumer distinguish an observed 0 hit rate / @@ -1803,11 +1803,11 @@ type ReplicaSnapshot struct { // zero-valued HitRate/Pressure/CacheMemoryBytes/LastUpdate. The CacheIndex // status projection uses it to leave the cluster-aggregate replica hitRate // nil rather than fabricating "0" (see internal/controller). - StatsReported bool `json:"statsReported,omitempty"` + StatsReported bool // T2HitTokens / T2QueryTokens carry the replica's cumulative tier-2 // (external offload) reload token counters across the /snapshot wire. - T2HitTokens int64 `json:"t2HitTokens,omitempty"` - T2QueryTokens int64 `json:"t2QueryTokens,omitempty"` + T2HitTokens int64 + T2QueryTokens int64 } // TenantSnapshot is the aggregate footprint for one tenant. @@ -1825,9 +1825,9 @@ type ReplicaSnapshot struct { // docs/design/crd-contract.md and docs/concepts/cachetenant-identity-and-quota.md // for the enforcement-boundary rationale. type TenantSnapshot struct { - TenantID string `json:"tenantId"` - IndexEntries int64 `json:"indexEntries"` - HitRate float32 `json:"hitRate"` + TenantID string + IndexEntries int64 + HitRate float32 // HitRateReported is true once at least one replica of this tenant has // reported stats (the hit-rate average had n > 0 samples). It is the // presence bit that distinguishes an observed mean hit rate of 0 from "no @@ -1836,9 +1836,9 @@ type TenantSnapshot struct { // and a zero-valued HitRate. The CacheIndex status projection uses it to // leave the cluster-aggregate tenant hitRate nil rather than fabricating // "0" (see internal/controller). - HitRateReported bool `json:"hitRateReported,omitempty"` + HitRateReported bool // Deprecated: always 0; read ReplicaSnapshot.CacheMemoryBytes instead. - MemoryUsed int64 `json:"memoryUsed"` + MemoryUsed int64 } // Snapshot returns the current cluster-wide aggregate. Replicas use the latest diff --git a/pkg/index/index_test.go b/internal/index/index_test.go similarity index 98% rename from pkg/index/index_test.go rename to internal/index/index_test.go index 08faa7d2..52267b72 100644 --- a/pkg/index/index_test.go +++ b/internal/index/index_test.go @@ -27,7 +27,7 @@ func hash(s string) []byte { return []byte(s) } // Stage A lookup still finds them) but invisible to the cap accounting, // aggregate, snapshot, and per-model entry-count gauge — so a probe in flight // cannot displace real workload state via the cap sweep AND cannot leak -// into observability surfaces. Mirrors TestProberRun* in pkg/server, but +// into observability surfaces. Mirrors TestProberRun* in internal/server, but // from the index's perspective. func TestReservedTenantHiddenFromCapAndAggregate(t *testing.T) { const reserved = "inferencecache.io/probe" @@ -487,47 +487,6 @@ func TestSnapshotPresenceBitsDistinguishAbsentFromZero(t *testing.T) { } } -// TestSnapshotJSONRoundtripPreservesTenantAndPrefixFields guards the wire -// shape of /snapshot. The controller decodes the JSON into the same -// Snapshot type, so a silent rename of one of the JSON tags (e.g. someone -// dropping `Tenant` to `omitempty` and writing a tenant-less replica) -// would still compile but break per-backend attribution downstream. This -// test JSON-encodes a snapshot with all the new fields set and asserts -// they survive the round-trip. -func TestSnapshotJSONRoundtripPreservesTenantAndPrefixFields(t *testing.T) { - idx := New() - idx.Ingest(Update{ - ReplicaID: "vllm-0", Model: "m", Tenant: "ns-a", HashScheme: "vllm", - Prefixes: []PrefixRef{{PrefixHash: hash("p"), TokenCount: 1}}, - Stats: &ReplicaStats{CacheMemoryBytes: 100, HitRate: 0.5, Pressure: 0.2}, - }) - - raw, err := json.Marshal(idx.Snapshot()) - if err != nil { - t.Fatalf("encode snapshot: %v", err) - } - var decoded Snapshot - if err := json.Unmarshal(raw, &decoded); err != nil { - t.Fatalf("decode snapshot: %v", err) - } - if len(decoded.Replicas) != 1 { - t.Fatalf("replicas = %d, want 1", len(decoded.Replicas)) - } - r := decoded.Replicas[0] - if r.ReplicaID != "vllm-0" || r.Tenant != "ns-a" { - t.Fatalf("identity round-trip lost: replicaId=%q tenant=%q", r.ReplicaID, r.Tenant) - } - if r.PrefixCount != 1 { - t.Fatalf("prefixCount round-trip = %d, want 1", r.PrefixCount) - } - if r.LastEventAt.IsZero() { - t.Fatal("lastEventAt round-trip lost (zero)") - } - if r.CacheMemoryBytes != 100 || r.HitRate != 0.5 || r.Pressure != 0.2 { - t.Fatalf("stats round-trip lost: %+v", r) - } -} - func TestReadyReflectsStartAndStop(t *testing.T) { idx := New(WithSweepInterval(10 * time.Millisecond)) if idx.Ready() { diff --git a/pkg/index/lfu_eviction_test.go b/internal/index/lfu_eviction_test.go similarity index 100% rename from pkg/index/lfu_eviction_test.go rename to internal/index/lfu_eviction_test.go diff --git a/pkg/index/tenant_quota_test.go b/internal/index/tenant_quota_test.go similarity index 100% rename from pkg/index/tenant_quota_test.go rename to internal/index/tenant_quota_test.go diff --git a/pkg/server/adapter_partition_test.go b/internal/server/adapter_partition_test.go similarity index 85% rename from pkg/server/adapter_partition_test.go rename to internal/server/adapter_partition_test.go index 4525121c..2ec9e84c 100644 --- a/pkg/server/adapter_partition_test.go +++ b/internal/server/adapter_partition_test.go @@ -8,9 +8,9 @@ import ( "context" "testing" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/subscriber" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" ) // End-to-end round trip of adapter identity: engine KV event (carrying vLLM's @@ -34,7 +34,7 @@ func TestLookupRouteAdapterPartitionRoundTrip(t *testing.T) { toks := tokenSeq(1_000, blockTok) key := fingerprint.Bytes(fingerprint.PrefixHashes(toks, blockTok)[0]) - cfg := engine.Config{ + cfg := subscriber.Config{ ReplicaID: "vllm-engine-cs1", ModelID: modelID, TenantID: tenantID, @@ -44,10 +44,10 @@ func TestLookupRouteAdapterPartitionRoundTrip(t *testing.T) { AdapterNames: map[int64]string{sqlID: "sql-lora", chatID: "chat-lora"}, } client, stop := runEngineReporterCfgAgainstServer(t, cfg, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, - &engine.EventBatch{Events: []engine.Event{ - engine.BlockStored{BlockHashes: [][]byte{be8(1)}, TokenIDs: toks, BlockSize: blockTok, LoRAID: &sqlID}, - engine.BlockStored{BlockHashes: [][]byte{be8(2)}, TokenIDs: toks, BlockSize: blockTok, LoRAID: &chatID}, + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, + &subscriber.EventBatch{Events: []subscriber.Event{ + subscriber.BlockStored{BlockHashes: [][]byte{be8(1)}, TokenIDs: toks, BlockSize: blockTok, LoRAID: &sqlID}, + subscriber.BlockStored{BlockHashes: [][]byte{be8(2)}, TokenIDs: toks, BlockSize: blockTok, LoRAID: &chatID}, }}, ) defer stop() @@ -106,9 +106,9 @@ func TestLookupRouteWithoutAdapterIsUnchanged(t *testing.T) { key := fingerprint.Bytes(fingerprint.PrefixHashes(toks, blockTok)[0]) client, stop := runEngineReporterAgainstServer(t, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, - &engine.EventBatch{Events: []engine.Event{ - engine.BlockStored{BlockHashes: [][]byte{be8(1)}, TokenIDs: toks, BlockSize: blockTok}, + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, + &subscriber.EventBatch{Events: []subscriber.Event{ + subscriber.BlockStored{BlockHashes: [][]byte{be8(1)}, TokenIDs: toks, BlockSize: blockTok}, }}, ) defer stop() diff --git a/pkg/server/affinity_routing_test.go b/internal/server/affinity_routing_test.go similarity index 95% rename from pkg/server/affinity_routing_test.go rename to internal/server/affinity_routing_test.go index 5882a759..151eff8b 100644 --- a/pkg/server/affinity_routing_test.go +++ b/internal/server/affinity_routing_test.go @@ -10,8 +10,9 @@ import ( "testing" "time" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" ) // newServiceWithReplicas builds the service against a fresh index that @@ -64,7 +65,7 @@ func TestLookupRouteAffinityHintOnNoMatchEnabled(t *testing.T) { func TestLookupRouteAffinityHintDisabledReturnsNoHint(t *testing.T) { svc, _, store := newServiceWithReplicas(t, "tenantA", "modelX", []string{"r-1", "r-2"}) fal := false - store.Replace([]ResolvedPolicy{{Namespace: "tenantA", AffinityRouting: &fal}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenantA", AffinityRouting: &fal}}) req := &icpb.LookupRouteRequest{ TenantId: "tenantA", @@ -178,7 +179,7 @@ func TestLookupRouteAffinityHintBypassesMinimumPrefixTokens(t *testing.T) { svc, _, store := newServiceWithReplicas(t, "tenantA", "modelX", []string{"r-1", "r-2", "r-3"}) // Set a high minimumPrefixTokens floor that would short-circuit a // small request. Affinity should still fire. - store.Replace([]ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024}}) req := &icpb.LookupRouteRequest{ TenantId: "tenantA", @@ -207,7 +208,7 @@ func TestLookupRouteAffinityHintBypassesMinimumPrefixTokens(t *testing.T) { func TestLookupRouteMinimumPrefixTokensStillFiltersWhenAffinityDisabled(t *testing.T) { svc, _, store := newServiceWithReplicas(t, "tenantA", "modelX", []string{"r-1", "r-2"}) fal := false - store.Replace([]ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024, AffinityRouting: &fal}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024, AffinityRouting: &fal}}) req := &icpb.LookupRouteRequest{ TenantId: "tenantA", @@ -249,7 +250,7 @@ func TestLookupRouteAffinityHintPreservesUnknownHashSchemePrecedence(t *testing. store := NewPolicyStore() // minimumPrefixTokens=1024 means the request below would short-circuit // pre-lookup if it weren't for the precedence guard. - store.Replace([]ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024}}) svc := newInferenceCacheService(idx, newServerMetrics(), store) req := &icpb.LookupRouteRequest{ @@ -287,7 +288,7 @@ func TestLookupRouteMinimumPrefixTokensDowngradesPrefixMatchWhenAffinityEnabled( // minimumPrefixTokens=1024 — the request below is well under the gate. // minimumMatchedTokens=0 — disable the matched-tokens floor so the only // thing in the way of PREFIX_MATCH is the minimumPrefixTokens downgrade. - store.Replace([]ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024, MinimumMatchedTokens: 0}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024, MinimumMatchedTokens: 0}}) svc := newInferenceCacheService(idx, newServerMetrics(), store) req := &icpb.LookupRouteRequest{ @@ -334,7 +335,7 @@ func TestLookupRouteMinimumPrefixTokensDowngradesTenantHotWhenAffinityEnabled(t }) store := NewPolicyStore() // MinimumPrefixTokens=1024 — way above the request's claimed 1 token. - store.Replace([]ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenantA", MinimumPrefixTokens: 1024}}) svc := newInferenceCacheService(idx, newServerMetrics(), store) req := &icpb.LookupRouteRequest{ diff --git a/pkg/server/auth/audience.go b/internal/server/auth/audience.go similarity index 100% rename from pkg/server/auth/audience.go rename to internal/server/auth/audience.go diff --git a/pkg/server/auth/doc.go b/internal/server/auth/doc.go similarity index 100% rename from pkg/server/auth/doc.go rename to internal/server/auth/doc.go diff --git a/pkg/server/auth/integration_test.go b/internal/server/auth/integration_test.go similarity index 98% rename from pkg/server/auth/integration_test.go rename to internal/server/auth/integration_test.go index d02ae6c4..b6e764cc 100644 --- a/pkg/server/auth/integration_test.go +++ b/internal/server/auth/integration_test.go @@ -21,7 +21,7 @@ import ( logf "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/log/zap" - "github.com/cachebox-project/inference-cache/pkg/server/auth" + "github.com/cachebox-project/inference-cache/internal/server/auth" ) // TestAuthMiddleware_AgainstEnvtestAPIServer exercises the middleware against diff --git a/pkg/server/auth/middleware.go b/internal/server/auth/middleware.go similarity index 100% rename from pkg/server/auth/middleware.go rename to internal/server/auth/middleware.go diff --git a/pkg/server/auth/middleware_test.go b/internal/server/auth/middleware_test.go similarity index 100% rename from pkg/server/auth/middleware_test.go rename to internal/server/auth/middleware_test.go diff --git a/pkg/server/diagnostics_test.go b/internal/server/diagnostics_test.go similarity index 96% rename from pkg/server/diagnostics_test.go rename to internal/server/diagnostics_test.go index eb72fdac..afe0add5 100644 --- a/pkg/server/diagnostics_test.go +++ b/internal/server/diagnostics_test.go @@ -13,8 +13,9 @@ import ( "github.com/prometheus/client_golang/prometheus/promhttp" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" ) // TestLookupRouteEmitsUnknownHashScheme pins the scheme-mismatch wire shape: @@ -119,7 +120,7 @@ func TestLookupRouteGenuineMissStillNoHint(t *testing.T) { // behavior on the same scenario is covered in // affinity_routing_test.go. fal := false - svc.policies.Replace([]ResolvedPolicy{{Namespace: "t", AffinityRouting: &fal}}) + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "t", AffinityRouting: &fal}}) // Use Stats: nil so the warm-replica TENANT_HOT path can't fire — we // want to isolate the "real miss" branch end-to-end. svc.index.Ingest(index.Update{ diff --git a/pkg/server/doc.go b/internal/server/doc.go similarity index 72% rename from pkg/server/doc.go rename to internal/server/doc.go index 2273b3ea..8a56015c 100644 --- a/pkg/server/doc.go +++ b/internal/server/doc.go @@ -4,6 +4,5 @@ // Package server implements the inference-cache server binary's gRPC and HTTP // surfaces. It is an in-repository implementation package, not a supported -// extension API; it remains under pkg only while the staged internal/server -// migration is completed. +// extension API. package server diff --git a/pkg/server/inferencecache_service.go b/internal/server/inferencecache_service.go similarity index 98% rename from pkg/server/inferencecache_service.go rename to internal/server/inferencecache_service.go index 19a252af..66c99a89 100644 --- a/pkg/server/inferencecache_service.go +++ b/internal/server/inferencecache_service.go @@ -12,9 +12,10 @@ import ( "math" "time" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" "github.com/cachebox-project/inference-cache/pkg/tokenize" ) @@ -202,7 +203,7 @@ func (s *inferenceCacheService) LookupRoute(ctx context.Context, req *icpb.Looku // would require a schema change owned by the standalone F-series // metric work.) The legitimate probe path uses index.LookupRoute // directly, not the gRPC handler. - if tenant == ProbeTenantID { + if tenant == controlplaneapi.ProbeTenantID { resp := &icpb.LookupRouteResponse{ReasonCode: reasonNoHint} s.metrics.observeLookup(model, resp.ReasonCode, false, 0) return resp, nil @@ -576,7 +577,7 @@ func (s *inferenceCacheService) buildLookupResponse(req *icpb.LookupRouteRequest if result.Strategy == index.StrategyPrefixMatch { if floor := s.policyRoutingFloorScore(tenant); floor > 0 && len(result.Scores) > 0 { // Scores are sorted descending by Score (see - // sortScoresDescByScoreThenID in pkg/index), so the first + // sortScoresDescByScoreThenID in internal/index), so the first // element is the best surviving replica. if result.Scores[0].Score < floor { // Drop the hits map by constructing a fresh result — @@ -694,7 +695,7 @@ func (s *inferenceCacheService) tryAffinityResponse(req *icpb.LookupRouteRequest // CachePolicy.spec.affinityRouting + PolicyStore.AffinityRoutingEnabled. func (s *inferenceCacheService) affinityRoutingEnabled(tenant string) bool { if s.policies == nil { - return DefaultAffinityRoutingEnabled + return controlplaneapi.DefaultAffinityRoutingEnabled } return s.policies.AffinityRoutingEnabled(tenant) } @@ -912,21 +913,21 @@ func (s *inferenceCacheService) policyRoutingFloorScore(tenant string) float32 { func (s *inferenceCacheService) policyChainMatchingEnabled(tenant string) bool { if s.policies == nil { - return DefaultEnableChainMatching + return controlplaneapi.DefaultEnableChainMatching } return s.policies.ChainMatchingEnabled(tenant) } func (s *inferenceCacheService) policyChainRequired(tenant string) bool { if s.policies == nil { - return DefaultRequireChain + return controlplaneapi.DefaultRequireChain } return s.policies.ChainRequired(tenant) } func (s *inferenceCacheService) policyTenantHotEnabled(tenant string) bool { if s.policies == nil { - return DefaultEnableTenantHot + return controlplaneapi.DefaultEnableTenantHot } return s.policies.TenantHotEnabled(tenant) } @@ -973,7 +974,7 @@ func (*inferenceCacheService) LookupPDRoute(context.Context, *icpb.LookupPDRoute // reads the cluster-wide aggregate via /snapshot, which also filters reserved // tenants. func (s *inferenceCacheService) GetCacheState(_ context.Context, req *icpb.GetCacheStateRequest) (*icpb.GetCacheStateResponse, error) { - if req.GetTenantId() == ProbeTenantID { + if req.GetTenantId() == controlplaneapi.ProbeTenantID { return &icpb.GetCacheStateResponse{Summary: &icpb.CacheSummary{}}, nil } replicas, totalPrefixes := s.index.CacheState(req.GetTenantId(), req.GetModelId()) @@ -1014,7 +1015,7 @@ func (s *inferenceCacheService) ReportCacheState(stream icpb.InferenceCache_Repo } return err } - if update.GetTenantId() == ProbeTenantID { + if update.GetTenantId() == controlplaneapi.ProbeTenantID { continue } s.index.Ingest(updateFromProto(update)) @@ -1028,7 +1029,7 @@ func (s *inferenceCacheService) ReportCacheState(stream icpb.InferenceCache_Repo // regardless, but the silent drop keeps the public gRPC contract from // touching server-internal state. func (s *inferenceCacheService) PublishEvent(_ context.Context, ev *icpb.CacheEvent) (*icpb.Ack, error) { - if ev.GetTenantId() == ProbeTenantID { + if ev.GetTenantId() == controlplaneapi.ProbeTenantID { return &icpb.Ack{Accepted: true}, nil } if t := eventTypeFromProto(ev.GetType()); t != 0 { diff --git a/pkg/server/lfu_credit_test.go b/internal/server/lfu_credit_test.go similarity index 93% rename from pkg/server/lfu_credit_test.go rename to internal/server/lfu_credit_test.go index b5f3e2de..186e78e7 100644 --- a/pkg/server/lfu_credit_test.go +++ b/internal/server/lfu_credit_test.go @@ -9,8 +9,9 @@ import ( "testing" "time" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" ) // lfuCreditService builds a service whose index runs LFU for tenant "t" with a @@ -18,7 +19,7 @@ import ( func lfuCreditService(t *testing.T) *inferenceCacheService { t.Helper() policies := NewPolicyStore() - policies.Replace([]ResolvedPolicy{{Namespace: "t", Eviction: "lfu", LookupTimeoutMs: 5}}) + policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "t", Eviction: "lfu", LookupTimeoutMs: 5}}) idx := index.New( index.WithTTL(time.Hour), index.WithMaxEntries(1), diff --git a/pkg/server/lmcache_offload_integration_test.go b/internal/server/lmcache_offload_integration_test.go similarity index 83% rename from pkg/server/lmcache_offload_integration_test.go rename to internal/server/lmcache_offload_integration_test.go index b85366ed..8af8e4ad 100644 --- a/pkg/server/lmcache_offload_integration_test.go +++ b/internal/server/lmcache_offload_integration_test.go @@ -10,9 +10,9 @@ import ( "testing" "time" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/subscriber" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" ) // L2 offload regression: the cache-stress benchmark returned NO_HINT on every @@ -35,9 +35,9 @@ import ( // hashes are 8-byte big-endian to match the on-the-wire shape the // subscriber's hashToBytes produces from vLLM's integer-hash variant — the // canonical L2 offload shape. -func runEngineReporterAgainstServer(t *testing.T, opts []engine.ReporterOption, batches ...*engine.EventBatch) (client icpb.InferenceCacheClient, stop func()) { +func runEngineReporterAgainstServer(t *testing.T, opts []subscriber.ReporterOption, batches ...*subscriber.EventBatch) (client icpb.InferenceCacheClient, stop func()) { t.Helper() - return runEngineReporterCfgAgainstServer(t, engine.Config{ + return runEngineReporterCfgAgainstServer(t, subscriber.Config{ ReplicaID: "vllm-engine-cs1", ModelID: "vllm-model", TenantID: "ic-smoke", @@ -48,16 +48,16 @@ func runEngineReporterAgainstServer(t *testing.T, opts []engine.ReporterOption, // runEngineReporterCfgAgainstServer is runEngineReporterAgainstServer with an // explicit subscriber Config, so a test can vary the engine-side identity (e.g. // supply Config.AdapterNames to exercise LoRA index partitioning end to end). -func runEngineReporterCfgAgainstServer(t *testing.T, cfg engine.Config, opts []engine.ReporterOption, batches ...*engine.EventBatch) (client icpb.InferenceCacheClient, stop func()) { +func runEngineReporterCfgAgainstServer(t *testing.T, cfg subscriber.Config, opts []subscriber.ReporterOption, batches ...*subscriber.EventBatch) (client icpb.InferenceCacheClient, stop func()) { t.Helper() conn, _, stopServer := startInProcessServerConn(t) client = icpb.NewInferenceCacheClient(conn) // Short flush window so the Run loop drains promptly when the input closes. - opts = append([]engine.ReporterOption{engine.WithWindow(10 * time.Millisecond)}, opts...) - reporter := engine.NewReporter(client, cfg, opts...) + opts = append([]subscriber.ReporterOption{subscriber.WithWindow(10 * time.Millisecond)}, opts...) + reporter := subscriber.NewReporter(client, cfg, opts...) - in := make(chan *engine.EventBatch, len(batches)) + in := make(chan *subscriber.EventBatch, len(batches)) for _, b := range batches { in <- b } @@ -114,8 +114,8 @@ func TestLMCacheOffloadKeepsRoutingHintWithIgnoreBlockRemoved(t *testing.T) { toks := tokenSeq(1000, 128) // one 128-token block // The index keys on our content fingerprint; the gateway queries with the same. our := fingerprint.Bytes(fingerprint.PrefixHashes(toks, 128)[0]) - stored := engine.BlockStored{BlockHashes: [][]byte{h}, TokenIDs: toks, BlockSize: 128} - removed := engine.BlockRemoved{BlockHashes: [][]byte{h}} + stored := subscriber.BlockStored{BlockHashes: [][]byte{h}, TokenIDs: toks, BlockSize: 128} + removed := subscriber.BlockRemoved{BlockHashes: [][]byte{h}} // The T2 re-report anchors the entry's freshness at the eviction event's // timestamp (it's when reload-ability was last confirmed) — production vLLM @@ -124,9 +124,9 @@ func TestLMCacheOffloadKeepsRoutingHintWithIgnoreBlockRemoved(t *testing.T) { // a property of the test clock, not the offload-pinning behavior under test.) now := float64(time.Now().Unix()) client, stop := runEngineReporterAgainstServer(t, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, - &engine.EventBatch{TimestampSeconds: now, Events: []engine.Event{stored}}, - &engine.EventBatch{TimestampSeconds: now, Events: []engine.Event{removed}}, + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, + &subscriber.EventBatch{TimestampSeconds: now, Events: []subscriber.Event{stored}}, + &subscriber.EventBatch{TimestampSeconds: now, Events: []subscriber.Event{removed}}, ) defer stop() @@ -158,12 +158,12 @@ func TestDefaultForwardsBlockRemovedAndIndexLosesHint(t *testing.T) { h := be8(0xC0FFEE0011223344) toks := tokenSeq(2000, 128) our := fingerprint.Bytes(fingerprint.PrefixHashes(toks, 128)[0]) - stored := engine.BlockStored{BlockHashes: [][]byte{h}, TokenIDs: toks, BlockSize: 128} - removed := engine.BlockRemoved{BlockHashes: [][]byte{h}} + stored := subscriber.BlockStored{BlockHashes: [][]byte{h}, TokenIDs: toks, BlockSize: 128} + removed := subscriber.BlockRemoved{BlockHashes: [][]byte{h}} client, stop := runEngineReporterAgainstServer(t, nil, // default reporter - &engine.EventBatch{TimestampSeconds: 0, Events: []engine.Event{stored}}, - &engine.EventBatch{TimestampSeconds: 2.0, Events: []engine.Event{removed}}, + &subscriber.EventBatch{TimestampSeconds: 0, Events: []subscriber.Event{stored}}, + &subscriber.EventBatch{TimestampSeconds: 2.0, Events: []subscriber.Event{removed}}, ) defer stop() @@ -192,11 +192,11 @@ func TestContentHashRoundTripViaReporterAndLookupRoute(t *testing.T) { // floor, so the assertion is about the hash round-trip, not the floor. toks := tokenSeq(3000, 128) our := fingerprint.Bytes(fingerprint.PrefixHashes(toks, 128)[0]) - stored := engine.BlockStored{BlockHashes: [][]byte{h}, TokenIDs: toks, BlockSize: 128} + stored := subscriber.BlockStored{BlockHashes: [][]byte{h}, TokenIDs: toks, BlockSize: 128} client, stop := runEngineReporterAgainstServer(t, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, - &engine.EventBatch{TimestampSeconds: 0, Events: []engine.Event{stored}}, + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, + &subscriber.EventBatch{TimestampSeconds: 0, Events: []subscriber.Event{stored}}, ) defer stop() diff --git a/pkg/server/logging.go b/internal/server/logging.go similarity index 100% rename from pkg/server/logging.go rename to internal/server/logging.go diff --git a/pkg/server/logging_test.go b/internal/server/logging_test.go similarity index 98% rename from pkg/server/logging_test.go rename to internal/server/logging_test.go index e0596dac..2aa99b12 100644 --- a/pkg/server/logging_test.go +++ b/internal/server/logging_test.go @@ -12,7 +12,7 @@ import ( "strings" "testing" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) func TestParseLogFormat(t *testing.T) { diff --git a/pkg/server/matched_tokens_floor_test.go b/internal/server/matched_tokens_floor_test.go similarity index 93% rename from pkg/server/matched_tokens_floor_test.go rename to internal/server/matched_tokens_floor_test.go index 91ffa5f8..4c98d771 100644 --- a/pkg/server/matched_tokens_floor_test.go +++ b/internal/server/matched_tokens_floor_test.go @@ -14,8 +14,9 @@ import ( "github.com/prometheus/client_golang/prometheus/promhttp" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" ) // Server-side matched_tokens floor for LookupRoute. Trivial @@ -41,8 +42,8 @@ import ( // that hasn't installed a CachePolicy — which is the common case. func TestPolicyStoreMinimumMatchedTokensFallsBackToDefaultWhenNoPolicy(t *testing.T) { store := NewPolicyStore() - if got := store.MinimumMatchedTokens("never-configured"); got != DefaultMinimumMatchedTokens { - t.Fatalf("MinimumMatchedTokens(no-policy) = %d, want DefaultMinimumMatchedTokens (%d)", got, DefaultMinimumMatchedTokens) + if got := store.MinimumMatchedTokens("never-configured"); got != controlplaneapi.DefaultMinimumMatchedTokens { + t.Fatalf("MinimumMatchedTokens(no-policy) = %d, want controlplaneapi.DefaultMinimumMatchedTokens (%d)", got, controlplaneapi.DefaultMinimumMatchedTokens) } } @@ -53,7 +54,7 @@ func TestPolicyStoreMinimumMatchedTokensFallsBackToDefaultWhenNoPolicy(t *testin // to >=64 and remove the disable-the-floor primitive. func TestPolicyStoreMinimumMatchedTokensRespectsPolicyValue(t *testing.T) { store := NewPolicyStore() - store.Replace([]ResolvedPolicy{ + store.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "ns-strict", MinimumMatchedTokens: 256}, {Namespace: "ns-disabled", MinimumMatchedTokens: 0}, }) @@ -73,7 +74,7 @@ func TestPolicyStoreMinimumMatchedTokensRespectsPolicyValue(t *testing.T) { // instead of clamping to the safest interpretation. func TestPolicyStoreMinimumMatchedTokensClampsNegativeToZero(t *testing.T) { store := NewPolicyStore() - store.Replace([]ResolvedPolicy{{Namespace: "ns-bad", MinimumMatchedTokens: -1}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "ns-bad", MinimumMatchedTokens: -1}}) if got := store.MinimumMatchedTokens("ns-bad"); got != 0 { t.Fatalf("negative floor = %d, want 0 (clamped)", got) } @@ -93,10 +94,10 @@ func TestLookupRouteAppliesDefaultMatchedTokensFloorWhenNoPolicy(t *testing.T) { // MinimumMatchedTokens (64) explicitly because Replace overwrites // the no-policy fallback the resolver would otherwise pick. fal := false - svc.policies.Replace([]ResolvedPolicy{{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{ Namespace: "no-policy-tenant", AffinityRouting: &fal, - MinimumMatchedTokens: DefaultMinimumMatchedTokens, + MinimumMatchedTokens: controlplaneapi.DefaultMinimumMatchedTokens, }}) svc.index.Ingest(index.Update{ ReplicaID: "r", Model: "m", Tenant: "no-policy-tenant", HashScheme: "vllm", @@ -112,7 +113,7 @@ func TestLookupRouteAppliesDefaultMatchedTokensFloorWhenNoPolicy(t *testing.T) { } if resp.GetReasonCode() != "NO_HINT" { t.Fatalf("reason = %q, want NO_HINT — 16-token match below default floor (%d) should not surface as PREFIX_MATCH", - resp.GetReasonCode(), DefaultMinimumMatchedTokens) + resp.GetReasonCode(), controlplaneapi.DefaultMinimumMatchedTokens) } if len(resp.GetReplicaScores()) != 0 { t.Fatalf("sub-floor match must downgrade to NO_HINT with empty scores, got %+v", resp.GetReplicaScores()) @@ -128,7 +129,7 @@ func TestLookupRouteKeepsPrefixMatchAtDefaultFloorWhenNoPolicy(t *testing.T) { svc := newTestService() svc.index.Ingest(index.Update{ ReplicaID: "r", Model: "m", Tenant: "no-policy-tenant", HashScheme: "vllm", - Prefixes: []index.PrefixRef{{PrefixHash: []byte("p"), TokenCount: DefaultMinimumMatchedTokens}}, + Prefixes: []index.PrefixRef{{PrefixHash: []byte("p"), TokenCount: controlplaneapi.DefaultMinimumMatchedTokens}}, }) resp, err := svc.LookupRoute(context.Background(), &icpb.LookupRouteRequest{ @@ -141,8 +142,8 @@ func TestLookupRouteKeepsPrefixMatchAtDefaultFloorWhenNoPolicy(t *testing.T) { if resp.GetReasonCode() != "PREFIX_MATCH" { t.Fatalf("reason = %q, want PREFIX_MATCH — match exactly at the floor should pass (>=, not >)", resp.GetReasonCode()) } - if got := resp.GetReplicaScores()[0].GetMatchedTokens(); got != DefaultMinimumMatchedTokens { - t.Fatalf("matched_tokens = %d, want %d (the boundary value)", got, DefaultMinimumMatchedTokens) + if got := resp.GetReplicaScores()[0].GetMatchedTokens(); got != controlplaneapi.DefaultMinimumMatchedTokens { + t.Fatalf("matched_tokens = %d, want %d (the boundary value)", got, controlplaneapi.DefaultMinimumMatchedTokens) } } @@ -156,7 +157,7 @@ func TestLookupRoutePolicyMatchedTokensFloorOverridesDefault(t *testing.T) { fal := false // Disable affinity to keep the floor downgrade as NO_HINT (see // the sibling test for rationale). - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-strict", MinimumMatchedTokens: 256, AffinityRouting: &fal}, }) svc.index.Ingest(index.Update{ @@ -185,7 +186,7 @@ func TestLookupRoutePolicyMatchedTokensFloorOverridesDefault(t *testing.T) { // ranker's raw recall. func TestLookupRoutePolicyMatchedTokensFloorZeroDisablesEnforcement(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "raw-recall", MinimumMatchedTokens: 0}, }) svc.index.Ingest(index.Update{ @@ -215,7 +216,7 @@ func TestLookupRoutePolicyMatchedTokensFloorZeroDisablesEnforcement(t *testing.T // for every well-warmed peer in the same response. func TestLookupRouteMatchedTokensFloorFiltersBelowFloorReplicasKeepsTheRest(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "mixed-warmth", MinimumMatchedTokens: 64}, }) // Both replicas hold the same chain head; A holds the leading 4 blocks @@ -270,10 +271,10 @@ func TestLookupRouteSubFloorMatchEmitsNoHintMetric(t *testing.T) { // MinimumMatchedTokens explicitly (Replace overrides the // no-policy fallback the resolver would otherwise pick). fal := false - svc.policies.Replace([]ResolvedPolicy{{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{ Namespace: "no-policy-tenant", AffinityRouting: &fal, - MinimumMatchedTokens: DefaultMinimumMatchedTokens, + MinimumMatchedTokens: controlplaneapi.DefaultMinimumMatchedTokens, }}) svc.index.Ingest(index.Update{ ReplicaID: "r", Model: "m", Tenant: "no-policy-tenant", HashScheme: "vllm", @@ -343,7 +344,7 @@ func TestLookupRouteSubFloorMatchEmitsNoHintMetric(t *testing.T) { // rB-keyed entry under b1, so b1 itself stays via rA). func TestLookupRouteFloorPrunesLFUHitsForFilteredReplicas(t *testing.T) { policies := NewPolicyStore() - policies.Replace([]ResolvedPolicy{ + policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "t", Eviction: "lfu", MinimumMatchedTokens: 64}, }) idx := index.New( diff --git a/pkg/server/metrics.go b/internal/server/metrics.go similarity index 99% rename from pkg/server/metrics.go rename to internal/server/metrics.go index 076c0878..f23207ae 100644 --- a/pkg/server/metrics.go +++ b/internal/server/metrics.go @@ -11,7 +11,7 @@ import ( "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/collectors" - "github.com/cachebox-project/inference-cache/pkg/server/auth" + "github.com/cachebox-project/inference-cache/internal/server/auth" ) // metricNamespace is the prefix for this project's own metrics (tech spec diff --git a/pkg/server/metrics_test.go b/internal/server/metrics_test.go similarity index 100% rename from pkg/server/metrics_test.go rename to internal/server/metrics_test.go diff --git a/pkg/server/policy.go b/internal/server/policy.go similarity index 63% rename from pkg/server/policy.go rename to internal/server/policy.go index 3a757c83..ad6573f1 100644 --- a/pkg/server/policy.go +++ b/internal/server/policy.go @@ -14,161 +14,6 @@ import ( "github.com/cachebox-project/inference-cache/internal/controlplaneapi" ) -// PolicyPropagationVersion identifies the schema of the /policy snapshot the -// server accepts. Bumped on a schema change so version skew is observable -// (the controller writes the same constant on each push). -// -// v2 added the Tenants slice (CacheTenant quota propagation). v3 added -// ResolvedPolicy.Eviction (per-namespace cap-eviction algorithm). v4 added -// ResolvedPolicy.MinimumMatchedTokens (the result-side matched-tokens floor). -// v5 added ResolvedPolicy.RoutingFloorScore (the per-namespace post-score -// floor for the distinguishing-power-aware LookupRoute ranker). v6 added -// ResolvedPolicy.Strategy (per-namespace LookupRoute strategy gates). v7 added -// ResolvedPolicy.AffinityRouting (the per-namespace toggle for consistent- -// hash fallback routing on the NO_HINT path). -// -// Rollout asymmetry — the bump is "additive when the new field can be -// defaulted; rejected when it can't": -// -// - **Newer server / older body.** A v7 server accepts a v3, v4, v5, or v6 body and -// normalizes each policy's missing MinimumMatchedTokens to -// DefaultMinimumMatchedTokens, missing RoutingFloorScore to -// DefaultRoutingFloorScore, missing Strategy to the historical gates -// (chain matching on, chain not required, tenant-hot on), and missing -// AffinityRouting to DefaultAffinityRoutingEnabled — so a -// server-first rollout does NOT drop existing CachePolicy state (TTL, -// timeouts, eviction, prefix gates, quotas). The normalized result is -// identical to a no-CachePolicy fallback for the new fields while every -// prior knob stays enforced. The lenience window is bounded by -// PolicyMinimumAcceptedVersion (the oldest body this server still -// understands); bodies older than that are still rejected as "unsupported". -// - **Older server / newer body.** The reverse — a v6 server receiving a -// v7 push — still hard-fails. Because the handler decodes the body -// before checking version, DisallowUnknownFields is the FIRST line of -// defense: the new affinityRouting field on each policy is unknown to the v6 -// Go struct, so decode rejects the body with -// `decode policy snapshot: json: unknown field "affinityRouting"`. -// Even on a hypothetical breaking change where the field rename or -// removal slips past DisallowUnknownFields, the explicit version-band -// check below catches it with `unsupported policy snapshot version`. -// Both diagnostics are fail-loud; the operator sees one specific message, -// not silent state loss. -const PolicyPropagationVersion = controlplaneapi.PolicyPropagationVersion - -// PolicyMinimumAcceptedVersion is the oldest /policy schema this server -// understands. Bodies below this version are rejected outright; bodies at -// or above are accepted with the new-field normalization described on -// PolicyPropagationVersion. Bump this in lockstep with PolicyPropagationVersion -// whenever a schema change is NOT additive-defaultable — anything load-bearing -// for a tenant, anything whose missing value cannot be safely synthesized. -const PolicyMinimumAcceptedVersion = controlplaneapi.PolicyMinimumAcceptedVersion - -// DefaultMinimumMatchedTokens is the server-side fallback floor on -// MATCHED prefix tokens applied when a tenant has no CachePolicy at all. -// Mirrors the +kubebuilder:default on CachePolicySpec.MinimumMatchedTokens -// so the "no policy" and "policy with default value" paths both behave -// identically — see PolicyStore.MinimumMatchedTokens. 64 ≈ 4 KV blocks at -// the typical 16-token block size: substantially above the chat-template -// framing tokens identical across every replica, well below any -// useful real-prompt overlap. -const DefaultMinimumMatchedTokens = controlplaneapi.DefaultMinimumMatchedTokens - -// DefaultRoutingFloorScore is the server-wide fallback the LookupRoute -// handler applies to PREFIX_MATCH responses when no CachePolicy is installed -// for the requesting tenant. Calibrated as a near-zero floor: the -// distinguishing-power factor collapses to 0 for the trivial-overlap shape -// (every replica holds the prefix), producing score=0; the floor also -// catches the next slice of near-zero scores — heavy diffusion combined -// with low matched_tokens (small partial overlaps), high pressure -// (pressure_factor near 0), or near-expired freshness — which the gateway -// gains little from routing on. Any substantive routing decision (a -// uniquely-held prefix of any meaningful token count and freshness) sees a -// score well above 0.1. Without this default the trivial-match-as- -// PREFIX_MATCH bug would persist for every namespace that has not -// installed a policy CR. Tunable per-namespace via -// CachePolicy.spec.routingFloorScore. -const DefaultRoutingFloorScore = controlplaneapi.DefaultRoutingFloorScore - -const ( - DefaultEnableChainMatching = controlplaneapi.DefaultEnableChainMatching - DefaultRequireChain = controlplaneapi.DefaultRequireChain - DefaultEnableTenantHot = controlplaneapi.DefaultEnableTenantHot -) - -// DefaultAffinityRoutingEnabled is the server-wide fallback applied when a -// tenant has no CachePolicy, or has one whose affinityRouting field was -// omitted. Mirrors the +kubebuilder:default=Enabled marker on -// CachePolicySpec.AffinityRouting so the "no policy" and "policy with -// default value" paths both behave identically. -// -// Diffuse single-turn workloads (chatbot, RAG with distinct corpus -// chunks per query) are common and the consistent-hash fallback is -// near-free, so default ON is the correct safety posture. Operators -// can disable explicitly via affinityRouting: Disabled when they want -// pure round-robin (raw-recall benchmarking, gateway debugging). -const DefaultAffinityRoutingEnabled = controlplaneapi.DefaultAffinityRoutingEnabled - -// ResolvedPolicy is the slice of CachePolicy the server actually enforces: -// only the fields the policy server needs at lookup/sweep time. The CRD -// types live in api/v1alpha1; the controller flattens them into this shape -// before pushing so pkg/server has no dependency on the CRD package. -// -// Zero values mean "unset / use server default" for most fields: -// - EvictionTTL <= 0 → fall back to index.DefaultTTL (via the global -// WithTTL the binary configured). -// - MinimumPrefixTokens <= 0 → no threshold (every prefix-hash hit returns). -// - MinimumMatchedTokens <= 0 → floor disabled for THIS namespace (every -// matched_tokens count, even 1-block trivial overlap, is reported as -// PREFIX_MATCH). A negative pointer round-trips as 0, which is the -// intentional opt-out. A tenant with no ResolvedPolicy at all instead -// falls back to DefaultMinimumMatchedTokens (the server-side default -// floor) via PolicyStore.MinimumMatchedTokens. -// - LookupTimeoutMs <= 0 → no deadline (lookup runs to completion). -// - Eviction == "" → LRU (the index default and the kubebuilder default). -// -// RoutingFloorScore is the EXCEPTION and uses a pointer to distinguish three -// distinct shapes on the wire: -// - nil → field was OMITTED on the wire body (legacy / hand-crafted / -// un-defaulted CR). Server applies DefaultRoutingFloorScore for safety. -// - &0 → operator EXPLICITLY set "0" (the opt-out — raw-recall -// benchmarking, ranker debugging). Server applies no floor for this -// namespace. -// - &x → operator set a specific threshold. Server applies x as-is. -// -// A flat float32 field with omitempty would conflate nil and &0 (both -// produce no field on the wire), so a controller pushing a CR whose -// kubebuilder-defaulted value was overridden to "0" by the operator would -// be indistinguishable from a hand-crafted body that simply omitted the -// field — the safe interpretation differs between those two cases. -type ResolvedPolicy = controlplaneapi.ResolvedPolicy - -// ResolvedLookupStrategy carries the server-enforced LookupRoute strategy -// gates flattened from CachePolicy.spec.strategy. -type ResolvedLookupStrategy = controlplaneapi.ResolvedLookupStrategy - -// ResolvedTenant is the slice of a CacheTenant the server enforces at ingest -// time: the tenant's external identity plus its index-entry budget. The CRD -// types live in api/v1alpha1; the controller flattens them into this shape so -// pkg/server has no dependency on the CRD package (mirrors ResolvedPolicy). -// -// Identity note: TenantID is the CacheTenant's spec.tenantID — the same value -// a CacheStateUpdate carries in tenant_id — NOT the CR's metadata.name. That is -// the join key the index matches an ingest against. -// -// There is deliberately no memory budget: the engine KV cache is a shared, -// tenant-unaware pool, so the control plane can neither enforce nor honestly -// attribute bytes per tenant. Only the index entry table — which the server -// owns — is enforceable. -type ResolvedTenant = controlplaneapi.ResolvedTenant - -// PolicySnapshot is the full set of CachePolicies + CacheTenants the controller -// pushes on each reconcile. Pushed via POST to /policy (PUT is accepted too for -// callers that prefer it). Replace-on-write: the controller is the source of -// truth, so the server discards its prior state and adopts the new snapshot. A -// CachePolicy/CacheTenant that disappears between snapshots reverts that -// namespace/tenant to the server default (no policy / no quota). -type PolicySnapshot = controlplaneapi.PolicySnapshot - // PolicyStore is the server-side cache of resolved policies (indexed by // namespace) and resolved tenant quotas (indexed by tenant ID). Reads take // the read lock; pushes from /policy (POST or PUT) take the write lock and @@ -181,8 +26,8 @@ type PolicySnapshot = controlplaneapi.PolicySnapshot // so they live in separate maps under the same lock. type PolicyStore struct { mu sync.RWMutex - policies map[string]ResolvedPolicy - tenants map[string]ResolvedTenant + policies map[string]controlplaneapi.ResolvedPolicy + tenants map[string]controlplaneapi.ResolvedTenant } // NewPolicyStore returns an empty store. Until the controller pushes a @@ -190,8 +35,8 @@ type PolicyStore struct { // and every TenantQuota reports "no quota" (= unbounded, fail open). func NewPolicyStore() *PolicyStore { return &PolicyStore{ - policies: make(map[string]ResolvedPolicy), - tenants: make(map[string]ResolvedTenant), + policies: make(map[string]controlplaneapi.ResolvedPolicy), + tenants: make(map[string]controlplaneapi.ResolvedTenant), } } @@ -200,7 +45,7 @@ func NewPolicyStore() *PolicyStore { // ReplaceSnapshot(policies, nil). Retained as a convenience for callers that // don't exercise the tenant-quota axis (mostly tests); it delegates so it can // never leave a stale tenant table behind. Idempotent. -func (s *PolicyStore) Replace(policies []ResolvedPolicy) { +func (s *PolicyStore) Replace(policies []controlplaneapi.ResolvedPolicy) { s.ReplaceSnapshot(policies, nil) } @@ -210,15 +55,15 @@ func (s *PolicyStore) Replace(policies []ResolvedPolicy) { // uses; the policies-only Replace delegates here with nil tenants. // Replace-on-write: a tenant absent from the new snapshot reverts to "no quota" // (unbounded, fail open). -func (s *PolicyStore) ReplaceSnapshot(policies []ResolvedPolicy, tenants []ResolvedTenant) { - nextPolicies := make(map[string]ResolvedPolicy, len(policies)) +func (s *PolicyStore) ReplaceSnapshot(policies []controlplaneapi.ResolvedPolicy, tenants []controlplaneapi.ResolvedTenant) { + nextPolicies := make(map[string]controlplaneapi.ResolvedPolicy, len(policies)) for _, p := range policies { if p.Namespace == "" { continue // see Replace: an unkeyed entry can't be routed. } nextPolicies[p.Namespace] = p } - nextTenants := make(map[string]ResolvedTenant, len(tenants)) + nextTenants := make(map[string]controlplaneapi.ResolvedTenant, len(tenants)) for _, t := range tenants { if t.TenantID == "" { // Defensive: a quota with no tenant ID can't be matched against any @@ -246,7 +91,7 @@ func (s *PolicyStore) ReplaceSnapshot(policies []ResolvedPolicy, tenants []Resol // Lookup returns the resolved policy for a namespace and whether one was // configured (false → caller should use server defaults). -func (s *PolicyStore) Lookup(namespace string) (ResolvedPolicy, bool) { +func (s *PolicyStore) Lookup(namespace string) (controlplaneapi.ResolvedPolicy, bool) { s.mu.RLock() defer s.mu.RUnlock() p, ok := s.policies[namespace] @@ -255,9 +100,9 @@ func (s *PolicyStore) Lookup(namespace string) (ResolvedPolicy, bool) { // Snapshot returns a copy of the current policies, sorted by namespace for // deterministic test output and /policy GET (if added later). -func (s *PolicyStore) Snapshot() []ResolvedPolicy { +func (s *PolicyStore) Snapshot() []controlplaneapi.ResolvedPolicy { s.mu.RLock() - out := make([]ResolvedPolicy, 0, len(s.policies)) + out := make([]controlplaneapi.ResolvedPolicy, 0, len(s.policies)) for _, p := range s.policies { out = append(out, p) } @@ -312,7 +157,7 @@ func (s *PolicyStore) MinimumMatchedTokens(tenant string) int32 { } return p.MinimumMatchedTokens } - return DefaultMinimumMatchedTokens + return controlplaneapi.DefaultMinimumMatchedTokens } // RoutingFloorScore returns the per-namespace post-score floor applied to @@ -352,12 +197,12 @@ func (s *PolicyStore) MinimumMatchedTokens(tenant string) int32 { func (s *PolicyStore) RoutingFloorScore(tenant string) float32 { p, ok := s.Lookup(tenant) if !ok { - return DefaultRoutingFloorScore + return controlplaneapi.DefaultRoutingFloorScore } if p.RoutingFloorScore == nil { // Policy is installed but did not carry this field (legacy / hand- // crafted body). Apply the safety floor, not the opt-out. - return DefaultRoutingFloorScore + return controlplaneapi.DefaultRoutingFloorScore } if *p.RoutingFloorScore < 0 { return 0 @@ -384,10 +229,10 @@ func (s *PolicyStore) RoutingFloorScore(tenant string) float32 { func (s *PolicyStore) AffinityRoutingEnabled(namespace string) bool { p, ok := s.Lookup(namespace) if !ok { - return DefaultAffinityRoutingEnabled + return controlplaneapi.DefaultAffinityRoutingEnabled } if p.AffinityRouting == nil { - return DefaultAffinityRoutingEnabled + return controlplaneapi.DefaultAffinityRoutingEnabled } return *p.AffinityRouting } @@ -407,7 +252,7 @@ func (s *PolicyStore) LookupTimeout(tenant string) time.Duration { func (s *PolicyStore) ChainMatchingEnabled(tenant string) bool { p, ok := s.Lookup(tenant) if !ok || p.Strategy == nil || p.Strategy.EnableChainMatching == nil { - return DefaultEnableChainMatching + return controlplaneapi.DefaultEnableChainMatching } return *p.Strategy.EnableChainMatching } @@ -418,7 +263,7 @@ func (s *PolicyStore) ChainMatchingEnabled(tenant string) bool { func (s *PolicyStore) ChainRequired(tenant string) bool { p, ok := s.Lookup(tenant) if !ok || p.Strategy == nil || p.Strategy.RequireChain == nil { - return DefaultRequireChain + return controlplaneapi.DefaultRequireChain } return *p.Strategy.RequireChain } @@ -428,7 +273,7 @@ func (s *PolicyStore) ChainRequired(tenant string) bool { func (s *PolicyStore) TenantHotEnabled(tenant string) bool { p, ok := s.Lookup(tenant) if !ok || p.Strategy == nil || p.Strategy.EnableTenantHot == nil { - return DefaultEnableTenantHot + return controlplaneapi.DefaultEnableTenantHot } return *p.Strategy.EnableTenantHot } @@ -447,7 +292,7 @@ func (s *PolicyStore) TenantHotEnabled(tenant string) bool { // The probe is server-internal state under a server-controlled tenant id; no // operator-supplied CacheTenant should govern it. func (s *PolicyStore) TenantQuota(tenant string) (maxEntries int64, ok bool) { - if tenant == ProbeTenantID { + if tenant == controlplaneapi.ProbeTenantID { return 0, false } s.mu.RLock() @@ -461,9 +306,9 @@ func (s *PolicyStore) TenantQuota(tenant string) (maxEntries int64, ok bool) { // TenantQuotas returns a copy of the current tenant quotas, sorted by tenant ID // for deterministic test output. -func (s *PolicyStore) TenantQuotas() []ResolvedTenant { +func (s *PolicyStore) TenantQuotas() []controlplaneapi.ResolvedTenant { s.mu.RLock() - out := make([]ResolvedTenant, 0, len(s.tenants)) + out := make([]controlplaneapi.ResolvedTenant, 0, len(s.tenants)) for _, t := range s.tenants { out = append(out, t) } @@ -504,7 +349,7 @@ func policyHandler(store *PolicyStore) http.HandlerFunc { defer func() { _ = body.Close() }() dec := json.NewDecoder(body) dec.DisallowUnknownFields() - var snap PolicySnapshot + var snap controlplaneapi.PolicySnapshot if err := dec.Decode(&snap); err != nil { http.Error(w, "decode policy snapshot: "+err.Error()+"\n", http.StatusBadRequest) return @@ -517,7 +362,7 @@ func policyHandler(store *PolicyStore) http.HandlerFunc { // which surfaces as a `decode policy snapshot: json: unknown field "..."` // — also fail-loud, just attributed to the decoder rather than this // branch). Both outcomes give the operator a specific diagnostic. - if snap.Version < PolicyMinimumAcceptedVersion || snap.Version > PolicyPropagationVersion { + if snap.Version < controlplaneapi.PolicyMinimumAcceptedVersion || snap.Version > controlplaneapi.PolicyPropagationVersion { http.Error(w, "unsupported policy snapshot version\n", http.StatusBadRequest) return } @@ -568,14 +413,14 @@ func policyHandler(store *PolicyStore) http.HandlerFunc { // Bodies already at PolicyPropagationVersion are returned untouched so an // operator's explicit opt-out (e.g. `routingFloorScore: 0` for raw-recall // benchmarking, or `enableTenantHot: false`, or `affinityRouting: false`) reaches the store as written. -func normalizePolicySnapshotForVersion(snap *PolicySnapshot) { - if snap.Version >= PolicyPropagationVersion { +func normalizePolicySnapshotForVersion(snap *controlplaneapi.PolicySnapshot) { + if snap.Version >= controlplaneapi.PolicyPropagationVersion { return } if snap.Version < 4 { for i := range snap.Policies { if snap.Policies[i].MinimumMatchedTokens == 0 { - snap.Policies[i].MinimumMatchedTokens = DefaultMinimumMatchedTokens + snap.Policies[i].MinimumMatchedTokens = controlplaneapi.DefaultMinimumMatchedTokens } } } @@ -588,7 +433,7 @@ func normalizePolicySnapshotForVersion(snap *PolicySnapshot) { // branch only fires for the missing-field case. for i := range snap.Policies { if snap.Policies[i].RoutingFloorScore == nil { - v := DefaultRoutingFloorScore + v := controlplaneapi.DefaultRoutingFloorScore snap.Policies[i].RoutingFloorScore = &v } } @@ -605,7 +450,7 @@ func normalizePolicySnapshotForVersion(snap *PolicySnapshot) { // An operator's explicit `affinityRouting: Disabled` is already a // non-nil &false and reaches the store as written; the nil branch // only fires for the missing-field case. - def := DefaultAffinityRoutingEnabled + def := controlplaneapi.DefaultAffinityRoutingEnabled for i := range snap.Policies { if snap.Policies[i].AffinityRouting == nil { v := def @@ -615,20 +460,20 @@ func normalizePolicySnapshotForVersion(snap *PolicySnapshot) { } } -func applyResolvedLookupStrategyDefaults(p *ResolvedPolicy) { +func applyResolvedLookupStrategyDefaults(p *controlplaneapi.ResolvedPolicy) { if p.Strategy == nil { - p.Strategy = &ResolvedLookupStrategy{} + p.Strategy = &controlplaneapi.ResolvedLookupStrategy{} } if p.Strategy.EnableChainMatching == nil { - v := DefaultEnableChainMatching + v := controlplaneapi.DefaultEnableChainMatching p.Strategy.EnableChainMatching = &v } if p.Strategy.RequireChain == nil { - v := DefaultRequireChain + v := controlplaneapi.DefaultRequireChain p.Strategy.RequireChain = &v } if p.Strategy.EnableTenantHot == nil { - v := DefaultEnableTenantHot + v := controlplaneapi.DefaultEnableTenantHot p.Strategy.EnableTenantHot = &v } } diff --git a/pkg/server/policy_test.go b/internal/server/policy_test.go similarity index 89% rename from pkg/server/policy_test.go rename to internal/server/policy_test.go index 1217e0da..ec6f1c22 100644 --- a/pkg/server/policy_test.go +++ b/internal/server/policy_test.go @@ -15,6 +15,8 @@ import ( "sync/atomic" "testing" "time" + + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" ) func TestPolicyStoreLookupReturnsDefaultsWhenUnset(t *testing.T) { @@ -35,7 +37,7 @@ func TestPolicyStoreLookupReturnsDefaultsWhenUnset(t *testing.T) { func TestPolicyStoreReplaceIsAtomicAndDropsStale(t *testing.T) { s := NewPolicyStore() - s.Replace([]ResolvedPolicy{ + s.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", EvictionTTL: 15 * time.Minute, MinimumPrefixTokens: 32, LookupTimeoutMs: 25}, {Namespace: "team-b", EvictionTTL: time.Hour}, }) @@ -51,7 +53,7 @@ func TestPolicyStoreReplaceIsAtomicAndDropsStale(t *testing.T) { } // Replace with a snapshot that omits team-b — that namespace must revert. - s.Replace([]ResolvedPolicy{ + s.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", EvictionTTL: 5 * time.Minute}, }) if _, ok := s.Lookup("team-b"); ok { @@ -64,7 +66,7 @@ func TestPolicyStoreReplaceIsAtomicAndDropsStale(t *testing.T) { func TestPolicyStoreReplaceDropsEmptyNamespace(t *testing.T) { s := NewPolicyStore() - s.Replace([]ResolvedPolicy{ + s.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "", EvictionTTL: time.Hour}, // bogus — must be dropped {Namespace: "ok", EvictionTTL: time.Minute}, }) @@ -81,7 +83,7 @@ func TestPolicyStoreReplaceDropsEmptyNamespace(t *testing.T) { // reads never observe a partial state catches missing locks. func TestPolicyStoreConcurrentReadsWithWriter(t *testing.T) { s := NewPolicyStore() - s.Replace([]ResolvedPolicy{{Namespace: "t", EvictionTTL: time.Hour}}) + s.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "t", EvictionTTL: time.Hour}}) stop := make(chan struct{}) var wg sync.WaitGroup @@ -110,7 +112,7 @@ func TestPolicyStoreConcurrentReadsWithWriter(t *testing.T) { } for i := 0; i < 500; i++ { - s.Replace([]ResolvedPolicy{ + s.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "t", EvictionTTL: time.Duration(i+1) * time.Minute}, }) } @@ -126,9 +128,9 @@ func TestPolicyHandlerReplacesSnapshot(t *testing.T) { srv := httptest.NewServer(policyHandler(s)) defer srv.Close() - body, _ := json.Marshal(PolicySnapshot{ - Version: PolicyPropagationVersion, - Policies: []ResolvedPolicy{ + body, _ := json.Marshal(controlplaneapi.PolicySnapshot{ + Version: controlplaneapi.PolicyPropagationVersion, + Policies: []controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", EvictionTTL: 7 * time.Minute, MinimumPrefixTokens: 16}, }, }) @@ -155,9 +157,9 @@ func TestPolicySnapshotRoundTripCarriesEviction(t *testing.T) { srv := httptest.NewServer(policyHandler(s)) defer srv.Close() - body, _ := json.Marshal(PolicySnapshot{ - Version: PolicyPropagationVersion, - Policies: []ResolvedPolicy{ + body, _ := json.Marshal(controlplaneapi.PolicySnapshot{ + Version: controlplaneapi.PolicyPropagationVersion, + Policies: []controlplaneapi.ResolvedPolicy{ {Namespace: "team-lfu", Eviction: "lfu"}, {Namespace: "team-lru", Eviction: "lru"}, {Namespace: "team-default"}, // no eviction set @@ -195,7 +197,7 @@ func TestPolicySnapshotRoundTripCarriesEviction(t *testing.T) { func TestPolicyHandlerRejectsBadVersion(t *testing.T) { srv := httptest.NewServer(policyHandler(NewPolicyStore())) defer srv.Close() - body, _ := json.Marshal(PolicySnapshot{Version: 99, Policies: []ResolvedPolicy{}}) + body, _ := json.Marshal(controlplaneapi.PolicySnapshot{Version: 99, Policies: []controlplaneapi.ResolvedPolicy{}}) resp, err := http.Post(srv.URL, "application/json", bytes.NewReader(body)) if err != nil { t.Fatalf("POST: %v", err) @@ -237,13 +239,13 @@ func TestPolicyHandlerCapsBodySize(t *testing.T) { srv := httptest.NewServer(policyHandler(NewPolicyStore())) defer srv.Close() // Build a snapshot that comfortably exceeds the 1 MiB cap. - policies := make([]ResolvedPolicy, 0, 20000) + policies := make([]controlplaneapi.ResolvedPolicy, 0, 20000) for i := 0; i < 20000; i++ { - policies = append(policies, ResolvedPolicy{ + policies = append(policies, controlplaneapi.ResolvedPolicy{ Namespace: fmt.Sprintf("ns-%d-padded-with-bytes-to-exceed-cap", i), }) } - body, _ := json.Marshal(PolicySnapshot{Version: PolicyPropagationVersion, Policies: policies}) + body, _ := json.Marshal(controlplaneapi.PolicySnapshot{Version: controlplaneapi.PolicyPropagationVersion, Policies: policies}) resp, err := http.Post(srv.URL, "application/json", bytes.NewReader(body)) if err != nil { t.Fatalf("POST: %v", err) @@ -263,7 +265,7 @@ func TestPolicyStoreAffinityRoutingEnabled(t *testing.T) { } tru, fal := true, false - store.Replace([]ResolvedPolicy{ + store.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "ns-nil"}, {Namespace: "ns-true", AffinityRouting: &tru}, {Namespace: "ns-false", AffinityRouting: &fal}, diff --git a/pkg/server/probe.go b/internal/server/probe.go similarity index 78% rename from pkg/server/probe.go rename to internal/server/probe.go index 4dd9c77a..7d515f4f 100644 --- a/pkg/server/probe.go +++ b/internal/server/probe.go @@ -17,7 +17,7 @@ import ( "time" "github.com/cachebox-project/inference-cache/internal/controlplaneapi" - "github.com/cachebox-project/inference-cache/pkg/index" + "github.com/cachebox-project/inference-cache/internal/index" ) // Functional-probe machinery. @@ -75,111 +75,6 @@ import ( // this file's tests are explicitly carved out (per the project's "no inert // field" rule — wired today, or names the follow-up that will wire it). -// ProbeTenantID is the reserved tenant id every probe synthesizes its state -// under. Real workload tenants (CacheTenant.spec.tenantID) are MinLength=1 and -// arbitrary; the slash-delimited `inferencecache.io/probe` form is in the -// project-canonical `inferencecache.io/...` namespace so a real tenant cannot -// accidentally collide. The /probe HTTP handler does not accept a caller- -// supplied tenant — the reservation is enforced server-side, never trusted -// from the request — so a real workload cannot read or write probe entries by -// spoofing the tenant id. -const ProbeTenantID = controlplaneapi.ProbeTenantID - -// ProbeReplicaPrefix is the literal prefix every probe replica id starts with. -// Real subscribers set replica_id = pod-name; Kubernetes pod names are RFC -// 1123 subdomain labels that disallow underscores entirely, so the reserved -// `__probe-` prefix (with its leading underscores) is collision-free with any -// legitimate replica id. Cleanup keys on this prefix (and the probe tenant + -// backend-derived suffix) to wipe ONLY the probe's own state on each Run, -// never a real replica's entries. -const ProbeReplicaPrefix = controlplaneapi.ProbeReplicaPrefix - -// ProbeTokenCount is the per-block token count carried by the synthesized -// BlockStored. 16 is the smallest unit a vLLM-class engine reports -// (one KV block) — small enough that the probe payload stays trivial in the -// index even if cleanup is somehow skipped, but non-zero so the LookupRoute -// ranker treats it as a real prefix hit. -const ProbeTokenCount = controlplaneapi.ProbeTokenCount - -// ProbeStageResult is the per-stage outcome encoded in the JSON ProbeResult -// the controller reads. Strings (not enums) so the wire stays stable when -// new outcomes appear — same forward-compat reasoning as gRPC reason_code. -type ProbeStageResult = controlplaneapi.ProbeStageResult - -// Possible per-stage outcomes. "skipped" applies to a stage the probe chose -// not to run (T2 on a non-LMCache backend, or any downstream stage when an -// upstream stage failed — running them would surface a cascade of false -// failures that masks the real one). -const ( - ProbeStageOK = controlplaneapi.ProbeStageOK - ProbeStageFailed = controlplaneapi.ProbeStageFailed - ProbeStageSkipped = controlplaneapi.ProbeStageSkipped -) - -// Stage names appear verbatim in ProbeStageError.Stage and (Stage 2) in the -// inferencecache_backend_probe_result_total metric `stage` label. Stage A's wire -// name is `ingest` — it exercises in-process index.Ingest, not the wire -// ReportCacheState handler the real subscriber uses (the handler drops -// probe-tenant messages by design). See the file-top doc for what a Stage -// A pass/fail does and does not prove. -const ( - ProbeStageIngest = controlplaneapi.ProbeStageIngest - ProbeStageRouting = controlplaneapi.ProbeStageRouting - ProbeStageT2 = controlplaneapi.ProbeStageT2 -) - -// BackendTypeLMCache is the spec.type value that gates Stage C. The probe -// only runs the T2 put/get against LMCache-backed CacheBackends — Memory -// and External backends carry no tier-2 client to drive. The string MUST -// agree with the CacheBackend.spec.type enum value in api/v1alpha1; using -// the literal here (rather than importing the CRD types) keeps pkg/server -// dependency-free of the CRD package, matching the policy/tenant pattern in -// pkg/server/policy.go. -// -// An empty BackendType on a ProbeRequest is treated as LMCache to match the -// CacheBackend CRD's defaulter (spec.type defaults to LMCache via the -// kubebuilder marker). Operators who run the probe by hand against a non- -// LMCache backend must set BackendType explicitly; the CacheBackend -// reconciler always reads spec.type from the CR and never sends empty. -const BackendTypeLMCache = controlplaneapi.BackendTypeLMCache - -// ProbeRequest carries the parameters the probe needs to synthesize a -// deterministic round-trip. The tenant_id is NOT a request field — it is -// always ProbeTenantID, fixed server-side. -// -// Backend uniquely identifies which CacheBackend the probe is running -// against AND is interpolated into the reserved replica id -// (__probe-) and the deterministic probe hash. To prevent -// same-name CacheBackends in different namespaces from colliding in the -// reserved replica id, callers MUST pass a globally-unique form — the -// canonical shape is `/` (matching K8s resource identity). -// The CacheBackend reconciler always sends `/`; the -// HTTP handler validates that the field is non-empty but does not enforce -// the slash format, since hand-invoked probes on a single-namespace -// install can use any unique string. -// -// Model + HashScheme pin the engine domain the synthesized state lives -// under (so a probe for the vllm adapter cannot collide with a probe for -// the sglang adapter on the same backend). BackendType decides whether -// Stage C runs. -type ProbeRequest = controlplaneapi.ProbeRequest - -// ProbeResult is the per-stage outcome returned to the controller. The -// CacheBackend reconciler maps a stage's failed result onto the -// corresponding FunctionalProbeOK condition reason: -// -// ingest failed → ProbeIngestFailed -// routing failed → ProbeRoutingFailed -// t2 failed → ProbeT2Failed -// -// Errors carries a stage-keyed message so the operator-visible condition -// surfaces a concrete diagnostic, not just "something failed". -type ProbeResult = controlplaneapi.ProbeResult - -// ProbeStageError names one stage's failure mode in operator-readable form. -// Stage is one of ProbeStageIngest / ProbeStageRouting / ProbeStageT2. -type ProbeStageError = controlplaneapi.ProbeStageError - // T2Prober drives a put/get round trip against an external tier-2 backend // (today: LMCache). The controller-side caller is already wired (see // internal/controller/cachebackend_probe.go), but the server boots WITHOUT @@ -315,7 +210,7 @@ func (p *Prober) lockForRun(backend, model string) *sync.Mutex { // real lookup" — can construct the same id without re-deriving the // prefix-plus-backend rule. func ProbeReplicaID(backend string) string { - return ProbeReplicaPrefix + backend + return controlplaneapi.ProbeReplicaPrefix + backend } // ProbeHash returns the deterministic 32-byte SHA-256 of a canonical input @@ -391,7 +286,7 @@ type hashWriter interface { // before cleanup (panic, ctx done) still leaves only reserved-tenant entries // the index's TTL sweep will eventually reap — the reserved naming makes the // residue invisible to real workload lookups regardless. -func (p *Prober) Run(ctx context.Context, req ProbeRequest) ProbeResult { +func (p *Prober) Run(ctx context.Context, req controlplaneapi.ProbeRequest) controlplaneapi.ProbeResult { replicaID := ProbeReplicaID(req.Backend) probeHash := ProbeHash(req.Backend, req.Model, req.HashScheme) @@ -404,7 +299,7 @@ func (p *Prober) Run(ctx context.Context, req ProbeRequest) ProbeResult { mu.Lock() defer mu.Unlock() - result := ProbeResult{Backend: req.Backend} + result := controlplaneapi.ProbeResult{Backend: req.Backend} // Cleanup ALWAYS runs — even on early return from a Stage-A failure or a // panic in Stage B/C — so a flaky probe can't leak reserved-tenant entries @@ -424,12 +319,12 @@ func (p *Prober) Run(ctx context.Context, req ProbeRequest) ProbeResult { update := index.Update{ ReplicaID: replicaID, Model: req.Model, - Tenant: ProbeTenantID, + Tenant: controlplaneapi.ProbeTenantID, HashScheme: req.HashScheme, Timestamp: p.now(), Prefixes: []index.PrefixRef{{ BlockHashes: [][]byte{probeHash}, - BlockTokenCounts: []int32{ProbeTokenCount}, + BlockTokenCounts: []int32{controlplaneapi.ProbeTokenCount}, }}, Stats: &index.ReplicaStats{ ReplicaID: replicaID, @@ -441,26 +336,26 @@ func (p *Prober) Run(ctx context.Context, req ProbeRequest) ProbeResult { p.ingestFn(update) directReq := index.LookupRequest{ - Tenant: ProbeTenantID, + Tenant: controlplaneapi.ProbeTenantID, Model: req.Model, HashScheme: req.HashScheme, BlockHashes: [][]byte{probeHash}, - BlockTokenCounts: []int32{ProbeTokenCount}, + BlockTokenCounts: []int32{controlplaneapi.ProbeTokenCount}, } if !replicaInScores(p.index.Lookup(directReq), replicaID) { - result.Ingest = ProbeStageFailed - result.Errors = append(result.Errors, ProbeStageError{ - Stage: ProbeStageIngest, + result.Ingest = controlplaneapi.ProbeStageFailed + result.Errors = append(result.Errors, controlplaneapi.ProbeStageError{ + Stage: controlplaneapi.ProbeStageIngest, Message: "synthesized probe event did not land in the index — in-process index ingest path is broken (Stage A calls index.Ingest directly; the wire ReportCacheState handler is not exercised here)", }) // An entry that never landed cannot route, so Stage B is undefined; // skip it so the controller's condition pinpoints the upstream stage // instead of also flagging a routing failure that's just a cascade. - result.Routing = ProbeStageSkipped - result.T2 = ProbeStageSkipped + result.Routing = controlplaneapi.ProbeStageSkipped + result.T2 = controlplaneapi.ProbeStageSkipped return result } - result.Ingest = ProbeStageOK + result.Ingest = controlplaneapi.ProbeStageOK // Stage B — index routing. Call index.LookupRoute (the orchestrated // ranking entrypoint: PREFIX_MATCH / TENANT_HOT / NO_HINT) against the @@ -477,16 +372,16 @@ func (p *Prober) Run(ctx context.Context, req ProbeRequest) ProbeResult { // (minimumPrefixTokens), lookupTimeoutMs deadline, proto→domain // translation in updateFromProto/effectivePrefixTokens — are NOT // covered by Stage B and require their own tests (handler unit tests - // exist for each gate in pkg/server/server_test.go). A Stage B pass + // exist for each gate in internal/server/server_test.go). A Stage B pass // proves the index orchestration ranks the probe's hash correctly; it // is not proof the public LookupRoute gRPC handler is healthy // end-to-end. routeRes := p.routeFn(directReq) switch { case routeRes.Strategy != index.StrategyPrefixMatch: - result.Routing = ProbeStageFailed - result.Errors = append(result.Errors, ProbeStageError{ - Stage: ProbeStageRouting, + result.Routing = controlplaneapi.ProbeStageFailed + result.Errors = append(result.Errors, controlplaneapi.ProbeStageError{ + Stage: controlplaneapi.ProbeStageRouting, Message: fmt.Sprintf("LookupRoute returned %s, expected PREFIX_MATCH — index routing/key-derivation regression (this stage does not exercise the gRPC handler; see probe.go)", reasonForStrategy(routeRes.Strategy)), }) case !replicaInScores(routeRes.Scores, replicaID): @@ -499,23 +394,23 @@ func (p *Prober) Run(ctx context.Context, req ProbeRequest) ProbeResult { // to the reserved replica id. Name the expected replica explicitly // so the operator's condition message points at a probe-id- // derivation regression, not a vague "wrong reason code". - result.Routing = ProbeStageFailed - result.Errors = append(result.Errors, ProbeStageError{ - Stage: ProbeStageRouting, + result.Routing = controlplaneapi.ProbeStageFailed + result.Errors = append(result.Errors, controlplaneapi.ProbeStageError{ + Stage: controlplaneapi.ProbeStageRouting, Message: fmt.Sprintf("LookupRoute returned PREFIX_MATCH but probe replica %q is not among the scored replicas — possible probe-id or reserved-replica collision", replicaID), }) } - if result.Routing == ProbeStageFailed { + if result.Routing == controlplaneapi.ProbeStageFailed { // Skip Stage C on a routing failure for the same reason Stage B was // skipped on a Stage-A failure: running a downstream stage when an // upstream one is broken cascades the diagnostic — the controller's // FunctionalProbeOK condition would then have to disentangle whether // a T2 fail was real or a side-effect of routing being broken. // Surface only the upstream stage; the operator fixes that first. - result.T2 = ProbeStageSkipped + result.T2 = controlplaneapi.ProbeStageSkipped return result } - result.Routing = ProbeStageOK + result.Routing = controlplaneapi.ProbeStageOK // Stage C — T2 cycle. Skip on non-LMCache backends (no tier-2 to test) and // when no prober is wired (Stage 1 default). Empty BackendType is treated as @@ -527,20 +422,20 @@ func (p *Prober) Run(ctx context.Context, req ProbeRequest) ProbeResult { // a successful round-trip is observable as a byte match on the receiving // side; the probe doesn't care about the payload's content, only that what // went in came out. - runT2 := req.BackendType == BackendTypeLMCache || req.BackendType == "" + runT2 := req.BackendType == controlplaneapi.BackendTypeLMCache || req.BackendType == "" if !runT2 || p.t2 == nil { - result.T2 = ProbeStageSkipped + result.T2 = controlplaneapi.ProbeStageSkipped return result } if err := p.t2.ProbePutGet(ctx, req.Backend, probePayload(probeHash)); err != nil { - result.T2 = ProbeStageFailed - result.Errors = append(result.Errors, ProbeStageError{ - Stage: ProbeStageT2, + result.T2 = controlplaneapi.ProbeStageFailed + result.Errors = append(result.Errors, controlplaneapi.ProbeStageError{ + Stage: controlplaneapi.ProbeStageT2, Message: fmt.Sprintf("T2 put/get cycle failed: %s", t2ErrorMessage(err)), }) return result } - result.T2 = ProbeStageOK + result.T2 = controlplaneapi.ProbeStageOK return result } @@ -556,7 +451,7 @@ func (p *Prober) cleanup(model, replicaID string) { Type: index.EventAllCleared, ReplicaID: replicaID, Model: model, - Tenant: ProbeTenantID, + Tenant: controlplaneapi.ProbeTenantID, Timestamp: p.now(), }) } @@ -621,7 +516,7 @@ func probeHandler(prober *Prober) http.HandlerFunc { defer func() { _ = body.Close() }() dec := json.NewDecoder(body) dec.DisallowUnknownFields() - var req ProbeRequest + var req controlplaneapi.ProbeRequest if err := dec.Decode(&req); err != nil { http.Error(w, "decode probe request: "+err.Error()+"\n", http.StatusBadRequest) return diff --git a/pkg/server/probe_test.go b/internal/server/probe_test.go similarity index 85% rename from pkg/server/probe_test.go rename to internal/server/probe_test.go index 7d33b631..0be35a38 100644 --- a/pkg/server/probe_test.go +++ b/internal/server/probe_test.go @@ -20,7 +20,8 @@ import ( "google.golang.org/grpc/test/bufconn" authnv1 "k8s.io/api/authentication/v1" - "github.com/cachebox-project/inference-cache/pkg/index" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" ) // fakeT2Prober is the Stage-C fake. ProbePutGet returns whatever err the @@ -150,10 +151,10 @@ func TestProbeHashEncodingIsInjective(t *testing.T) { // collision-free namespace that keeps probe + workload state disjoint. func TestProbeReplicaIDReservedPrefix(t *testing.T) { got := ProbeReplicaID("my-backend") - if !strings.HasPrefix(got, ProbeReplicaPrefix) { - t.Fatalf("ProbeReplicaID = %q, want prefix %q", got, ProbeReplicaPrefix) + if !strings.HasPrefix(got, controlplaneapi.ProbeReplicaPrefix) { + t.Fatalf("ProbeReplicaID = %q, want prefix %q", got, controlplaneapi.ProbeReplicaPrefix) } - if got == ProbeReplicaPrefix { + if got == controlplaneapi.ProbeReplicaPrefix { t.Fatalf("ProbeReplicaID(%q) returned bare prefix — backend suffix dropped", "my-backend") } } @@ -164,19 +165,19 @@ func TestProbeReplicaIDReservedPrefix(t *testing.T) { // same as "every stage succeeded" — without this guard, the controller- // wiring follow-up could flip FunctionalProbeOK True on an empty response. func TestProbeResultAllPassedZeroValueFailsClosed(t *testing.T) { - if (ProbeResult{}).AllPassed() { - t.Fatal("ProbeResult{}.AllPassed() = true, want false — zero-value must not pass") + if (controlplaneapi.ProbeResult{}).AllPassed() { + t.Fatal("controlplaneapi.ProbeResult{}.AllPassed() = true, want false — zero-value must not pass") } // A partially-populated result also fails: two stages ok + one // zero-value field is still "no information" for that stage. - partial := ProbeResult{Ingest: ProbeStageOK, Routing: ProbeStageOK} + partial := controlplaneapi.ProbeResult{Ingest: controlplaneapi.ProbeStageOK, Routing: controlplaneapi.ProbeStageOK} if partial.AllPassed() { - t.Fatal("ProbeResult with zero-value T2 returned AllPassed=true; want false") + t.Fatal("controlplaneapi.ProbeResult with zero-value T2 returned AllPassed=true; want false") } // The all-explicit-ok case still passes. - all := ProbeResult{Ingest: ProbeStageOK, Routing: ProbeStageOK, T2: ProbeStageSkipped} + all := controlplaneapi.ProbeResult{Ingest: controlplaneapi.ProbeStageOK, Routing: controlplaneapi.ProbeStageOK, T2: controlplaneapi.ProbeStageSkipped} if !all.AllPassed() { - t.Fatal("explicit ok/ok/skipped ProbeResult should report AllPassed=true") + t.Fatal("explicit ok/ok/skipped controlplaneapi.ProbeResult should report AllPassed=true") } } @@ -187,21 +188,21 @@ func TestProbeResultAllPassedZeroValueFailsClosed(t *testing.T) { // yet. func TestProberRunHappyPathStageABCSkippedT2(t *testing.T) { prober, _ := newProberForTest(t, nil) - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-happy", Model: "llama-3-8b", HashScheme: "vllm", - BackendType: BackendTypeLMCache, + BackendType: controlplaneapi.BackendTypeLMCache, }) - if result.Ingest != ProbeStageOK { - t.Errorf("Ingest = %q, want %q", result.Ingest, ProbeStageOK) + if result.Ingest != controlplaneapi.ProbeStageOK { + t.Errorf("Ingest = %q, want %q", result.Ingest, controlplaneapi.ProbeStageOK) } - if result.Routing != ProbeStageOK { - t.Errorf("Routing = %q, want %q", result.Routing, ProbeStageOK) + if result.Routing != controlplaneapi.ProbeStageOK { + t.Errorf("Routing = %q, want %q", result.Routing, controlplaneapi.ProbeStageOK) } - if result.T2 != ProbeStageSkipped { - t.Errorf("T2 = %q, want %q when no T2Prober is wired", result.T2, ProbeStageSkipped) + if result.T2 != controlplaneapi.ProbeStageSkipped { + t.Errorf("T2 = %q, want %q when no T2Prober is wired", result.T2, controlplaneapi.ProbeStageSkipped) } if !result.AllPassed() { t.Fatalf("AllPassed() = false; result = %+v", result) @@ -219,14 +220,14 @@ func TestProberRunHappyPathStageABCSkippedT2(t *testing.T) { func TestProberRunStageCSkippedForNonLMCache(t *testing.T) { t2 := &fakeT2Prober{} prober, _ := newProberForTest(t, t2) - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-mem", Model: "m", HashScheme: "vllm", BackendType: "Memory", // not LMCache }) - if result.T2 != ProbeStageSkipped { - t.Fatalf("T2 = %q, want %q for non-LMCache backend", result.T2, ProbeStageSkipped) + if result.T2 != controlplaneapi.ProbeStageSkipped { + t.Fatalf("T2 = %q, want %q for non-LMCache backend", result.T2, controlplaneapi.ProbeStageSkipped) } if t2.calls != 0 { t.Fatalf("T2Prober was called %d times for non-LMCache backend, want 0", t2.calls) @@ -241,7 +242,7 @@ func TestProberRunStageCSkippedForNonLMCache(t *testing.T) { func TestProberRunStageCTreatsEmptyBackendTypeAsLMCache(t *testing.T) { t2 := &fakeT2Prober{} prober, _ := newProberForTest(t, t2) - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-default", Model: "m", HashScheme: "vllm", @@ -249,8 +250,8 @@ func TestProberRunStageCTreatsEmptyBackendTypeAsLMCache(t *testing.T) { // the CR omitted spec.type (the kubebuilder defaulter writes LMCache, // but a hand-rolled request can still arrive empty). }) - if result.T2 != ProbeStageOK { - t.Errorf("T2 = %q, want %q — empty BackendType must run Stage C (CRD default)", result.T2, ProbeStageOK) + if result.T2 != controlplaneapi.ProbeStageOK { + t.Errorf("T2 = %q, want %q — empty BackendType must run Stage C (CRD default)", result.T2, controlplaneapi.ProbeStageOK) } if t2.calls != 1 { t.Errorf("T2Prober calls = %d, want 1 — empty BackendType silently skipped Stage C", t2.calls) @@ -265,14 +266,14 @@ func TestProberRunStageCTreatsEmptyBackendTypeAsLMCache(t *testing.T) { func TestProberRunStageCRunsForLMCacheWithProber(t *testing.T) { t2 := &fakeT2Prober{} prober, _ := newProberForTest(t, t2) - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-lm", Model: "m", HashScheme: "vllm", - BackendType: BackendTypeLMCache, + BackendType: controlplaneapi.BackendTypeLMCache, }) - if result.T2 != ProbeStageOK { - t.Errorf("T2 = %q, want %q", result.T2, ProbeStageOK) + if result.T2 != controlplaneapi.ProbeStageOK { + t.Errorf("T2 = %q, want %q", result.T2, controlplaneapi.ProbeStageOK) } if t2.calls != 1 { t.Errorf("T2Prober calls = %d, want 1", t2.calls) @@ -294,22 +295,22 @@ func TestProberRunStageAFailsWhenIngestNoOps(t *testing.T) { prober, _ := newProberForTest(t, nil) prober.ingestFn = func(index.Update) {} // simulate a write that never lands - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-1", Model: "m", HashScheme: "vllm", }) - if result.Ingest != ProbeStageFailed { - t.Errorf("Ingest = %q, want %q", result.Ingest, ProbeStageFailed) + if result.Ingest != controlplaneapi.ProbeStageFailed { + t.Errorf("Ingest = %q, want %q", result.Ingest, controlplaneapi.ProbeStageFailed) } - if result.Routing != ProbeStageSkipped { - t.Errorf("Routing = %q, want %q (cascade from failed Stage A)", result.Routing, ProbeStageSkipped) + if result.Routing != controlplaneapi.ProbeStageSkipped { + t.Errorf("Routing = %q, want %q (cascade from failed Stage A)", result.Routing, controlplaneapi.ProbeStageSkipped) } - if result.T2 != ProbeStageSkipped { - t.Errorf("T2 = %q, want %q (cascade from failed Stage A)", result.T2, ProbeStageSkipped) + if result.T2 != controlplaneapi.ProbeStageSkipped { + t.Errorf("T2 = %q, want %q (cascade from failed Stage A)", result.T2, controlplaneapi.ProbeStageSkipped) } if result.AllPassed() { t.Fatal("AllPassed() = true despite Ingest failed") } - if !stageErrorPresent(result.Errors, ProbeStageIngest) { + if !stageErrorPresent(result.Errors, controlplaneapi.ProbeStageIngest) { t.Errorf("expected ingest stage error, got %+v", result.Errors) } } @@ -332,17 +333,17 @@ func TestProberRunStageBFailsWhenRouteReturnsNoHint(t *testing.T) { return index.LookupResult{Strategy: index.StrategyNone} } - result := prober.Run(t.Context(), ProbeRequest{ - Backend: "cb-1", Model: "m", HashScheme: "vllm", BackendType: BackendTypeLMCache, + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ + Backend: "cb-1", Model: "m", HashScheme: "vllm", BackendType: controlplaneapi.BackendTypeLMCache, }) - if result.Ingest != ProbeStageOK { - t.Errorf("Ingest = %q, want %q — direct Lookup is unaffected by the routeFn override", result.Ingest, ProbeStageOK) + if result.Ingest != controlplaneapi.ProbeStageOK { + t.Errorf("Ingest = %q, want %q — direct Lookup is unaffected by the routeFn override", result.Ingest, controlplaneapi.ProbeStageOK) } - if result.Routing != ProbeStageFailed { - t.Errorf("Routing = %q, want %q", result.Routing, ProbeStageFailed) + if result.Routing != controlplaneapi.ProbeStageFailed { + t.Errorf("Routing = %q, want %q", result.Routing, controlplaneapi.ProbeStageFailed) } - if result.T2 != ProbeStageSkipped { - t.Errorf("T2 = %q, want %q — must skip on Stage-B failure to avoid cascading diagnostic", result.T2, ProbeStageSkipped) + if result.T2 != controlplaneapi.ProbeStageSkipped { + t.Errorf("T2 = %q, want %q — must skip on Stage-B failure to avoid cascading diagnostic", result.T2, controlplaneapi.ProbeStageSkipped) } if t2.calls != 0 { t.Errorf("T2Prober was called %d times after Stage B failed; want 0 (cascade prevention)", t2.calls) @@ -350,7 +351,7 @@ func TestProberRunStageBFailsWhenRouteReturnsNoHint(t *testing.T) { if result.AllPassed() { t.Fatal("AllPassed() = true despite Routing failed") } - if !stageErrorPresent(result.Errors, ProbeStageRouting) { + if !stageErrorPresent(result.Errors, controlplaneapi.ProbeStageRouting) { t.Errorf("expected routing stage error, got %+v", result.Errors) } } @@ -371,13 +372,13 @@ func TestProberRunStageBDistinguishesWrongReplicaFromWrongStrategy(t *testing.T) } } - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-1", Model: "m", HashScheme: "vllm", }) - if result.Routing != ProbeStageFailed { - t.Fatalf("Routing = %q, want %q", result.Routing, ProbeStageFailed) + if result.Routing != controlplaneapi.ProbeStageFailed { + t.Fatalf("Routing = %q, want %q", result.Routing, controlplaneapi.ProbeStageFailed) } - msg := stageErrorMessage(result.Errors, ProbeStageRouting) + msg := stageErrorMessage(result.Errors, controlplaneapi.ProbeStageRouting) expectedReplica := ProbeReplicaID("cb-1") if !strings.Contains(msg, "not among the scored replicas") { t.Errorf("routing error %q should distinguish wrong-replica from wrong-strategy", msg) @@ -405,17 +406,17 @@ func TestProberRunStageCDistinguishesPutFromGet(t *testing.T) { t.Run(tc.name, func(t *testing.T) { t2 := &fakeT2Prober{err: tc.err} prober, _ := newProberForTest(t, t2) - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-1", Model: "m", HashScheme: "vllm", - BackendType: BackendTypeLMCache, + BackendType: controlplaneapi.BackendTypeLMCache, }) - if result.T2 != ProbeStageFailed { - t.Fatalf("T2 = %q, want %q", result.T2, ProbeStageFailed) + if result.T2 != controlplaneapi.ProbeStageFailed { + t.Fatalf("T2 = %q, want %q", result.T2, controlplaneapi.ProbeStageFailed) } - if !stageErrorPresent(result.Errors, ProbeStageT2) { + if !stageErrorPresent(result.Errors, controlplaneapi.ProbeStageT2) { t.Fatalf("expected t2 stage error, got %+v", result.Errors) } - msg := stageErrorMessage(result.Errors, ProbeStageT2) + msg := stageErrorMessage(result.Errors, controlplaneapi.ProbeStageT2) if !strings.Contains(msg, tc.wantSubstr) { t.Errorf("T2 error message %q does not contain %q", msg, tc.wantSubstr) } @@ -430,9 +431,9 @@ func TestProberRunStageCDistinguishesPutFromGet(t *testing.T) { // reconcile pass, polluting both /snapshot and the entry-count metric. func TestProberRunLeavesNoStateInIndex(t *testing.T) { prober, idx := newProberForTest(t, nil) - _ = prober.Run(t.Context(), ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) + _ = prober.Run(t.Context(), controlplaneapi.ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) - replicas, totalPrefixes := idx.CacheState(ProbeTenantID, "m") + replicas, totalPrefixes := idx.CacheState(controlplaneapi.ProbeTenantID, "m") if totalPrefixes != 0 { t.Errorf("totalPrefixes after probe = %d, want 0 — cleanup did not run", totalPrefixes) } @@ -451,10 +452,10 @@ func TestProberRunSerializesConcurrentProbesForSameBackend(t *testing.T) { prober, idx := newProberForTest(t, nil) const concurrent = 8 - results := make(chan ProbeResult, concurrent) + results := make(chan controlplaneapi.ProbeResult, concurrent) for i := 0; i < concurrent; i++ { go func() { - results <- prober.Run(t.Context(), ProbeRequest{ + results <- prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-shared", Model: "m", HashScheme: "vllm", }) }() @@ -468,7 +469,7 @@ func TestProberRunSerializesConcurrentProbesForSameBackend(t *testing.T) { } // All Runs serialized correctly, so the final state is clean. - _, totalPrefixes := idx.CacheState(ProbeTenantID, "m") + _, totalPrefixes := idx.CacheState(controlplaneapi.ProbeTenantID, "m") if totalPrefixes != 0 { t.Errorf("after %d concurrent runs, totalPrefixes = %d, want 0", concurrent, totalPrefixes) } @@ -481,12 +482,12 @@ func TestProberRunSerializesConcurrentProbesForSameBackend(t *testing.T) { func TestProberRunIsIdempotent(t *testing.T) { prober, idx := newProberForTest(t, nil) for i := 0; i < 3; i++ { - result := prober.Run(t.Context(), ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) if !result.AllPassed() { t.Fatalf("iteration %d: AllPassed() = false; result = %+v", i, result) } } - _, totalPrefixes := idx.CacheState(ProbeTenantID, "m") + _, totalPrefixes := idx.CacheState(controlplaneapi.ProbeTenantID, "m") if totalPrefixes != 0 { t.Fatalf("after 3 idempotent runs, totalPrefixes = %d, want 0", totalPrefixes) } @@ -504,7 +505,7 @@ func TestProberRunDoesNotEvictRealWorkloadOnFullIndex(t *testing.T) { // Index sized exactly to one entry, with the probe tenant reserved. idx := index.New( index.WithMaxEntries(1), - index.WithReservedTenants(ProbeTenantID), + index.WithReservedTenants(controlplaneapi.ProbeTenantID), ) idx.Start(t.Context()) prober := NewProber(idx, nil) @@ -518,7 +519,7 @@ func TestProberRunDoesNotEvictRealWorkloadOnFullIndex(t *testing.T) { // Run the probe. With WithReservedTenants, the probe-tenant entry is // cap-invisible: enforceCapLocked sees effectiveTotal=1 (the real entry) // even though totalEntries is 2 momentarily. No eviction. - result := prober.Run(t.Context(), ProbeRequest{ + result := prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-1", Model: "m", HashScheme: "vllm", }) if !result.AllPassed() { @@ -546,7 +547,7 @@ func TestProberRunDoesNotEvictRealWorkloadOnFullIndex(t *testing.T) { func TestProberRunConcurrentWithRealWorkloadDoesNotEvict(t *testing.T) { idx := index.New( index.WithMaxEntries(1), - index.WithReservedTenants(ProbeTenantID), + index.WithReservedTenants(controlplaneapi.ProbeTenantID), ) idx.Start(t.Context()) prober := NewProber(idx, nil) @@ -560,7 +561,7 @@ func TestProberRunConcurrentWithRealWorkloadDoesNotEvict(t *testing.T) { done := make(chan struct{}, concurrent*2) for i := 0; i < concurrent; i++ { go func() { - _ = prober.Run(t.Context(), ProbeRequest{ + _ = prober.Run(t.Context(), controlplaneapi.ProbeRequest{ Backend: "cb-1", Model: "m", HashScheme: "vllm", }) done <- struct{}{} @@ -604,7 +605,7 @@ func TestProberRunReservedTenantUsedRegardlessOfRequest(t *testing.T) { Prefixes: []index.PrefixRef{{PrefixHash: []byte("p"), TokenCount: 64}}, }) - _ = prober.Run(t.Context(), ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) + _ = prober.Run(t.Context(), controlplaneapi.ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) // The real workload entry must survive. If the probe somehow leaked into // real-tenant's scope, ALL_CLEARED cleanup would have removed it. @@ -623,7 +624,7 @@ func TestProbeHandlerHappyPath(t *testing.T) { prober, _ := newProberForTest(t, nil) handler := NewProbeHTTPHandler(prober) - body, err := json.Marshal(ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) + body, err := json.Marshal(controlplaneapi.ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) if err != nil { t.Fatalf("marshal: %v", err) } @@ -638,7 +639,7 @@ func TestProbeHandlerHappyPath(t *testing.T) { if got := rr.Header().Get("Content-Type"); got != "application/json" { t.Errorf("Content-Type = %q, want application/json", got) } - var result ProbeResult + var result controlplaneapi.ProbeResult if err := json.Unmarshal(rr.Body.Bytes(), &result); err != nil { t.Fatalf("decode body: %v (body=%s)", err, rr.Body.String()) } @@ -807,7 +808,7 @@ func TestControllerAuth_ProbeRejectsUnauthenticated(t *testing.T) { // required fields). func emptyProbeRequestBody(t *testing.T) string { t.Helper() - b, err := json.Marshal(ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) + b, err := json.Marshal(controlplaneapi.ProbeRequest{Backend: "cb-1", Model: "m", HashScheme: "vllm"}) if err != nil { t.Fatalf("marshal probe request: %v", err) } @@ -837,7 +838,7 @@ func postJSON(t *testing.T, url, token, body string) int { return resp.StatusCode } -func stageErrorPresent(errs []ProbeStageError, stage string) bool { +func stageErrorPresent(errs []controlplaneapi.ProbeStageError, stage string) bool { for _, e := range errs { if e.Stage == stage { return true @@ -846,7 +847,7 @@ func stageErrorPresent(errs []ProbeStageError, stage string) bool { return false } -func stageErrorMessage(errs []ProbeStageError, stage string) string { +func stageErrorMessage(errs []controlplaneapi.ProbeStageError, stage string) string { for _, e := range errs { if e.Stage == stage { return e.Message diff --git a/pkg/server/route_dual_input_test.go b/internal/server/route_dual_input_test.go similarity index 95% rename from pkg/server/route_dual_input_test.go rename to internal/server/route_dual_input_test.go index b608b829..80bba71c 100644 --- a/pkg/server/route_dual_input_test.go +++ b/internal/server/route_dual_input_test.go @@ -11,10 +11,11 @@ import ( "testing" "time" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" + "github.com/cachebox-project/inference-cache/internal/subscriber" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" "github.com/cachebox-project/inference-cache/pkg/tokenize" ) @@ -103,7 +104,7 @@ func (b blockingTokenizer) EncodeText(context.Context, string, string, tokenize. // budget context, not before it). func TestLookupRoutePromptTextSlowTokenizerTimesOut(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{{Namespace: "tenant-x", LookupTimeoutMs: 20}}) + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenant-x", LookupTimeoutMs: 20}}) release := make(chan struct{}) defer close(release) // unblock the tokenizer goroutine at test end svc.tokenizer = blockingTokenizer{release: release} @@ -146,16 +147,16 @@ func TestLookupRouteTokenIDsEqualsExplicitChain(t *testing.T) { ) tokens := tokenSeq(1_000, 64) // 4 blocks, matched_tokens=64 clears the default floor - batch := &engine.EventBatch{ + batch := &subscriber.EventBatch{ TimestampSeconds: 0, - Events: []engine.Event{engine.BlockStored{ + Events: []subscriber.Event{subscriber.BlockStored{ BlockHashes: [][]byte{be8(1), be8(2), be8(3), be8(4)}, TokenIDs: tokens, BlockSize: blockSz, }}, } client, stop := runEngineReporterAgainstServer(t, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, batch) + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, batch) defer stop() ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -254,11 +255,11 @@ func TestLookupRoutePromptTextDefaultTokenizeTimeout(t *testing.T) { // callers, not just pre-fingerprinted block_hashes requests. func TestLookupRouteTokenIDsNovelAffinityFallback(t *testing.T) { stored := tokenSeq(1_000, 64) - batch := &engine.EventBatch{Events: []engine.Event{engine.BlockStored{ + batch := &subscriber.EventBatch{Events: []subscriber.Event{subscriber.BlockStored{ BlockHashes: [][]byte{be8(1), be8(2), be8(3), be8(4)}, TokenIDs: stored, BlockSize: 16, }}} client, stop := runEngineReporterAgainstServer(t, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, batch) + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, batch) defer stop() resp, err := client.LookupRoute(context.Background(), &icpb.LookupRouteRequest{ @@ -390,11 +391,11 @@ func TestLookupRouteOversizedTokenizerOutputFailsOpen(t *testing.T) { // overrode it the server would fingerprint the novel tokens and miss. func TestLookupRouteExplicitChainBeatsTokenIDs(t *testing.T) { stored := tokenSeq(1_000, 64) - batch := &engine.EventBatch{Events: []engine.Event{engine.BlockStored{ + batch := &subscriber.EventBatch{Events: []subscriber.Event{subscriber.BlockStored{ BlockHashes: [][]byte{be8(1), be8(2), be8(3), be8(4)}, TokenIDs: stored, BlockSize: 16, }}} client, stop := runEngineReporterAgainstServer(t, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, batch) + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, batch) defer stop() bh, btc := fingerprint.Chain(stored, 16) @@ -486,7 +487,7 @@ func TestLookupRouteTokenIDsHonorsConfiguredBlockSize(t *testing.T) { func TestLookupRoutePromptTextEchoesTokensOnMinPrefixGate(t *testing.T) { tokens := tokenSeq(2_500_000, 64) // 64 effective tokens, below the 1000 gate svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{{Namespace: "tenant-x", MinimumPrefixTokens: 1000}}) + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "tenant-x", MinimumPrefixTokens: 1000}}) svc.tokenizer = fakeTokenizer{tokens: tokens} resp, err := svc.LookupRoute(context.Background(), &icpb.LookupRouteRequest{ diff --git a/pkg/server/route_lookup_hitmiss_test.go b/internal/server/route_lookup_hitmiss_test.go similarity index 89% rename from pkg/server/route_lookup_hitmiss_test.go rename to internal/server/route_lookup_hitmiss_test.go index 047ce9d8..fc07aa7c 100644 --- a/pkg/server/route_lookup_hitmiss_test.go +++ b/internal/server/route_lookup_hitmiss_test.go @@ -8,9 +8,9 @@ import ( "context" "testing" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/subscriber" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" ) // A healthy routing index must yield a *mix* of hits and misses on a designed @@ -38,14 +38,14 @@ func TestRouteLookupMixedHitMiss(t *testing.T) { ) // Store K distinct single-block prefixes and remember each one's content key. - var batches []*engine.EventBatch + var batches []*subscriber.EventBatch keys := make([][]byte, k) for i := 0; i < k; i++ { toks := tokenSeq(1_000+i*10_000, blockTok) // far-apart ranges → distinct content keys[i] = fingerprint.Bytes(fingerprint.PrefixHashes(toks, blockTok)[0]) - batches = append(batches, &engine.EventBatch{ + batches = append(batches, &subscriber.EventBatch{ TimestampSeconds: 0, // 0 = "now" server-side; a real epoch ts would be past the freshness TTL - Events: []engine.Event{engine.BlockStored{ + Events: []subscriber.Event{subscriber.BlockStored{ BlockHashes: [][]byte{be8(uint64(i) + 1)}, TokenIDs: toks, BlockSize: blockTok, @@ -54,7 +54,7 @@ func TestRouteLookupMixedHitMiss(t *testing.T) { } client, stop := runEngineReporterAgainstServer(t, - []engine.ReporterOption{engine.WithIgnoreBlockRemoved(true)}, batches...) + []subscriber.ReporterOption{subscriber.WithIgnoreBlockRemoved(true)}, batches...) defer stop() match := func(key []byte) bool { diff --git a/pkg/server/routing_floor_response_test.go b/internal/server/routing_floor_response_test.go similarity index 93% rename from pkg/server/routing_floor_response_test.go rename to internal/server/routing_floor_response_test.go index 8a4f0c50..d256489d 100644 --- a/pkg/server/routing_floor_response_test.go +++ b/internal/server/routing_floor_response_test.go @@ -8,8 +8,9 @@ import ( "context" "testing" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" ) // LookupRoute-level tests for the routing floor score. The PolicyStore @@ -47,7 +48,7 @@ func TestLookupRouteRoutingFloorScoreDowngradesWhenAllReplicasHoldPrefix(t *test // orthogonal to the routing-floor → NO_HINT invariant this test // pins. fal := false - svc.policies.Replace([]ResolvedPolicy{{Namespace: "no-policy", AffinityRouting: &fal}}) + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "no-policy", AffinityRouting: &fal}}) for _, rid := range []string{"r0", "r1", "r2"} { svc.index.Ingest(index.Update{ ReplicaID: rid, Model: "m", Tenant: "no-policy", HashScheme: "vllm", @@ -116,7 +117,7 @@ func TestLookupRouteRoutingFloorScorePolicyOverride(t *testing.T) { // TestLookupRouteRoutingFloorScoreDowngradesWhenAllReplicasHoldPrefix // comment for the rationale. fal := false - svc.policies.Replace([]ResolvedPolicy{{Namespace: "strict", RoutingFloorScore: f32Ptr(100), AffinityRouting: &fal}}) + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "strict", RoutingFloorScore: f32Ptr(100), AffinityRouting: &fal}}) svc.index.Ingest(index.Update{ ReplicaID: "r0", Model: "m", Tenant: "strict", HashScheme: "vllm", Prefixes: []index.PrefixRef{{PrefixHash: []byte("unique"), TokenCount: 64}}, @@ -150,13 +151,13 @@ func TestLookupRouteRoutingFloorScorePolicyOverride(t *testing.T) { // to surface every trivial match — the operator would have to set both // to 0 / "0". The point of the assertion here is the routing-floor opt- // out semantics in isolation; the matched-tokens opt-out is tested by -// the matched-tokens floor suite (pkg/server/matched_tokens_floor_test.go). +// the matched-tokens floor suite (internal/server/matched_tokens_floor_test.go). func TestLookupRouteRoutingFloorScoreZeroDisablesFloor(t *testing.T) { svc := newTestService() // MinimumMatchedTokens defaults to 0 in the struct, so this Replace // installs a policy with both floors off — exercising the routing- // floor opt-out in isolation. - svc.policies.Replace([]ResolvedPolicy{{Namespace: "raw", RoutingFloorScore: f32Ptr(0)}}) + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "raw", RoutingFloorScore: f32Ptr(0)}}) for _, rid := range []string{"r0", "r1", "r2"} { svc.index.Ingest(index.Update{ ReplicaID: rid, Model: "m", Tenant: "raw", HashScheme: "vllm", diff --git a/pkg/server/routing_floor_score_test.go b/internal/server/routing_floor_score_test.go similarity index 82% rename from pkg/server/routing_floor_score_test.go rename to internal/server/routing_floor_score_test.go index 250eea8f..6a0f8c3b 100644 --- a/pkg/server/routing_floor_score_test.go +++ b/internal/server/routing_floor_score_test.go @@ -7,6 +7,8 @@ package server import ( "math" "testing" + + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" ) func f32Ptr(v float32) *float32 { return &v } @@ -28,8 +30,8 @@ func approxFloorEq(a, b float32) bool { return math.Abs(float64(a-b)) <= floorTo // CachePolicy CR — server defaults are deliberately sane). func TestPolicyStoreRoutingFloorScoreFallsBackToDefault(t *testing.T) { store := NewPolicyStore() - if got := store.RoutingFloorScore("never-configured"); !approxFloorEq(got, DefaultRoutingFloorScore) { - t.Fatalf("RoutingFloorScore(no-policy) = %v, want DefaultRoutingFloorScore (%v)", got, DefaultRoutingFloorScore) + if got := store.RoutingFloorScore("never-configured"); !approxFloorEq(got, controlplaneapi.DefaultRoutingFloorScore) { + t.Fatalf("RoutingFloorScore(no-policy) = %v, want controlplaneapi.DefaultRoutingFloorScore (%v)", got, controlplaneapi.DefaultRoutingFloorScore) } } @@ -42,7 +44,7 @@ func TestPolicyStoreRoutingFloorScoreFallsBackToDefault(t *testing.T) { // regression-testing the ranker. func TestPolicyStoreRoutingFloorScoreRespectsPolicyValue(t *testing.T) { store := NewPolicyStore() - store.Replace([]ResolvedPolicy{ + store.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "ns-strict", RoutingFloorScore: f32Ptr(5.0)}, {Namespace: "ns-disabled", RoutingFloorScore: f32Ptr(0)}, }) @@ -50,7 +52,7 @@ func TestPolicyStoreRoutingFloorScoreRespectsPolicyValue(t *testing.T) { t.Fatalf("strict floor = %v, want 5.0", got) } if got := store.RoutingFloorScore("ns-disabled"); got != 0 { - t.Fatalf("disabled floor = %v, want exactly 0 (explicit opt-out, not DefaultRoutingFloorScore)", got) + t.Fatalf("disabled floor = %v, want exactly 0 (explicit opt-out, not controlplaneapi.DefaultRoutingFloorScore)", got) } } @@ -65,15 +67,15 @@ func TestPolicyStoreRoutingFloorScoreRespectsPolicyValue(t *testing.T) { // operator never expressed an opt-out intent. func TestPolicyStoreRoutingFloorScoreNilFieldFallsBackToDefault(t *testing.T) { store := NewPolicyStore() - store.Replace([]ResolvedPolicy{ + store.Replace([]controlplaneapi.ResolvedPolicy{ // CachePolicy present in the store but with RoutingFloorScore=nil // (the wire body omitted the field — pre-defaulting body, manual // crafter, legacy CR). {Namespace: "ns-bare", RoutingFloorScore: nil}, }) - if got := store.RoutingFloorScore("ns-bare"); !approxFloorEq(got, DefaultRoutingFloorScore) { - t.Fatalf("bare-field-present ns = %v, want DefaultRoutingFloorScore (%v) — absent field MUST NOT be inferred as opt-out", - got, DefaultRoutingFloorScore) + if got := store.RoutingFloorScore("ns-bare"); !approxFloorEq(got, controlplaneapi.DefaultRoutingFloorScore) { + t.Fatalf("bare-field-present ns = %v, want controlplaneapi.DefaultRoutingFloorScore (%v) — absent field MUST NOT be inferred as opt-out", + got, controlplaneapi.DefaultRoutingFloorScore) } } @@ -83,7 +85,7 @@ func TestPolicyStoreRoutingFloorScoreNilFieldFallsBackToDefault(t *testing.T) { // floor<0 never fires) instead of the safest interpretation. Clamp to 0. func TestPolicyStoreRoutingFloorScoreClampsNegative(t *testing.T) { store := NewPolicyStore() - store.Replace([]ResolvedPolicy{{Namespace: "ns-bad", RoutingFloorScore: f32Ptr(-1.0)}}) + store.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "ns-bad", RoutingFloorScore: f32Ptr(-1.0)}}) if got := store.RoutingFloorScore("ns-bad"); got != 0 { t.Fatalf("negative floor = %v, want 0 (clamped)", got) } diff --git a/pkg/server/server.go b/internal/server/server.go similarity index 97% rename from pkg/server/server.go rename to internal/server/server.go index 0a52360f..f562e23c 100644 --- a/pkg/server/server.go +++ b/internal/server/server.go @@ -21,9 +21,10 @@ import ( healthpb "google.golang.org/grpc/health/grpc_health_v1" "google.golang.org/grpc/reflection" - "github.com/cachebox-project/inference-cache/pkg/index" - "github.com/cachebox-project/inference-cache/pkg/server/auth" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" + "github.com/cachebox-project/inference-cache/internal/server/auth" "github.com/cachebox-project/inference-cache/pkg/tokenize" ) @@ -147,7 +148,7 @@ func New(opts ...Option) *Service { // - reservedEntries) and the victim candidate set, so the probe path's // "never mutates real workload state" invariant holds even under // concurrent real-workload writes on a saturated index. - index.WithReservedTenants(ProbeTenantID), + index.WithReservedTenants(controlplaneapi.ProbeTenantID), ) publicMux := http.NewServeMux() @@ -187,7 +188,7 @@ func New(opts ...Option) *Service { return } w.Header().Set("Content-Type", "application/json") - if err := json.NewEncoder(w).Encode(idx.Snapshot()); err != nil { + if err := json.NewEncoder(w).Encode(snapshotForControlPlane(idx.Snapshot())); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } }) diff --git a/pkg/server/server_test.go b/internal/server/server_test.go similarity index 96% rename from pkg/server/server_test.go rename to internal/server/server_test.go index 9cff5bfd..c90604e1 100644 --- a/pkg/server/server_test.go +++ b/internal/server/server_test.go @@ -24,9 +24,10 @@ import ( "google.golang.org/grpc/test/bufconn" authnv1 "k8s.io/api/authentication/v1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - "github.com/cachebox-project/inference-cache/pkg/index" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" "github.com/cachebox-project/inference-cache/pkg/tokenize" ) @@ -41,7 +42,7 @@ func newTestService() *inferenceCacheService { policies := NewPolicyStore() idx := index.New( index.WithTTLResolver(policies), - index.WithReservedTenants(ProbeTenantID), + index.WithReservedTenants(controlplaneapi.ProbeTenantID), ) return newInferenceCacheService(idx, newServerMetrics(), policies) } @@ -501,10 +502,11 @@ func getString(t *testing.T, url string) (int, string) { return resp.StatusCode, string(body) } -// TestSnapshotEndpointReflectsIngest ingests state over gRPC and confirms the +// TestSnapshotEndpointMapsIndexDomainToWireDTO ingests state over gRPC and confirms the // internal /snapshot HTTP endpoint reflects it as JSON (the controller scrapes -// this to populate the CacheIndex status). -func TestSnapshotEndpointReflectsIngest(t *testing.T) { +// this to populate the CacheIndex status). Decoding into the control-plane DTO +// exercises the explicit index-domain-to-wire mapping at the endpoint boundary. +func TestSnapshotEndpointMapsIndexDomainToWireDTO(t *testing.T) { conn, _, snapshotURL, stop := startInProcessServerConnFull(t) grpcClient := icpb.NewInferenceCacheClient(conn) defer stop() @@ -534,7 +536,7 @@ func TestSnapshotEndpointReflectsIngest(t *testing.T) { if code != http.StatusOK { t.Fatalf("/snapshot status = %d, want 200", code) } - var snap index.Snapshot + var snap controlplaneapi.Snapshot if err := json.Unmarshal([]byte(body), &snap); err != nil { t.Fatalf("decode snapshot JSON: %v (body=%s)", err, body) } @@ -597,14 +599,14 @@ func TestReportCacheState_AcceptsClientVersionOnReplicaStats(t *testing.T) { if code != http.StatusOK { t.Fatalf("/snapshot status = %d, want 200", code) } - var snap index.Snapshot + var snap controlplaneapi.Snapshot if err := json.Unmarshal([]byte(body), &snap); err != nil { t.Fatalf("decode snapshot JSON: %v (body=%s)", err, body) } if snap.TotalPrefixes != 1 { t.Fatalf("totalPrefixes = %d, want 1 — ingestion path dropped the update", snap.TotalPrefixes) } - var got *index.ReplicaSnapshot + var got *controlplaneapi.ReplicaSnapshot for i := range snap.Replicas { if snap.Replicas[i].ReplicaID == "replica-cv" { got = &snap.Replicas[i] @@ -672,9 +674,9 @@ func TestPolicyServedOnSnapshotListener(t *testing.T) { // replace-on-write payload. func emptyPolicySnapshotBody(t *testing.T) string { t.Helper() - b, err := json.Marshal(PolicySnapshot{Version: PolicyPropagationVersion}) + b, err := json.Marshal(controlplaneapi.PolicySnapshot{Version: controlplaneapi.PolicyPropagationVersion}) if err != nil { - t.Fatalf("marshal empty PolicySnapshot: %v", err) + t.Fatalf("marshal empty controlplaneapi.PolicySnapshot: %v", err) } return string(b) } @@ -744,7 +746,7 @@ func TestControllerAuth_RejectsUnauthenticated(t *testing.T) { // Audience left empty here — this test exercises the auth middleware's // expectedSA / cache / 401-on-missing-bearer matrix end-to-end against // real listeners for BOTH /snapshot AND /policy. The audience-binding - // path is covered by the envtest integration in pkg/server/auth (real + // path is covered by the envtest integration in internal/server/auth (real // apiserver mints audience-bound tokens); a fake reviewer can't // faithfully model audience enforcement. svc := New(WithControllerAuth(reviewer, sa, "")) @@ -1215,12 +1217,12 @@ func TestLookupRouteFailsOpenForReservedProbeTenant(t *testing.T) { svc := newTestService() // Seed something in the probe scope so a leak would actually surface. svc.index.Ingest(index.Update{ - ReplicaID: ProbeReplicaID("cb-1"), Model: "m", Tenant: ProbeTenantID, HashScheme: "vllm", + ReplicaID: ProbeReplicaID("cb-1"), Model: "m", Tenant: controlplaneapi.ProbeTenantID, HashScheme: "vllm", Prefixes: []index.PrefixRef{{PrefixHash: []byte("p"), TokenCount: 32}}, }) resp, err := svc.LookupRoute(context.Background(), &icpb.LookupRouteRequest{ - ModelId: "m", TenantId: ProbeTenantID, HashScheme: "vllm", PrefixHash: []byte("p"), + ModelId: "m", TenantId: controlplaneapi.ProbeTenantID, HashScheme: "vllm", PrefixHash: []byte("p"), }) if err != nil { t.Fatalf("LookupRoute: %v", err) @@ -1243,7 +1245,7 @@ func TestLookupRouteFailsOpenForReservedProbeTenant(t *testing.T) { func TestLookupRouteEmitsMetricForReservedProbeTenantNoHint(t *testing.T) { svc := newTestService() if _, err := svc.LookupRoute(context.Background(), &icpb.LookupRouteRequest{ - ModelId: "m", TenantId: ProbeTenantID, HashScheme: "vllm", PrefixHash: []byte("p"), + ModelId: "m", TenantId: controlplaneapi.ProbeTenantID, HashScheme: "vllm", PrefixHash: []byte("p"), }); err != nil { t.Fatalf("LookupRoute: %v", err) } @@ -1300,13 +1302,13 @@ func lookupCallsValueFromService(t *testing.T, svc *inferenceCacheService, model func TestGetCacheStateReturnsEmptyForReservedProbeTenant(t *testing.T) { svc := newTestService() svc.index.Ingest(index.Update{ - ReplicaID: ProbeReplicaID("cb-1"), Model: "m", Tenant: ProbeTenantID, HashScheme: "vllm", + ReplicaID: ProbeReplicaID("cb-1"), Model: "m", Tenant: controlplaneapi.ProbeTenantID, HashScheme: "vllm", Prefixes: []index.PrefixRef{{PrefixHash: []byte("p"), TokenCount: 32}}, Stats: &index.ReplicaStats{ReplicaID: ProbeReplicaID("cb-1"), CacheMemoryBytes: 1234, HitRate: 1.0}, }) resp, err := svc.GetCacheState(context.Background(), &icpb.GetCacheStateRequest{ - ModelId: "m", TenantId: ProbeTenantID, + ModelId: "m", TenantId: controlplaneapi.ProbeTenantID, }) if err != nil { t.Fatalf("GetCacheState: %v", err) @@ -1332,7 +1334,7 @@ func TestSnapshotFiltersReservedProbeTenant(t *testing.T) { Stats: &index.ReplicaStats{ReplicaID: "real-r", CacheMemoryBytes: 5000, HitRate: 0.5}, }) svc.index.Ingest(index.Update{ - ReplicaID: ProbeReplicaID("cb-1"), Model: "m", Tenant: ProbeTenantID, HashScheme: "vllm", + ReplicaID: ProbeReplicaID("cb-1"), Model: "m", Tenant: controlplaneapi.ProbeTenantID, HashScheme: "vllm", Prefixes: []index.PrefixRef{{PrefixHash: []byte("pp"), TokenCount: 16}}, Stats: &index.ReplicaStats{ReplicaID: ProbeReplicaID("cb-1"), CacheMemoryBytes: 1234, HitRate: 1.0}, }) @@ -1343,12 +1345,12 @@ func TestSnapshotFiltersReservedProbeTenant(t *testing.T) { t.Errorf("TotalPrefixes = %d, want 1 — reserved tenant must not contribute to the cluster total", snap.TotalPrefixes) } for _, r := range snap.Replicas { - if r.Tenant == ProbeTenantID || strings.HasPrefix(r.ReplicaID, ProbeReplicaPrefix) { + if r.Tenant == controlplaneapi.ProbeTenantID || strings.HasPrefix(r.ReplicaID, controlplaneapi.ProbeReplicaPrefix) { t.Errorf("Snapshot exposed reserved replica row: %+v", r) } } for _, tn := range snap.Tenants { - if tn.TenantID == ProbeTenantID { + if tn.TenantID == controlplaneapi.ProbeTenantID { t.Errorf("Snapshot exposed reserved tenant row: %+v", tn) } } @@ -1365,7 +1367,7 @@ func TestReportCacheStateDropsReservedProbeTenant(t *testing.T) { stream := &fakeReportStream{updates: []*icpb.CacheStateUpdate{{ ReplicaId: "spoofed", ModelId: "m", - TenantId: ProbeTenantID, + TenantId: controlplaneapi.ProbeTenantID, HashScheme: "vllm", Prefixes: []*icpb.PrefixEntry{{PrefixHash: []byte("p"), TokenCount: 32}}, }}} @@ -1373,7 +1375,7 @@ func TestReportCacheStateDropsReservedProbeTenant(t *testing.T) { t.Fatalf("ReportCacheState: %v", err) } scores := svc.index.Lookup(index.LookupRequest{ - Tenant: ProbeTenantID, Model: "m", HashScheme: "vllm", PrefixHash: []byte("p"), + Tenant: controlplaneapi.ProbeTenantID, Model: "m", HashScheme: "vllm", PrefixHash: []byte("p"), }) if len(scores) != 0 { t.Fatalf("external ingest under reserved probe tenant landed in the index: %+v", scores) @@ -1388,12 +1390,12 @@ func TestReportCacheStateDropsReservedProbeTenant(t *testing.T) { func TestPublishEventDropsReservedProbeTenant(t *testing.T) { svc := newTestService() svc.index.Ingest(index.Update{ - ReplicaID: "real", Model: "m", Tenant: ProbeTenantID, HashScheme: "vllm", + ReplicaID: "real", Model: "m", Tenant: controlplaneapi.ProbeTenantID, HashScheme: "vllm", Prefixes: []index.PrefixRef{{PrefixHash: []byte("p"), TokenCount: 32}}, }) ack, err := svc.PublishEvent(context.Background(), &icpb.CacheEvent{ Type: icpb.CacheEvent_ALL_CLEARED, ReplicaId: "real", - ModelId: "m", TenantId: ProbeTenantID, + ModelId: "m", TenantId: controlplaneapi.ProbeTenantID, }) if err != nil { t.Fatalf("PublishEvent: %v", err) @@ -1403,7 +1405,7 @@ func TestPublishEventDropsReservedProbeTenant(t *testing.T) { } // The seeded entry must survive — the ALL_CLEARED was dropped before the index saw it. scores := svc.index.Lookup(index.LookupRequest{ - Tenant: ProbeTenantID, Model: "m", HashScheme: "vllm", PrefixHash: []byte("p"), + Tenant: controlplaneapi.ProbeTenantID, Model: "m", HashScheme: "vllm", PrefixHash: []byte("p"), }) if len(scores) != 1 || scores[0].ReplicaID != "real" { t.Fatalf("external CacheEvent against probe tenant disturbed reserved state: %+v", scores) @@ -1487,7 +1489,7 @@ func TestMicrosToTime(t *testing.T) { // the index lookup and returns the normal PREFIX_MATCH response. func TestLookupRouteAboveMinimumPrefixTokensProceedsToLookup(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", MinimumPrefixTokens: 50}, }) svc.index.Ingest(index.Update{ @@ -1525,7 +1527,7 @@ func TestLookupRouteAboveMinimumPrefixTokensProceedsToLookup(t *testing.T) { func TestLookupRouteBelowMinimumPrefixTokensReturnsNoHintWithoutTouchingIndex(t *testing.T) { svc := newTestService() fal := false - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", MinimumPrefixTokens: 200, AffinityRouting: &fal}, }) svc.lookupFn = func(index.LookupRequest) index.LookupResult { @@ -1581,7 +1583,7 @@ func TestLookupRouteReturnsTimeoutWhenCallerDeadlineBreached(t *testing.T) { // select pseudorandom choice could leak stale scores as PREFIX_MATCH. func TestLookupRouteReturnsTimeoutEvenIfLookupRacesPastDeadline(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", LookupTimeoutMs: 5}, }) // Lookup deliberately exceeds the budget before returning a hit. @@ -1615,7 +1617,7 @@ func TestLookupRouteReturnsTimeoutEvenIfLookupRacesPastDeadline(t *testing.T) { // reason_code:TIMEOUT. func TestLookupRouteBoundsWallTimeWhenLookupBlocks(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", LookupTimeoutMs: 20}, }) @@ -1662,7 +1664,7 @@ func TestLookupRouteAppliesPolicyTimeoutBudget(t *testing.T) { // deterministically by TestLookupRouteReturnsTimeoutEvenIfLookupRacesPastDeadline // and TestLookupRouteBoundsWallTimeWhenLookupBlocks, which inject a lookup // that overruns the budget by a large, jitter-proof margin. - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", LookupTimeoutMs: 1000}, }) svc.index.Ingest(index.Update{ @@ -1684,7 +1686,7 @@ func TestLookupRouteAppliesPolicyTimeoutBudget(t *testing.T) { func TestLookupRouteUnaffectedByPolicyForUnknownTenant(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", MinimumPrefixTokens: 200, LookupTimeoutMs: 1}, }) // TokenCount=128 keeps the realized match above the server-wide @@ -1773,7 +1775,7 @@ func TestLookupRouteChainReturnsPartialPrefixMatch(t *testing.T) { // test scope to the request-side gate alone, matching its name. func TestLookupRouteAboveMinimumPrefixTokensViaChainCounts(t *testing.T) { svc := newTestService() - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", MinimumPrefixTokens: 32, MinimumMatchedTokens: 0}, }) hashes := [][]byte{[]byte("b1"), []byte("b2"), []byte("b3")} @@ -1803,7 +1805,7 @@ func TestLookupRouteAboveMinimumPrefixTokensViaChainCounts(t *testing.T) { func TestLookupRouteBelowMinimumPrefixTokensViaChainCounts(t *testing.T) { svc := newTestService() fal := false - svc.policies.Replace([]ResolvedPolicy{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{ {Namespace: "team-a", MinimumPrefixTokens: 200, AffinityRouting: &fal}, }) svc.lookupFn = func(index.LookupRequest) index.LookupResult { @@ -1853,9 +1855,9 @@ func TestLookupRouteRequireChainGateReturnsPolicyReason(t *testing.T) { t.Run(tc.name, func(t *testing.T) { svc := newTestService() reqChain := true - svc.policies.Replace([]ResolvedPolicy{{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{ Namespace: "team-a", - Strategy: &ResolvedLookupStrategy{RequireChain: &reqChain}, + Strategy: &controlplaneapi.ResolvedLookupStrategy{RequireChain: &reqChain}, }}) svc.lookupFn = func(index.LookupRequest) index.LookupResult { t.Fatal("index lookup should not run when policy requires a carried chain and request lacks one") @@ -1965,9 +1967,9 @@ func TestLookupRouteDisableChainMatchingUsesExactPrefixHash(t *testing.T) { svc.tokenizer = tc.tokenizer } enableChain := false - svc.policies.Replace([]ResolvedPolicy{{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{ Namespace: "team-a", - Strategy: &ResolvedLookupStrategy{EnableChainMatching: &enableChain}, + Strategy: &controlplaneapi.ResolvedLookupStrategy{EnableChainMatching: &enableChain}, }}) var got index.LookupRequest var called bool @@ -2016,10 +2018,10 @@ func TestLookupRouteDisableChainMatchingMinPrefixIgnoresChainCounts(t *testing.T svc := newTestService() enableChain := false affDisabled := false - svc.policies.Replace([]ResolvedPolicy{{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{ Namespace: "team-a", MinimumPrefixTokens: 100, - Strategy: &ResolvedLookupStrategy{EnableChainMatching: &enableChain}, + Strategy: &controlplaneapi.ResolvedLookupStrategy{EnableChainMatching: &enableChain}, // affinity Disabled so the below-threshold request still short-circuits // to NO_HINT without an index lookup (affinity Enabled would run the // full lookup to classify diagnostics before any fallback). @@ -2054,9 +2056,9 @@ func TestLookupRouteDisableTenantHotDowngradesToNoHint(t *testing.T) { svc := newTestService() enableTenantHot := false affDisabled := false - svc.policies.Replace([]ResolvedPolicy{{ + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{ Namespace: "t", - Strategy: &ResolvedLookupStrategy{EnableTenantHot: &enableTenantHot}, + Strategy: &controlplaneapi.ResolvedLookupStrategy{EnableTenantHot: &enableTenantHot}, // affinity Disabled so the tenant-hot downgrade surfaces as NO_HINT in // isolation (affinity Enabled would pick up the StrategyNone result and // return AFFINITY_HINT — covered by the affinity tests). @@ -2173,7 +2175,7 @@ func TestLookupRouteChainNoOverlapNeverFallsThroughToTenantHot(t *testing.T) { // (covered in affinity_routing_test.go) but orthogonal to the chain // vs TENANT_HOT invariant this test pins. fal := false - svc.policies.Replace([]ResolvedPolicy{{Namespace: "t", AffinityRouting: &fal}}) + svc.policies.Replace([]controlplaneapi.ResolvedPolicy{{Namespace: "t", AffinityRouting: &fal}}) svc.index.Ingest(index.Update{ ReplicaID: "warm-r", Model: "m", Tenant: "t", HashScheme: "vllm", diff --git a/internal/server/snapshot.go b/internal/server/snapshot.go new file mode 100644 index 00000000..7318bd15 --- /dev/null +++ b/internal/server/snapshot.go @@ -0,0 +1,52 @@ +// SPDX-FileCopyrightText: 2026 The inference-cache Authors +// +// SPDX-License-Identifier: Apache-2.0 + +package server + +import ( + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" +) + +// snapshotForControlPlane maps the index-owned domain snapshot to the private +// controller/server HTTP DTO. Keep this field-by-field: aliases would make a +// mutable index refactor an accidental wire-contract change. +func snapshotForControlPlane(src index.Snapshot) controlplaneapi.Snapshot { + dst := controlplaneapi.Snapshot{ + TotalPrefixes: src.TotalPrefixes, + HotPrefixes: src.HotPrefixes, + } + if src.Replicas != nil { + dst.Replicas = make([]controlplaneapi.ReplicaSnapshot, len(src.Replicas)) + for i, replica := range src.Replicas { + dst.Replicas[i] = controlplaneapi.ReplicaSnapshot{ + ReplicaID: replica.ReplicaID, + Tenant: replica.Tenant, + CacheMemoryBytes: replica.CacheMemoryBytes, + HitRate: replica.HitRate, + Pressure: replica.Pressure, + LastUpdate: replica.LastUpdate, + PrefixCount: replica.PrefixCount, + LastEventAt: replica.LastEventAt, + StatsReported: replica.StatsReported, + T2HitTokens: replica.T2HitTokens, + T2QueryTokens: replica.T2QueryTokens, + } + } + } + if src.Tenants != nil { + dst.Tenants = make([]controlplaneapi.TenantSnapshot, len(src.Tenants)) + for i, tenant := range src.Tenants { + dst.Tenants[i] = controlplaneapi.TenantSnapshot{ + TenantID: tenant.TenantID, + IndexEntries: tenant.IndexEntries, + HitRate: tenant.HitRate, + HitRateReported: tenant.HitRateReported, + // MemoryUsed is intentionally omitted: the deprecated wire key + // remains present through its non-omitempty JSON tag, with value 0. + } + } + } + return dst +} diff --git a/internal/server/snapshot_test.go b/internal/server/snapshot_test.go new file mode 100644 index 00000000..1decccd0 --- /dev/null +++ b/internal/server/snapshot_test.go @@ -0,0 +1,75 @@ +// SPDX-FileCopyrightText: 2026 The inference-cache Authors +// +// SPDX-License-Identifier: Apache-2.0 + +package server + +import ( + "reflect" + "testing" + "time" + + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/index" +) + +func TestSnapshotForControlPlaneMapsEveryField(t *testing.T) { + lastUpdate := time.Unix(1_700_000_000, 0).UTC() + lastEvent := lastUpdate.Add(time.Minute) + src := index.Snapshot{ + TotalPrefixes: 11, + HotPrefixes: 2, + Replicas: []index.ReplicaSnapshot{{ + ReplicaID: "replica-a", + Tenant: "tenant-a", + CacheMemoryBytes: 4096, + HitRate: 0.75, + Pressure: 0.25, + LastUpdate: lastUpdate, + PrefixCount: 7, + LastEventAt: lastEvent, + StatsReported: true, + T2HitTokens: 600, + T2QueryTokens: 1000, + }}, + Tenants: []index.TenantSnapshot{{ + TenantID: "tenant-a", + IndexEntries: 11, + HitRate: 0.75, + HitRateReported: true, + MemoryUsed: 0, + }}, + } + + want := controlplaneapi.Snapshot{ + TotalPrefixes: 11, + HotPrefixes: 2, + Replicas: []controlplaneapi.ReplicaSnapshot{{ + ReplicaID: "replica-a", + Tenant: "tenant-a", + CacheMemoryBytes: 4096, + HitRate: 0.75, + Pressure: 0.25, + LastUpdate: lastUpdate, + PrefixCount: 7, + LastEventAt: lastEvent, + StatsReported: true, + T2HitTokens: 600, + T2QueryTokens: 1000, + }}, + Tenants: []controlplaneapi.TenantSnapshot{{ + TenantID: "tenant-a", + IndexEntries: 11, + HitRate: 0.75, + HitRateReported: true, + MemoryUsed: 0, + }}, + } + + if got := snapshotForControlPlane(src); !reflect.DeepEqual(got, want) { + t.Fatalf("snapshot mapping = %+v, want %+v", got, want) + } + if got := snapshotForControlPlane(index.Snapshot{}); got.Replicas != nil || got.Tenants != nil { + t.Fatalf("zero snapshot changed nil-slice wire semantics: %+v", got) + } +} diff --git a/pkg/server/tenant_quota_test.go b/internal/server/tenant_quota_test.go similarity index 80% rename from pkg/server/tenant_quota_test.go rename to internal/server/tenant_quota_test.go index f2fd8a6c..6534c24d 100644 --- a/pkg/server/tenant_quota_test.go +++ b/internal/server/tenant_quota_test.go @@ -10,6 +10,8 @@ import ( "net/http" "net/http/httptest" "testing" + + "github.com/cachebox-project/inference-cache/internal/controlplaneapi" ) // TestTenantQuotaExemptsProbeTenant pins the server-internal defense against @@ -23,12 +25,12 @@ import ( // govern it. func TestTenantQuotaExemptsProbeTenant(t *testing.T) { store := NewPolicyStore() - store.ReplaceSnapshot(nil, []ResolvedTenant{ - {TenantID: ProbeTenantID, MaxIndexEntries: 0, IsolationMode: "Fairness"}, + store.ReplaceSnapshot(nil, []controlplaneapi.ResolvedTenant{ + {TenantID: controlplaneapi.ProbeTenantID, MaxIndexEntries: 0, IsolationMode: "Fairness"}, // A normal tenant still gets its quota honored. {TenantID: "team-a", MaxIndexEntries: 1000}, }) - if _, ok := store.TenantQuota(ProbeTenantID); ok { + if _, ok := store.TenantQuota(controlplaneapi.ProbeTenantID); ok { t.Fatal("TenantQuota(probe tenant) reported a quota; want exemption (fail open)") } if max, ok := store.TenantQuota("team-a"); !ok || max != 1000 { @@ -48,8 +50,8 @@ func TestTenantQuotaExemptsProbeTenant(t *testing.T) { // version mismatch outside the accepted band is rejected with a clear // "unsupported version" rather than a decode error. func TestPolicyPropagationVersionIsV7(t *testing.T) { - if PolicyPropagationVersion != 7 { - t.Fatalf("PolicyPropagationVersion = %d, want 7", PolicyPropagationVersion) + if controlplaneapi.PolicyPropagationVersion != 7 { + t.Fatalf("controlplaneapi.PolicyPropagationVersion = %d, want 7", controlplaneapi.PolicyPropagationVersion) } // PolicyMinimumAcceptedVersion bounds the lenience window for older bodies. // v3, v4, and v5 must be accepted so a server-first rollout does not drop @@ -57,8 +59,8 @@ func TestPolicyPropagationVersionIsV7(t *testing.T) { // (normalizePolicySnapshotForVersion fills the missing fields with their // server-side defaults); bodies below v3 are still rejected — there is no // documented path to normalize the older Tenants / Eviction shapes. - if PolicyMinimumAcceptedVersion != 3 { - t.Fatalf("PolicyMinimumAcceptedVersion = %d, want 3", PolicyMinimumAcceptedVersion) + if controlplaneapi.PolicyMinimumAcceptedVersion != 3 { + t.Fatalf("controlplaneapi.PolicyMinimumAcceptedVersion = %d, want 3", controlplaneapi.PolicyMinimumAcceptedVersion) } } @@ -107,24 +109,24 @@ func TestPolicySnapshotV3AcceptedWithFloorDefault(t *testing.T) { // The v3-missing fields must be normalized to their server-side defaults — // otherwise a server-first rollout silently disables the floors for every // namespace that already had a CR. - if pA.MinimumMatchedTokens != DefaultMinimumMatchedTokens { - t.Fatalf("team-a MinimumMatchedTokens after v3 push = %d, want DefaultMinimumMatchedTokens (%d) — v3 → v4 matched-tokens normalization missing", pA.MinimumMatchedTokens, DefaultMinimumMatchedTokens) + if pA.MinimumMatchedTokens != controlplaneapi.DefaultMinimumMatchedTokens { + t.Fatalf("team-a MinimumMatchedTokens after v3 push = %d, want controlplaneapi.DefaultMinimumMatchedTokens (%d) — v3 → v4 matched-tokens normalization missing", pA.MinimumMatchedTokens, controlplaneapi.DefaultMinimumMatchedTokens) } if pA.RoutingFloorScore == nil { - t.Fatalf("team-a RoutingFloorScore after v3 push is nil — v3 → v5 routing-floor normalization missing (must synthesize DefaultRoutingFloorScore)") + t.Fatalf("team-a RoutingFloorScore after v3 push is nil — v3 → v5 routing-floor normalization missing (must synthesize controlplaneapi.DefaultRoutingFloorScore)") } - if *pA.RoutingFloorScore != DefaultRoutingFloorScore { - t.Fatalf("team-a RoutingFloorScore after v3 push = %v, want DefaultRoutingFloorScore (%v) — v3 → v5 routing-floor normalization synthesized the wrong value", *pA.RoutingFloorScore, DefaultRoutingFloorScore) + if *pA.RoutingFloorScore != controlplaneapi.DefaultRoutingFloorScore { + t.Fatalf("team-a RoutingFloorScore after v3 push = %v, want controlplaneapi.DefaultRoutingFloorScore (%v) — v3 → v5 routing-floor normalization synthesized the wrong value", *pA.RoutingFloorScore, controlplaneapi.DefaultRoutingFloorScore) } if !store.ChainMatchingEnabled("team-a") || store.ChainRequired("team-a") || !store.TenantHotEnabled("team-a") { t.Fatalf("team-a strategy defaults after v3 push = chain=%v require=%v tenantHot=%v, want true/false/true", store.ChainMatchingEnabled("team-a"), store.ChainRequired("team-a"), store.TenantHotEnabled("team-a")) } if pA.AffinityRouting == nil { - t.Fatalf("team-a AffinityRouting after v3 push is nil — v3 → v7 affinity-routing normalization missing (must synthesize DefaultAffinityRoutingEnabled)") + t.Fatalf("team-a AffinityRouting after v3 push is nil — v3 → v7 affinity-routing normalization missing (must synthesize controlplaneapi.DefaultAffinityRoutingEnabled)") } - if *pA.AffinityRouting != DefaultAffinityRoutingEnabled { - t.Fatalf("team-a AffinityRouting after v3 push = %v, want DefaultAffinityRoutingEnabled (%v) — v3 → v7 affinity-routing normalization synthesized the wrong value", *pA.AffinityRouting, DefaultAffinityRoutingEnabled) + if *pA.AffinityRouting != controlplaneapi.DefaultAffinityRoutingEnabled { + t.Fatalf("team-a AffinityRouting after v3 push = %v, want controlplaneapi.DefaultAffinityRoutingEnabled (%v) — v3 → v7 affinity-routing normalization synthesized the wrong value", *pA.AffinityRouting, controlplaneapi.DefaultAffinityRoutingEnabled) } // Every other knob the v3 body carried must reach the store unchanged. if pA.EvictionTTL != 900_000_000_000 || pA.MinimumPrefixTokens != 32 || pA.LookupTimeoutMs != 25 || pA.Eviction != "lfu" { @@ -135,18 +137,18 @@ func TestPolicySnapshotV3AcceptedWithFloorDefault(t *testing.T) { if !ok { t.Fatal("team-b policy missing from store after v3 push") } - if pB.MinimumMatchedTokens != DefaultMinimumMatchedTokens { - t.Fatalf("team-b MinimumMatchedTokens after v3 push = %d, want DefaultMinimumMatchedTokens (%d)", pB.MinimumMatchedTokens, DefaultMinimumMatchedTokens) + if pB.MinimumMatchedTokens != controlplaneapi.DefaultMinimumMatchedTokens { + t.Fatalf("team-b MinimumMatchedTokens after v3 push = %d, want controlplaneapi.DefaultMinimumMatchedTokens (%d)", pB.MinimumMatchedTokens, controlplaneapi.DefaultMinimumMatchedTokens) } - if pB.RoutingFloorScore == nil || *pB.RoutingFloorScore != DefaultRoutingFloorScore { - t.Fatalf("team-b RoutingFloorScore after v3 push = %v, want &DefaultRoutingFloorScore (%v)", pB.RoutingFloorScore, DefaultRoutingFloorScore) + if pB.RoutingFloorScore == nil || *pB.RoutingFloorScore != controlplaneapi.DefaultRoutingFloorScore { + t.Fatalf("team-b RoutingFloorScore after v3 push = %v, want &controlplaneapi.DefaultRoutingFloorScore (%v)", pB.RoutingFloorScore, controlplaneapi.DefaultRoutingFloorScore) } if !store.ChainMatchingEnabled("team-b") || store.ChainRequired("team-b") || !store.TenantHotEnabled("team-b") { t.Fatalf("team-b strategy defaults after v3 push = chain=%v require=%v tenantHot=%v, want true/false/true", store.ChainMatchingEnabled("team-b"), store.ChainRequired("team-b"), store.TenantHotEnabled("team-b")) } - if pB.AffinityRouting == nil || *pB.AffinityRouting != DefaultAffinityRoutingEnabled { - t.Fatalf("team-b AffinityRouting after v3 push = %v, want &DefaultAffinityRoutingEnabled (%v)", pB.AffinityRouting, DefaultAffinityRoutingEnabled) + if pB.AffinityRouting == nil || *pB.AffinityRouting != controlplaneapi.DefaultAffinityRoutingEnabled { + t.Fatalf("team-b AffinityRouting after v3 push = %v, want &controlplaneapi.DefaultAffinityRoutingEnabled (%v)", pB.AffinityRouting, controlplaneapi.DefaultAffinityRoutingEnabled) } // Tenant quotas survive the version normalization unchanged. @@ -215,8 +217,8 @@ func TestPolicySnapshotV4ExplicitZeroPreservedAndRoutingFloorNormalized(t *testi if p.RoutingFloorScore == nil { t.Fatal("RoutingFloorScore after v4 push is nil — v4 → v5 routing-floor normalization missing") } - if *p.RoutingFloorScore != DefaultRoutingFloorScore { - t.Fatalf("RoutingFloorScore after v4 push = %v, want DefaultRoutingFloorScore (%v) — v4 → v5 routing-floor normalization synthesized the wrong value", *p.RoutingFloorScore, DefaultRoutingFloorScore) + if *p.RoutingFloorScore != controlplaneapi.DefaultRoutingFloorScore { + t.Fatalf("RoutingFloorScore after v4 push = %v, want controlplaneapi.DefaultRoutingFloorScore (%v) — v4 → v5 routing-floor normalization synthesized the wrong value", *p.RoutingFloorScore, controlplaneapi.DefaultRoutingFloorScore) } if !store.ChainMatchingEnabled("raw-recall") || store.ChainRequired("raw-recall") || !store.TenantHotEnabled("raw-recall") { t.Fatalf("strategy defaults after v4 push = chain=%v require=%v tenantHot=%v, want true/false/true", @@ -229,8 +231,8 @@ func TestPolicySnapshotV4ExplicitZeroPreservedAndRoutingFloorNormalized(t *testi if p.AffinityRouting == nil { t.Fatal("AffinityRouting after v4 push is nil — v4 → v7 affinity normalization missing") } - if *p.AffinityRouting != DefaultAffinityRoutingEnabled { - t.Fatalf("AffinityRouting after v4 push = %v, want DefaultAffinityRoutingEnabled (%v) — v4 → v7 affinity normalization synthesized the wrong value", *p.AffinityRouting, DefaultAffinityRoutingEnabled) + if *p.AffinityRouting != controlplaneapi.DefaultAffinityRoutingEnabled { + t.Fatalf("AffinityRouting after v4 push = %v, want controlplaneapi.DefaultAffinityRoutingEnabled (%v) — v4 → v7 affinity normalization synthesized the wrong value", *p.AffinityRouting, controlplaneapi.DefaultAffinityRoutingEnabled) } } @@ -252,9 +254,9 @@ func TestPolicySnapshotV5ExplicitRoutingFloorZeroPreserved(t *testing.T) { defer srv.Close() zero := float32(0) - body, err := json.Marshal(PolicySnapshot{ + body, err := json.Marshal(controlplaneapi.PolicySnapshot{ Version: 5, // literal v5 — must reach the store byte-for-byte even on a v7 server. - Policies: []ResolvedPolicy{ + Policies: []controlplaneapi.ResolvedPolicy{ {Namespace: "raw-recall", RoutingFloorScore: &zero}, }, }) @@ -287,11 +289,11 @@ func TestPolicySnapshotExplicitStrategyPreserved(t *testing.T) { enable := true disable := false require := true - body, err := json.Marshal(PolicySnapshot{ - Version: PolicyPropagationVersion, - Policies: []ResolvedPolicy{{ + body, err := json.Marshal(controlplaneapi.PolicySnapshot{ + Version: controlplaneapi.PolicyPropagationVersion, + Policies: []controlplaneapi.ResolvedPolicy{{ Namespace: "strict", - Strategy: &ResolvedLookupStrategy{ + Strategy: &controlplaneapi.ResolvedLookupStrategy{ EnableChainMatching: &enable, RequireChain: &require, EnableTenantHot: &disable, @@ -332,7 +334,7 @@ func TestPolicySnapshotVersionTooOldRejected(t *testing.T) { } _ = resp.Body.Close() if resp.StatusCode != http.StatusBadRequest { - t.Fatalf("v2 body status = %d, want 400 (below PolicyMinimumAcceptedVersion)", resp.StatusCode) + t.Fatalf("v2 body status = %d, want 400 (below controlplaneapi.PolicyMinimumAcceptedVersion)", resp.StatusCode) } } @@ -341,10 +343,10 @@ func TestPolicySnapshotRoundTripCarriesPoliciesAndTenants(t *testing.T) { srv := httptest.NewServer(NewPolicyHTTPHandler(store)) defer srv.Close() - snap := PolicySnapshot{ - Version: PolicyPropagationVersion, - Policies: []ResolvedPolicy{{Namespace: "team-a", MinimumPrefixTokens: 16}}, - Tenants: []ResolvedTenant{ + snap := controlplaneapi.PolicySnapshot{ + Version: controlplaneapi.PolicyPropagationVersion, + Policies: []controlplaneapi.ResolvedPolicy{{Namespace: "team-a", MinimumPrefixTokens: 16}}, + Tenants: []controlplaneapi.ResolvedTenant{ {TenantID: "team-a", MaxIndexEntries: 1000, IsolationMode: "Fairness"}, {TenantID: "team-b", MaxIndexEntries: 0}, }, @@ -387,7 +389,7 @@ func TestPolicySnapshotRoundTripCarriesPoliciesAndTenants(t *testing.T) { func TestReplaceSnapshotRevertsRemovedTenant(t *testing.T) { store := NewPolicyStore() - store.ReplaceSnapshot(nil, []ResolvedTenant{{TenantID: "team-a", MaxIndexEntries: 5}}) + store.ReplaceSnapshot(nil, []controlplaneapi.ResolvedTenant{{TenantID: "team-a", MaxIndexEntries: 5}}) if _, ok := store.TenantQuota("team-a"); !ok { t.Fatal("team-a quota should be present after first push") } @@ -400,7 +402,7 @@ func TestReplaceSnapshotRevertsRemovedTenant(t *testing.T) { func TestReplaceSnapshotDropsEmptyTenantID(t *testing.T) { store := NewPolicyStore() - store.ReplaceSnapshot(nil, []ResolvedTenant{{TenantID: "", MaxIndexEntries: 5}}) + store.ReplaceSnapshot(nil, []controlplaneapi.ResolvedTenant{{TenantID: "", MaxIndexEntries: 5}}) if _, ok := store.TenantQuota(""); ok { t.Fatal("an empty tenant ID must not be stored (would shadow empty-tenant lookups)") } @@ -413,7 +415,7 @@ func TestReplaceSnapshotDropsEmptyTenantID(t *testing.T) { // ReplaceSnapshot must clamp it to the design minimum of 0 (admit nothing). func TestReplaceSnapshotClampsNegativeBudget(t *testing.T) { store := NewPolicyStore() - store.ReplaceSnapshot(nil, []ResolvedTenant{{TenantID: "team-a", MaxIndexEntries: -1}}) + store.ReplaceSnapshot(nil, []controlplaneapi.ResolvedTenant{{TenantID: "team-a", MaxIndexEntries: -1}}) max, ok := store.TenantQuota("team-a") if !ok { t.Fatal("a negative budget must still register an (enforced) quota, not fail open") diff --git a/pkg/server/tls.go b/internal/server/tls.go similarity index 100% rename from pkg/server/tls.go rename to internal/server/tls.go diff --git a/pkg/server/tls_test.go b/internal/server/tls_test.go similarity index 99% rename from pkg/server/tls_test.go rename to internal/server/tls_test.go index 9eb866aa..98fe036b 100644 --- a/pkg/server/tls_test.go +++ b/internal/server/tls_test.go @@ -27,7 +27,7 @@ import ( "google.golang.org/grpc/credentials/insecure" healthpb "google.golang.org/grpc/health/grpc_health_v1" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) // genCertPEM mints a short-lived self-signed certificate for dnsName with the diff --git a/pkg/adapters/engine/config.go b/internal/subscriber/config.go similarity index 99% rename from pkg/adapters/engine/config.go rename to internal/subscriber/config.go index cda73d88..a6ea27ee 100644 --- a/pkg/adapters/engine/config.go +++ b/internal/subscriber/config.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "fmt" diff --git a/pkg/adapters/engine/coverage_test.go b/internal/subscriber/coverage_test.go similarity index 97% rename from pkg/adapters/engine/coverage_test.go rename to internal/subscriber/coverage_test.go index d4b0ce70..7ee11b5d 100644 --- a/pkg/adapters/engine/coverage_test.go +++ b/internal/subscriber/coverage_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "context" @@ -18,7 +18,7 @@ import ( "google.golang.org/grpc/status" "google.golang.org/grpc/test/bufconn" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) func TestReporterOptionsApplied(t *testing.T) { diff --git a/pkg/adapters/engine/doc.go b/internal/subscriber/doc.go similarity index 74% rename from pkg/adapters/engine/doc.go rename to internal/subscriber/doc.go index 4dd604e5..e344d7c8 100644 --- a/pkg/adapters/engine/doc.go +++ b/internal/subscriber/doc.go @@ -2,8 +2,9 @@ // // SPDX-License-Identifier: Apache-2.0 -// Package engine is the KV-event subscriber. It runs as a sidecar next to a vLLM -// engine replica, subscribes to the engine's KV cache events over ZMQ, decodes +// Package subscriber implements KV-event ingestion for the kvevent-subscriber +// sidecar. It runs next to a vLLM engine replica, subscribes to the engine's +// KV cache events over ZMQ, decodes // them, and reports cache state to the inferencecache-server over gRPC. // // Two independent paths share one gRPC client: @@ -16,5 +17,5 @@ // // Metadata only — never KV tensors or prompt text. Fail-soft on both paths: // neither a ZMQ drop nor a scrape failure can stall the engine. The package is -// built into the kvevent-subscriber binary (cmd/kvevent-subscriber). -package engine +// private to the kvevent-subscriber binary (cmd/kvevent-subscriber). +package subscriber diff --git a/pkg/adapters/engine/events.go b/internal/subscriber/events.go similarity index 99% rename from pkg/adapters/engine/events.go rename to internal/subscriber/events.go index 4c02222c..300a90d0 100644 --- a/pkg/adapters/engine/events.go +++ b/internal/subscriber/events.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "encoding/binary" diff --git a/pkg/adapters/engine/events_test.go b/internal/subscriber/events_test.go similarity index 99% rename from pkg/adapters/engine/events_test.go rename to internal/subscriber/events_test.go index fb9d1c46..95d1c5b4 100644 --- a/pkg/adapters/engine/events_test.go +++ b/internal/subscriber/events_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "bytes" diff --git a/pkg/adapters/engine/forwarder.go b/internal/subscriber/forwarder.go similarity index 99% rename from pkg/adapters/engine/forwarder.go rename to internal/subscriber/forwarder.go index 9c3126ec..707ff81c 100644 --- a/pkg/adapters/engine/forwarder.go +++ b/internal/subscriber/forwarder.go @@ -2,14 +2,14 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "context" "log/slog" "time" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) // Reporter forwards decoded KV-cache events to the policy server over gRPC. diff --git a/pkg/adapters/engine/forwarder_test.go b/internal/subscriber/forwarder_test.go similarity index 99% rename from pkg/adapters/engine/forwarder_test.go rename to internal/subscriber/forwarder_test.go index 861433fc..4c41f5e6 100644 --- a/pkg/adapters/engine/forwarder_test.go +++ b/internal/subscriber/forwarder_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "bytes" @@ -17,8 +17,8 @@ import ( "google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/test/bufconn" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" ) // recordingServer captures what the Reporter sends, over a real gRPC connection. diff --git a/pkg/adapters/engine/lora_adapter_test.go b/internal/subscriber/lora_adapter_test.go similarity index 99% rename from pkg/adapters/engine/lora_adapter_test.go rename to internal/subscriber/lora_adapter_test.go index 4eea9934..bbcafb78 100644 --- a/pkg/adapters/engine/lora_adapter_test.go +++ b/internal/subscriber/lora_adapter_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "testing" diff --git a/pkg/adapters/engine/mapper.go b/internal/subscriber/mapper.go similarity index 97% rename from pkg/adapters/engine/mapper.go rename to internal/subscriber/mapper.go index c69727e3..3723a79b 100644 --- a/pkg/adapters/engine/mapper.go +++ b/internal/subscriber/mapper.go @@ -2,10 +2,10 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) // This file stamps the replica/model/tenant/hash_scheme identity onto the gRPC diff --git a/pkg/adapters/engine/mapper_test.go b/internal/subscriber/mapper_test.go similarity index 94% rename from pkg/adapters/engine/mapper_test.go rename to internal/subscriber/mapper_test.go index dfd4cbf4..c59497b3 100644 --- a/pkg/adapters/engine/mapper_test.go +++ b/internal/subscriber/mapper_test.go @@ -2,13 +2,13 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "bytes" "testing" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) func testConfig() Config { diff --git a/pkg/adapters/engine/metrics_scraper.go b/internal/subscriber/metrics_scraper.go similarity index 99% rename from pkg/adapters/engine/metrics_scraper.go rename to internal/subscriber/metrics_scraper.go index eee9058e..a461ba49 100644 --- a/pkg/adapters/engine/metrics_scraper.go +++ b/internal/subscriber/metrics_scraper.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "context" @@ -17,7 +17,7 @@ import ( "github.com/prometheus/common/expfmt" "github.com/prometheus/common/model" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) // CacheTier selects which vLLM cache-usage gauge feeds cache_memory_bytes. diff --git a/pkg/adapters/engine/metrics_scraper_test.go b/internal/subscriber/metrics_scraper_test.go similarity index 99% rename from pkg/adapters/engine/metrics_scraper_test.go rename to internal/subscriber/metrics_scraper_test.go index 23f6a50c..71d4ff8a 100644 --- a/pkg/adapters/engine/metrics_scraper_test.go +++ b/internal/subscriber/metrics_scraper_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "context" diff --git a/pkg/adapters/engine/positional.go b/internal/subscriber/positional.go similarity index 98% rename from pkg/adapters/engine/positional.go rename to internal/subscriber/positional.go index e5524fd9..802af6af 100644 --- a/pkg/adapters/engine/positional.go +++ b/internal/subscriber/positional.go @@ -2,11 +2,11 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" "github.com/cachebox-project/inference-cache/pkg/fingerprint" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" ) // positionalIndex turns the engine's incremental, parent-chained block-store diff --git a/pkg/adapters/engine/positional_test.go b/internal/subscriber/positional_test.go similarity index 99% rename from pkg/adapters/engine/positional_test.go rename to internal/subscriber/positional_test.go index 93718755..2e32ccec 100644 --- a/pkg/adapters/engine/positional_test.go +++ b/internal/subscriber/positional_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "encoding/binary" diff --git a/pkg/adapters/engine/sglang_wire_test.go b/internal/subscriber/sglang_wire_test.go similarity index 99% rename from pkg/adapters/engine/sglang_wire_test.go rename to internal/subscriber/sglang_wire_test.go index 345e514f..c82b17a2 100644 --- a/pkg/adapters/engine/sglang_wire_test.go +++ b/internal/subscriber/sglang_wire_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "bytes" diff --git a/pkg/adapters/engine/stats_reporter.go b/internal/subscriber/stats_reporter.go similarity index 98% rename from pkg/adapters/engine/stats_reporter.go rename to internal/subscriber/stats_reporter.go index ec0af707..7fdb31de 100644 --- a/pkg/adapters/engine/stats_reporter.go +++ b/internal/subscriber/stats_reporter.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "context" @@ -12,7 +12,7 @@ import ( "log/slog" "time" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) // statsScraper is the dependency a StatsReporter takes. *MetricsScraper diff --git a/pkg/adapters/engine/stats_reporter_test.go b/internal/subscriber/stats_reporter_test.go similarity index 99% rename from pkg/adapters/engine/stats_reporter_test.go rename to internal/subscriber/stats_reporter_test.go index 7d3132d2..5c38f140 100644 --- a/pkg/adapters/engine/stats_reporter_test.go +++ b/internal/subscriber/stats_reporter_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "bytes" @@ -19,7 +19,7 @@ import ( "google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/test/bufconn" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" ) // stubScraper returns the configured stats (or error) and counts calls. diff --git a/pkg/adapters/engine/subscriber.go b/internal/subscriber/subscriber.go similarity index 99% rename from pkg/adapters/engine/subscriber.go rename to internal/subscriber/subscriber.go index 05b67d01..696e6362 100644 --- a/pkg/adapters/engine/subscriber.go +++ b/internal/subscriber/subscriber.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "context" diff --git a/pkg/adapters/engine/subscriber_test.go b/internal/subscriber/subscriber_test.go similarity index 99% rename from pkg/adapters/engine/subscriber_test.go rename to internal/subscriber/subscriber_test.go index 29e77b49..22fa1212 100644 --- a/pkg/adapters/engine/subscriber_test.go +++ b/internal/subscriber/subscriber_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine +package subscriber import ( "context" diff --git a/pkg/adapters/engine/testdata/vllm_metrics_cpu.txt b/internal/subscriber/testdata/vllm_metrics_cpu.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_cpu.txt rename to internal/subscriber/testdata/vllm_metrics_cpu.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_cpu_tick2.txt b/internal/subscriber/testdata/vllm_metrics_cpu_tick2.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_cpu_tick2.txt rename to internal/subscriber/testdata/vllm_metrics_cpu_tick2.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_gpu.txt b/internal/subscriber/testdata/vllm_metrics_gpu.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_gpu.txt rename to internal/subscriber/testdata/vllm_metrics_gpu.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_legacy_cpu.txt b/internal/subscriber/testdata/vllm_metrics_legacy_cpu.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_legacy_cpu.txt rename to internal/subscriber/testdata/vllm_metrics_legacy_cpu.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_multimodel.txt b/internal/subscriber/testdata/vllm_metrics_multimodel.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_multimodel.txt rename to internal/subscriber/testdata/vllm_metrics_multimodel.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_openmetrics.txt b/internal/subscriber/testdata/vllm_metrics_openmetrics.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_openmetrics.txt rename to internal/subscriber/testdata/vllm_metrics_openmetrics.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_openmetrics_tick2.txt b/internal/subscriber/testdata/vllm_metrics_openmetrics_tick2.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_openmetrics_tick2.txt rename to internal/subscriber/testdata/vllm_metrics_openmetrics_tick2.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_partial.txt b/internal/subscriber/testdata/vllm_metrics_partial.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_partial.txt rename to internal/subscriber/testdata/vllm_metrics_partial.txt diff --git a/pkg/adapters/engine/testdata/vllm_metrics_t2.txt b/internal/subscriber/testdata/vllm_metrics_t2.txt similarity index 100% rename from pkg/adapters/engine/testdata/vllm_metrics_t2.txt rename to internal/subscriber/testdata/vllm_metrics_t2.txt diff --git a/pkg/adapters/engine/wire_test.go b/internal/subscriber/wire_test.go similarity index 84% rename from pkg/adapters/engine/wire_test.go rename to internal/subscriber/wire_test.go index 3e761b81..9a7fe6ee 100644 --- a/pkg/adapters/engine/wire_test.go +++ b/internal/subscriber/wire_test.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package engine_test +package subscriber_test // End-to-end wire test: the StatsReporter scrapes a synthetic /metrics endpoint, // emits stats-only CacheStateUpdates against the real policy server, and the @@ -27,10 +27,10 @@ import ( "google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/test/bufconn" - "github.com/cachebox-project/inference-cache/pkg/adapters/engine" - "github.com/cachebox-project/inference-cache/pkg/index" - "github.com/cachebox-project/inference-cache/pkg/server" - icpb "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1" + icpb "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1" + controlplaneapi "github.com/cachebox-project/inference-cache/internal/controlplaneapi" + "github.com/cachebox-project/inference-cache/internal/server" + "github.com/cachebox-project/inference-cache/internal/subscriber" ) func TestStatsReporterPopulatesSnapshotReplicas(t *testing.T) { @@ -76,15 +76,15 @@ func TestStatsReporterPopulatesSnapshotReplicas(t *testing.T) { // 3. Drive the StatsReporter against the live server. const oneGiB = 1 << 30 - scraper := engine.NewMetricsScraper(metrics.Client(), engine.ScraperConfig{ + scraper := subscriber.NewMetricsScraper(metrics.Client(), subscriber.ScraperConfig{ URL: metrics.URL, - Tier: engine.CacheTierAuto, + Tier: subscriber.CacheTierAuto, CacheSizeBytes: oneGiB, MaxConcurrencyCeiling: 256, }, nil) - reporter := engine.NewStatsReporter(icpb.NewInferenceCacheClient(conn), scraper, - engine.Config{ReplicaID: "vllm-0", ModelID: "Qwen/Qwen2.5-0.5B-Instruct", TenantID: "tenant-a", HashScheme: "vllm"}, - engine.WithStatsInterval(20*time.Millisecond), + reporter := subscriber.NewStatsReporter(icpb.NewInferenceCacheClient(conn), scraper, + subscriber.Config{ReplicaID: "vllm-0", ModelID: "Qwen/Qwen2.5-0.5B-Instruct", TenantID: "tenant-a", HashScheme: "vllm"}, + subscriber.WithStatsInterval(20*time.Millisecond), ) ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -95,7 +95,7 @@ func TestStatsReporterPopulatesSnapshotReplicas(t *testing.T) { // auth scoping); the test reads it directly since no auth is configured. baseURL := "http://" + snapLis.Addr().String() deadline := time.Now().Add(5 * time.Second) - var snap index.Snapshot + var snap controlplaneapi.Snapshot for time.Now().Before(deadline) { code, body := getSnapshot(t, baseURL) if code == http.StatusOK && json.Unmarshal([]byte(body), &snap) == nil && len(snap.Replicas) > 0 { diff --git a/internal/testutil/doc.go b/internal/testutil/doc.go new file mode 100644 index 00000000..b2b0bdcd --- /dev/null +++ b/internal/testutil/doc.go @@ -0,0 +1,7 @@ +// SPDX-FileCopyrightText: 2026 The inference-cache Authors +// +// SPDX-License-Identifier: Apache-2.0 + +// Package testutil provides envtest helpers used only by this repository's +// test suites. +package testutil diff --git a/pkg/testing/envtest_setup.go b/internal/testutil/envtest_setup.go similarity index 98% rename from pkg/testing/envtest_setup.go rename to internal/testutil/envtest_setup.go index 5b3f1643..9809ef7c 100644 --- a/pkg/testing/envtest_setup.go +++ b/internal/testutil/envtest_setup.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package testing +package testutil import ( "context" diff --git a/pkg/version/doc.go b/internal/version/doc.go similarity index 68% rename from pkg/version/doc.go rename to internal/version/doc.go index 157b8aa2..4dcbdae9 100644 --- a/pkg/version/doc.go +++ b/internal/version/doc.go @@ -2,6 +2,5 @@ // // SPDX-License-Identifier: Apache-2.0 -// Package version exposes build metadata to repository binaries. It is an -// implementation detail and is awaiting migration to internal/version. +// Package version exposes build metadata to repository binaries. package version diff --git a/pkg/version/version.go b/internal/version/version.go similarity index 100% rename from pkg/version/version.go rename to internal/version/version.go diff --git a/internal/webhook/pod/envtest_integration_test.go b/internal/webhook/pod/envtest_integration_test.go index c74bf414..3a756a86 100644 --- a/internal/webhook/pod/envtest_integration_test.go +++ b/internal/webhook/pod/envtest_integration_test.go @@ -27,7 +27,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/webhook" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" - adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" + builtinruntime "github.com/cachebox-project/inference-cache/internal/adapters/builtin/runtime" + "github.com/cachebox-project/inference-cache/internal/enginebinding" ) // TestWebhookOnEnvtest_EndToEnd boots a real apiserver via envtest, installs @@ -99,10 +100,8 @@ func TestWebhookOnEnvtest_EndToEnd(t *testing.T) { // by passing --kvevent-subscriber-image to the controller. mgr.GetWebhookServer().Register(WebhookPath, &webhook.Admission{ Handler: &EngineInjector{ - Reader: mgr.GetAPIReader(), - Registry: newVLLMRegistry( - adapterruntime.WithSubscriberImage(adapterruntime.DefaultSubscriberImage), - ), + Reader: mgr.GetAPIReader(), + Registry: newVLLMRegistry(builtinruntime.SubscriberConfig{Image: testSubscriberImage}), }, }) @@ -172,7 +171,7 @@ func TestWebhookOnEnvtest_EndToEnd(t *testing.T) { }, Spec: corev1.PodSpec{ Containers: []corev1.Container{{ - Name: adapterruntime.EngineContainerName, + Name: testVLLMEngineContainerName, Image: "vllm/vllm-openai-cpu:latest", Args: []string{"--model", "Qwen/Qwen2.5-0.5B-Instruct"}, }}, @@ -187,8 +186,8 @@ func TestWebhookOnEnvtest_EndToEnd(t *testing.T) { t.Fatalf("get pod after create: %v", err) } - mustHaveContainerEnv(t, &got, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) - mustHaveContainerEnv(t, &got, adapterruntime.EnvVLLMUseV1, "1") + mustHaveContainerEnv(t, &got, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveContainerEnv(t, &got, testEnvVLLMUseV1, "1") if got.Annotations[AnnotationInjectedBy] != ns+"/"+cb.Name { t.Fatalf("annotation %s: got %q want %q", AnnotationInjectedBy, got.Annotations[AnnotationInjectedBy], ns+"/"+cb.Name) @@ -215,7 +214,7 @@ func TestWebhookOnEnvtest_EndToEnd(t *testing.T) { if len(got.Spec.Containers) != 2 { t.Fatalf("expected 2 containers (engine + subscriber); got %d: %v", len(got.Spec.Containers), envtestContainerNames(&got)) } - sub := envtestFindContainer(&got, adapterruntime.SubscriberContainerName) + sub := envtestFindContainer(&got, enginebinding.SubscriberContainerName) if sub == nil { t.Fatalf("subscriber sidecar missing; containers = %v", envtestContainerNames(&got)) } @@ -244,7 +243,7 @@ func TestWebhookOnEnvtest_EndToEnd(t *testing.T) { if err := mgr.GetAPIReader().Get(ctx, types.NamespacedName{Namespace: ns, Name: pod2.Name}, &got2); err != nil { t.Fatalf("get second pod: %v", err) } - mustHaveContainerEnv(t, &got2, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveContainerEnv(t, &got2, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) skipped := &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ @@ -255,7 +254,7 @@ func TestWebhookOnEnvtest_EndToEnd(t *testing.T) { }, Spec: corev1.PodSpec{ Containers: []corev1.Container{{ - Name: adapterruntime.EngineContainerName, + Name: testVLLMEngineContainerName, Image: "vllm/vllm-openai-cpu:latest", Args: []string{"--model", "Qwen/Qwen2.5-0.5B-Instruct"}, }}, @@ -274,9 +273,9 @@ func TestWebhookOnEnvtest_EndToEnd(t *testing.T) { if got := gotSkipped.Annotations[AnnotationInjectedBy]; got != "" { t.Fatalf("annotation %s: got %q want absent on skipped pod", AnnotationInjectedBy, got) } - if envtestHasContainerEnv(&gotSkipped, adapterruntime.EnvLMCacheRemoteURL) { + if envtestHasContainerEnv(&gotSkipped, testEnvLMCacheRemoteURL) { t.Fatalf("skipped pod unexpectedly has %s env; webhook must not inject engine wiring when %s=true", - adapterruntime.EnvLMCacheRemoteURL, AnnotationSkip) + testEnvLMCacheRemoteURL, AnnotationSkip) } } diff --git a/internal/webhook/pod/podinjector.go b/internal/webhook/pod/podinjector.go index 06c3a896..69f70dae 100644 --- a/internal/webhook/pod/podinjector.go +++ b/internal/webhook/pod/podinjector.go @@ -218,7 +218,7 @@ func (h *EngineInjector) Handle(ctx context.Context, req admission.Request) admi extra := "" if storage != nil && storage.Ownership == cachev1alpha1.CacheBackendRemoteStorageOwnershipExternal { missingField = "spec.remoteStorage.endpoint" - if err := adapterruntime.ValidateExternalEndpoint(storage.Provider, storage.Endpoint); err != nil { + if err := backendadapter.ValidateExternalEndpoint(storage.Provider, storage.Endpoint); err != nil { extra = ": " + err.Error() } } @@ -323,11 +323,11 @@ func (h *EngineInjector) Handle(ctx context.Context, req admission.Request) admi // otherwise block an events-only engine in strict mode, or be trusted by // the controller (which keys off the container name), both of which // contradict the mode's "no connector, no kernel tier" contract. - if icp, ok := adapter.(adapterruntime.InitContainerProvider); ok { + if icp, ok := adapter.(enginebinding.InitContainerProvider); ok { if cache.Spec.IsEventsOnly() { - if removed := removeContainerByName(&mutated.Spec.InitContainers, adapterruntime.LMCacheKernelCheckContainerName); removed { + if removed := removeContainerByName(&mutated.Spec.InitContainers, enginebinding.LMCacheKernelCheckContainerName); removed { log.V(1).Info("kernel-check init container removed (events-only: no connector, no kernel tier)", - "runtime", string(runtimeID), "container", adapterruntime.LMCacheKernelCheckContainerName) + "runtime", string(runtimeID), "container", enginebinding.LMCacheKernelCheckContainerName) } } else if initC, iErr := icp.KernelCheckInitContainer(cache, mutated); iErr != nil { log.V(1).Info("fail-open: kernel-check init container rejected", @@ -340,7 +340,7 @@ func (h *EngineInjector) Handle(ctx context.Context, req admission.Request) admi } log.V(1).Info("kernel-check init container injected", "runtime", string(runtimeID), "container", initC.Name) - } else if removed := removeContainerByName(&mutated.Spec.InitContainers, adapterruntime.LMCacheKernelCheckContainerName); removed { + } else if removed := removeContainerByName(&mutated.Spec.InitContainers, enginebinding.LMCacheKernelCheckContainerName); removed { // The adapter DECLINED to inject (mode=off, or auto on a non-GPU / // unresolvable pod). The webhook is authoritative for this // container, so strip any pre-existing same-name init container: a @@ -350,7 +350,7 @@ func (h *EngineInjector) Handle(ctx context.Context, req admission.Request) admi // the explicit decline strips; a transient adapter error above is // fail-open and leaves the pod untouched. log.V(1).Info("kernel-check init container removed (adapter declined to inject)", - "runtime", string(runtimeID), "container", adapterruntime.LMCacheKernelCheckContainerName) + "runtime", string(runtimeID), "container", enginebinding.LMCacheKernelCheckContainerName) } } @@ -616,7 +616,7 @@ func effectiveEndpoint(cache *cachev1alpha1.CacheBackend) string { if ep == "" { return "" } - if err := adapterruntime.ValidateExternalEndpoint(storage.Provider, storage.Endpoint); err != nil { + if err := backendadapter.ValidateExternalEndpoint(storage.Provider, storage.Endpoint); err != nil { return "" } return ep diff --git a/internal/webhook/pod/podinjector_test.go b/internal/webhook/pod/podinjector_test.go index d28f2a90..1da3f256 100644 --- a/internal/webhook/pod/podinjector_test.go +++ b/internal/webhook/pod/podinjector_test.go @@ -32,16 +32,76 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" builtinadapters "github.com/cachebox-project/inference-cache/internal/adapters/builtin" builtinruntime "github.com/cachebox-project/inference-cache/internal/adapters/builtin/runtime" + "github.com/cachebox-project/inference-cache/internal/enginebinding" backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) -func newVLLMRegistry(opts ...adapterruntime.Option) *adapterruntime.Registry { +const ( + testVLLMEngineContainerName = "vllm" + testSubscriberImage = "subscriber:test" + testEnvLMCacheRemoteURL = "LMCACHE_REMOTE_URL" + testEnvLMCacheChunkSize = "LMCACHE_CHUNK_SIZE" + testEnvVLLMUseV1 = "VLLM_USE_V1" + testEnvPythonHashSeed = "PYTHONHASHSEED" + testReferenceCacheEndpoint = "INFERENCECACHE_CACHE_ENDPOINT" + testRuntimeReference = adapterruntime.RuntimeID("reference") +) + +func newVLLMRegistry(configs ...builtinruntime.SubscriberConfig) *adapterruntime.Registry { + var config builtinruntime.SubscriberConfig + if len(configs) > 0 { + config = configs[0] + } registry := adapterruntime.NewRegistry() - registry.Register(builtinruntime.NewVLLMLMCacheAdapter(opts...)) + registry.Register(builtinruntime.NewVLLMLMCacheAdapter(config)) return registry } +// referenceRuntimeAdapter is a webhook-local fixture for the public runtime +// extension contract. It deliberately renders no observation sidecar. +type referenceRuntimeAdapter struct{} + +func (referenceRuntimeAdapter) Supports(runtime adapterruntime.RuntimeID, cache *cachev1alpha1.CacheBackend) bool { + return cache != nil && runtime == testRuntimeReference +} + +func (referenceRuntimeAdapter) SupportsBinding(binding *backendadapter.Binding) bool { + return binding != nil && binding.Protocol != "" +} + +func (referenceRuntimeAdapter) InjectEngineConfig(pod *corev1.PodSpec, binding *backendadapter.Binding, _ *cachev1alpha1.CacheBackend) error { + if pod == nil || binding == nil { + return errors.New("reference fixture requires pod and binding") + } + for i := range pod.Containers { + pod.Containers[i].Env = referenceUpsertEnv(pod.Containers[i].Env, corev1.EnvVar{Name: testReferenceCacheEndpoint, Value: binding.Endpoint}) + } + return nil +} + +func (referenceRuntimeAdapter) InjectRouterConfig(*corev1.PodSpec, *backendadapter.Binding, *cachev1alpha1.CacheBackend) error { + return nil +} + +func (referenceRuntimeAdapter) ObservationSidecar(*cachev1alpha1.CacheBackend, *corev1.Pod) (*corev1.Container, error) { + return nil, nil +} + +func (referenceRuntimeAdapter) ReservedArgs() []string { return nil } +func (referenceRuntimeAdapter) ReservedEnv() []string { return nil } +func (referenceRuntimeAdapter) EngineContainerName() string { return "" } + +func referenceUpsertEnv(env []corev1.EnvVar, want corev1.EnvVar) []corev1.EnvVar { + for i := range env { + if env[i].Name == want.Name { + env[i] = want + return env + } + } + return append(env, want) +} + func externalLMCacheStorage(endpoint string) *cachev1alpha1.CacheBackendRemoteStorageSpec { return &cachev1alpha1.CacheBackendRemoteStorageSpec{ Provider: cachev1alpha1.CacheBackendRemoteStorageProviderLMCacheServer, @@ -131,7 +191,7 @@ func vllmEnginePod(name string, labels map[string]string) *corev1.Pod { }, Spec: corev1.PodSpec{ Containers: []corev1.Container{{ - Name: adapterruntime.EngineContainerName, + Name: testVLLMEngineContainerName, Image: "vllm/vllm-openai-cpu:latest", Env: []corev1.EnvVar{ {Name: "USER_FLAG", Value: "preserved"}, @@ -203,7 +263,7 @@ func newHandler(t *testing.T, objs ...client.Object) *EngineInjector { c := fake.NewClientBuilder().WithScheme(s).WithObjects(objs...).Build() return &EngineInjector{ Reader: c, - Registry: builtinadapters.New().Runtime, + Registry: builtinadapters.New(builtinadapters.Options{}).Runtime, Log: logr.Discard(), } } @@ -217,9 +277,7 @@ func newHandlerWithSubscriber(t *testing.T, objs ...client.Object) *EngineInject t.Helper() s := newScheme(t) c := fake.NewClientBuilder().WithScheme(s).WithObjects(objs...).Build() - reg := newVLLMRegistry( - adapterruntime.WithSubscriberImage(adapterruntime.DefaultSubscriberImage), - ) + reg := newVLLMRegistry(builtinruntime.SubscriberConfig{Image: testSubscriberImage}) return &EngineInjector{ Reader: c, Registry: reg, @@ -244,9 +302,9 @@ func TestHandle_MatchAndInject(t *testing.T) { mutated := applyPatches(t, req.Object.Raw, resp) mustHaveEnv(t, mutated, "USER_FLAG", "preserved") - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) - mustHaveEnv(t, mutated, adapterruntime.EnvVLLMUseV1, "1") + mustHaveEnv(t, mutated, testEnvVLLMUseV1, "1") if got, want := mutated.Annotations[AnnotationInjectedBy], ns+"/"+cb.Name; got != want { t.Fatalf("annotation %s: got %q, want %q", AnnotationInjectedBy, got, want) } @@ -310,11 +368,11 @@ func TestHandle_MatchAndInject_SGLang(t *testing.T) { } } for _, e := range c.Env { - if e.Name == adapterruntime.EnvVLLMUseV1 || e.Name == adapterruntime.EnvPythonHashSeed { + if e.Name == testEnvVLLMUseV1 || e.Name == testEnvPythonHashSeed { t.Fatalf("SGLang pod got vLLM-only env %q (SGLang injects neither)", e.Name) } - if e.Name == adapterruntime.EnvLMCacheRemoteURL { - t.Fatalf("SGLang MP wire must not inject %s", adapterruntime.EnvLMCacheRemoteURL) + if e.Name == testEnvLMCacheRemoteURL { + t.Fatalf("SGLang MP wire must not inject %s", testEnvLMCacheRemoteURL) } } } @@ -469,8 +527,8 @@ func TestHandle_MooncakeBackend_InjectsMooncakeStoreEndpoint(t *testing.T) { // The defining difference from the LMCache path: the remote URL carries // the mooncakestore:// scheme, pointed at the master RPC endpoint. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "mooncakestore://"+cb.Status.Endpoint) - mustHaveEnv(t, mutated, adapterruntime.EnvVLLMUseV1, "1") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "mooncakestore://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvVLLMUseV1, "1") mustHaveArgFlag(t, mutated, "--kv-transfer-config") // User-set engine arg survives the merge (merge, not clobber). mustHaveArgPair(t, mutated, "--model", "Qwen/Qwen2.5-0.5B-Instruct") @@ -480,7 +538,7 @@ func TestHandle_MooncakeBackend_InjectsMooncakeStoreEndpoint(t *testing.T) { // The kvevent-subscriber sidecar is appended on the Mooncake path too // (same shared builder; vLLM's KV-event stream is store-independent). - sub := findContainer(mutated, adapterruntime.SubscriberContainerName) + sub := findContainer(mutated, enginebinding.SubscriberContainerName) if sub == nil { t.Fatalf("subscriber sidecar missing on Mooncake path; containers = %v", containerNames(mutated)) } @@ -545,7 +603,7 @@ func TestHandle_MooncakeBackend_EngineHostNetworkIsOptIn(t *testing.T) { } // The rest of the Mooncake wiring still lands — the opt-in gates the // networking rewrite only, never the connector env. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "mooncakestore://mc.engines.svc.cluster.local:50051") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "mooncakestore://mc.engines.svc.cluster.local:50051") }) t.Run("InjectedWhenOperatorOptsIn", func(t *testing.T) { @@ -566,7 +624,7 @@ func TestHandle_MooncakeBackend_EngineHostNetworkIsOptIn(t *testing.T) { if got, want := mutated.Spec.DNSPolicy, corev1.DNSClusterFirstWithHostNet; got != want { t.Fatalf("dnsPolicy: got %q, want %q (hostNetwork pods lose cluster DNS without it, and status.endpoint is a DNS name)", got, want) } - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "mooncakestore://mc.engines.svc.cluster.local:50051") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "mooncakestore://mc.engines.svc.cluster.local:50051") }) } @@ -618,7 +676,7 @@ func TestHandle_MooncakeBackend_HostNetworkNeverGrantedToAnUnwiredPod(t *testing } for _, c := range mutated.Spec.Containers { for _, e := range c.Env { - if e.Name == adapterruntime.EnvLMCacheRemoteURL { + if e.Name == testEnvLMCacheRemoteURL { t.Fatalf("connector env %s injected without an endpoint", e.Name) } } @@ -683,7 +741,7 @@ func TestHandle_AppendsObservationSidecar(t *testing.T) { if len(mutated.Spec.Containers) != 2 { t.Fatalf("expected 2 containers (engine + subscriber), got %d: %v", len(mutated.Spec.Containers), containerNames(mutated)) } - sub := findContainer(mutated, adapterruntime.SubscriberContainerName) + sub := findContainer(mutated, enginebinding.SubscriberContainerName) if sub == nil { t.Fatalf("subscriber sidecar missing; containers = %v", containerNames(mutated)) } @@ -698,7 +756,7 @@ func TestHandle_AppendsObservationSidecar(t *testing.T) { } // The engine container is still wired with LMCache env — appending the // sidecar must not regress the engine-side injection. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) } func TestHandle_AppendsObservationSidecar_SGLang(t *testing.T) { @@ -721,8 +779,9 @@ func TestHandle_AppendsObservationSidecar_SGLang(t *testing.T) { s := newScheme(t) c := fake.NewClientBuilder().WithScheme(s).WithObjects(cb).Build() - reg := newVLLMRegistry(adapterruntime.WithSubscriberImage(adapterruntime.DefaultSubscriberImage)) - reg.Register(builtinruntime.NewSGLangLMCacheAdapter(adapterruntime.WithSubscriberImage(adapterruntime.DefaultSubscriberImage))) + config := builtinruntime.SubscriberConfig{Image: testSubscriberImage} + reg := newVLLMRegistry(config) + reg.Register(builtinruntime.NewSGLangLMCacheAdapter(config)) h := &EngineInjector{Reader: c, Registry: reg, Log: logr.Discard()} pod := sglangEnginePod("sg-engine-a", map[string]string{"app": "sglang"}) @@ -736,7 +795,7 @@ func TestHandle_AppendsObservationSidecar_SGLang(t *testing.T) { if len(mutated.Spec.Containers) != 2 { t.Fatalf("expected 2 containers (sglang engine + subscriber), got %d: %v", len(mutated.Spec.Containers), containerNames(mutated)) } - sub := findContainer(mutated, adapterruntime.SubscriberContainerName) + sub := findContainer(mutated, enginebinding.SubscriberContainerName) if sub == nil { t.Fatalf("subscriber sidecar missing; containers = %v", containerNames(mutated)) } @@ -831,7 +890,7 @@ func TestHandle_EventsOnly_EmptyEndpoint_InjectsSubscriberWithoutConnector(t *te t.Fatalf("expected 2 containers (engine + subscriber), got %d: %v", len(mutated.Spec.Containers), containerNames(mutated)) } - sub := findContainer(mutated, adapterruntime.SubscriberContainerName) + sub := findContainer(mutated, enginebinding.SubscriberContainerName) if sub == nil { t.Fatalf("subscriber sidecar missing; containers = %v", containerNames(mutated)) } @@ -851,7 +910,7 @@ func TestHandle_EventsOnly_EmptyEndpoint_InjectsSubscriberWithoutConnector(t *te // The engine container gets NO KV connector wiring: events-only loads no // connector (a hybrid-attention model's KV-cache manager would be disabled // by one). The user's own env/args survive untouched. - engine := findContainer(mutated, adapterruntime.EngineContainerName) + engine := findContainer(mutated, testVLLMEngineContainerName) if engine == nil { t.Fatalf("engine container missing; containers = %v", containerNames(mutated)) } @@ -936,7 +995,7 @@ func TestHandle_EventsOnly_NoSubscriberImage_InjectsNothingNoStamp(t *testing.T) // No subscriber container appended. mutated := applyPatches(t, req.Object.Raw, resp) - if c := findContainer(mutated, adapterruntime.SubscriberContainerName); c != nil { + if c := findContainer(mutated, enginebinding.SubscriberContainerName); c != nil { t.Fatalf("no subscriber image configured — must NOT append a sidecar; found %+v", c) } // No injected-by / injected-by-uid stamped — nothing was wired. @@ -994,7 +1053,7 @@ func TestHandle_EventsOnly_PrebakedSubscriber_NotClaimedNoStamp(t *testing.T) { h := newHandlerWithSubscriber(t, cb) // subscriber image IS configured pod := vllmEnginePod("engine-a", map[string]string{"app": "vllm"}) pod.Spec.Containers = append(pod.Spec.Containers, corev1.Container{ - Name: adapterruntime.SubscriberContainerName, + Name: enginebinding.SubscriberContainerName, Image: "operator/hand-baked-subscriber:wrong", }) req := newRequest(t, pod, ns) @@ -1008,7 +1067,7 @@ func TestHandle_EventsOnly_PrebakedSubscriber_NotClaimedNoStamp(t *testing.T) { // Idempotent: still exactly one subscriber-named container (no duplicate append). count := 0 for i := range mutated.Spec.Containers { - if mutated.Spec.Containers[i].Name == adapterruntime.SubscriberContainerName { + if mutated.Spec.Containers[i].Name == enginebinding.SubscriberContainerName { count++ } } @@ -1017,7 +1076,7 @@ func TestHandle_EventsOnly_PrebakedSubscriber_NotClaimedNoStamp(t *testing.T) { count, containerNames(mutated)) } // The hand-baked container is left as-is — the subscriber is NOT webhook-authoritative. - if sub := findContainer(mutated, adapterruntime.SubscriberContainerName); sub == nil || sub.Image != "operator/hand-baked-subscriber:wrong" { + if sub := findContainer(mutated, enginebinding.SubscriberContainerName); sub == nil || sub.Image != "operator/hand-baked-subscriber:wrong" { t.Fatalf("hand-baked subscriber must be left untouched; got %+v", sub) } // NOT claimed: the webhook authored no wiring, so it stamps no injected-by. @@ -1058,7 +1117,7 @@ func TestHandle_EventsOnly_EngineOverrides_DoNotTouchEngineContainer(t *testing. } mutated := applyPatches(t, req.Object.Raw, resp) - engine := findContainer(mutated, adapterruntime.EngineContainerName) + engine := findContainer(mutated, testVLLMEngineContainerName) if engine == nil { t.Fatalf("engine container missing; containers = %v", containerNames(mutated)) } @@ -1081,7 +1140,7 @@ func TestHandle_EventsOnly_EngineOverrides_DoNotTouchEngineContainer(t *testing. // The subscriber IS still injected (image configured) and the pod is wired, // so injected-by is stamped — confirms the engine-untouched guarantee is // independent of the sidecar-append path. - if sub := findContainer(mutated, adapterruntime.SubscriberContainerName); sub == nil { + if sub := findContainer(mutated, enginebinding.SubscriberContainerName); sub == nil { t.Fatalf("subscriber sidecar must still attach for a configured events-only backend; containers = %v", containerNames(mutated)) } if got, want := mutated.Annotations[AnnotationInjectedBy], ns+"/"+cb.Name; got != want { @@ -1133,8 +1192,8 @@ func TestHandle_ExternalBackend_InjectsOperatorEndpoint(t *testing.T) { // LMCACHE_REMOTE_URL must be the operator-supplied endpoint with the // lm:// scheme prepended, identical to what the managed adapter // would write for the same endpoint. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+endpoint) - mustHaveEnv(t, mutated, adapterruntime.EnvVLLMUseV1, "1") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+endpoint) + mustHaveEnv(t, mutated, testEnvVLLMUseV1, "1") // User --model arg survives the merge — the adapter only adds; it // never clobbers user-set args. if !containsArgPairLocal(mutated.Spec.Containers[0].Args, "--model", "Qwen/Qwen2.5-0.5B-Instruct") { @@ -1142,7 +1201,7 @@ func TestHandle_ExternalBackend_InjectsOperatorEndpoint(t *testing.T) { } // The external-ownership path attaches no observation sidecar — the // controller has no observability seam into an operator-managed cache. - if c := findContainer(mutated, adapterruntime.SubscriberContainerName); c != nil { + if c := findContainer(mutated, enginebinding.SubscriberContainerName); c != nil { t.Fatalf("External backend must NOT get a subscriber sidecar; found %+v", c) } if mutated.Annotations[AnnotationInjectedBy] != ns+"/ext" { @@ -1278,7 +1337,7 @@ func TestHandle_ExternalBackend_StatusEmpty_UsesSpecDirectly(t *testing.T) { t.Fatalf("expected Allowed, got %+v", resp.Result) } mutated := applyPatches(t, req.Object.Raw, resp) - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+endpoint) + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+endpoint) } func TestHandle_ExternalBackend_PrefersSpecOverStaleStatus(t *testing.T) { @@ -1323,9 +1382,9 @@ func TestHandle_ExternalBackend_PrefersSpecOverStaleStatus(t *testing.T) { } mutated := applyPatches(t, req.Object.Raw, resp) // Must use spec.remoteStorage.endpoint, NOT the stale status.endpoint. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+freshSpec) + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+freshSpec) for _, e := range mutated.Spec.Containers[0].Env { - if e.Name == adapterruntime.EnvLMCacheRemoteURL && e.Value == "lm://"+staleStatus { + if e.Name == testEnvLMCacheRemoteURL && e.Value == "lm://"+staleStatus { t.Fatalf("pod wired to stale status.endpoint %q; should be spec.remoteStorage.endpoint %q", staleStatus, freshSpec) } } @@ -1372,7 +1431,7 @@ func TestHandle_ExternalBackend_UpperCaseSchemeNormalised(t *testing.T) { mutated := applyPatches(t, req.Object.Raw, resp) // Must be the canonical lower-case scheme, with the original // host portion preserved verbatim. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://cache.example.com:8200") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://cache.example.com:8200") } func TestHandle_WhitespaceStatusEndpointFailsOpen(t *testing.T) { @@ -1408,7 +1467,7 @@ func TestHandle_WhitespaceStatusEndpointFailsOpen(t *testing.T) { } mutated := applyPatches(t, req.Object.Raw, resp) for _, e := range mutated.Spec.Containers[0].Env { - if e.Name == adapterruntime.EnvLMCacheRemoteURL { + if e.Name == testEnvLMCacheRemoteURL { t.Fatalf("whitespace status.endpoint must not become injected env; got %s=%q", e.Name, e.Value) } } @@ -1450,7 +1509,7 @@ func TestHandle_ManagedBackend_StatusEmpty_FailsOpen(t *testing.T) { t.Fatalf("pod has no containers after admission") } for _, e := range mutated.Spec.Containers[0].Env { - if e.Name == adapterruntime.EnvLMCacheRemoteURL { + if e.Name == testEnvLMCacheRemoteURL { t.Fatalf("managed CR with no status.endpoint must NOT trigger injection; got %s=%q", e.Name, e.Value) } } @@ -1475,11 +1534,11 @@ func TestHandle_ExternalBackend_NoSidecar(t *testing.T) { // without appending a kvevent-subscriber container. const ns = "engines" cb := readyCacheBackend("primary", ns, map[string]string{"app": "vllm"}) - cb.Spec.Runtime = cachev1alpha1.CacheBackendRuntime(adapterruntime.RuntimeReference) + cb.Spec.Runtime = cachev1alpha1.CacheBackendRuntime(testRuntimeReference) s := newScheme(t) c := fake.NewClientBuilder().WithScheme(s).WithObjects(cb).Build() reg := adapterruntime.NewRegistry() - reg.Register(adapterruntime.NewReferenceAdapter()) + reg.Register(referenceRuntimeAdapter{}) h := &EngineInjector{Reader: c, Registry: reg, Log: logr.Discard()} pod := vllmEnginePod("engine-a", map[string]string{"app": "vllm"}) req := newRequest(t, pod, ns) @@ -1489,7 +1548,7 @@ func TestHandle_ExternalBackend_NoSidecar(t *testing.T) { t.Fatalf("expected Allowed, got %+v", resp.Result) } mutated := applyPatches(t, req.Object.Raw, resp) - if c := findContainer(mutated, adapterruntime.SubscriberContainerName); c != nil { + if c := findContainer(mutated, enginebinding.SubscriberContainerName); c != nil { t.Fatalf("External-style backend must NOT get a subscriber sidecar; found %+v", c) } } @@ -1512,10 +1571,10 @@ func TestHandle_SidecarOptInDefaultsToNoSidecar(t *testing.T) { t.Fatalf("engine injection must still happen; Allowed=%v patches=%d", resp.Allowed, len(resp.Patches)) } mutated := applyPatches(t, req.Object.Raw, resp) - if c := findContainer(mutated, adapterruntime.SubscriberContainerName); c != nil { + if c := findContainer(mutated, enginebinding.SubscriberContainerName); c != nil { t.Fatalf("default install must NOT auto-attach the sidecar; got %+v", c) } - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) } func TestHandle_SidecarSkippedWithoutModel(t *testing.T) { @@ -1533,10 +1592,10 @@ func TestHandle_SidecarSkippedWithoutModel(t *testing.T) { t.Fatalf("engine injection must still happen; Allowed=%v patches=%d", resp.Allowed, len(resp.Patches)) } mutated := applyPatches(t, req.Object.Raw, resp) - if c := findContainer(mutated, adapterruntime.SubscriberContainerName); c != nil { + if c := findContainer(mutated, enginebinding.SubscriberContainerName); c != nil { t.Fatalf("sidecar must be skipped without a model id; got %+v", c) } - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) } func TestHandle_SidecarErrorIsFailOpen(t *testing.T) { @@ -1561,7 +1620,7 @@ func TestHandle_SidecarErrorIsFailOpen(t *testing.T) { } mutated := applyPatches(t, req.Object.Raw, resp) mustHaveEnv(t, mutated, "STUB_INJECTED", "yes") - if c := findContainer(mutated, adapterruntime.SubscriberContainerName); c != nil { + if c := findContainer(mutated, enginebinding.SubscriberContainerName); c != nil { t.Fatalf("sidecar errored — webhook must not append a partial container, got %+v", c) } } @@ -1573,7 +1632,7 @@ func TestHandle_PreExistingSidecar_NotDuplicated(t *testing.T) { h := newHandlerWithSubscriber(t, cb) pod := vllmEnginePod("engine-a", map[string]string{"app": "vllm"}) pod.Spec.Containers = append(pod.Spec.Containers, corev1.Container{ - Name: adapterruntime.SubscriberContainerName, + Name: enginebinding.SubscriberContainerName, Image: "operator/pre-baked:tag", }) req := newRequest(t, pod, ns) @@ -1585,13 +1644,13 @@ func TestHandle_PreExistingSidecar_NotDuplicated(t *testing.T) { mutated := applyPatches(t, req.Object.Raw, resp) subs := 0 for _, c := range mutated.Spec.Containers { - if c.Name == adapterruntime.SubscriberContainerName { + if c.Name == enginebinding.SubscriberContainerName { subs++ } } if subs != 1 { t.Fatalf("expected exactly one %s container after admission, got %d: %v", - adapterruntime.SubscriberContainerName, subs, containerNames(mutated)) + enginebinding.SubscriberContainerName, subs, containerNames(mutated)) } } @@ -1627,7 +1686,7 @@ func (sidecarErrorAdapter) ObservationSidecar(*cachev1alpha1.CacheBackend, *core func (sidecarErrorAdapter) ReservedArgs() []string { return nil } func (sidecarErrorAdapter) ReservedEnv() []string { return nil } -func (sidecarErrorAdapter) EngineContainerName() string { return adapterruntime.EngineContainerName } +func (sidecarErrorAdapter) EngineContainerName() string { return testVLLMEngineContainerName } func findContainer(pod *corev1.Pod, name string) *corev1.Container { for i := range pod.Spec.Containers { @@ -1712,7 +1771,7 @@ func TestHandle_PartialEnvOnly_StillConverges(t *testing.T) { h := newHandler(t, cb) pod := vllmEnginePod("engine-a", map[string]string{"app": "vllm"}) pod.Spec.Containers[0].Env = append(pod.Spec.Containers[0].Env, corev1.EnvVar{ - Name: adapterruntime.EnvLMCacheRemoteURL, + Name: testEnvLMCacheRemoteURL, Value: "lm://stale.example:65432", }) req := newRequest(t, pod, ns) @@ -1724,8 +1783,8 @@ func TestHandle_PartialEnvOnly_StillConverges(t *testing.T) { mutated := applyPatches(t, req.Object.Raw, resp) // The stale URL is overwritten with the canonical one for the matched // backend, and the missing pieces are added. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) - mustHaveEnv(t, mutated, adapterruntime.EnvVLLMUseV1, "1") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvVLLMUseV1, "1") mustHaveArgFlag(t, mutated, "--kv-transfer-config") } @@ -1951,11 +2010,11 @@ func TestHandle_RegistryOverride_UsedInsteadOfDefault(t *testing.T) { const ns = "engines" cb := readyCacheBackend("primary", ns, map[string]string{"app": "vllm"}) cb.Spec.Runtime = "" - cb.Spec.Runtime = cachev1alpha1.CacheBackendRuntime(adapterruntime.RuntimeReference) + cb.Spec.Runtime = cachev1alpha1.CacheBackendRuntime(testRuntimeReference) s := newScheme(t) c := fake.NewClientBuilder().WithScheme(s).WithObjects(cb).Build() reg := adapterruntime.NewRegistry() - reg.Register(adapterruntime.NewReferenceAdapter()) + reg.Register(referenceRuntimeAdapter{}) h := &EngineInjector{Reader: c, Registry: reg, Log: logr.Discard()} pod := vllmEnginePod("engine-a", map[string]string{"app": "vllm"}) req := newRequest(t, pod, ns) @@ -1965,7 +2024,7 @@ func TestHandle_RegistryOverride_UsedInsteadOfDefault(t *testing.T) { t.Fatalf("expected Allowed with patches; got Allowed=%v patches=%d", resp.Allowed, len(resp.Patches)) } mutated := applyPatches(t, req.Object.Raw, resp) - mustHaveEnv(t, mutated, adapterruntime.EnvCacheEndpoint, cb.Status.Endpoint) + mustHaveEnv(t, mutated, testReferenceCacheEndpoint, cb.Status.Endpoint) } func TestHandle_PodNamespaceDefaultedFromRequest(t *testing.T) { @@ -2129,7 +2188,7 @@ func TestHandle_EngineOverrides_EnvUpsertAndArgAppend(t *testing.T) { {Name: "FOO", Value: "bar"}, // Override a tunable canonical env value, which is allowed // because LMCACHE_CHUNK_SIZE is NOT reserved. - {Name: adapterruntime.EnvLMCacheChunkSize, Value: "512"}, + {Name: testEnvLMCacheChunkSize, Value: "512"}, }, } h := newHandler(t, cb) @@ -2146,10 +2205,10 @@ func TestHandle_EngineOverrides_EnvUpsertAndArgAppend(t *testing.T) { mustHaveEnv(t, mutated, "FOO", "bar") // Override wins for the tunable name (LMCACHE_CHUNK_SIZE is an // adapter-owned canonical entry — the override surface can touch it). - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheChunkSize, "512") + mustHaveEnv(t, mutated, testEnvLMCacheChunkSize, "512") // Canonical reserved env still landed unchanged. - mustHaveEnv(t, mutated, adapterruntime.EnvVLLMUseV1, "1") - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvVLLMUseV1, "1") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) // User-template env preserved. mustHaveEnv(t, mutated, "USER_FLAG", "preserved") @@ -2194,7 +2253,7 @@ func TestHandle_EngineOverrides_DoNotMutateUserTemplate(t *testing.T) { // User-owned env untouched by the CR-driven override + suppress. mustHaveEnv(t, mutated, "USER_FLAG", "preserved") // Canonical injection still landed. - mustHaveEnv(t, mutated, adapterruntime.EnvVLLMUseV1, "1") + mustHaveEnv(t, mutated, testEnvVLLMUseV1, "1") mustHaveArgFlag(t, mutated, "--kv-transfer-config") } @@ -2232,8 +2291,8 @@ func TestHandle_EngineOverrides_NoOverride_ByteIdenticalToBaseline(t *testing.T) mutated := applyPatches(t, req.Object.Raw, resp) // Sanity: canonical injection lands as expected — so a green test // is meaningful (not green by producing an empty patch set). - mustHaveEnv(t, mutated, adapterruntime.EnvVLLMUseV1, "1") - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvVLLMUseV1, "1") + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) mustHaveArgFlag(t, mutated, "--kv-transfer-config") raw, err := json.Marshal(mutated) @@ -2348,17 +2407,17 @@ func TestHandle_KernelCheckInitContainer_AppendedOnGPUPod(t *testing.T) { // The kernel-check init container must be present. found := false for _, ic := range mutated.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { found = true break } } if !found { t.Fatalf("kernel-check init container %q missing from Spec.InitContainers; got: %v", - adapterruntime.LMCacheKernelCheckContainerName, initContainerNames(mutated)) + enginebinding.LMCacheKernelCheckContainerName, initContainerNames(mutated)) } // Engine-side injection must still have landed. - mustHaveEnv(t, mutated, adapterruntime.EnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) + mustHaveEnv(t, mutated, testEnvLMCacheRemoteURL, "lm://"+cb.Status.Endpoint) } // TestHandle_KernelCheckInitContainer_Idempotent verifies that a second @@ -2382,7 +2441,7 @@ func TestHandle_KernelCheckInitContainer_Idempotent(t *testing.T) { count := 0 for _, ic := range injected.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { count++ } } @@ -2400,7 +2459,7 @@ func TestHandle_KernelCheckInitContainer_Idempotent(t *testing.T) { count = 0 for _, ic := range readmitted.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { count++ } } @@ -2423,7 +2482,7 @@ func TestHandle_EventsOnly_StripsPreexistingKernelCheckInitContainer(t *testing. pod := vllmEnginePod("engine-a", map[string]string{"app": "vllm"}) pod.Spec.InitContainers = append(pod.Spec.InitContainers, corev1.Container{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, Image: "stale-hand-baked-kernel-check:latest", }) req := newRequest(t, pod, ns) @@ -2435,13 +2494,13 @@ func TestHandle_EventsOnly_StripsPreexistingKernelCheckInitContainer(t *testing. mutated := applyPatches(t, req.Object.Raw, resp) for _, ic := range mutated.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { t.Fatalf("stale kernel-check init container survived events-only admission; init containers = %v", initContainerNames(mutated)) } } // The subscriber sidecar is still wired (this is a normal events-only inject). - if findContainer(mutated, adapterruntime.SubscriberContainerName) == nil { + if findContainer(mutated, enginebinding.SubscriberContainerName) == nil { t.Fatalf("subscriber sidecar missing; containers = %v", containerNames(mutated)) } } @@ -2458,7 +2517,7 @@ func TestHandle_KernelCheckInitContainer_ReplacesForged(t *testing.T) { // A hand-authored / forged same-name init container that would bypass the // real check if the webhook merely skipped injection (e.g. a fake "OK"). pod.Spec.InitContainers = []corev1.Container{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, Image: "attacker/fake:latest", Command: []string{"echo", "OK"}, }} @@ -2469,7 +2528,7 @@ func TestHandle_KernelCheckInitContainer_ReplacesForged(t *testing.T) { var got *corev1.Container count := 0 for i := range injected.Spec.InitContainers { - if injected.Spec.InitContainers[i].Name == adapterruntime.LMCacheKernelCheckContainerName { + if injected.Spec.InitContainers[i].Name == enginebinding.LMCacheKernelCheckContainerName { got = &injected.Spec.InitContainers[i] count++ } @@ -2498,7 +2557,7 @@ func TestHandle_KernelCheckInitContainer_StrippedWhenAdapterDeclines(t *testing. // the container by name), preserving "auto on a non-GPU pod = absent". pod := vllmEnginePod("engine-cpu", map[string]string{"app": "vllm"}) pod.Spec.InitContainers = []corev1.Container{{ - Name: adapterruntime.LMCacheKernelCheckContainerName, + Name: enginebinding.LMCacheKernelCheckContainerName, Image: "attacker/fake:latest", Command: []string{"echo", "OK"}, }} @@ -2507,7 +2566,7 @@ func TestHandle_KernelCheckInitContainer_StrippedWhenAdapterDeclines(t *testing. injected := applyPatches(t, newRequest(t, pod, ns).Object.Raw, resp) for _, ic := range injected.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { t.Fatalf("forged kernel-check init container survived on a non-GPU (auto) pod; want it stripped: %v", initContainerNames(injected)) } @@ -2545,7 +2604,7 @@ func TestHandle_KernelCheckInitContainer_SkipAnnotationSuppresses(t *testing.T) t.Fatalf("annotation %s = %q, want %q", AnnotationInjectSkipped, got, InjectSkippedReasonSkipAnnotation) } for _, ic := range mutated.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { t.Fatalf("kernel-check init container must be absent when skip annotation is set; found %+v", ic) } } @@ -2564,7 +2623,7 @@ func TestHandle_KernelCheckInitContainer_SkippedForEventsOnly(t *testing.T) { // Offload backend this would inject (and, on a strict failure, block the // pod). Events-only must skip it regardless. cb.Annotations = map[string]string{ - adapterruntime.AnnotationLMCacheKernelCheck: adapterruntime.KernelCheckModeStrict, + enginebinding.AnnotationLMCacheKernelCheck: enginebinding.KernelCheckModeStrict, } h := newHandlerWithSubscriber(t, cb) @@ -2584,15 +2643,15 @@ func TestHandle_KernelCheckInitContainer_SkippedForEventsOnly(t *testing.T) { // No kernel-check init container for events-only. for _, ic := range mutated.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { t.Fatalf("events-only pod must NOT get the %q init container; init containers = %v", - adapterruntime.LMCacheKernelCheckContainerName, initContainerNames(mutated)) + enginebinding.LMCacheKernelCheckContainerName, initContainerNames(mutated)) } } // Sanity: the events-only wiring still happened (subscriber sidecar // appended), so this is not a fail-open no-op masquerading as a skip. - if sub := findContainer(mutated, adapterruntime.SubscriberContainerName); sub == nil { + if sub := findContainer(mutated, enginebinding.SubscriberContainerName); sub == nil { t.Fatalf("events-only subscriber sidecar missing; containers = %v", containerNames(mutated)) } } @@ -2605,7 +2664,7 @@ func TestHandle_KernelCheckInitContainer_AppendedOnOffloadStrict(t *testing.T) { const ns = "engines" cb := readyCacheBackend("primary", ns, map[string]string{"app": "vllm"}) cb.Annotations = map[string]string{ - adapterruntime.AnnotationLMCacheKernelCheck: adapterruntime.KernelCheckModeStrict, + enginebinding.AnnotationLMCacheKernelCheck: enginebinding.KernelCheckModeStrict, } h := newHandler(t, cb) @@ -2623,14 +2682,14 @@ func TestHandle_KernelCheckInitContainer_AppendedOnOffloadStrict(t *testing.T) { found := false for _, ic := range mutated.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { found = true break } } if !found { t.Fatalf("Offload (strict) pod must still get the %q init container; init containers = %v", - adapterruntime.LMCacheKernelCheckContainerName, initContainerNames(mutated)) + enginebinding.LMCacheKernelCheckContainerName, initContainerNames(mutated)) } } @@ -2673,9 +2732,7 @@ func TestHandle_EventsOnlyExternal_NoConnectorWiring(t *testing.T) { c := fake.NewClientBuilder().WithScheme(s).WithObjects(cb).Build() // Configure the shipping vLLM adapter's subscriber image so the events-only // sidecar path is live. - reg := newVLLMRegistry( - adapterruntime.WithSubscriberImage(adapterruntime.DefaultSubscriberImage), - ) + reg := newVLLMRegistry(builtinruntime.SubscriberConfig{Image: testSubscriberImage}) h := &EngineInjector{Reader: c, Registry: reg, Log: logr.Discard()} pod := vllmEnginePod("engine-a", map[string]string{"app": "vllm"}) @@ -2687,7 +2744,7 @@ func TestHandle_EventsOnlyExternal_NoConnectorWiring(t *testing.T) { } mutated := applyPatches(t, req.Object.Raw, resp) - engine := findContainer(mutated, adapterruntime.EngineContainerName) + engine := findContainer(mutated, testVLLMEngineContainerName) if engine == nil { t.Fatalf("engine container missing; containers = %v", containerNames(mutated)) } @@ -2705,7 +2762,7 @@ func TestHandle_EventsOnlyExternal_NoConnectorWiring(t *testing.T) { } // No kernel-check init container either; assert the contract end-to-end. for _, ic := range mutated.Spec.InitContainers { - if ic.Name == adapterruntime.LMCacheKernelCheckContainerName { + if ic.Name == enginebinding.LMCacheKernelCheckContainerName { t.Fatalf("events-only+External pod must NOT get the kernel-check init container; init containers = %v", initContainerNames(mutated)) } diff --git a/internal/webhook/v1alpha1/cachebackend_webhook.go b/internal/webhook/v1alpha1/cachebackend_webhook.go index f8d93b2e..b38a9e8c 100644 --- a/internal/webhook/v1alpha1/cachebackend_webhook.go +++ b/internal/webhook/v1alpha1/cachebackend_webhook.go @@ -27,6 +27,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/webhook/admission" cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" + "github.com/cachebox-project/inference-cache/internal/enginebinding" backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) @@ -216,7 +217,7 @@ func validateCacheHierarchy(cb *cachev1alpha1.CacheBackend) field.ErrorList { if strings.TrimSpace(storage.Endpoint) == "" { errs = append(errs, field.Required(storagePath.Child("endpoint"), "required when remoteStorage.ownership=External")) - } else if err := adapterruntime.ValidateExternalEndpoint(storage.Provider, storage.Endpoint); err != nil { + } else if err := backendadapter.ValidateExternalEndpoint(storage.Provider, storage.Endpoint); err != nil { errs = append(errs, field.Invalid(storagePath.Child("endpoint"), storage.Endpoint, err.Error())) } } @@ -527,17 +528,17 @@ func rejectSGLangRedisL2ScaleOut(cb *cachev1alpha1.CacheBackend) field.ErrorList // observability, with no signal to the operator. Validate it at admission // instead. An unset annotation (or an explicit empty value) is accepted. func rejectInvalidKernelCheckAnnotation(cb *cachev1alpha1.CacheBackend) field.ErrorList { - v, ok := cb.Annotations[adapterruntime.AnnotationLMCacheKernelCheck] - if !ok || adapterruntime.IsValidKernelCheckMode(v) { + v, ok := cb.Annotations[enginebinding.AnnotationLMCacheKernelCheck] + if !ok || enginebinding.IsValidKernelCheckMode(v) { return nil } return field.ErrorList{ field.Invalid( - field.NewPath("metadata", "annotations").Key(adapterruntime.AnnotationLMCacheKernelCheck), + field.NewPath("metadata", "annotations").Key(enginebinding.AnnotationLMCacheKernelCheck), v, fmt.Sprintf("must be one of %q, %q, %q, %q (or unset)", - adapterruntime.KernelCheckModeAuto, adapterruntime.KernelCheckModeReportOnly, - adapterruntime.KernelCheckModeStrict, adapterruntime.KernelCheckModeOff), + enginebinding.KernelCheckModeAuto, enginebinding.KernelCheckModeReportOnly, + enginebinding.KernelCheckModeStrict, enginebinding.KernelCheckModeOff), ), } } @@ -679,7 +680,7 @@ func warnMooncakeEngineHostNetwork(cb *cachev1alpha1.CacheBackend) admission.War if !usesMooncakeStorage(cb) { return nil } - if adapterruntime.EngineHostNetworkRequested(cb) { + if enginebinding.EngineHostNetworkRequested(cb) { // Opted in: the pod webhook moves engine pods onto the host network, so the // data plane is complete and there is nothing left to warn about. return nil @@ -699,7 +700,7 @@ func warnMooncakeEngineHostNetwork(cb *cachev1alpha1.CacheBackend) admission.War // changed the pod's networking. hostNetwork is a privilege — a no-op that *looks* // like it granted one is worse than a rejection, so reject at the door. func rejectEngineHostNetworkOnBackendThatDoesNotNeedIt(cb *cachev1alpha1.CacheBackend) field.ErrorList { - if !adapterruntime.EngineHostNetworkRequested(cb) || + if !enginebinding.EngineHostNetworkRequested(cb) || usesMooncakeStorage(cb) { return nil } diff --git a/internal/webhook/v1alpha1/cachebackend_webhook_test.go b/internal/webhook/v1alpha1/cachebackend_webhook_test.go index 5f03d57e..0725648f 100644 --- a/internal/webhook/v1alpha1/cachebackend_webhook_test.go +++ b/internal/webhook/v1alpha1/cachebackend_webhook_test.go @@ -20,6 +20,7 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" builtinruntime "github.com/cachebox-project/inference-cache/internal/adapters/builtin/runtime" + "github.com/cachebox-project/inference-cache/internal/enginebinding" backendadapter "github.com/cachebox-project/inference-cache/pkg/adapters/backend" adapterruntime "github.com/cachebox-project/inference-cache/pkg/adapters/runtime" ) @@ -53,9 +54,9 @@ func i32p(v int32) *int32 { return &v } func defaultShippingRegistry() *adapterruntime.Registry { registry := adapterruntime.NewRegistry() - registry.Register(builtinruntime.NewVLLMLMCacheAdapter()) - registry.Register(builtinruntime.NewSGLangLMCacheAdapter()) - registry.Register(builtinruntime.NewSGLangHiCacheAdapter()) + registry.Register(builtinruntime.NewVLLMLMCacheAdapter(builtinruntime.SubscriberConfig{})) + registry.Register(builtinruntime.NewSGLangLMCacheAdapter(builtinruntime.SubscriberConfig{})) + registry.Register(builtinruntime.NewSGLangHiCacheAdapter(builtinruntime.SubscriberConfig{})) return registry } @@ -1009,19 +1010,19 @@ func TestValidator_InvalidKernelCheckAnnotationRejected(t *testing.T) { // A typo for "strict" would silently fall back to "auto" (report-only) and // disable the fail-closed gate — reject it at admission instead. bad := newBackend() - bad.Annotations = map[string]string{adapterruntime.AnnotationLMCacheKernelCheck: "strcit"} + bad.Annotations = map[string]string{enginebinding.AnnotationLMCacheKernelCheck: "strcit"} requireInvalidWithCause(t, v, bad, "metadata.annotations[inferencecache.io/lmcache-kernel-check]", "must be one of") // Every known value — and an unset annotation — is accepted. for _, val := range []string{ - adapterruntime.KernelCheckModeAuto, - adapterruntime.KernelCheckModeReportOnly, - adapterruntime.KernelCheckModeStrict, - adapterruntime.KernelCheckModeOff, + enginebinding.KernelCheckModeAuto, + enginebinding.KernelCheckModeReportOnly, + enginebinding.KernelCheckModeStrict, + enginebinding.KernelCheckModeOff, "", // explicit empty == unset } { ok := newBackend() - ok.Annotations = map[string]string{adapterruntime.AnnotationLMCacheKernelCheck: val} + ok.Annotations = map[string]string{enginebinding.AnnotationLMCacheKernelCheck: val} if _, err := v.ValidateCreate(context.Background(), ok); err != nil { t.Fatalf("valid kernel-check annotation %q rejected: %v", val, err) } @@ -2028,7 +2029,7 @@ func (stubVLLMLMCacheAdapter) EngineContainerName() string { return "vllm" } // it is a remote-storage binding property, not a separate cache type. func stubRegistry() *adapterruntime.Registry { r := adapterruntime.NewRegistry() - r.Register(builtinruntime.NewVLLMLMCacheAdapter()) + r.Register(builtinruntime.NewVLLMLMCacheAdapter(builtinruntime.SubscriberConfig{})) return r } diff --git a/pkg/adapters/runtime/wire_contract.go b/pkg/adapters/backend/endpoint.go similarity index 80% rename from pkg/adapters/runtime/wire_contract.go rename to pkg/adapters/backend/endpoint.go index b88a4810..115d2f75 100644 --- a/pkg/adapters/runtime/wire_contract.go +++ b/pkg/adapters/backend/endpoint.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -package runtime +package backend import ( "fmt" @@ -13,27 +13,6 @@ import ( cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" ) -// Engine-side wire names are public so admission, controllers, tests, and -// out-of-tree adapters can share the exact protocol spellings without -// importing a built-in implementation. -const ( - EnvLMCacheRemoteURL = "LMCACHE_REMOTE_URL" - EnvLMCacheRemoteSerde = "LMCACHE_REMOTE_SERDE" - EnvLMCacheChunkSize = "LMCACHE_CHUNK_SIZE" - EnvLMCacheLocalCPU = "LMCACHE_LOCAL_CPU" - EnvLMCacheMaxLocalCPU = "LMCACHE_MAX_LOCAL_CPU_SIZE" - EnvVLLMUseV1 = "VLLM_USE_V1" - EnvInferenceCacheFailOpen = "INFERENCECACHE_FAIL_OPEN" - EnvPythonHashSeed = "PYTHONHASHSEED" - EngineContainerName = "vllm" -) - -// EngineHostNetworkRequested reports whether the operator opted an engine pod -// using a Mooncake remote binding into host networking. -func EngineHostNetworkRequested(cache *cachev1alpha1.CacheBackend) bool { - return cache != nil && cache.Spec.Integration != nil && cache.Spec.Integration.EngineHostNetwork -} - // ValidateLMCacheEndpoint validates a bare host:port or lm://host:port. The // port must be a decimal integer in the TCP range 1-65535. func ValidateLMCacheEndpoint(value string) error { diff --git a/pkg/adapters/engineclient/grpc.go b/pkg/adapters/engineclient/grpc.go deleted file mode 100644 index f40813d8..00000000 --- a/pkg/adapters/engineclient/grpc.go +++ /dev/null @@ -1,19 +0,0 @@ -// SPDX-FileCopyrightText: 2026 The inference-cache Authors -// -// SPDX-License-Identifier: Apache-2.0 - -package engineclient - -import "context" - -// GRPCTokenizedClient is a placeholder for sending pre-tokenized input over -// vLLM's gRPC frontend (the TokenizedInput{input_ids} message SMG uses). The -// reference stack runs the OpenAI HTTP server, not the gRPC frontend, so this -// is stubbed behind the EngineClient interface until an engine that exposes the -// gRPC frontend is in scope. Implement here without touching callers. -type GRPCTokenizedClient struct{} - -// Complete reports ErrNotImplemented. -func (GRPCTokenizedClient) Complete(context.Context, string, string, []uint32, CompletionParams) (Completion, error) { - return Completion{}, ErrNotImplemented -} diff --git a/pkg/adapters/runtime/adapter.go b/pkg/adapters/runtime/adapter.go index 969219eb..feae9f1f 100644 --- a/pkg/adapters/runtime/adapter.go +++ b/pkg/adapters/runtime/adapter.go @@ -178,7 +178,7 @@ func (p SupportedPair) String() string { // when it can enumerate the concrete (runtime, backend) pairs it accepts. // Adapters that match a single canonical pair (the vLLM+LMCache adapter, the // future SGLang HiCache adapter) implement it; permissive adapters that -// accept arbitrary backends (e.g. the in-tree reference adapter) leave it +// accept arbitrary backends can leave it // off and simply do not contribute to [Registry.SupportedPairs]. type PairLister interface { SupportedPairs() []SupportedPair @@ -224,44 +224,3 @@ func ResolveRuntimeID(cache *cachev1alpha1.CacheBackend) RuntimeID { } return RuntimeID(strings.ToLower(string(cache.Spec.Runtime))) } - -// Options configures runtime adapters and is passed through by the built-in -// production composition. Zero values are -// valid: empty PolicyServerGRPCAddress falls back to the package default, and -// empty SubscriberImage disables sidecar auto-attach (see the field doc for -// why). -type Options struct { - // SubscriberImage is the image reference the vLLM/LMCache adapter uses for - // the kvevent-subscriber sidecar across remote bindings (the KV-event stream - // is engine-side, not - // store-specific). Empty (the zero value) - // **disables** sidecar auto-attach — the adapter returns no sidecar - // at all. Auto-attach is opt-in by design: a nonexistent default - // image would put the sidecar container into ImagePullBackOff and - // keep the engine pod from going Ready. See [DefaultSubscriberImage] - // for the build-tag operators pin to (or a digest-pinned production - // image), passed through the controller's --kvevent-subscriber-image - // flag. - SubscriberImage string - - // PolicyServerGRPCAddress overrides the host:port the kvevent- - // subscriber sidecar dials to ReportCacheState. Empty selects the - // package default ([DefaultPolicyServerGRPCAddress]), which assumes the - // in-cluster Service produced by config/server installed into the - // inference-cache-system namespace. - PolicyServerGRPCAddress string -} - -// Option mutates [Options] for callers that prefer the functional-option -// style. Either Options{...} or a chain of Option helpers work. -type Option func(*Options) - -// WithSubscriberImage sets [Options.SubscriberImage]. -func WithSubscriberImage(image string) Option { - return func(o *Options) { o.SubscriberImage = image } -} - -// WithPolicyServerGRPCAddress sets [Options.PolicyServerGRPCAddress]. -func WithPolicyServerGRPCAddress(addr string) Option { - return func(o *Options) { o.PolicyServerGRPCAddress = addr } -} diff --git a/pkg/adapters/runtime/kvevent_subscriber.go b/pkg/adapters/runtime/kvevent_subscriber.go deleted file mode 100644 index bb1ccdf3..00000000 --- a/pkg/adapters/runtime/kvevent_subscriber.go +++ /dev/null @@ -1,162 +0,0 @@ -// SPDX-FileCopyrightText: 2026 The inference-cache Authors -// -// SPDX-License-Identifier: Apache-2.0 - -package runtime - -import ( - "fmt" - - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/resource" - - cachev1alpha1 "github.com/cachebox-project/inference-cache/api/v1alpha1" -) - -// SubscriberSidecarParams carries the per-adapter inputs to -// [RenderSubscriberSidecar]. Image and ServerAddr come from the controller -// flags (operator-supplied); Cache + Pod are the admission inputs; HashScheme -// and EngineZMQPortStr are the engine-specific wiring each adapter pins so the -// one subscriber binary speaks the right engine's dialect. -type SubscriberSidecarParams struct { - // Image is the kvevent-subscriber sidecar image. Empty disables - // auto-attach (the builder returns (nil, nil)) — see [DefaultSubscriberImage]. - Image string - // ServerAddr is the policy-server gRPC address the sidecar dials. Empty - // falls back to [DefaultPolicyServerGRPCAddress]. - ServerAddr string - // Cache + Pod are the admission inputs the identity flags derive from so - // the CR is the single source of truth (no operator-supplied identity). - Cache *cachev1alpha1.CacheBackend - Pod *corev1.Pod - // HashScheme is the engine prefix-hash domain the sidecar tags every report - // with ("vllm", "sglang"). The index keys on it so engines stay disjoint - // (no cross-engine false hits on identical prefix bytes). - HashScheme string - // EngineZMQPortStr is the port the engine's KV-event ZMQ PUB endpoint binds - // (both vLLM and SGLang default to 5557; parameterised so a future engine - // can differ without touching this builder). - EngineZMQPortStr string -} - -// RenderSubscriberSidecar renders the kvevent-subscriber sidecar the Pod webhook -// appends to an engine pod so its KV-cache events flow to the policy server with -// no out-of-band bring-up. It is shared by every adapter whose engine emits the -// vLLM-style ZMQ KV-event stream — the vLLM+LMCache adapter across its remote bindings -// (HashScheme "vllm") and the SGLang+LMCache adapter (HashScheme "sglang") today -// — because the stream is produced by the engine itself, independent of which L2 -// store the engine offloads to. The engine dialect (HashScheme + EngineZMQPortStr) -// is the only thing that varies across those adapters, so the rendered container -// is otherwise identical for a given integration mode and lives here as the -// single source of truth. -// -// The container shares the engine pod's network namespace, so the subscriber -// dials the engine over 127.0.0.1 (the ZMQ PUB endpoint on EngineZMQPortStr); -// identity flags are derived from Cache + Pod (--replica-id from pod.Name via the -// downward API, --tenant-id from pod.Namespace ditto, --model-id from -// spec.observation.modelID, --hash-scheme from -// HashScheme) so the CR is the single source of truth. -// -// The flag surface here is deliberately the intersection of what the shipped -// kvevent-subscriber binary accepts: passing flags the binary doesn't know -// would crash the sidecar on startup (Go's flag package rejects unknown flags). -// Stats-path flags (--engine-metrics-url, --stats-interval, etc.) are added -// when the binary itself learns to scrape and emit ReplicaStats. -// -// Returns (nil, nil) when Image is empty (auto-attach is opt-in — a nonexistent -// default image would put the sidecar into ImagePullBackOff and keep the engine -// pod from going Ready, turning the cache into a serving dependency the fail-open -// posture exists to avoid) or when the served model id is not derivable from the -// CR (the subscriber's --model-id flag is required, so emitting a container that -// would CrashLoopBackOff is worse than skipping; the webhook logs the skip and -// the next admission picks it up once the operator sets the observation model). -// ServerAddr falls back to [DefaultPolicyServerGRPCAddress] when empty. -func RenderSubscriberSidecar(p SubscriberSidecarParams) (*corev1.Container, error) { - if p.Cache == nil { - return nil, fmt.Errorf("observation sidecar: cache is nil") - } - if p.Pod == nil { - return nil, fmt.Errorf("observation sidecar: pod is nil") - } - if p.Image == "" { - return nil, nil - } - modelID := p.Cache.Spec.EffectiveObservationModelID() - if modelID == "" { - return nil, nil - } - serverAddr := p.ServerAddr - if serverAddr == "" { - serverAddr = DefaultPolicyServerGRPCAddress - } - - // Eviction-forwarding policy is mode-dependent (engine-agnostic — the same - // for vLLM and SGLang). In Offload mode the paired L2 tier (LMCache, or a - // Mooncake store) retains a block after the engine evicts it from GPU, so the - // engine's BlockRemoved does NOT mean the prefix is gone — forwarding it as - // PREFIX_EVICTED would drop a routing hint the replica can still cheaply serve - // from L2, so suppress it (--ignore-block-removed=true) and let the hint age - // out on its freshness TTL. In EventsOnly mode there is NO L2 retaining blocks, - // so a BlockRemoved genuinely means the prefix is gone and the hint MUST be - // pruned — omit the flag (the subscriber binary defaults it to false). Soft - // state means a stale hint is a cache miss at worst, while a missing one routes - // the request away from its warm replica — the opposite risk in each mode, - // hence the opposite default. (Mooncake is always an L2 store, so it takes the - // Offload branch unless an operator sets EventsOnly, which is a contradiction - // the admission validator already rejects.) - args := []string{ - "--engine-endpoint=tcp://127.0.0.1:" + p.EngineZMQPortStr, - "--server=" + serverAddr, - "--replica-id=$(POD_NAME)", - "--tenant-id=$(POD_NAMESPACE)", - "--model-id=" + modelID, - "--hash-scheme=" + p.HashScheme, - } - if !p.Cache.Spec.IsEventsOnly() { - args = append(args, "--ignore-block-removed=true") - } - - nonRoot := true - noPrivEsc := false - readOnlyRoot := true - uid := int64(65532) - return &corev1.Container{ - Name: SubscriberContainerName, - Image: p.Image, - ImagePullPolicy: corev1.PullIfNotPresent, - // pod.Name is empty at admission for generateName pods; resolve - // via the downward API so the value is filled in at container - // start. K8s expands $(VAR) references in args from the - // container's own env, which lets the literal CR-derived fields - // (model id, hash scheme) live next to the dynamically resolved - // ones in one place. - Env: []corev1.EnvVar{ - { - Name: "POD_NAME", - ValueFrom: &corev1.EnvVarSource{FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.name"}}, - }, - { - Name: "POD_NAMESPACE", - ValueFrom: &corev1.EnvVarSource{FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.namespace"}}, - }, - }, - Args: args, - Resources: corev1.ResourceRequirements{ - Requests: corev1.ResourceList{ - corev1.ResourceCPU: resource.MustParse("10m"), - corev1.ResourceMemory: resource.MustParse("64Mi"), - }, - Limits: corev1.ResourceList{ - corev1.ResourceCPU: resource.MustParse("200m"), - corev1.ResourceMemory: resource.MustParse("128Mi"), - }, - }, - SecurityContext: &corev1.SecurityContext{ - RunAsNonRoot: &nonRoot, - RunAsUser: &uid, - AllowPrivilegeEscalation: &noPrivEsc, - ReadOnlyRootFilesystem: &readOnlyRoot, - Capabilities: &corev1.Capabilities{Drop: []corev1.Capability{"ALL"}}, - }, - }, nil -} diff --git a/pkg/adapters/runtime/lmcache_shared.go b/pkg/adapters/runtime/lmcache_shared.go deleted file mode 100644 index 5ee934b5..00000000 --- a/pkg/adapters/runtime/lmcache_shared.go +++ /dev/null @@ -1,24 +0,0 @@ -// SPDX-FileCopyrightText: 2026 The inference-cache Authors -// -// SPDX-License-Identifier: Apache-2.0 - -package runtime - -// Shared kvevent-subscriber sidecar defaults. Vendor-neutral; production -// should set the image to a digest-pinned reference and the policy-server -// address to the in-cluster Service DNS the operator's server exposes. -const ( - // SubscriberContainerName is the well-known name for the - // kvevent-subscriber sidecar. Webhook callers use it to short-circuit - // re-admission, and operators can address the sidecar without guessing. - SubscriberContainerName = "kvevent-subscriber" - - // DefaultSubscriberImage is the well-known dev tag the Makefile's - // subscriber-image target emits. Auto-attach remains opt-in; a missing - // image must not put an otherwise healthy engine pod in ImagePullBackOff. - DefaultSubscriberImage = "ghcr.io/cachebox-project/inference-cache-subscriber:dev" - - // DefaultPolicyServerGRPCAddress is the in-cluster Service DNS the - // kvevent-subscriber sidecar dials by default. - DefaultPolicyServerGRPCAddress = "inference-cache-server.inference-cache-system.svc.cluster.local:9090" -) diff --git a/pkg/adapters/runtime/reference.go b/pkg/adapters/runtime/reference_test.go similarity index 100% rename from pkg/adapters/runtime/reference.go rename to pkg/adapters/runtime/reference_test.go diff --git a/pkg/adapters/engineclient/engineclient.go b/pkg/engineclient/engineclient.go similarity index 71% rename from pkg/adapters/engineclient/engineclient.go rename to pkg/engineclient/engineclient.go index 174d2ec1..2e9b9409 100644 --- a/pkg/adapters/engineclient/engineclient.go +++ b/pkg/engineclient/engineclient.go @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: Apache-2.0 -// Package engineclient sends a PRE-TOKENIZED prompt (token IDs) to an inference +// Package engineclient sends a pre-tokenized prompt (token IDs) to an inference // engine. It is the "pass tokens to the engine" half of server-side tokenization: // the engine caches exactly the tokens the router fingerprinted, so the routing // key and the engine's cache key match by construction — no tokenizer-parity @@ -12,17 +12,13 @@ // it on the hot path. A gateway, benchmark, or canary drives the flow // (tokenize → fingerprint → LookupRoute → pick replica → Complete). // -// Placement: it lives under pkg/adapters/ because it adapts to an inference -// engine's request API (the OpenAI /v1/completions and, later, vLLM gRPC -// surfaces). Unlike the engine subscriber under pkg/adapters/engine (which a -// binary owns for KV-event ingest), this egress client belongs to no binary — -// it is harness/demonstrator code used by the canary and future gateway clients. +// The supported boundary is deliberately narrow: EngineClient, CompletionParams, +// Completion, OpenAIClient, NewOpenAI, and the pre-tokenized OpenAI-compatible +// /v1/completions mapping. It does not promise authentication, retries, endpoint +// discovery, load balancing, streaming, tracing, or a complete OpenAI API SDK. package engineclient -import ( - "context" - "errors" -) +import "context" // CompletionParams carries the sampling knobs a caller sets per request. Kept // minimal on purpose — this is a routing/cache demonstrator, not a full @@ -48,6 +44,3 @@ type EngineClient interface { // re-tokenization) so the cached prefix equals the fingerprinted tokens. Complete(ctx context.Context, endpoint, model string, tokenIDs []uint32, p CompletionParams) (Completion, error) } - -// ErrNotImplemented is returned by clients whose transport is not wired yet. -var ErrNotImplemented = errors.New("engineclient: not implemented") diff --git a/pkg/adapters/engineclient/openai.go b/pkg/engineclient/openai.go similarity index 100% rename from pkg/adapters/engineclient/openai.go rename to pkg/engineclient/openai.go diff --git a/pkg/adapters/engineclient/openai_test.go b/pkg/engineclient/openai_test.go similarity index 89% rename from pkg/adapters/engineclient/openai_test.go rename to pkg/engineclient/openai_test.go index 3c5e3365..7812e557 100644 --- a/pkg/adapters/engineclient/openai_test.go +++ b/pkg/engineclient/openai_test.go @@ -7,7 +7,6 @@ package engineclient import ( "context" "encoding/json" - "errors" "net/http" "net/http/httptest" "testing" @@ -97,15 +96,6 @@ func TestOpenAIErrorsOnNoChoices(t *testing.T) { } } -// The gRPC TokenizedInput client is a stub for now — it must report that -// clearly rather than silently doing nothing. -func TestGRPCClientNotImplemented(t *testing.T) { - var c EngineClient = GRPCTokenizedClient{} - if _, err := c.Complete(context.Background(), "engine:8000", "m", []uint32{1}, CompletionParams{}); !errors.Is(err, ErrNotImplemented) { - t.Fatalf("err = %v, want ErrNotImplemented", err) - } -} - func equalU32(a, b []uint32) bool { if len(a) != len(b) { return false diff --git a/pkg/fingerprint/chain.go b/pkg/fingerprint/chain.go index d73a8124..fc9e46d1 100644 --- a/pkg/fingerprint/chain.go +++ b/pkg/fingerprint/chain.go @@ -10,7 +10,7 @@ package fingerprint // (block i is the i-th full block, so block 0 covers the first blockSize tokens), // and blockTokenCounts[i] is the per-block token count (always blockSize for a // full block). It is the query-side mirror of the subscriber's per-block PrefixEntry -// ingest (pkg/adapters/engine/positional.go) — so a lookup built from the same +// ingest (internal/subscriber/positional.go) — so a lookup built from the same // tokens the engine cached matches the ingested keys by construction. // // Partial trailing tokens (fewer than blockSize) are discarded, matching engines diff --git a/pkg/index/doc.go b/pkg/index/doc.go deleted file mode 100644 index 100468d6..00000000 --- a/pkg/index/doc.go +++ /dev/null @@ -1,14 +0,0 @@ -// SPDX-FileCopyrightText: 2026 The inference-cache Authors -// -// SPDX-License-Identifier: Apache-2.0 - -// Package index is part of inferencecache-server: the cluster cache-state aggregator -// (the CacheIndex), populated from engine KV events and queried by LookupRoute. -// Observability and routing input only — not a routing-decision substrate. -// -// The index engine (the in-memory store, ingestion, eviction, ranking) runs only -// in the server binary. The Snapshot* types are the one deliberate exception: -// they are the read-only wire contract for the server's /snapshot HTTP endpoint -// and are intentionally shared — the controller's CacheIndex poller imports them -// to decode that endpoint when reflecting the aggregate into CacheIndex status. -package index diff --git a/pkg/render/doc.go b/pkg/render/doc.go index e4347a39..9b7e621f 100644 --- a/pkg/render/doc.go +++ b/pkg/render/doc.go @@ -2,8 +2,11 @@ // // SPDX-License-Identifier: Apache-2.0 -// Package render is the mutable-slot prompt rendering engine (the "wedge"): it turns -// templated prompts into stable cache keys so a gateway's cache-aware routing matches -// on real prompts. Used by inferencecache-server's RenderTemplate RPC; kept importable -// as a standalone library for the OSS standardization play. +// Package render reserves the public import path for a planned reusable +// RenderTemplate implementation. +// +// The package currently has no production implementation or stable Go API, +// and the inference-cache server does not depend on it. No compatibility +// guarantee is made until a concrete implementation requirement defines the +// contract. package render diff --git a/pkg/testing/doc.go b/pkg/testing/doc.go deleted file mode 100644 index 2ca7f282..00000000 --- a/pkg/testing/doc.go +++ /dev/null @@ -1,8 +0,0 @@ -// SPDX-FileCopyrightText: 2026 The inference-cache Authors -// -// SPDX-License-Identifier: Apache-2.0 - -// Package testing provides envtest helpers used by this repository's test -// suites. It is not a supported external testing API and is awaiting migration -// to internal/testutil. -package testing diff --git a/proto/inferencecache/v1alpha1/inferencecache.proto b/proto/inferencecache/v1alpha1/inferencecache.proto index 948b0371..1d0f080a 100644 --- a/proto/inferencecache/v1alpha1/inferencecache.proto +++ b/proto/inferencecache/v1alpha1/inferencecache.proto @@ -6,7 +6,7 @@ syntax = "proto3"; package inferencecache.v1alpha1; -option go_package = "github.com/cachebox-project/inference-cache/pkg/server/proto/inferencecache/v1alpha1;inferencecachev1alpha1pb"; +option go_package = "github.com/cachebox-project/inference-cache/gen/inferencecache/v1alpha1;inferencecachev1alpha1pb"; // InferenceCache is the cache-policy control-plane API. See // docs/design/grpc-contract.md. diff --git a/site/content/en/docs/administration/troubleshooting.md b/site/content/en/docs/administration/troubleshooting.md index 31e829e7..55f42b12 100644 --- a/site/content/en/docs/administration/troubleshooting.md +++ b/site/content/en/docs/administration/troubleshooting.md @@ -40,7 +40,7 @@ the condition `.message` first — the controller embeds the server's stage diag | Reason | Stage | Meaning | First response | |---|---|---|---| -| `ProbeIngestFailed` | ingest | The server's in-process index ingest path is dropping writes. (Not a subscriber problem — the probe bypasses the gRPC ingest surface by design.) | Read `.message`; check `inferencecache_backend_probe_result_total{stage="ingest",result="failed"}`; confirm `inferencecache_server_up == 1`; inspect server `pkg/index` logs. | +| `ProbeIngestFailed` | ingest | The server's in-process index ingest path is dropping writes. (Not a subscriber problem — the probe bypasses the gRPC ingest surface by design.) | Read `.message`; check `inferencecache_backend_probe_result_total{stage="ingest",result="failed"}`; confirm `inferencecache_server_up == 1`; inspect server `internal/index` logs. | | `ProbeRoutingFailed` | routing | `LookupRoute` did not return a clean `PREFIX_MATCH` for the probe's reserved replica — usually an internal `hash_scheme` regression that dropped the probe's scheme on ingest, or a lookup-filter regression. | Read `.message` (the server names the failure mode); check the `stage="routing"` probe counter; inspect server lookup-path logs. | | `ProbeT2Failed` | tier-2 | The tier-2 put/get cycle failed. Only reachable once a tier-2 prober is wired — none ships today, so this does not appear on a clean install. | Not actionable today. | diff --git a/site/content/en/docs/concepts/architecture.md b/site/content/en/docs/concepts/architecture.md index 56b1969b..cccadfa2 100644 --- a/site/content/en/docs/concepts/architecture.md +++ b/site/content/en/docs/concepts/architecture.md @@ -45,8 +45,9 @@ The gRPC + HTTP server. It: aggregate), `/policy` (controller writes resolved policy), `/probe` (functional self-test). All three are gated by ServiceAccount bearer auth + a `NetworkPolicy`. -Owns the index (`pkg/index`), the mutable-slot render pipeline (`pkg/render`), and the -deterministic content fingerprint (`pkg/fingerprint`). +Owns the index (`internal/index`) and uses the deterministic content fingerprint +(`pkg/fingerprint`). The `pkg/render` path is reserved for a planned reusable renderer; +the server does not depend on it today. The server **fails closed**: without `--allowed-controller-sa` or `--insecure-disable-auth` it exits rather than silently shipping unauthenticated endpoints. @@ -58,7 +59,7 @@ event stream, computes the content fingerprint in-pod, and calls the server's `ReportCacheState`. It sets `replica_id = ` and also runs a stats reporter that derives `cache_memory_bytes` from a scraped usage percentage. Auto-injection is opt-in — the controller injects it only when started with a `--kvevent-subscriber-image`. -The subscriber binary owns the engine event adapters in `pkg/adapters/engine`. +The subscriber binary owns the engine event adapters in `internal/subscriber`. ### `inferencecache` CLI diff --git a/site/content/en/docs/developer-guide/_index.md b/site/content/en/docs/developer-guide/_index.md index 80f4a0ed..736ba0fb 100644 --- a/site/content/en/docs/developer-guide/_index.md +++ b/site/content/en/docs/developer-guide/_index.md @@ -27,15 +27,15 @@ The repository is one operator split across two binaries plus the CRDs. In short |---|---| | A CRD field / new API type | `api/v1alpha1/` → `make manifests generate` | | Controller / reconciler logic | `internal/controller/` | -| gRPC handlers, server wiring | `pkg/server/` | -| Cache-state index logic | `pkg/index/` | -| Mutable-slot rendering | `pkg/render/` | +| gRPC handlers, server wiring | `internal/server/` | +| Cache-state index logic | `internal/index/` | +| Planned reusable rendering API (reserved; not implemented) | `pkg/render/` | | Built-in runtime / storage adapters | `internal/adapters/builtin/{runtime,storage}/` | | Public adapter extension contracts | `pkg/adapters/{runtime,backend}/` | | The gRPC contract | `proto/` → `make proto-gen` | Generated code (`config/crd/`, `config/rbac/role.yaml`, `zz_generated*.go`, -`pkg/server/proto/`) is committed but never hand-edited — regenerate and commit it with the +`gen/`) is committed but never hand-edited — regenerate and commit it with the source change. Each package's `doc.go` states which binary it belongs to. ## Design docs