feat(dev): add local model serving tool

[binaryaaron]

Summary

• Add a source-tree developer tool that compiles and manages local vLLM inference services from pinned TOML profiles.
• Support stock Hugging Face generation models and NVIDIA GLiNER or GLiNER2 through the pinned external vLLM Factory integration.
• Preserve pure compilation, digest-protected plans, task-aware capability probes, PID-reuse protection, and bounded process-group cleanup.
• Preserve vLLM API-key authentication on the Factory detector compatibility route.
• Ship seven profiles: NVIDIA GLiNER, GLiNER2, TinyLlama, GPT-OSS 20B and 120B, Qwen3 30B A3B Instruct, and NVIDIA Nemotron 3.5 Lightning.
• Add deployment guidance and a checked-in NVIDIA GPU container image that runs the same lifecycle inside the container boundary.

Scope and lifecycle

The tool owns one deployment domain: a managed local-process vLLM server.

• The v2 input schema has four sections: task, model, vllm, and local.
• Lifecycle commands are compile, launch, probe, status, and stop.
• Factory-backed detection and stock vLLM generation share one compiler and lifecycle.
• The Factory detector route enforces VLLM_API_KEY before request parsing or inference and returns 401 for missing, malformed, or incorrect credentials.
• The supplied container runs that unchanged local-process lifecycle. It does not add a second compiler path.
• Docker-specific compiler branches, native Transformers GLiNER serving, and Hugging Face cache discovery remain outside the design because no shipped profile requires them.
• Plan SHA-256 digests detect accidental storage or transport changes. They do not authenticate plan writers or prove semantic validity.

The tool remains under tools/ and is not part of the distributable Anonymizer package. It does not attach to existing endpoints or manage remote compute.

Validation

Validation on current HEAD 8562bdd:

• Focused inference-service and vLLM Factory tests: 58 passed and 3 expected skips.
• make format-check: passed.
• make typecheck: passed.
• make test: 1,341 passed and 3 expected skips.

Earlier full-branch validation on rebased HEAD 836a13f:

• make check: passed, including formatting, lint, ty, lock consistency, and SPDX checks.
• make docs-build: strict build passed.
• Range-diff matched all 17 product commits exactly after rebasing onto current main.
• Every rebased product commit has a DCO trailer and a valid repository ED25519 signature.

Runtime evidence from the patch-equivalent pre-rebase tree:

• Built tools/inference_service.Dockerfile as a real GPU image. The packaged CLI exposed all lifecycle commands and the vLLM Factory compatibility doctor passed.
• Co-hosted pinned NVIDIA GLiNER and GPT-OSS 20B services on one A100 80GB. Capability probes passed for dynamic labels, offsets, scores, and chat completions.
• Processed all 25 records in docs/data/NVIDIA_synthetic_biographies.csv with zero workflow failures.
• Published a sealed PR-versus-main pair to the W&B scorecard. The PR candidate completed in 458.09 seconds versus 558.92 seconds for the August 14 main baseline.
• Both managed services stopped through their receipts with cleanup_complete=true; final A100 memory use was 0 MiB.

Fresh NVSkills validation remains requested so generated skill artifacts and signatures bind to the current commit chain.

Fixes #252

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Post-rebase A100 dogfood completed on HEAD 1dd931ea8b7cf558c22b3cd2c2916d746492ca19 using the managed compiled plans and local endpoints for nvidia/gliner-pii plus pinned nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (33268dc8a6da85a56be2b12241453e4e1237bbe1).

• Full benchmark: 50 records across biographies redact+evaluate and legal hash+evaluate
• Both cases completed on their first attempt
• Model requests: 200 total, 200 successful, 0 failed
• Pipeline errors: 0 stage, 0 NDD workflow, 0 model workflow
• Replacement completeness: 0 missing final entities and 0 missing final values
• Managed shutdown: both services terminated with cleanup_complete: true
• Final A100 state: 0 MiB used

The analyzer reproduced the seven previously verified measurement false positives (1 biographies, 6 legal). No leak-metric changes are included in this hoster PR.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added pinned GPT-OSS 120B, GPT-OSS 20B, and Qwen3 30B A3B Instruct profiles on rebased HEAD 9d9536b990738e409e2ea335dd7d4b051dcd9c74. GPT-OSS 120B loaded successfully on the A100 through vLLM 0.27.1; the managed capability probe passed after accounting for reasoning-token output, and cleanup returned GPU use to 0 MiB. Fresh validation: 1,260 local tests passed, make check passed, strict docs build passed, and GitHub tests pass on Python 3.11, 3.12, and 3.13. NVSkills live evaluation is running.

[binaryaaron]

Medium hoster simplification is now on ccfedc8a4aeeb353d90bfc7692b4cc10a2c2c887.

• Removed Docker, native Transformers GLiNER, and cache-discovery branches.
• Compacted profiles to intent schema v2 with task, model, vllm, and local sections.
• Kept compile/launch/probe/inspect/cancel, plan digests, receipts, secrets, PID identity checks, bounded cleanup, LoRA, vLLM controls, and Factory-backed detection.
• Net hoster reduction: 988 production lines. Dedicated tests: 1,705 to 852 lines.
• Local validation: 1,226 tests passed, 35 focused inference-host tests passed, make check passed, and strict docs passed.
• A100 dogfood: local GPT-OSS 20B plus NVIDIA GLiNER passed managed probes; a two-record Anonymizer run found and redacted four PII entities with zero failed records; both receipt-owned services canceled cleanly; final GPU use was 0 MiB.

The implementation began as an Engineer Arc candidate. Parent review rejected its over-aggressive test reduction, restored lifecycle-focused behavioral coverage, removed a stray compiler branch, and reran all validation before push.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added the local-model deployment docs and checked-in GPU container image in commit 0034b8f. Validation: full suite 1,226 passed; make check passed; strict docs build passed; Docker image build passed; packaged CLI and vLLM Factory compatibility doctor passed inside the image.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Modern Python boundary refactor is now on HEAD ff40e2b.

• Closed FactoryPlugin typing now flows through intent, compiler, runtime, Factory integration, and adapter boundaries.
• Exhaustive match/assert_never dispatch covers owned task, environment, and plugin unions.
• Internal multi-field products are frozen slotted dataclasses; Factory metadata is one immutable exhaustive specification.
• CommandSpec owns redacted rendering, secret-source inventory, and strict execution-time resolution.
• HTTP client ownership is context-managed, compiler argument construction is more declarative, and obsolete delegation/test-loader seams are removed.
• CLI translation remains narrow while preserving exit 125 for known filesystem, transport, integrity, runtime, validation, TOML, and empty-source-revision failures.

A Terra implementation and independent-review cycle found and resolved plugin-spec duplication, secret ownership, positional chunk tuples, and stale delegation seams. Primary-agent review then caught the empty-source-revision exit regression and the remaining manual HTTP ownership flag before push.

Validation on the exact signed commit chain:

• focused inference-service tests: 36 passed, 3 skipped
• make check: passed, including ty
• make test: 1,227 passed, 3 skipped
• all four commits have DCO signoff and valid repository ED25519 signatures

No bundled skill files changed, so no new /nvskills-ci request is needed.

[binaryaaron]

Semantic cleanup is now on HEAD 7cc0c58037e64ae31f8f8ccfdfb5e5be3f817d9a.

• Reframed the draft input as LocalInferenceServiceSpec; profiles now use inference-service.local-spec/v2 in place, with no compatibility layer or schema-v3 migration.
• Renamed endpoint and compatibility concepts to EndpointAddress and CompatibilityAssessment; lifecycle commands are now status and stop.
• Removed redundant ServiceCompilation, CommandArgument, Generation.chat, RunPlan.runtime, intent_digest, declared_capabilities, and duplicate readiness coordinates.
• Preserved the pure spec-to-plan compiler, plan digest, secret-safe command resolution, task-aware readiness probing, process-group cleanup, and separate parent/child runtime boundaries.
• Tightened process identity: launch now fails closed and cleans up if a PID start marker cannot be recorded.
• Updated the deployment docs, README, devnote, profiles, and bundled Anonymizer skill to the new vocabulary.

A Terra reviewed-engineering Arc performed implementation, independent review, bounded fix-up, and confirmation. Review initially blocked on the PID-marker fallback and stale shipped instructions; both were fixed, and confirmation accepted the exact final tree with no findings.

Validation on that tree:

• focused inference-service tests: 24 passed
• make test: 1,229 passed, 3 skipped
• make check: passed, including formatting, lint, ty, lock, and SPDX checks
• make docs-build: passed
• final commit has DCO signoff and a valid repository ED25519 signature

[binaryaaron]

/nvskills-ci

[binaryaaron]

Follow-up council warnings addressed in abf8a7e:

• Bound the missing-start-marker cleanup path across TERM and KILL waits; an unreapable child now returns the normal diagnostic with cleanup_complete=false instead of hanging.
• Made plan validity explicit: SHA-256 detects accidental storage/transport changes only. The developer tool trusts plan writers and does not authenticate, sign, or recompile plans for semantic proof.
• Replaced stale reconnectable/proves/intent wording in CLI help, runtime/compiler docstrings, README, and deployment docs.

Validation:

• focused inference-service tests: 25 passed
• make check: passed
• make docs-build: passed
• commit has ED25519 signature and Signed-off-by trailer

[binaryaaron]

Post-fix A100 dogfood passed on HEAD abf8a7e.

• Managed services: pinned NVIDIA GLiNER on 127.0.0.1:8001 and pinned GPT-OSS 20B on 127.0.0.1:8000; GPT-OSS used the documented co-hosting cap of 0.75 GPU memory utilization.
• Launch receipts recorded exact PID/start markers. GLiNER passed dynamic-labels, offsets, and scores; GPT-OSS passed chat-completions.
• Required preview: 2 records, 14 entities across 8 labels, 0 failed records.
• Full Anonymizer Replace + Substitute run: 2/2 records transformed, 0 failed records, and no detected source value remained verbatim in substituted text.
• Local serving handled 24 chat-completion requests with no ERROR, Traceback, or Exception entries in either service log.
• Receipt-based stop: both services terminated with cleanup_complete=true; both status receipts report stopped; final A100 usage is 0 MiB.

Run artifacts are retained locally under /tmp/anonymizer-abf8a7e-full-run-U7pB7p.

[greptile-apps]

Failed requests retain workers

When one chunk of a multi-chunk detector request receives a non-200 or malformed pooling response, asyncio.gather propagates the error without cancelling the sibling workers. The middleware then returns HTTP 400 while those workers continue submitting the failed request's remaining chunks through the shared semaphore, delaying unrelated detector requests.

[lipikaramaswamy]

I agree this should use structured cancellation, such as TaskGroup, or explicitly cancel and await the remaining tasks.

[lipikaramaswamy]

vLLM registers its authentication middleware before custom middleware, while Starlette makes later-added middleware outermost. Because this handler returns directly for /v1/chat/completions without calling call_next, it appears that requests handled here never reach vLLM’s API-key check.

This matters because the deployment guidance suggests configuring api_key_env; users may reasonably expect that to protect detector requests. Could we either enforce the configured bearer token in this middleware or add an integration test showing that missing/incorrect tokens receive 401 while the correct token succeeds?

[binaryaaron]

Fixed in 8562bdd.

• The compatibility middleware now validates VLLM_API_KEY before request parsing or inference.
• Missing, malformed, and incorrect credentials return vLLM's 401 {"error":"Unauthorized"} response.
• Correct Bearer credentials and disabled authentication continue to succeed.
• Regression tests cover all five cases.

Validation: focused tests passed (58 passed, 3 skipped), make format-check passed, make typecheck passed, and the full suite passed (1,341 passed, 3 skipped).

[lipikaramaswamy]

After sending SIGTERM to the process group, the shutdown loop calls is_handle_running(handle), which checks the recorded leader PID. If the leader exits while one of its vLLM worker processes remains alive, cleanup can report cleanup_complete=True and skip the SIGKILL fallback.

Could we verify that the entire process group has exited before reporting successful cleanup? This seems particularly important for GPU workers, since a surviving child could continue holding GPU memory even though stop reports success.

[lipikaramaswamy]

Can we preserve the native PyTorch GLiNER serving path as an optional, installable component rather than deleting it entirely?

The vLLM Factory path makes sense for Linux/NVIDIA GPU deployments, but it does not work for the existing Mac/MPS and CPU use cases. I'm thinking about this from the perspective of fixing up the tutorial notebooks, where we want users to install something like nemo-anonymizer[notebooks] and launch a local GLiNER runtime directly from Python. On Colab this could use CUDA, on Apple Silicon MPS, and otherwise CPU.

Since tools/serve_gliner.py is not included in the wheel, perhaps its portable server implementation could move somewhere under src/anonymizer/ and be exposed through a helper such as start_runtime(gliner_device="auto"). The vLLM Factory implementation should remain the production deployment path though.