Skip to content

feat(dev): add managed local model serving tool #252

Description

@binaryaaron

Priority Level

Medium

Task Summary

Add a source-tree developer tool that deploys and manages local models through one vLLM process lifecycle. The tool should let contributors run Anonymizer against pinned generation and detection models on a GPU host or inside the supplied container without introducing a second deployment domain.

Technical Details & Implementation Plan

The implementation should:

  • Compile pinned TOML profiles into normalized plans with task, model, vllm, and local sections.
  • Provide compile, launch, probe, status, and stop commands.
  • Support stock Hugging Face generation models and NVIDIA GLiNER or GLiNER2 through vLLM Factory.
  • Preserve process identity with launch receipts, PID start markers, and process-group ownership checks.
  • Probe task-specific capabilities rather than treating a listening port as readiness.
  • Bound termination and cleanup so failed or stale processes cannot hang the tool indefinitely.
  • Run the same compiler and local-process lifecycle on a GPU host and inside the supplied container.
  • Document the deployment workflow and provide representative pinned profiles.

Acceptance criteria:

  • Compilation is pure and produces deterministic, digest-protected plans.
  • Plan digests are documented as accidental corruption detection, not authentication or semantic proof.
  • Generation and GLiNER profiles share one lifecycle implementation.
  • Capability probes validate chat completions or GLiNER dynamic labels, offsets, and scores as appropriate.
  • Status and stop operations reject stale or reused process identities.
  • Cleanup is bounded and reports whether it completed.
  • Tests cover lifecycle invariants and behavior rather than profile inventory tables.
  • Documentation covers both direct GPU-host deployment and the supplied container.
  • A full Anonymizer dataset run succeeds with locally hosted GLiNER and a generation model, and both services clean up successfully.

Out of scope:

  • Public package APIs for deployment.
  • Remote compute orchestration or attachment to existing endpoints.
  • Docker-specific compiler branches.
  • Native Transformers GLiNER serving.
  • Model-cache discovery.
  • Cryptographic authentication of plans.

Implementation is tracked in PR #212.

Dependencies

vLLM Factory integration is pinned by the repository's local-model dependency group.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    taskDevelopment task

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions