feat: Add batch processing for multiple packages

[rdwj]

Summary

veripak currently processes one package at a time via veripak check <package>. Users auditing a project's full dependency tree need to run the command repeatedly. Batch processing would allow checking multiple packages in a single invocation.

Real-world use case: auditing a list of 400 packages. This rules out synchronous MCP tool calls and simple chunking — we need an async architecture.

Proposed Enhancement

Shared batch engine

The CLI and MCP server should share the same batch processing engine. The difference is the interface, not the execution.

Core engine responsibilities:

• Parse dependency file formats (requirements.txt, pyproject.toml, package.json, go.mod, etc.)
• Run checks in parallel with configurable concurrency
• Respect API rate limits (NVD: 5 req/10s without key, 50 req/30s with key)
• Persist job state to disk (~/.veripak/jobs/) so jobs survive process restarts
• Report progress and partial results as packages complete
• Handle partial failures (some packages fail, others succeed)

CLI batch mode

veripak check --from requirements.txt
veripak check --from package-lock.json --concurrency 10
veripak check --from packages.txt --format json --output report.json

• Streams results to stdout as packages complete
• Progress bar or periodic status line
• Exit code reflects worst-case finding (0 = clean, 1 = CVEs found, 2 = errors)

MCP batch tools (async job pattern)

For MCP-only agents (no bash access), synchronous tool calls won't work at scale. A 400-package batch could take 10-20 minutes. MCP connections time out, agents abandon long-running calls, and context windows fill up.

Three tools:

veripak_batch_start — Kicks off a batch job, returns immediately with a job ID.

// Input
{
  "packages": [
    {"package": "requests", "ecosystem": "python", "versions_in_use": ["2.31.0"]},
    {"package": "lodash", "ecosystem": "javascript"}
  ],
  "skip_cves": false,
  "skip_download": true
}

// Response (immediate)
{
  "job_id": "batch-20260330-a1b2c3",
  "total_packages": 400,
  "status": "running"
}

veripak_batch_status — Poll for progress. Returns completed results so far plus overall status. The agent can call this periodically and process partial results without waiting for the full batch.

// Input
{"job_id": "batch-20260330-a1b2c3"}

// Response
{
  "job_id": "batch-20260330-a1b2c3",
  "status": "running",
  "completed": 147,
  "failed": 3,
  "total": 400,
  "results_since_last_poll": [
    {"package": "requests", "ecosystem": "python", ...},
    {"package": "flask", "ecosystem": "python", ...}
  ],
  "errors": [
    {"package": "some-obscure-lib", "error": "Registry timeout"}
  ]
}

Returning results_since_last_poll (rather than all results every time) keeps token usage manageable. The agent accumulates results across polls.

veripak_batch_cancel — Cancel a running job.

Job persistence

Jobs are stored in ~/.veripak/jobs/<job_id>/:

• manifest.json — input parameters, timestamps, status
• results/ — one JSON file per completed package
• errors.json — failed packages with error details

This means:

• Jobs survive MCP server restarts
• The CLI can read MCP-initiated job results and vice versa
• Old jobs can be cleaned up with veripak jobs prune

Why not simpler approaches

Approach	Problem at 400 packages
N parallel MCP tool calls	Most clients don't support 400 parallel calls; rate limits would still serialize them
Synchronous batch tool	10-20 min execution; connection timeout, agent gives up
Chunked batches (e.g., 25/chunk)	16 round trips; agent still has to orchestrate; token-heavy
SSE progress heartbeats	Keeps connection alive but still blocks the agent for the full duration

The async job pattern lets the agent fire-and-forget, do other work, and poll when convenient. It also handles the case where the agent's context fills up or it gets restarted — it can pick the job back up with the job ID.

Considerations

• Shared LLM token budget across all packages (relevant for agent pipeline mode; deterministic-only mode has no LLM cost)
• Concurrency defaults: 10 for registry APIs, 3 for LLM calls, 1 for NVD (rate-limited)
• results_since_last_poll design keeps MCP response sizes bounded regardless of batch size
• Job TTL: auto-cleanup after 24h or configurable retention
• MCP transport type (stdio vs streamable-http) doesn't matter with async pattern since no single call blocks for long

[rdwj]

Impact analysis completed and documented in research/batch-processing-impact.md.

Key findings:

• Two high-severity concurrency blockers to fix first (global usage tracking, NVD rate limiter race)
• Rate limits at scale make NVD, GitHub API, and Repology the tightest bottlenecks
• Significant test coverage gaps in the orchestration layer
• Recommended three-PR incremental approach: concurrency prep → batch engine + CLI → MCP async jobs