venv_resolve auto-provisioning installs unpinned `pip install lingtai`, causing kernel-version skew for CPR-launched agents

[huangzesen]

Summary

When resolve_venv() auto-creates the managed runtime venv at ~/.lingtai-tui/runtime/venv, _create_venv() runs pip install lingtai with no version constraint (src/lingtai/venv_resolve.py:181-184). A child agent launched through Agent._cpr_agent or cli.run therefore gets whatever version is newest on PyPI at provisioning time — potentially a different kernel than the parent that spawned it. The .lingtai-env.json marker only records OS/arch/Python identity, so this version skew is never detected or healed. The install should be pinned to the running kernel's version (lingtai.__version__), with a graceful fallback for dev builds, and the installed version should be recorded in the marker.

Narrative

The lingtai runtime is designed to be self-sufficient: an agent directory only needs an init.json, and the machinery figures out how to run it. The venv resolution chain in src/lingtai/venv_resolve.py implements this in three steps (docstring, lines 3-6):

1. init.json → venv_path → test → use if working
2. ~/.lingtai-tui/runtime/venv/ → test → use if working
3. Neither → create ~/.lingtai-tui/runtime/venv/ automatically

Step 3 is where the problem lives. _create_venv (src/lingtai/venv_resolve.py:156-186) finds a Python ≥ 3.11, creates a venv, and then:

print("Installing lingtai...", file=sys.stderr)
subprocess.run(
    [pip, "install", "lingtai"],
    check=True,
)

There is no version constraint. Whoever triggers auto-provisioning gets the latest release on PyPI, regardless of which kernel version the triggering process is running.

Consider how an agent actually hits this path today. A parent agent running lingtai 0.4.x decides to resuscitate a suspended peer via CPR. Agent._cpr_agent (src/lingtai/agent.py:733-770) reads the target's init.json, calls resolve_venv(target_data), and launches python -m lingtai run <target> with the resolved interpreter. If the machine has no working runtime venv yet — fresh host, a wiped ~/.lingtai-tui, or a marker mismatch that caused _remove_mismatched_managed_venv (src/lingtai/venv_resolve.py:353-355) to delete and recreate the venv — the CPR call silently provisions a brand-new venv with whatever lingtai happens to be newest, say 0.6.0. The child agent now boots on a kernel two minor versions ahead of the parent. Mail schemas, heartbeat conventions, signal-file semantics, or init.json fields may have drifted between those versions. The parent believes it launched "another one of itself"; it actually launched a stranger.

The same happens through cli.run (src/lingtai/cli.py:212-225): it calls resolve_venv(data) and then writes the resolved path back into init.json (data["venv_path"] = str(venv_dir)), making the skewed venv sticky for every future boot of that agent.

Why doesn't the existing self-healing machinery catch this? Because the environment marker was designed to detect platform drift, not package drift. _current_process_env_marker (src/lingtai/venv_resolve.py:234-251) records os, arch, and a Python identity block; _env_marker_matches (src/lingtai/venv_resolve.py:271-295) compares exactly those fields plus schema versions:

for key in (
    "sys_platform",
    "machine",
    "sysconfig_platform",
    "implementation",
    "version_major",
    "version_minor",
):
    if python.get(key) != current_python.get(key):
        return False

The installed lingtai version appears nowhere in the marker, the probe (_PYTHON_ENV_PROBE, lines 30-71), or the match logic. Nothing ever asks "does this venv contain the same kernel I am running?" So the skew is not a transient race — it persists until a human notices misbehavior and manually rebuilds the runtime.

The fix is straightforward because the running version is already exposed: src/lingtai/__init__.py:3-5 defines __version__ = _pkg_version("lingtai") via importlib.metadata. Better looks like: _create_venv pins the install to lingtai=={__version__}, falls back to unpinned install when the pinned version is unavailable on PyPI (a dev/editable build whose local version was never published), and the marker records the installed lingtai version so future diagnostics can see what the venv contains. Recording the version in the marker as informational (not a hard mismatch trigger) avoids destructive rebuild loops while giving env-marker check real observability.

Current behavior

• _create_venv (src/lingtai/venv_resolve.py:156-186) installs lingtai unpinned; the provisioned venv tracks PyPI-latest, not the provisioning process's version.
• _current_process_env_marker / _current_venv_env_marker (src/lingtai/venv_resolve.py:234-268) and the _PYTHON_ENV_PROBE script (lines 30-71) capture only OS/arch/Python identity; no lingtai package version.
• _env_marker_matches (src/lingtai/venv_resolve.py:271-295) and _env_marker_status_detail (lines 303-334) can therefore never flag kernel-version skew; _test_venv_detail (lines 124-153) only checks import lingtai succeeds.
• Callers that can trigger auto-provisioning: Agent._cpr_agent (src/lingtai/agent.py:733-770, resolve_venv at line 768), the avatar/venv path in src/lingtai/agent.py around line 1661, and cli.run (src/lingtai/cli.py:212-225), which also persists the skewed path into init.json.
• lingtai.__version__ exists (src/lingtai/__init__.py:5) but is unused by provisioning.

Detailed implementation plan

1. Add a version-resolution helper in src/lingtai/venv_resolve.py. Keep it dependency-light (do not import the lingtai package top-level object; query metadata directly to avoid import cycles):

   def _running_lingtai_version() -> str | None:
       try:
           from importlib.metadata import version, PackageNotFoundError
           return version("lingtai")
       except Exception:
           return None

2. Pin the install in _create_venv (src/lingtai/venv_resolve.py:156-186). Try the pinned spec first; on CalledProcessError, fall back to unpinned with a stderr warning (covers dev builds like 0.5.0.dev3+g1234abc that were never published, and yanked releases):

   version = _running_lingtai_version()
   specs = []
   if version and not _is_local_dev_version(version):
       specs.append(f"lingtai=={version}")
   specs.append("lingtai")
   
   for spec in specs:
       result = subprocess.run([pip, "install", spec], ...)
       if result.returncode == 0:
           break
       print(f"warning: install of {spec} failed; trying fallback", file=sys.stderr)
   else:
       raise subprocess.CalledProcessError(...)

   Add _is_local_dev_version(version: str) -> bool returning True when the version contains + (PEP 440 local segment) or .dev — skip the doomed pinned attempt entirely in that case rather than paying a failed network round-trip.

3. Record the installed lingtai version in the marker. Extend _PYTHON_ENV_PROBE (lines 30-71) to include the venv's own installed version:

   try:
       from importlib.metadata import version as _v
       lingtai_version = _v("lingtai")
   except Exception:
       lingtai_version = None
   # add "lingtai_version": lingtai_version to the printed JSON

   Mirror the field in _current_process_env_marker (lines 234-251) using _running_lingtai_version().

4. Schema handling — do NOT bump _ENV_MARKER_SCHEMA_VERSION. _env_marker_status_detail (lines 316-319) treats an unfamiliar schema_version as _MARKER_ERROR, and older kernels reading a bumped marker would refuse it; adding an optional field is backward-compatible under the existing version-1 schema. Existing markers without lingtai_version must continue to parse and match.

5. Keep _env_marker_matches (lines 271-295) unchanged with respect to lingtai_version — version skew must not be treated as _MARKER_MISMATCH, because for the managed runtime dir a mismatch triggers _remove_mismatched_managed_venv (lines 353-355), and rebuilding the venv on every parent/child version difference would cause rebuild churn (e.g., a newer venv being destroyed by an older parent). Instead, surface skew as a diagnostic: in _env_marker_status_detail, when everything else matches but lingtai_version differs, return _MARKER_MATCH with a non-empty detail string (or add an "info" key to the env-marker check payload in _env_marker_main, lines 362-396) such as "lingtai version skew: venv has 0.6.0, process runs 0.4.2".

6. Refresh the marker after installs. _write_env_marker (lines 337-343) already re-probes the venv, so it picks up lingtai_version for free once step 3 lands — verify _test_venv_detail's legacy-marker rewrite path (line 148) also stamps the new field.

7. No changes needed in src/lingtai/agent.py or src/lingtai/cli.py — both go through resolve_venv, which inherits the fix. Optionally add a one-line log in Agent._cpr_agent after resolve_venv when the check payload reports version skew, so CPR logs make skew visible.

Risks and alternatives

• Pinned version missing from PyPI. The running version may be unreleased (dev build, RC, or freshly yanked). The fallback-to-unpinned step plus the _is_local_dev_version short-circuit handles this; provisioning never gets worse than today.
• Treating version skew as a hard mismatch (alternative, rejected). Adding lingtai_version to _env_marker_matches would make _remove_mismatched_managed_venv destroy and rebuild the shared runtime venv whenever any process at a different version touches it — two agents at different versions on one host would fight, rebuilding the venv back and forth. Informational-only recording avoids that while still fixing the provisioning-time skew, which is the actual bug.
• Upgrading existing venvs in place (alternative, deferred). One could pip install lingtai==<running> into an existing venv when skew is detected. That is a behavior change with real blast radius (downgrades, dependency resolution mid-flight under a running agent) and deserves its own issue; this plan only fixes what gets installed at creation time and makes skew observable.
• Schema-version bump (rejected). Bumping _ENV_MARKER_SCHEMA_VERSION would make old kernels report _MARKER_ERROR on new markers and vice versa (lines 316-319), breaking mixed-version hosts — exactly the scenario this issue is about. Optional-field addition is strictly safer.
• Network behavior change. Pinned installs can fail where unpinned would succeed (e.g., index caching lag right after a release). The fallback covers this; worst case is one extra pip invocation.

Testing plan

Extend tests/test_venv_resolve.py (monkeypatch subprocess.run to capture pip argv; no network):

• test_create_venv_pins_running_version — with _running_lingtai_version patched to return "0.4.2", assert the first pip invocation is [pip, "install", "lingtai==0.4.2"].
• test_create_venv_falls_back_to_unpinned_when_pin_unavailable — first pip call returns nonzero, assert a second call with bare "lingtai" follows and _create_venv succeeds.
• test_create_venv_skips_pin_for_dev_version — with version "0.5.0.dev3+g1234abc", assert exactly one pip call, unpinned.
• test_is_local_dev_version — +local and .dev versions return True; "0.4.2", "1.0.0rc1" return expected values.
• test_env_marker_records_lingtai_version — after _write_env_marker against a stub venv whose probe reports lingtai_version, assert the marker JSON contains it.
• test_env_marker_without_version_field_still_matches — a legacy marker lacking lingtai_version yields _MARKER_MATCH (back-compat).
• test_env_marker_version_skew_is_not_mismatch — marker lingtai_version differs from probe/process; status is _MARKER_MATCH (or match-with-detail), and _remove_mismatched_managed_venv does not delete the directory.
• test_env_marker_check_reports_version_skew_detail — _env_marker_main(["env-marker", "check", "--venv", ...]) payload includes the skew detail string when versions differ.

Generated by a Claude Fable 5 deep review of the lingtai-kernel codebase.