Published wheels are not reproducible from source; `toolshield auto` is broken in PyPI 0.1.2

[VascoSch92]

Summary

The published PyPI wheels do not match what a clean checkout builds from the repository.

At current main (6eec2d20700566700d367d6576792e490c35a56e), the repo only tracks one bundled experience JSON and does not track toolshield/mcp_scan.py. However, the published wheels include many additional
bundled experience files, and the 0.1.1 wheel also included mcp_scan.py.

This causes two issues:

• a clean source build is missing most bundled safety experiences advertised in the README;
• the current PyPI release, 0.1.2, is missing toolshield/mcp_scan.py, so toolshield auto crashes.

Package contents mismatch

Current git main contains:

• Bundled experience JSON files: 1
• toolshield/mcp_scan.py: absent

PyPI toolshield==0.1.1 contains:

• Bundled experience JSON files: 30
• toolshield/mcp_scan.py: present

PyPI toolshield==0.1.2 contains:

• Bundled experience JSON files: 31
• toolshield/mcp_scan.py: absent

The only tracked bundled experience in git is: toolshield/experiences/claude-sonnet-4.5/gmail-mcp.json

But the published wheels include additional bundled experiences such as:

  filesystem-mcp.json
  terminal-mcp.json
  postgres-mcp.json
  playwright-mcp.json
  notion-mcp.json

Reproduction

1. A clean source build is missing bundled files

  git clone https://github.com/CHATS-Lab/ToolShield
  cd ToolShield
  git checkout 6eec2d20700566700d367d6576792e490c35a56e

  uv build --wheel

  python - <<'PY'
  import glob
  import zipfile

  z = zipfile.ZipFile(glob.glob("dist/*.whl")[0])
  names = z.namelist()

  print("experiences:", sum(
      x.startswith("toolshield/experiences/") and x.endswith(".json")
      for x in names
  ))
  print("mcp_scan.py:", sum(x.endswith("mcp_scan.py") for x in names))
  PY

Observed:

experiences: 1
mcp_scan.py: 0

Git confirms this is what is tracked:

  git ls-files toolshield/experiences
  # toolshield/experiences/claude-sonnet-4.5/gmail-mcp.json

  git ls-files toolshield/mcp_scan.py
  # no output

  git grep -n "mcp_scan"
  # toolshield/cli.py:532 imports toolshield.mcp_scan

2. toolshield auto is broken in PyPI 0.1.2

pip install "toolshield>=0.1.1,<0.2" currently resolves to 0.1.2.

python -c "from toolshield.mcp_scan import main" -> ModuleNotFoundError: No module named 'toolshield.mcp_scan'

The CLI also fails:

  toolshield auto --agent claude_code

Observed:

  File ".../toolshield/cli.py", line 532, in auto_discover
      from toolshield.mcp_scan import main as scan_main
  ModuleNotFoundError: No module named 'toolshield.mcp_scan'

3. Clean source builds cannot run advertised bundled import examples

The README advertises commands such as:

  toolshield import --exp-file terminal-mcp.json --agent claude_code
  toolshield import --exp-file filesystem-mcp.json --agent claude_code
  toolshield import --exp-file postgres-mcp.json --agent claude_code

But a wheel built from a clean checkout only includes gmail-mcp.json.

Example:

  pip install dist/toolshield-0.1.1-py3-none-any.whl
  toolshield import --exp-file terminal-mcp.json --agent claude_code

Observed:

  Experience file not found: 'terminal-mcp.json'
  Available bundled experiences for claude-sonnet-4.5: gmail-mcp.json

4. Version numbers are desynchronized

For PyPI 0.1.2:

  python -c "import importlib.metadata as m; print(m.version('toolshield'))"
  # 0.1.2

  python -c "import toolshield; print(toolshield.__version__)"
  # 0.1.0

At current main, pyproject.toml still says:

version = "0.1.1"

So there are three different version values:

  PyPI metadata:             0.1.2
  pyproject.toml on main:    0.1.1
  toolshield.__version__:    0.1.0

CI gap

Current CI only runs:

ruff check toolshield/
pytest tests/ -v --tb=short

The pytest step is conditional and there is currently no tests/ directory, so it is skipped.

CI does not currently:

• build the wheel from a clean checkout;
• inspect wheel contents;
• install the built wheel;
• smoke-test the installed CLI.

Likely root cause

The release wheels appear to have been built from local working trees containing files that were not committed. Hatch includes package files that are present on disk at build time, so untracked local files can
end up in a wheel, while a clean checkout silently omits them.

That explains why:

• PyPI wheels contain many experience files not present in git;
• mcp_scan.py appeared in 0.1.1 but disappeared in 0.1.2;
• clean source builds differ from published wheels.

Suggested fixes

1. Commit the bundled experience JSON files and toolshield/mcp_scan.py, or generate them through a committed, reproducible build step.

2. Add CI that builds a wheel from a clean checkout and asserts expected contents.

3. Smoke-test installed-wheel commands in CI, for example:

   • toolshield list
   • toolshield import --all --agent claude_code --source_location
   • toolshield auto --help

4. Single-source the package version, for example from package metadata or Hatch’s version configuration.

5. Add a release workflow that builds and publishes from a clean tag.

Impact

Users installing the current PyPI release cannot run toolshield auto.

Users building from source get a different wheel than PyPI and cannot use most README examples for bundled experiences.

This makes downstream integrations unreliable because package contents vary depending on how the wheel was produced.

[xli04]

Thanks for the very precise bug report, @VascoSch92 — you were right on every count. Fixed and published as toolshield==0.1.3.

What was wrong (PyPI 0.1.2 was built from a local working tree that had files on disk but not in git, so the published wheel didn't match the repo):

• toolshield/mcp_scan.py — untracked, but imported by cli.py:auto_discover → why toolshield auto crashed.
• toolshield/experience_store.py — untracked, but is the documented public API (from toolshield import ExperienceStore).
• 30/31 of the toolshield/experiences/ JSON files — untracked.
• __version__ was hardcoded to "0.1.0" in __init__.py while pyproject.toml said "0.1.3".

What changed:

commit	what
ea023f4	Tracked the missing files. Per your suggestion, we're shipping only the claude-sonnet-4.5 experiences (the model we have the most validation on); added an explicit exclude list in pyproject.toml so other research artefacts can't accidentally leak into a future release. __version__ now reads from installed package metadata so it can't desync from pyproject.toml again.
f9bc16d	New wheel-contents CI job that builds from a clean git archive HEAD and asserts the wheel contains every file the public API imports, that the excluded experience subdirs do not leak in, and that pip install + import toolshield + ExperienceStore.load_bundled(...) round-trip. This is the same failure mode you reported, exercised on every push.
d938c3f	Lint fix: narrowed a bare except in mcp_scan.py to OSError (ruff E722 flagged it once mcp_scan.py was actually in the repo).

Your reproducer now passes:

```python

from toolshield import ExperienceStore
from toolshield.mcp_scan import scan_port
from toolshield.cli import main
ExperienceStore.list_bundled()
['filesystem-mcp', 'gmail-mcp', 'notion-mcp', 'playwright-mcp', 'postgres-mcp', 'terminal-mcp']
```

Live evidence (PyPI install, your smoke test, byte-identical local rebuild — sha256 matches PyPI for both wheel and sdist) is checked in on the SDK PR side under .pr/ since that's where the convention applies.

Please give it a try and let me know if anything still looks off — thanks again for the thorough report.