Skip to content

ci: verify adapter imports on a bare install - #7

Merged
yash161004 merged 1 commit into
mainfrom
ci/bare-install-guard
Jul 29, 2026
Merged

ci: verify adapter imports on a bare install#7
yash161004 merged 1 commit into
mainfrom
ci/bare-install-guard

Conversation

@yash161004

Copy link
Copy Markdown
Owner

Closes #6.

Problem

The test job installs .[dev], and dev pulls openeval-core[fixtura,langchain,openai]. Every optional dependency is therefore always present in CI, so the suite cannot detect an adapter that has started requiring one.

That is precisely how the bug fixed in #5 shipped. openeval/adapters/__init__.py eagerly imported the LangChain adapter, so pip install openeval-core could not import any adapter — while every test stayed green. #5 removed the eager import but added nothing to stop it returning.

Change

A bare-install job that reproduces what a real user gets (pip install ., no extras) and asserts four things:

Assertion Why
fixtura + openai adapters import They have no third-party dependency and must never gain one implicitly
LangChain adapter fails only on access, and cleanly Its dep is genuinely optional; lazy resolution must keep it that way
Unknown attribute still raises AttributeError The lazy __getattr__ must not swallow typos
The optional deps really are absent Otherwise the job passes vacuously if an adapter dep ever becomes a base dependency — which would "fix" a failure here the wrong way

Wired into all-tests-passed, the required check, so it actually gates rather than sitting decorative.

One assumption corrected while writing it: from_openai_messages imports fine with no extras. That adapter parses message dicts and never imports the openai package — openai is in the extras for tests and examples only. The guard asserts that reality rather than a symmetry that doesn't exist.

Verified in a clean venv

confirmed: no optional adapter dependencies present
all import OK
expected on access: No module named 'langchain_core'
expected: module 'openeval.adapters' has no attribute 'does_not_exist'

And — the part that matters — reinstating the old eager import makes it fail:

import openeval.adapters.fixtura
  File ".../openeval/adapters/__init__.py", line 1, in <module>
    from .langchain import from_langchain_run
ModuleNotFoundError: No module named 'langchain_core'
>>> guard correctly FAILS - regression caught

Restoring the lazy version makes it pass again. A guard that can't catch the regression it exists for is worse than none, so this was checked rather than assumed.

🤖 Generated with Claude Code

The `test` job installs .[dev], which pulls
openeval-core[fixtura,langchain,openai], so every optional dependency is always
present and the job cannot detect an adapter that has started requiring one.

That is how the eager `from .langchain import ...` in
openeval/adapters/__init__.py shipped: the suite stayed green the whole time
while `pip install openeval-core` could not import any adapter at all. The fix
in #5 removed the bug but left nothing to stop it coming back.

Add a bare-install job that reproduces what a real user gets and asserts:

- adapters with no third-party dependency import (fixtura, openai -- the openai
  adapter parses dicts and never imports the openai package)
- the LangChain adapter fails only on access, and cleanly
- an unknown attribute still raises AttributeError, so the lazy __getattr__
  cannot silently swallow typos
- the optional deps really are absent, so the job cannot pass vacuously if one
  ever becomes a base dependency

Wired into all-tests-passed, which is the required check, so it actually gates.

Verified against a clean venv: all assertions pass on current main, and
reinstating the eager import makes the import assertion fail as intended.

Closes #6

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@yash161004
yash161004 merged commit c513d63 into main Jul 29, 2026
7 checks passed
@yash161004
yash161004 deleted the ci/bare-install-guard branch July 29, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI never tests a bare install, so the lazy-adapter-import fix is unguarded against regression

1 participant