Repository navigation
test(ci): exercise the public API on a bare install (#189) - #211
Conversation
Every CI job installs the dev environment, so two whole classes of bug are invisible to it: a dependency declared only in an extra, and a code path the unit tests reach only through a mock. Both shipped in 1.11.0. Adds scripts/smoke_bare_install.py, which does what a user does — install the wheel, call enrich(), check the DataFrame — against a local stub of the OpenAI chat-completions API, so it needs no key and cannot flake. It covers pandas, Polars, a CSV path, and schema= structured output, and asserts on the returned values rather than the frame's shape: a totally failed run still has the right row count and column names. The new CI job installs the built wheel into a clean venv with no extras and runs it. The [mcp] leg catches extras-only import breakage; [all] is left out deliberately — the Install Profile job already proves it installs, and extras can only hide the bug class this job exists for. Also grows tests/unit/test_enrich.py an end-to-end class that patches the client factory instead of Pipeline.execute, so the loader, invocation, and parser stages actually run. Verified by mutation: all four fail when the stub client raises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughChangesEnrichment validation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant SmokeTest
participant Ondine
participant LocalHTTPStub
SmokeTest->>Ondine: Call enrich with dataframe, CSV, or schema
Ondine->>LocalHTTPStub: Send OpenAI-compatible completion request
LocalHTTPStub-->>Ondine: Return canned text, JSON-schema, or tool-call response
Ondine-->>SmokeTest: Return enriched result
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 97-107: Update the extras loop in the CI workflow to add a
conditional MCP smoke step when extras is "[mcp]". After the existing
smoke_bare_install.py invocation, import ondine.mcp and call create_server()
using the smoke environment’s Python so the MCP entrypoint and FastMCP wiring
are exercised.
- Around line 68-73: Update the bare-install-smoke job to explicitly set
job-level contents permission to read, and configure its actions/checkout@v6
step with persist-credentials disabled; do not rely on repository-level token
defaults.
In `@scripts/smoke_bare_install.py`:
- Around line 190-197: Update the stub instrumentation used by the schema
scenario around _assert_answered to record whether the request includes
response_format.type equal to "json_schema" or a tools payload, then assert that
structured-output evidence before or alongside validating the sentiment result.
Ensure the enrich(schema=Sentiment) case fails when neither request field is
observed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 80cef3fd-b141-4ecd-b2a4-b94e87724eaa
📒 Files selected for processing (3)
.github/workflows/ci.ymlscripts/smoke_bare_install.pytests/unit/test_enrich.py
| bare-install-smoke: | ||
| name: Bare Install Smoke | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v6 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow context =="
if [ -f .github/workflows/ci.yml ]; then
nl -ba .github/workflows/ci.yml | sed -n '1,140p'
else
echo "missing .github/workflows/ci.yml"
fi
echo
echo "== checkout usages in workflows =="
rg -n "actions/checkout|persist-credentials|permissions:" .github/workflows || true
echo
echo "== git diff stat/name =="
git diff --stat || true
git diff -- .github/workflows/ci.yml 2>/dev/null | nl -ba | sed -n '1,220p' || trueRepository: ptimizeroracle/ondine
Length of output: 222
🌐 Web query:
actions/checkout v6 default persist-credentials true README
💡 Result:
The default value for the persist-credentials input in actions/checkout v6 is true [1][2]. When set to true (the default), the action configures the authentication token or SSH key with the local git configuration [1][2]. Starting with v6, this process was updated to improve security: instead of storing credentials directly in the.git/config file, they are stored in a separate file under $RUNNER_TEMP [1][3][4]. This token is automatically removed during the post-job cleanup [1][5]. You can set persist-credentials to false if you wish to opt out of this behavior [1][5].
Citations:
- 1: https://github.com/actions/checkout/tree/v6.0.2?tab=readme-ov-file
- 2: https://raw.githubusercontent.com/actions/checkout/v6/action.yml
- 3: https://github.com/actions/checkout/?tab=readme-ov-file
- 4: https://github.com/actions/checkout/tree/v6.0.0
- 5: https://github.com/actions/checkout
🌐 Web query:
GitHub Actions workflow syntax default GITHUB_TOKEN permissions jobs job-name permissions contents read
💡 Result:
In GitHub Actions, you can control the permissions granted to the GITHUB_TOKEN by using the permissions key in your workflow file [1][2]. When you define the permissions key, any permission not explicitly listed is set to none [2][3]. To set the permissions for a specific job, you define the permissions key within that job's configuration [2]. The following example grants only read access to the repository contents for a single job: jobs: job-name: runs-on: ubuntu-latest permissions: contents: read steps: - uses: actions/checkout@v4 Key principles regarding GITHUB_TOKEN permissions include: 1. Hierarchy: If permissions are defined at the workflow level, they apply to all jobs in the workflow [4]. If defined at the job level, they override any workflow-level permissions for that specific job [2][4]. 2. Default Settings: If no permissions are explicitly set, the GITHUB_TOKEN inherits the default settings defined at the repository or organization level [5][2]. 3. Principle of Least Privilege: It is a security best practice to use the permissions key to grant only the minimum access required for each job [1][2]. For more information, see the official documentation on workflow syntax for GitHub Actions [6][2].
Citations:
- 1: https://docs.github.com/en/actions/tutorials/authenticate-with-github_token
- 2: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax?use_case=
- 3: https://github.blog/changelog/2021-04-20-github-actions-control-permissions-for-github_token/
- 4: https://adaptive-enforcement-lab.com/secure/github-actions-security/token-permissions/job-scoping/
- 5: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository
- 6: https://docs.github.com/actions/using-workflows/workflow-syntax-for-github-actions
Restrict the job token and disable persisted checkout credentials.
This job builds and executes checked-out code. actions/checkout defaults persist-credentials to true, so later commands can use the checkout credential.
Set contents: read for this job and set persist-credentials: false. Do not rely on repository-level default token permissions.
Proposed fix
bare-install-smoke:
name: Bare Install Smoke
runs-on: ubuntu-latest
+ permissions:
+ contents: read
steps:
- - uses: actions/checkout@v6
+ - uses: actions/checkout@v6
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| bare-install-smoke: | |
| name: Bare Install Smoke | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v6 | |
| bare-install-smoke: | |
| name: Bare Install Smoke | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| steps: | |
| - uses: actions/checkout@v6 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.28.0)
[warning] 73-73: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[warning] 68-111: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block
(excessive-permissions)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 68 - 73, Update the bare-install-smoke
job to explicitly set job-level contents permission to read, and configure its
actions/checkout@v6 step with persist-credentials disabled; do not rely on
repository-level token defaults.
Source: Linters/SAST tools
| for extras in "" "[mcp]"; do | ||
| echo "::group::install ondine${extras:-} (no dev environment)" | ||
| rm -rf "${RUNNER_TEMP}/smokeenv" | ||
| python -m venv "${RUNNER_TEMP}/smokeenv" | ||
| "${RUNNER_TEMP}/smokeenv/bin/python" -m pip install --quiet --upgrade pip | ||
| # Wheels only, and install the wheel by path: a source build would | ||
| # need a toolchain users do not have, and resolving by name could | ||
| # quietly pull the last release from PyPI instead of this commit. | ||
| "${RUNNER_TEMP}/smokeenv/bin/python" -m pip install --quiet \ | ||
| --only-binary=:all: "${wheel}${extras}" | ||
| "${RUNNER_TEMP}/smokeenv/bin/python" scripts/smoke_bare_install.py |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
# Find the public Ondine MCP import path before adding it to the smoke job.
rg -n -C 3 '\b(fastmcp|FastMCP)\b' ondine tests scriptsRepository: ptimizeroracle/ondine
Length of output: 6127
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '--- workflow outline ---\n'
wc -l .github/workflows/ci.yml
sed -n '80,115p' .github/workflows/ci.yml
printf '\n--- smoke script ---\n'
wc -l scripts/smoke_bare_install.py
sed -n '1,240p' scripts/smoke_bare_install.py
printf '\n--- mcp modules ---\n'
wc -l ondine/mcp/__init__.py ondine/mcp/server.py
sed -n '1,80p' ondine/mcp/__init__.py
sed -n '1,85p' ondine/mcp/server.py
printf '\n--- wheel metadata mentions mcp/fastmcp ---\n'
rg -n -C 2 -i '\b(mcp|fastmcp)\b' pyproject.toml setup.cfg setup.py ondine tests scripts 2>/dev/null || trueRepository: ptimizeroracle/ondine
Length of output: 250
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- workflow outline ---'
wc -l .github/workflows/ci.yml
sed -n '80,115p' .github/workflows/ci.yml
printf '%s\n' '\n--- smoke script ---'
wc -l scripts/smoke_bare_install.py
sed -n '1,240p' scripts/smoke_bare_install.py
printf '%s\n' '\n--- mcp modules ---'
wc -l ondine/mcp/__init__.py ondine/mcp/server.py
sed -n '1,80p' ondine/mcp/__init__.py
sed -n '1,85p' ondine/mcp/server.py
printf '%s\n' '\n--- wheel metadata mentions mcp/fastmcp ---'
rg -n -C 2 -i '\b(mcp|fastmcp)\b' pyproject.toml setup.cfg setup.py ondine tests scripts 2>/dev/null || trueRepository: ptimizeroracle/ondine
Length of output: 37231
Import and build the MCP server in the [mcp] iteration.
ondine[mcp] installs fastmcp, but smoke_bare_install.py does not exercise the MCP entrypoint, and ondine.mcp lazily defers create_server imports. Add a conditional MCP smoke step that imports ondine.mcp and calls create_server() so missing broken FastMCP wiring fails this CI leg.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 97 - 107, Update the extras loop in
the CI workflow to add a conditional MCP smoke step when extras is "[mcp]".
After the existing smoke_bare_install.py invocation, import ondine.mcp and call
create_server() using the smoke environment’s Python so the MCP entrypoint and
FastMCP wiring are exercised.
| _assert_answered( | ||
| ondine.enrich( | ||
| pd.DataFrame(rows), prompt, output_columns=["sentiment"], schema=Sentiment | ||
| ), | ||
| "sentiment", | ||
| 2, | ||
| "enrich(schema=)", | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Assert that schema= used a structured-output request.
This assertion passes if schema=Sentiment is ignored and enrich() uses the plain completion path. Both paths return "positive".
Record whether the stub received response_format.type == "json_schema" or tools. Fail the schema scenario when neither is present. This makes the smoke test detect a loss of structured-output propagation.
Proposed fix
class _ChatCompletionsStub(BaseHTTPRequestHandler):
+ saw_structured_request = False
+
def do_POST(self) -> None:
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])) or "{}")
response_format = body.get("response_format") or {}
+ if body.get("tools") or response_format.get("type") == "json_schema":
+ type(self).saw_structured_request = True
class Sentiment(BaseModel):
sentiment: str
+ _ChatCompletionsStub.saw_structured_request = False
_assert_answered(
ondine.enrich(
pd.DataFrame(rows), prompt, output_columns=["sentiment"], schema=Sentiment
),
"sentiment",
2,
"enrich(schema=)",
)
+ if not _ChatCompletionsStub.saw_structured_request:
+ raise AssertionError("enrich(schema=) did not request structured output")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/smoke_bare_install.py` around lines 190 - 197, Update the stub
instrumentation used by the schema scenario around _assert_answered to record
whether the request includes response_format.type equal to "json_schema" or a
tools payload, then assert that structured-output evidence before or alongside
validating the sentiment result. Ensure the enrich(schema=Sentiment) case fails
when neither request field is observed.
1 participant
Closes #189.
Why
Every CI job installs the dev environment (
uv sync --all-extras), so two whole classes of bug are invisible to it, and both shipped in 1.11.0:enrich(polars_df, …)→ModuleNotFoundError: pyarrow--all-extrasinstalls pyarrow, masking that it is not a core dependencyenrich(schema=…)→Either dataframe or source_path must be providedtest_enrich.pystubsPipeline.execute, so the loader stage never ranThe existing
Install Profilejobs verify the package imports. They do not verify it works.What
scripts/smoke_bare_install.py— does what a user does: install the wheel, callenrich(), look at the DataFrame. It talks to a local stub of the OpenAI chat-completions API, so it needs no key, costs nothing, and cannot flake on the network. Coversenrich()with a pandas frame, a Polars frame, a CSV path, andschema=structured output.It asserts on the values, not the shape: a run where every row failed still has the right row count and column names, which is exactly how the original bug looked green.
New
bare-install-smokeCI job — builds the wheel, installs it into a clean venv with--only-binary=:all:and no extras, runs the script. Added tobuild'sneeds.tests/unit/test_enrich.pygrows aTestEnrichEndToEndclass that patches the client factory instead ofPipeline.execute, so the loader, prompt, invocation, parser, and output-quality guard all run for real — closing the gap the issue calls out.Verification
pip install ondine==1.11.2from PyPI in a clean venv (no pyarrow present): all four checks pass.Two deliberate calls
[all]is not smoke-tested. The bug class this job exists for is only visible in the bare leg — every extra installed can only hide it.Install Profile (all)already proves[all]installs, and adding it here pulls torch via sentence-transformers for no extra detection power. The[mcp]leg is kept because it is cheap and catches extras-only import breakage.The stub is reached via
OPENAI_BASE_URL, not a parameter, becauseenrich()has nobase_url— that is #208. If that gap closes, this should pass it explicitly; there is a comment in the script saying so.Found in passing
#210 — the unit suite has a pre-existing flake:
test_mcp_server.pystarts background threads that outlive their tests, and a stragglerlitellmcall can land inside another test's patch window. Hit it once during this work, filed rather than folded in.ai assistance: i directed this work with help from claude code.
Summary by CodeRabbit
Tests
Chores