test(sdk): add Anthropic Messages integration tests against llm-katan

[yossiovadia]

Summary

• Add Anthropic Messages API SDK integration tests using the official anthropic Python SDK
• Tests run through Praxis against llm-katan echo backend
• Exercises the full Anthropic filter chain end-to-end: anthropic_messages_format → anthropic_validate → anthropic_messages_protocol → token_count (provider: anthropic)

What this covers

Test	What it validates
test_non_streaming_basic	Request/response cycle, content blocks, stop_reason
test_non_streaming_with_system	System message handling
test_streaming_basic	SSE event lifecycle (message_start → content_block_delta → message_stop)
test_streaming_collects_full_text	Text stream assembly
test_multi_turn	Multi-message conversation
test_usage_present	Anthropic-format token extraction (input_tokens/output_tokens)

Why

The existing Anthropic integration tests (tests/integration/tests/suite/anthropic_messages.rs) use Backend::fixed() to replay pre-recorded JSON fixtures. They verify Praxis passes through static data but never test against a server that speaks the actual Anthropic Messages protocol.

This test sends real Anthropic SDK requests through the full filter chain to a live Anthropic-format endpoint, validating protocol fidelity end-to-end — including streaming SSE and Anthropic-native token usage extraction.

6 tests, ~1.3 seconds total. Skips gracefully when llm-katan is unreachable.

Usage

cargo build -p praxis-ai-proxy
uv run tests/integration/sdk/anthropic/test_anthropic_messages_llmkatan.py -s -v

Relates to #600

Checklist

• I reviewed every changed line and can explain the change.
• Tests are added or updated when behavior changes.
• New capabilities include an example config and functional example test.
• User-facing behavior and generated documentation are updated.
• Performance-sensitive changes include appropriate benchmark or load-test evidence.
• Commits are signed and include a Signed-off-by trailer.

[aslakknutsen]

@yossiovadia What is the plan here? Are we starting up a local llm-katan instance at some point, or is this only to be ran manually?

[yossiovadia]

Re: the July sign-off callout — ec519c1 was replaced by c77ed28 (signed) when the branch was rebased; DCO is green.

[yossiovadia]

Good question. Current state: the suite is opt-in — it skips when LLM_KATAN_BASE_URL is unset/unreachable, so CI stays green without a backend, and it is meant to be run manually. It is self-contained (PEP 723 header, uv run), and I have a public instance it works against, so it is easy to verify right now:

LLM_KATAN_BASE_URL=https://3-147-232-199.sslip.io uv run tests/integration/sdk/anthropic/test_anthropic_messages_llmkatan.py

(needs a built praxis-ai binary — the test starts its own proxy; the katan URL is the only external dependency). I verified that endpoint serves the suite a few minutes ago.

On the plan (tracked in #600): running it in CI is future work rather than a promise in this PR. The natural shape is a dedicated workflow like vllm-integration.yaml — the echo backend is much lighter than the vLLM container (no model weights, no GPU, just an install and a start). The one prerequisite is pinning llm-katan from somewhere the org controls rather than a personal fork (it is public and Apache-2.0, so that is a repo move or a tagged release). Happy to take that up as a follow-up if the maintainers want these running in CI.

[praxis-bot]

PR Re-Review

Summary: Most findings resolved. The hardcoded IP default is gone (env var with skip), URL parsing consolidated into _parse_llm_katan_url(), event ordering now uses a list with sequence assertions, and inline config has a comment explaining why no example exists. Helper duplication (finding #2) remains open. One new medium finding on the error-path test implementation.

Severity	Count
Medium	1

[praxis-bot]

[Medium] This error-path test uses raw httpx instead of the Anthropic SDK client. The original finding requested testing through the SDK (anthropic_client.messages.create with pytest.raises(anthropic.BadRequestError)) to confirm the filter works correctly when exercised by the real SDK, not just raw HTTP.

The SDK may transform errors, add retries, or handle response parsing differently than a raw HTTP client. Testing via httpx bypasses SDK-level error handling entirely.

Add a test that exercises the SDK's error path, for example:

def test_missing_max_tokens_rejected_via_sdk(self, anthropic_client):
    """Verify anthropic_validate rejects missing max_tokens through the SDK."""
    import anthropic
    
    with pytest.raises(anthropic.BadRequestError) as exc_info:
        anthropic_client.messages.create(
            model=LLM_KATAN_MODEL,
            messages=[{"role": "user", "content": "test"}],
            # Missing max_tokens — required by anthropic_validate
        )
    
    assert exc_info.value.status_code == 400

This confirms the SDK correctly surfaces validation errors from the proxy.