[Help Needed] Testing with Claude API Key

[priyanshus]

Wanted to check if we can configure the API key and provider details for Anthropic directly in evaliphy.config.ts and verify that evaluations work with Anthropic models.

I’m trying something like this:

evaluate.use({
  llmAsJudgeConfig: {
    model: "claude-3.5-sonnet",
    provider: {
      type: "anthropic",
      apiKey: process.env.ANTHROPIC_KEY,
    },
    temperature: 0,
  },
  timeout: 15000,
});

evaluate(
  "RAG /api/chat: return policy answer is faithful",
  async ({ httpClient }) => {
    const query = "What is your return policy?";
    const res = await httpClient.post("/api/chat", { message: query });
    const data = await res.json<ChatResponse>();

    await expect({
      query,
      response: data.answer,
      context: data.context,
    }).toBeFaithful({ threshold: 0.8 });
  }
);

Just want to ensure Evaliphy supports direct integration with LLM provider APIs like Anthropic.

[quanticsoul4772]

Evaliphy uses the Vercel AI SDK under the hood, so you can use any provider it supports — including Claude directly.

Setup:

1. Install the Anthropic provider:
   ```bash
   npm install @ai-sdk/anthropic

   or: pnpm add @ai-sdk/anthropic

   ```

2. Set your API key:
   ```bash
   export ANTHROPIC_API_KEY=sk-ant-...
   ```

3. Pass the model to evaliphy:
   ```typescript
   import { anthropic } from '@ai-sdk/anthropic'
   import { createEvaluator } from 'evaliphy'

   const evaluator = createEvaluator({
   model: anthropic('claude-3-5-sonnet-20241022'),
   })
   ```

The @ai-sdk/anthropic provider reads ANTHROPIC_API_KEY from the environment automatically. If you need to pass it explicitly: anthropic({ apiKey: 'sk-ant-...' })('claude-3-5-sonnet-20241022').

[priyanshus]

Hey @quanticsoul4772, I wanted to check if we can configure the API key and provider details for Anthropic directly in evaliphy.config.ts and verify that evaluations work with Anthropic models.

I’m trying something like this:

evaluate.use({
  llmAsJudgeConfig: {
    model: "claude-3.5-sonnet",
    provider: {
      type: "anthropic",
      apiKey: process.env.ANTHROPIC_KEY,
    },
    temperature: 0,
  },
  timeout: 15000,
});

evaluate(
  "RAG /api/chat: return policy answer is faithful",
  async ({ httpClient }) => {
    const query = "What is your return policy?";
    const res = await httpClient.post("/api/chat", { message: query });
    const data = await res.json<ChatResponse>();

    await expect({
      query,
      response: data.answer,
      context: data.context,
    }).toBeFaithful({ threshold: 0.8 });
  }
);

Could you confirm whether this approach works i.e., using Anthropic directly without going through a gateway/provider abstraction?

Just want to ensure Evaliphy supports direct integration with LLM provider APIs like Anthropic.

[quanticsoul4772]

Yes — the shape you wrote is supported as-is. The llmAsJudgeConfig.provider union accepts { type: 'anthropic', apiKey } directly, and Evaliphy's @evaliphy/ai package resolves it via @ai-sdk/anthropic under the hood. You don't need to install or import the AI SDK yourself for the judge path.

Reference: packages/ai/src/providers/direct.ts maps type: 'anthropic' → createAnthropic({ apiKey })(modelId). packages/core/src/config/schema.ts defines the DirectProviderSchema that your config validates against.

Two small corrections to your example:

1. Model ID. "claude-3.5-sonnet" isn't a valid Anthropic model identifier — you need the versioned ID Anthropic expects (e.g. "claude-3-5-sonnet-20241022" or a newer 4.x model like "claude-sonnet-4-6"). Check Anthropic's model list for current IDs.

2. Env var name. If you ever omit apiKey and rely on the env fallback, Evaliphy looks for ANTHROPIC_API_KEY specifically (see apiKey.ts). Your explicit apiKey: process.env.ANTHROPIC_KEY is fine as long as the env var you read from is set on the machine running the evals.

So the corrected form of your snippet:

```ts
evaluate.use({
llmAsJudgeConfig: {
model: "claude-3-5-sonnet-20241022",
provider: {
type: "anthropic",
apiKey: process.env.ANTHROPIC_API_KEY,
},
temperature: 0,
},
timeout: 15000,
});
```

(Apologies for the earlier createEvaluator/@ai-sdk/anthropic answer — that's the programmatic-setup path, not the evaluate.use({ llmAsJudgeConfig }) path you're asking about.)

[priyanshus]

Thanks!