[System] Refactor AI Service & Backend to Support Multiple LLM Providers

[SagiEv]

Issue: Refactor AI Service & Backend to Support Multiple LLM Providers

Title

refactor(ai): introduce multi-provider LLM architecture with configurable provider/model selection per feature

---

Description

Currently, the AI Service only supports Groq as the LLM provider and the backend only stores a single Groq API key.

Refactor the AI infrastructure to support multiple LLM providers through a unified abstraction layer, allowing each AI feature to use a different provider and model.

The system should be extensible so that adding a new provider in the future requires minimal changes.

---

Motivation

Different LLM providers excel at different tasks.

Examples:

• OpenAI → CV Tailoring
• Claude → Cover Letter / Email generation
• Gemini → Long context analysis
• Groq → Fast inference
• Future providers (DeepSeek, Mistral, xAI, etc.)

Instead of hardcoding Groq everywhere, the application should allow selecting the provider/model per feature.

---

Goals

• Support multiple LLM providers
• Store multiple API keys
• Allow selecting provider & model for each AI feature
• Make provider implementations interchangeable
• Keep feature code provider-agnostic

---

Supported Providers (Initial)

• Groq
• OpenAI
• Claude (Anthropic)
• Gemini (Google)

Future providers should be easy to plug in.

---

Current Architecture

Frontend
    ↓
Backend
    ↓
AI Service
    ↓
Groq API

---

Desired Architecture

Frontend
      ↓
Backend
      ↓
AI Service
      ↓
LLM Router
      ↓
 ┌──────────────┬─────────────┬──────────────┬──────────────┐
 │ Groq         │ OpenAI      │ Claude       │ Gemini       │
 └──────────────┴─────────────┴──────────────┴──────────────┘

Feature code should never directly know which provider is being used.

---

Functional Requirements

1. Provider Abstraction

Create a common interface for all providers.

Example:

interface LLMProvider {
    chat(request): Promise<ChatResponse>;
    getAvailableModels(): Promise<Model[]>;
}

Each provider implements this interface.

Examples:

• GroqProvider
• OpenAIProvider
• ClaudeProvider
• GeminiProvider

---

2. LLM Router

Introduce a router/factory responsible for:

• Reading configuration
• Selecting provider
• Selecting model
• Executing request

Example:

router.generate({
    feature: "cvTailoring",
    ...
})

Router internally resolves:

Feature
    ↓
Provider
    ↓
Model

---

3. Provider Configuration

Store API credentials for every supported provider.

Example configuration:

{
  "groq": {
    "apiKey": "..."
  },
  "openai": {
    "apiKey": "..."
  },
  "claude": {
    "apiKey": "..."
  },
  "gemini": {
    "apiKey": "..."
  }
}

Missing API keys should disable that provider without affecting others.

---

4. Feature-Level Provider Selection

Each AI feature should have configurable:

• provider
• model

Initial configurable features:

• CV Tailoring
• Mail Creator
• Interview Insights Analyzer
• Job Search (future)

Example:

{
  "cvTailoring": {
    "provider": "openai",
    "model": "gpt-5"
  },
  "mailCreator": {
    "provider": "claude",
    "model": "claude-sonnet-4"
  },
  "interviewInsights": {
    "provider": "gemini",
    "model": "gemini-2.5-pro"
  }
}

No feature should hardcode provider names.

---

5. Backend Settings API

Extend the settings endpoints to support:

API Keys

• Groq API Key
• OpenAI API Key
• Claude API Key
• Gemini API Key

AI Feature Configuration

For every feature:

• Provider
• Model

---

6. Settings UI

Refactor the settings page.

API Keys Section

Groq API Key
OpenAI API Key
Claude API Key
Gemini API Key

AI Configuration

For each feature:

CV Tailoring

Provider:
▼ OpenAI

Model:
▼ GPT-5

Mail Creator

Provider:
▼ Claude

Model:
▼ Claude Sonnet 4

etc.

Model lists should update dynamically based on the selected provider (where supported).

---

7. Model Discovery

Where supported by the provider:

• Fetch available models dynamically.

Otherwise:

• Use predefined model lists.

---

8. Error Handling

Handle cases where:

• API key missing
• Invalid API key
• Provider unavailable
• Model unavailable
• Timeout
• Rate limit
• Unsupported provider

Errors should be provider-independent at the feature layer.

---

9. Extensibility

Adding a new provider should only require:

• New provider implementation
• Provider registration
• Optional UI updates (if needed)

No feature logic should require modification.

---

Non-Goals

• Prompt optimization
• Provider benchmarking
• Automatic provider fallback
• Load balancing between providers
• Cost optimization
• Streaming responses (unless already supported)

These can be addressed in future issues.

---

Suggested Project Structure

ai-service/

providers/
    groq/
    openai/
    claude/
    gemini/

interfaces/
    llm-provider.ts

router/
    llm-router.ts

config/
    provider-config.ts

features/
    cv-tailoring/
    mail-creator/
    interview-insights/

---

Acceptance Criteria

• Groq, OpenAI, Claude, and Gemini are supported through a common provider interface.
• Multiple API keys can be stored simultaneously.
• Backend exposes settings for all supported providers.
• Provider/model can be configured independently for each AI feature.
• AI feature implementations are provider-agnostic.
• A centralized LLM router resolves provider and model selection.
• Provider-specific errors are translated into common application errors.
• Existing Groq functionality continues to work after the refactor.
• Adding a new provider requires minimal code changes and no modifications to existing feature logic.

[SagiEv]

I like the per-feature provider selection direction. CV tailoring, cover letters, email generation, and long-context analysis have different quality, latency, and cost needs, so one global provider is usually too rigid.

A practical first layer is a feature-to-provider policy table that records model, provider, base URL, timeout, retry/fallback behavior, and usage metadata for each feature. That gives you enough data to decide which tasks can move to cheaper models without hurting output quality.

If this is still active, is the main bottleneck provider abstraction, feature-level routing, or comparing cost and quality across providers? I am testing an OpenAI-compatible multi-model API layer focused on official Chinese models, and job/CV workflows are useful for small A/B comparisons.

Yes, it's still active. The main goal isn't provider benchmarking as much as decoupling the feature logic from any specific LLM. Different models have different strengths, so we want the flexibility to choose the best provider for each use case without changing the application code.

For example, one model might produce better CV tailoring, another might write stronger cover letters or emails, while another is better suited for long-context analysis or offers lower latency. We don't want those decisions hardcoded into the features themselves.

The idea is to introduce a provider-agnostic abstraction with feature-level routing, so each feature can be configured with its own provider and model through settings. That also makes it straightforward to experiment with different providers or swap models over time as quality, pricing, or capabilities evolve. Cost and quality comparisons are definitely something we'd like to do later, but they're not the primary focus of this refactor.