feat: Intelligent Backchannel Interruption Handler - Pothamsetti Radha Krishna,NIT DURGAPUR

INTERRUPT_HANDLER_README.md

@@ -0,0 +1,190 @@
+# LiveKit Intelligent Interruption Handling
+
+## Overview
+
+This implementation adds **context-aware backchannel detection** to the LiveKit Agents framework. The agent can now distinguish between passive acknowledgements ("yeah", "ok", "hmm") and active interruptions ("stop", "wait", "no") based on whether the agent is currently speaking or silent.
+
+### The Problem
+
+LiveKit's default Voice Activity Detection (VAD) is too sensitive to user feedback. When the agent is explaining something and the user says "yeah" or "ok" to indicate they are listening, the agent interprets this as an interruption and abruptly stops speaking.
+
+### The Solution
+
+A **logic filtering layer** sits between the VAD/STT pipeline and the interruption engine. It validates the transcript content before allowing an interruption, without modifying the low-level VAD kernel.
+
+## How It Works
+
+### Logic Matrix
+
+| User Input | Agent State | Behavior |
+|---|---|---|
+| "Yeah / Ok / Hmm" | **Speaking** | **IGNORE** - Agent continues speaking seamlessly |
+| "Wait / Stop / No" | **Speaking** | **INTERRUPT** - Agent stops immediately |
+| "Yeah / Ok / Hmm" | **Silent** | **RESPOND** - Agent treats it as valid input |
+| "Yeah okay but wait" | **Speaking** | **INTERRUPT** - Contains non-backchannel word "wait" |
+
+### Architecture
+
+```
+User speaks → [VAD] detects speech (fast, ~20ms)
+                ↓
+            Agent speaking + backchannel filter enabled?
+            YES → Skip VAD-only interruption, wait for STT
+                    ↓
+              [STT] returns transcript (~200-500ms)
+                    ↓
+              All words are backchannel? → YES → IGNORE (no pause, no stutter)
+                                        → NO  → INTERRUPT immediately
+            NO → Normal interruption behavior
+```
+
+**Key design decision**: When backchannel filtering is active and the agent is speaking, VAD events alone do NOT trigger interruptions. The system waits for STT to deliver the actual transcript, then makes an informed decision. This prevents the "false start" problem where VAD would pause the agent before STT realizes the user only said "yeah".
+
+### Files Modified
+
+| File | Change |
+|---|---|
+| `livekit-agents/livekit/agents/voice/backchannel.py` | **NEW** - Backchannel detection module with configurable word list |
+| `livekit-agents/livekit/agents/voice/agent_activity.py` | Added `_is_agent_speaking()`, `_should_ignore_as_backchannel()`, modified `_interrupt_by_audio_activity()`, `on_vad_inference_done()`, `on_interim_transcript()`, `on_final_transcript()` |
+| `livekit-agents/livekit/agents/voice/agent_session.py` | Added `backchannel_words` parameter to `AgentSessionOptions` and `AgentSession.__init__()` |
+| `livekit-agents/livekit/agents/telemetry/traces.py` | Fixed opentelemetry SDK compatibility |
+| `interrupt_agent.py` | **NEW** - Demo agent with backchannel filtering enabled |
+| `requirements.txt` | Updated with all required dependencies |
+
+### Backchannel Module (`backchannel.py`)
+
+The `is_backchannel()` function checks if **every** word in the transcript is a backchannel word:
+
+- `"yeah ok hmm"` → all backchannel → returns `True` → ignored while speaking
+- `"yeah okay but wait"` → "but" and "wait" are NOT backchannel → returns `False` → triggers interruption
+- Multi-word phrases like "got it", "i see", "uh huh" are supported
+
+The `DEFAULT_BACKCHANNEL_WORDS` set includes: yeah, yep, yup, yes, ok, okay, hmm, hm, mhm, mm, uh-huh, uh huh, uhuh, ah, aha, right, sure, got it, i see, alright, cool.
+
+### Configurable Ignore List
+
+The word list is fully configurable via the `backchannel_words` parameter:
+
+```python
+from livekit.agents.voice.backchannel import DEFAULT_BACKCHANNEL_WORDS
+
+# Use defaults
+session = AgentSession(backchannel_words=DEFAULT_BACKCHANNEL_WORDS)
+
+# Extend with custom words
+custom_words = DEFAULT_BACKCHANNEL_WORDS | frozenset({"gotcha", "yea", "ya"})
+session = AgentSession(backchannel_words=custom_words)
+
+# Disable backchannel filtering entirely
+session = AgentSession(backchannel_words=None)
+```
+
+## Setup
+
+### Prerequisites
+
+- Python 3.10+
+- API keys for STT, LLM, and TTS providers
+
+### Installation
+
+```bash
+git clone https://github.com/<your-username>/agents-assignment.git
+cd agents-assignment
+git checkout feature/interrupt-handler-krish
+
+# Install from local repo (required for backchannel changes)
+pip install -e livekit-agents
+pip install -e livekit-plugins/livekit-plugins-silero
+pip install -e livekit-plugins/livekit-plugins-deepgram
+pip install -e livekit-plugins/livekit-plugins-openai
+pip install -e livekit-plugins/livekit-plugins-groq
+pip install -e livekit-plugins/livekit-plugins-cartesia
+pip install python-dotenv
+```
+
+### Environment Variables
+
+Create a `.env` file in the project root:
+
+```env
+# At minimum, you need one LLM provider (OpenAI or Groq)
+OPENAI_API_KEY="sk-..."       # Primary LLM (optional)
+GROQ_API_KEY="gsk_..."        # Fallback LLM (if no OpenAI key)
+
+# STT
+DEEPGRAM_API_KEY="..."        # Speech-to-Text
+
+# TTS
+CARTESIA_API_KEY="sk_car_..." # Text-to-Speech
+
+# LiveKit (required for dev mode, not needed for console mode)
+LIVEKIT_URL="wss://your-project.livekit.cloud"
+LIVEKIT_API_KEY="API..."
+LIVEKIT_API_SECRET="..."
+```
+
+The agent auto-selects providers based on available keys:
+- **LLM**: OpenAI (primary) → Groq (fallback)
+- **STT**: Deepgram (primary) → OpenAI → Groq
+- **TTS**: Cartesia (primary) → OpenAI → Groq
+
+## Running the Agent
+
+### Console Mode (local microphone, no LiveKit server needed)
+
+```bash
+python interrupt_agent.py console
+```
+
+### Dev Mode (connects to LiveKit Cloud, test via browser)
+
+```bash
+python interrupt_agent.py dev
+```
+
+Then open [agents-playground.livekit.io](https://agents-playground.livekit.io) and connect.
+
+## Testing
+
+### Test Scenarios
+
+**Scenario 1 - The Long Explanation**:
+Let the agent talk for a while. Say "okay... yeah... uh-huh" while it speaks. The agent should NOT stop, pause, or stutter.
+
+**Scenario 2 - The Passive Affirmation**:
+Wait for the agent to finish and go silent. Say "Yeah." The agent should respond (e.g., "Great, how can I help?").
+
+**Scenario 3 - The Correction**:
+While the agent is speaking, say "No, stop." The agent should cut off immediately.
+
+**Scenario 4 - The Mixed Input**:
+While the agent is speaking, say "Yeah okay but wait." The agent should stop because "but wait" is not a backchannel word.
+
+### Capturing Logs for Proof
+
+```bash
+# PowerShell
+python interrupt_agent.py dev 2>&1 | Tee-Object -FilePath test_log.txt
+
+# Bash / Git Bash
+python interrupt_agent.py dev 2>&1 | tee test_log.txt
+```
+
+Look for log lines like:
+```
+backchannel detected while agent speaking, ignoring {"transcript": "yeah"}
+backchannel in final transcript, ignoring interruption {"transcript": "ok"}
+```
+
+These confirm the agent is correctly filtering backchannel words.
+
+## Design Decisions
+
+1. **No VAD modification**: All filtering is done as a logic layer in `agent_activity.py`. The Silero VAD kernel is untouched.
+
+2. **VAD deferral to STT**: When backchannel filtering is enabled and the agent is speaking, VAD events alone do NOT trigger interruption. This prevents the agent from pausing before STT can determine what was said.
+
+3. **Zero-latency filtering**: The `is_backchannel()` function is a pure string operation (set lookups) running in microseconds. No network calls or model inference.
+
+4. **Backwards compatible**: Setting `backchannel_words=None` (the default) preserves the original behavior entirely. Existing agents are unaffected.

interrupt_agent.py

@@ -0,0 +1,116 @@
+import logging
+import os
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+from livekit.agents import Agent, AgentSession, JobContext, RunContext, cli, AgentServer
+from livekit.agents.voice.backchannel import DEFAULT_BACKCHANNEL_WORDS
+from livekit.plugins import silero
+
+HAS_OPENAI = bool(os.getenv("OPENAI_API_KEY"))
+HAS_GROQ = bool(os.getenv("GROQ_API_KEY"))
+HAS_DEEPGRAM = bool(os.getenv("DEEPGRAM_API_KEY"))
+HAS_CARTESIA = bool(os.getenv("CARTESIA_API_KEY"))
+
+if HAS_DEEPGRAM:
+    from livekit.plugins import deepgram
+if HAS_OPENAI:
+    from livekit.plugins import openai
+if HAS_GROQ:
+    from livekit.plugins import groq
+if HAS_CARTESIA:
+    from livekit.plugins import cartesia
+
+logger = logging.getLogger("interrupt-agent")
+
+
+def get_llm():
+    if HAS_OPENAI:
+        logger.info("LLM: Using OpenAI gpt-4o-mini")
+        return openai.LLM(model="gpt-4o-mini")
+    if HAS_GROQ:
+        logger.info("LLM: Using Groq llama-3.3-70b-versatile")
+        return groq.LLM(model="llama-3.3-70b-versatile")
+    raise RuntimeError("No LLM key found. Set OPENAI_API_KEY or GROQ_API_KEY")
+
+
+def get_stt():
+    if HAS_DEEPGRAM:
+        logger.info("STT: Using Deepgram nova-3")
+        return deepgram.STT(model="nova-3")
+    if HAS_OPENAI:
+        logger.info("STT: Using OpenAI whisper-1")
+        return openai.STT(model="whisper-1")
+    if HAS_GROQ:
+        logger.info("STT: Using Groq whisper-large-v3-turbo")
+        return groq.STT(model="whisper-large-v3-turbo")
+    raise RuntimeError("No STT key found. Set DEEPGRAM_API_KEY, OPENAI_API_KEY, or GROQ_API_KEY")
+
+
+def get_tts():
+    if HAS_CARTESIA:
+        logger.info("TTS: Using Cartesia")
+        return cartesia.TTS()
+    if HAS_OPENAI:
+        logger.info("TTS: Using OpenAI alloy")
+        return openai.TTS(voice="alloy")
+    if HAS_GROQ:
+        logger.info("TTS: Using Groq PlayAI")
+        return groq.TTS(model="playai-tts", voice="Arista-PlayAI")
+    raise RuntimeError("No TTS key found. Set CARTESIA_API_KEY, OPENAI_API_KEY, or GROQ_API_KEY")
+
+
+BACKCHANNEL_WORDS = DEFAULT_BACKCHANNEL_WORDS | frozenset(
+    {
+        "uh-huh",
+        "mm-hmm",
+        "gotcha",
+        "yea",
+        "ya",
+        "huh",
+    }
+)
+
+
+class InterruptAgent(Agent):
+    def __init__(self) -> None:
+        super().__init__(
+            instructions=(
+                "You are a helpful voice assistant. "
+                "Keep your responses concise and conversational. "
+                "Do not use emojis, asterisks, markdown, or special characters. "
+                "When the user gives short acknowledgements like 'yeah' or 'ok' "
+                "after being silent, treat them as valid responses and continue "
+                "the conversation naturally."
+            ),
+        )
+
+    async def on_enter(self):
+        self.session.generate_reply(
+            instructions="Greet the user and ask how you can help them today."
+        )
+
+
+server = AgentServer()
+
+
+@server.rtc_session()
+async def entrypoint(ctx: JobContext):
+    session = AgentSession(
+        stt=get_stt(),
+        llm=get_llm(),
+        tts=get_tts(),
+        vad=silero.VAD.load(),
+        allow_interruptions=True,
+        min_interruption_duration=0.5,
+        min_interruption_words=0,
+        backchannel_words=BACKCHANNEL_WORDS,
+    )
+
+    await session.start(agent=InterruptAgent(), room=ctx.room)
+
+
+if __name__ == "__main__":
+    cli.run_app(server)

livekit-agents/livekit/agents/telemetry/traces.py

@@ -16,12 +16,14 @@
 from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
 from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
 from opentelemetry.sdk._logs import (
-    LogData,
     LoggerProvider,
     LoggingHandler,
-    LogRecord,
     LogRecordProcessor,
+    ReadWriteLogRecord,
 )
+from opentelemetry.sdk._logs._internal import LogRecord
+
+LogData = ReadWriteLogRecord
 from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
 from opentelemetry.sdk.resources import SERVICE_NAME, Resource
 from opentelemetry.sdk.trace import SpanProcessor, TracerProvider