diff --git a/INTERRUPT_HANDLER_README.md b/INTERRUPT_HANDLER_README.md new file mode 100644 index 0000000000..2764d5fdd6 --- /dev/null +++ b/INTERRUPT_HANDLER_README.md @@ -0,0 +1,190 @@ +# LiveKit Intelligent Interruption Handling + +## Overview + +This implementation adds **context-aware backchannel detection** to the LiveKit Agents framework. The agent can now distinguish between passive acknowledgements ("yeah", "ok", "hmm") and active interruptions ("stop", "wait", "no") based on whether the agent is currently speaking or silent. + +### The Problem + +LiveKit's default Voice Activity Detection (VAD) is too sensitive to user feedback. When the agent is explaining something and the user says "yeah" or "ok" to indicate they are listening, the agent interprets this as an interruption and abruptly stops speaking. + +### The Solution + +A **logic filtering layer** sits between the VAD/STT pipeline and the interruption engine. It validates the transcript content before allowing an interruption, without modifying the low-level VAD kernel. + +## How It Works + +### Logic Matrix + +| User Input | Agent State | Behavior | +|---|---|---| +| "Yeah / Ok / Hmm" | **Speaking** | **IGNORE** - Agent continues speaking seamlessly | +| "Wait / Stop / No" | **Speaking** | **INTERRUPT** - Agent stops immediately | +| "Yeah / Ok / Hmm" | **Silent** | **RESPOND** - Agent treats it as valid input | +| "Yeah okay but wait" | **Speaking** | **INTERRUPT** - Contains non-backchannel word "wait" | + +### Architecture + +``` +User speaks → [VAD] detects speech (fast, ~20ms) + ↓ + Agent speaking + backchannel filter enabled? + YES → Skip VAD-only interruption, wait for STT + ↓ + [STT] returns transcript (~200-500ms) + ↓ + All words are backchannel? → YES → IGNORE (no pause, no stutter) + → NO → INTERRUPT immediately + NO → Normal interruption behavior +``` + +**Key design decision**: When backchannel filtering is active and the agent is speaking, VAD events alone do NOT trigger interruptions. The system waits for STT to deliver the actual transcript, then makes an informed decision. This prevents the "false start" problem where VAD would pause the agent before STT realizes the user only said "yeah". + +### Files Modified + +| File | Change | +|---|---| +| `livekit-agents/livekit/agents/voice/backchannel.py` | **NEW** - Backchannel detection module with configurable word list | +| `livekit-agents/livekit/agents/voice/agent_activity.py` | Added `_is_agent_speaking()`, `_should_ignore_as_backchannel()`, modified `_interrupt_by_audio_activity()`, `on_vad_inference_done()`, `on_interim_transcript()`, `on_final_transcript()` | +| `livekit-agents/livekit/agents/voice/agent_session.py` | Added `backchannel_words` parameter to `AgentSessionOptions` and `AgentSession.__init__()` | +| `livekit-agents/livekit/agents/telemetry/traces.py` | Fixed opentelemetry SDK compatibility | +| `interrupt_agent.py` | **NEW** - Demo agent with backchannel filtering enabled | +| `requirements.txt` | Updated with all required dependencies | + +### Backchannel Module (`backchannel.py`) + +The `is_backchannel()` function checks if **every** word in the transcript is a backchannel word: + +- `"yeah ok hmm"` → all backchannel → returns `True` → ignored while speaking +- `"yeah okay but wait"` → "but" and "wait" are NOT backchannel → returns `False` → triggers interruption +- Multi-word phrases like "got it", "i see", "uh huh" are supported + +The `DEFAULT_BACKCHANNEL_WORDS` set includes: yeah, yep, yup, yes, ok, okay, hmm, hm, mhm, mm, uh-huh, uh huh, uhuh, ah, aha, right, sure, got it, i see, alright, cool. + +### Configurable Ignore List + +The word list is fully configurable via the `backchannel_words` parameter: + +```python +from livekit.agents.voice.backchannel import DEFAULT_BACKCHANNEL_WORDS + +# Use defaults +session = AgentSession(backchannel_words=DEFAULT_BACKCHANNEL_WORDS) + +# Extend with custom words +custom_words = DEFAULT_BACKCHANNEL_WORDS | frozenset({"gotcha", "yea", "ya"}) +session = AgentSession(backchannel_words=custom_words) + +# Disable backchannel filtering entirely +session = AgentSession(backchannel_words=None) +``` + +## Setup + +### Prerequisites + +- Python 3.10+ +- API keys for STT, LLM, and TTS providers + +### Installation + +```bash +git clone https://github.com//agents-assignment.git +cd agents-assignment +git checkout feature/interrupt-handler-krish + +# Install from local repo (required for backchannel changes) +pip install -e livekit-agents +pip install -e livekit-plugins/livekit-plugins-silero +pip install -e livekit-plugins/livekit-plugins-deepgram +pip install -e livekit-plugins/livekit-plugins-openai +pip install -e livekit-plugins/livekit-plugins-groq +pip install -e livekit-plugins/livekit-plugins-cartesia +pip install python-dotenv +``` + +### Environment Variables + +Create a `.env` file in the project root: + +```env +# At minimum, you need one LLM provider (OpenAI or Groq) +OPENAI_API_KEY="sk-..." # Primary LLM (optional) +GROQ_API_KEY="gsk_..." # Fallback LLM (if no OpenAI key) + +# STT +DEEPGRAM_API_KEY="..." # Speech-to-Text + +# TTS +CARTESIA_API_KEY="sk_car_..." # Text-to-Speech + +# LiveKit (required for dev mode, not needed for console mode) +LIVEKIT_URL="wss://your-project.livekit.cloud" +LIVEKIT_API_KEY="API..." +LIVEKIT_API_SECRET="..." +``` + +The agent auto-selects providers based on available keys: +- **LLM**: OpenAI (primary) → Groq (fallback) +- **STT**: Deepgram (primary) → OpenAI → Groq +- **TTS**: Cartesia (primary) → OpenAI → Groq + +## Running the Agent + +### Console Mode (local microphone, no LiveKit server needed) + +```bash +python interrupt_agent.py console +``` + +### Dev Mode (connects to LiveKit Cloud, test via browser) + +```bash +python interrupt_agent.py dev +``` + +Then open [agents-playground.livekit.io](https://agents-playground.livekit.io) and connect. + +## Testing + +### Test Scenarios + +**Scenario 1 - The Long Explanation**: +Let the agent talk for a while. Say "okay... yeah... uh-huh" while it speaks. The agent should NOT stop, pause, or stutter. + +**Scenario 2 - The Passive Affirmation**: +Wait for the agent to finish and go silent. Say "Yeah." The agent should respond (e.g., "Great, how can I help?"). + +**Scenario 3 - The Correction**: +While the agent is speaking, say "No, stop." The agent should cut off immediately. + +**Scenario 4 - The Mixed Input**: +While the agent is speaking, say "Yeah okay but wait." The agent should stop because "but wait" is not a backchannel word. + +### Capturing Logs for Proof + +```bash +# PowerShell +python interrupt_agent.py dev 2>&1 | Tee-Object -FilePath test_log.txt + +# Bash / Git Bash +python interrupt_agent.py dev 2>&1 | tee test_log.txt +``` + +Look for log lines like: +``` +backchannel detected while agent speaking, ignoring {"transcript": "yeah"} +backchannel in final transcript, ignoring interruption {"transcript": "ok"} +``` + +These confirm the agent is correctly filtering backchannel words. + +## Design Decisions + +1. **No VAD modification**: All filtering is done as a logic layer in `agent_activity.py`. The Silero VAD kernel is untouched. + +2. **VAD deferral to STT**: When backchannel filtering is enabled and the agent is speaking, VAD events alone do NOT trigger interruption. This prevents the agent from pausing before STT can determine what was said. + +3. **Zero-latency filtering**: The `is_backchannel()` function is a pure string operation (set lookups) running in microseconds. No network calls or model inference. + +4. **Backwards compatible**: Setting `backchannel_words=None` (the default) preserves the original behavior entirely. Existing agents are unaffected. diff --git a/interrupt_agent.py b/interrupt_agent.py new file mode 100644 index 0000000000..fec671ca30 --- /dev/null +++ b/interrupt_agent.py @@ -0,0 +1,116 @@ +import logging +import os + +from dotenv import load_dotenv + +load_dotenv() + +from livekit.agents import Agent, AgentSession, JobContext, RunContext, cli, AgentServer +from livekit.agents.voice.backchannel import DEFAULT_BACKCHANNEL_WORDS +from livekit.plugins import silero + +HAS_OPENAI = bool(os.getenv("OPENAI_API_KEY")) +HAS_GROQ = bool(os.getenv("GROQ_API_KEY")) +HAS_DEEPGRAM = bool(os.getenv("DEEPGRAM_API_KEY")) +HAS_CARTESIA = bool(os.getenv("CARTESIA_API_KEY")) + +if HAS_DEEPGRAM: + from livekit.plugins import deepgram +if HAS_OPENAI: + from livekit.plugins import openai +if HAS_GROQ: + from livekit.plugins import groq +if HAS_CARTESIA: + from livekit.plugins import cartesia + +logger = logging.getLogger("interrupt-agent") + + +def get_llm(): + if HAS_OPENAI: + logger.info("LLM: Using OpenAI gpt-4o-mini") + return openai.LLM(model="gpt-4o-mini") + if HAS_GROQ: + logger.info("LLM: Using Groq llama-3.3-70b-versatile") + return groq.LLM(model="llama-3.3-70b-versatile") + raise RuntimeError("No LLM key found. Set OPENAI_API_KEY or GROQ_API_KEY") + + +def get_stt(): + if HAS_DEEPGRAM: + logger.info("STT: Using Deepgram nova-3") + return deepgram.STT(model="nova-3") + if HAS_OPENAI: + logger.info("STT: Using OpenAI whisper-1") + return openai.STT(model="whisper-1") + if HAS_GROQ: + logger.info("STT: Using Groq whisper-large-v3-turbo") + return groq.STT(model="whisper-large-v3-turbo") + raise RuntimeError("No STT key found. Set DEEPGRAM_API_KEY, OPENAI_API_KEY, or GROQ_API_KEY") + + +def get_tts(): + if HAS_CARTESIA: + logger.info("TTS: Using Cartesia") + return cartesia.TTS() + if HAS_OPENAI: + logger.info("TTS: Using OpenAI alloy") + return openai.TTS(voice="alloy") + if HAS_GROQ: + logger.info("TTS: Using Groq PlayAI") + return groq.TTS(model="playai-tts", voice="Arista-PlayAI") + raise RuntimeError("No TTS key found. Set CARTESIA_API_KEY, OPENAI_API_KEY, or GROQ_API_KEY") + + +BACKCHANNEL_WORDS = DEFAULT_BACKCHANNEL_WORDS | frozenset( + { + "uh-huh", + "mm-hmm", + "gotcha", + "yea", + "ya", + "huh", + } +) + + +class InterruptAgent(Agent): + def __init__(self) -> None: + super().__init__( + instructions=( + "You are a helpful voice assistant. " + "Keep your responses concise and conversational. " + "Do not use emojis, asterisks, markdown, or special characters. " + "When the user gives short acknowledgements like 'yeah' or 'ok' " + "after being silent, treat them as valid responses and continue " + "the conversation naturally." + ), + ) + + async def on_enter(self): + self.session.generate_reply( + instructions="Greet the user and ask how you can help them today." + ) + + +server = AgentServer() + + +@server.rtc_session() +async def entrypoint(ctx: JobContext): + session = AgentSession( + stt=get_stt(), + llm=get_llm(), + tts=get_tts(), + vad=silero.VAD.load(), + allow_interruptions=True, + min_interruption_duration=0.5, + min_interruption_words=0, + backchannel_words=BACKCHANNEL_WORDS, + ) + + await session.start(agent=InterruptAgent(), room=ctx.room) + + +if __name__ == "__main__": + cli.run_app(server) diff --git a/livekit-agents/livekit/agents/telemetry/traces.py b/livekit-agents/livekit/agents/telemetry/traces.py index 09b82363e2..2df490f83d 100644 --- a/livekit-agents/livekit/agents/telemetry/traces.py +++ b/livekit-agents/livekit/agents/telemetry/traces.py @@ -16,12 +16,14 @@ from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk._logs import ( - LogData, LoggerProvider, LoggingHandler, - LogRecord, LogRecordProcessor, + ReadWriteLogRecord, ) +from opentelemetry.sdk._logs._internal import LogRecord + +LogData = ReadWriteLogRecord from opentelemetry.sdk._logs.export import BatchLogRecordProcessor from opentelemetry.sdk.resources import SERVICE_NAME, Resource from opentelemetry.sdk.trace import SpanProcessor, TracerProvider diff --git a/livekit-agents/livekit/agents/voice/agent_activity.py b/livekit-agents/livekit/agents/voice/agent_activity.py index 0c3f7c743d..2aacc56c29 100644 --- a/livekit-agents/livekit/agents/voice/agent_activity.py +++ b/livekit-agents/livekit/agents/voice/agent_activity.py @@ -74,6 +74,7 @@ remove_instructions, update_instructions, ) +from .backchannel import is_backchannel from .speech_handle import SpeechHandle if TYPE_CHECKING: @@ -1166,12 +1167,29 @@ def _on_generation_created(self, ev: llm.GenerationCreatedEvent) -> None: ) self._schedule_speech(handle, SpeechHandle.SPEECH_PRIORITY_NORMAL) + def _is_agent_speaking(self) -> bool: + """Check if the agent is currently outputting speech.""" + return ( + self._current_speech is not None + and not self._current_speech.interrupted + and not self._current_speech.done() + ) + + def _should_ignore_as_backchannel(self, transcript: str) -> bool: + """Return True if the transcript is a backchannel input that should be + ignored while the agent is speaking.""" + bc_words = self._session.options.backchannel_words + if bc_words is None: + return False + if not self._is_agent_speaking(): + return False + return is_backchannel(transcript, bc_words) + def _interrupt_by_audio_activity(self) -> None: opt = self._session.options use_pause = opt.resume_false_interruption and opt.false_interruption_timeout is not None if isinstance(self.llm, llm.RealtimeModel) and self.llm.capabilities.turn_detection: - # ignore if realtime model has turn detection enabled return if ( @@ -1181,10 +1199,22 @@ def _interrupt_by_audio_activity(self) -> None: ): text = self._audio_recognition.current_transcript - # TODO(long): better word splitting for multi-language if len(split_words(text, split_character=True)) < opt.min_interruption_words: return + if ( + self._audio_recognition is not None + and opt.backchannel_words is not None + and self._is_agent_speaking() + ): + text = self._audio_recognition.current_transcript + if self._should_ignore_as_backchannel(text): + logger.debug( + "backchannel detected while agent speaking, ignoring", + extra={"transcript": text}, + ) + return + if self._rt_session is not None: self._rt_session.start_user_activity() @@ -1195,7 +1225,6 @@ def _interrupt_by_audio_activity(self) -> None: ): self._paused_speech = self._current_speech - # reset the false interruption timer if self._false_interruption_timer: self._false_interruption_timer.cancel() self._false_interruption_timer = None @@ -1237,15 +1266,21 @@ def on_end_of_speech(self, ev: vad.VADEvent | None) -> None: def on_vad_inference_done(self, ev: vad.VADEvent) -> None: if self._turn_detection in ("manual", "realtime_llm"): - # ignore vad inference done event if turn_detection is manual or realtime_llm return if ev.speech_duration >= self._session.options.min_interruption_duration: + if ( + self._session.options.backchannel_words is not None + and self._is_agent_speaking() + and self.stt is not None + ): + # when backchannel filtering is enabled and agent is speaking, + # skip VAD-only interruption — wait for STT transcript to decide + return self._interrupt_by_audio_activity() def on_interim_transcript(self, ev: stt.SpeechEvent, *, speaking: bool | None) -> None: if isinstance(self.llm, llm.RealtimeModel) and self.llm.capabilities.user_transcription: - # skip stt transcription if user_transcription is enabled on the realtime model return self._session._user_input_transcribed( @@ -1257,10 +1292,18 @@ def on_interim_transcript(self, ev: stt.SpeechEvent, *, speaking: bool | None) - ), ) - if ev.alternatives[0].text and self._turn_detection not in ( + transcript_text = ev.alternatives[0].text + if transcript_text and self._turn_detection not in ( "manual", "realtime_llm", ): + if self._should_ignore_as_backchannel(transcript_text): + logger.debug( + "backchannel in interim transcript, ignoring", + extra={"transcript": transcript_text}, + ) + return + self._interrupt_by_audio_activity() if ( @@ -1268,30 +1311,34 @@ def on_interim_transcript(self, ev: stt.SpeechEvent, *, speaking: bool | None) - and self._paused_speech and (timeout := self._session.options.false_interruption_timeout) is not None ): - # schedule a resume timer if interrupted after end_of_speech self._start_false_interruption_timer(timeout) def on_final_transcript(self, ev: stt.SpeechEvent, *, speaking: bool | None = None) -> None: if isinstance(self.llm, llm.RealtimeModel) and self.llm.capabilities.user_transcription: - # skip stt transcription if user_transcription is enabled on the realtime model return + transcript_text = ev.alternatives[0].text + self._session._user_input_transcribed( UserInputTranscribedEvent( language=ev.alternatives[0].language, - transcript=ev.alternatives[0].text, + transcript=transcript_text, is_final=True, speaker_id=ev.alternatives[0].speaker_id, ), ) - # agent speech might not be interrupted if VAD failed and a final transcript is received - # we call _interrupt_by_audio_activity (idempotent) to pause the speech, if possible - # which will also be immediately interrupted if self._audio_recognition and self._turn_detection not in ( "manual", "realtime_llm", ): + if self._should_ignore_as_backchannel(transcript_text): + logger.debug( + "backchannel in final transcript, ignoring interruption", + extra={"transcript": transcript_text}, + ) + return + self._interrupt_by_audio_activity() if ( @@ -1299,7 +1346,6 @@ def on_final_transcript(self, ev: stt.SpeechEvent, *, speaking: bool | None = No and self._paused_speech and (timeout := self._session.options.false_interruption_timeout) is not None ): - # schedule a resume timer if interrupted after end_of_speech self._start_false_interruption_timer(timeout) self._interrupt_paused_speech_task = asyncio.create_task( diff --git a/livekit-agents/livekit/agents/voice/agent_session.py b/livekit-agents/livekit/agents/voice/agent_session.py index 628718a6b2..812082bcbf 100644 --- a/livekit-agents/livekit/agents/voice/agent_session.py +++ b/livekit-agents/livekit/agents/voice/agent_session.py @@ -89,6 +89,7 @@ class AgentSessionOptions: preemptive_generation: bool tts_text_transforms: Sequence[TextTransforms] | None ivr_detection: bool + backchannel_words: frozenset[str] | None Userdata_T = TypeVar("Userdata_T") @@ -159,6 +160,7 @@ def __init__( tts_text_transforms: NotGivenOr[Sequence[TextTransforms] | None] = NOT_GIVEN, preemptive_generation: bool = False, ivr_detection: bool = False, + backchannel_words: frozenset[str] | None = None, conn_options: NotGivenOr[SessionConnectOptions] = NOT_GIVEN, loop: asyncio.AbstractEventLoop | None = None, # deprecated @@ -245,6 +247,12 @@ def __init__( Defaults to ``False``. ivr_detection (bool): Whether to detect if the agent is interacting with an IVR system. Default ``False``. + backchannel_words (frozenset[str], optional): Set of words treated as + passive acknowledgements (e.g. "yeah", "ok", "hmm"). When the agent + is speaking and the user says only these words, the agent continues + without interruption. When the agent is silent, the same words are + processed as normal input. Set to ``None`` to disable backchannel + filtering (default). conn_options (SessionConnectOptions, optional): Connection options for stt, llm, and tts. loop (asyncio.AbstractEventLoop, optional): Event loop to bind the @@ -288,6 +296,7 @@ def __init__( use_tts_aligned_transcript=use_tts_aligned_transcript if is_given(use_tts_aligned_transcript) else None, + backchannel_words=backchannel_words, ) self._conn_options = conn_options or SessionConnectOptions() self._started = False diff --git a/livekit-agents/livekit/agents/voice/backchannel.py b/livekit-agents/livekit/agents/voice/backchannel.py new file mode 100644 index 0000000000..2423188190 --- /dev/null +++ b/livekit-agents/livekit/agents/voice/backchannel.py @@ -0,0 +1,64 @@ +"""Backchannel detection for intelligent interruption handling. + +Distinguishes between passive acknowledgements ("yeah", "ok", "hmm") and +active interruptions ("stop", "wait", "no") based on whether the agent is +currently speaking or silent. +""" + +from __future__ import annotations + +import re +from collections.abc import Set + +DEFAULT_BACKCHANNEL_WORDS: frozenset[str] = frozenset( + { + "yeah", + "yep", + "yup", + "yes", + "ok", + "okay", + "hmm", + "hm", + "mhm", + "mm", + "uh-huh", + "uh huh", + "uhuh", + "ah", + "aha", + "right", + "sure", + "got it", + "i see", + "alright", + "cool", + } +) + +_WORD_SPLIT_RE = re.compile(r"[^\w'-]+", re.UNICODE) + + +def is_backchannel(transcript: str, backchannel_words: Set[str]) -> bool: + """Check if the entire transcript consists only of backchannel words. + + Returns True if every word/phrase in the transcript is a backchannel word, + meaning it should be ignored while the agent is speaking. + + For mixed inputs like "yeah okay but wait", returns False because + "but" and "wait" are not backchannel words. + """ + text = transcript.strip().lower() + if not text: + return False + + for phrase in backchannel_words: + if " " in phrase and phrase in text: + text = text.replace(phrase, " ") + + words = [w for w in _WORD_SPLIT_RE.split(text) if w] + if not words: + return True + + single_bc = {w for w in backchannel_words if " " not in w} + return all(w in single_bc for w in words) diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000..b5e1a03d9c --- /dev/null +++ b/requirements.txt @@ -0,0 +1,28 @@ +# Core framework with audio codec support (install from local repo with: pip install -e livekit-agents) +livekit-agents[codecs]>=1.0 + +# VAD - Voice Activity Detection (runs locally, no API key needed) +livekit-plugins-silero>=1.3.3 + +# STT - Speech to Text (uses DEEPGRAM_API_KEY) +livekit-plugins-deepgram>=1.3.3 + +# LLM - Primary: OpenAI (OPENAI_API_KEY), Fallback: Groq (GROQ_API_KEY) +livekit-plugins-openai>=1.3.3 +livekit-plugins-groq>=1.3.3 + +# TTS - Text to Speech (uses CARTESIA_API_KEY) +livekit-plugins-cartesia>=1.3.3 + +# Environment variable loading +python-dotenv>=1.0 + +# NOTE: For development with the backchannel interruption handler, +# install all packages from the local repo in editable mode: +# +# pip install -e livekit-agents +# pip install -e livekit-plugins/livekit-plugins-silero +# pip install -e livekit-plugins/livekit-plugins-deepgram +# pip install -e livekit-plugins/livekit-plugins-openai +# pip install -e livekit-plugins/livekit-plugins-groq +# pip install -e livekit-plugins/livekit-plugins-cartesia diff --git a/test_log.txt b/test_log.txt new file mode 100644 index 0000000000..dd05dfc669 --- /dev/null +++ b/test_log.txt @@ -0,0 +1,331 @@ +C:\Users\krish\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_fields.py:132: UserWarning: Field "model_name" in Metadata has conflict with protected namespace "model_". + +You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`. + warnings.warn( +C:\Users\krish\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_fields.py:132: UserWarning: Field "model_provider" in Metadata has conflict with protected namespace "model_". + +You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`. + warnings.warn( + 00:30:24 DEBUG asyncio Using proactor: IocpProactor + DEV livekit.agents Watching + C:\Users\krish\OneDrive\Desktop\salescod + eai\agents-assignment + DEV livekit.agents Watching + C:\Users\krish\OneDrive\Desktop\salescod + eai\agents-assignment\livekit-agents + DEV livekit.agents Watching + C:\Users\krish\OneDrive\Desktop\salescod + eai\agents-assignment\livekit-plugins\li + vekit-plugins-silero + DEV livekit.agents Watching + C:\Users\krish\OneDrive\Desktop\salescod + eai\agents-assignment\livekit-plugins\li + vekit-plugins-deepgram + DEV livekit.agents Watching + C:\Users\krish\OneDrive\Desktop\salescod + eai\agents-assignment\livekit-plugins\li + vekit-plugins-openai + DEV livekit.agents Watching + C:\Users\krish\OneDrive\Desktop\salescod + eai\agents-assignment\livekit-plugins\li + vekit-plugins-groq + DEV livekit.agents Watching + C:\Users\krish\OneDrive\Desktop\salescod + eai\agents-assignment\livekit-plugins\li + vekit-plugins-cartesia +C:\Users\krish\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_fields.py:132: UserWarning: Field "model_name" in Metadata has conflict with protected namespace "model_". + +You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`. + warnings.warn( +C:\Users\krish\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_fields.py:132: UserWarning: Field "model_provider" in Metadata has conflict with protected namespace "model_". + +You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`. + warnings.warn( + 00:30:27 DEBUG asyncio Using proactor: IocpProactor + INFO livekit.agents starting worker {"version": "1.3.3", + "rtc-version": "1.1.5"} + 00:30:28 INFO livekit.agents registered worker {"agent_name": "", "id": + "AW_Bvzy7W2pLvjB", + "url": + "wss://projectvoice-i4o… + "region": "India South", + "protocol": 17} + 00:31:57 INFO livekit.agents received job request {"job_id": + "AJ_rhbhqVnexstD", + "dispatch_id": + "AD_yZTuXNYYdBLg", + "room": + "playground-jfhs-4cY… + "room_id": + "RM_W8stbbJWr7sS", + "agent_name": "", + "resuming": false, + "enable_recording": + false} + DEBUG livekit.agents received assignment {"agent_name": "", + "room_id": + "RM_W8stbbJWr7sS", + "room": + "playground-jfhs-4cYB… + "job_id": + "AJ_rhbhqVnexstD", + "dispatch_id": + "AD_yZTuXNYYdBLg", + "enable_recording": + false} + INFO livekit.agents initializing job runner {"tid": 9008} + INFO livekit.agents job runner {"tid": 9008, + initialized "elapsed_time": 0.0} + DEBUG asyncio Using proactor: IocpProactor + INFO interrupt-agent STT: Using Deepgram nova-3 + INFO interrupt-agent LLM: Using Groq llama-3.3-70b-versatile + 00:31:58 INFO interrupt-agent TTS: Using Cartesia + DEBUG livekit.agents input stream {"participant": null, + attached "source": + "SOURCE_UNKNOWN", + "accepted_sources": + ["SOURCE_MICROPHONE"… + DEBUG livekit.agents http_session(): creating a new + httpclient ctx + DEBUG livekit.plugins… Established new {"cartesia_request_i… + Cartesia TTS "ab844c61-c677-459e-… + WebSocket connection + 00:31:59 DEBUG livekit.plugins… Established new {"headers": + Deepgram STT {"dg-project-id": + WebSocket "c6bb50b4-cc59-416d-… + connection: "dg-request-id": + "019d97ab-ea0b-75c1-… + "Date": "Thu, 16 Apr + 2026 19:01:59 GMT"}} + 00:32:00 DEBUG livekit.agents using audio io: `RoomIO` -> + `AgentSession` -> + `TranscriptSynchronizer` -> `RoomIO` + DEBUG livekit.agents using transcript io: `AgentSession` -> + `TranscriptSynchronizer` -> `RoomIO` + DEBUG livekit.agents start reading stream {"participant": + "identity-gK1w", + "source": + "SOURCE_MICROPHONE"} + INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:03 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:07 DEBUG livekit.agents received user {"user_transcript": + transcript "The history of World + War two in detail.", + "language": "en-US", + "transcript_delay": + 0.5324900150299072} + 00:32:08 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:13 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:17 DEBUG livekit.agents backchannel in {"transcript": + interim transcript, "Okay."} + ignoring + DEBUG livekit.agents backchannel in final {"transcript": + transcript, ignoring "Okay."} + interruption + DEBUG livekit.agents received user {"user_transcript": + transcript "Okay.", "language": + "en-US", + "transcript_delay": + 0.8926429748535156} + 00:32:18 DEBUG livekit.plugins… Established new {"cartesia_request_i… + Cartesia TTS "e6e236ff-9f4b-4c14-… + WebSocket connection + INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:23 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:28 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:30 DEBUG livekit.agents backchannel in final {"transcript": + transcript, ignoring "Right."} + interruption + DEBUG livekit.agents received user {"user_transcript": + transcript "Right.", "language": + "en-US", + "transcript_delay": + 0.6656558513641357} + 00:32:33 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:38 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:43 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:47 DEBUG livekit.agents backchannel in {"transcript": + interim transcript, "Yeah."} + ignoring + 00:32:48 DEBUG livekit.agents backchannel in final {"transcript": + transcript, ignoring "Yeah."} + interruption + DEBUG livekit.agents received user {"user_transcript": + transcript "Yeah.", "language": + "en-US", + "transcript_delay": + 0.7425458431243896} + INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:53 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:32:58 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:03 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:08 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:13 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:18 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:22 DEBUG livekit.agents received user {"user_transcript": + transcript "Hey.", "language": + "en-US", + "transcript_delay": + 0.6443328857421875} + 00:33:23 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:28 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:31 DEBUG livekit.agents received user {"user_transcript": + transcript "Stop.", "language": + "en-US", + "transcript_delay": + 0.5689189434051514} + WARNI… livekit.agents _SegmentSynchronizerImpl.resume called + after close + 00:33:33 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:38 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:43 DEBUG livekit.agents received user {"user_transcript": + transcript "Ask me before + starting your + countdown.", + "language": "en-US", + "transcript_delay": + 0.5404720306396484} + INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:46 DEBUG livekit.agents received user {"user_transcript": + transcript "Why is it", + "language": "en-US"} + WARNI… livekit.agents _SegmentSynchronizerImpl.resume called + after close + DEBUG livekit.plugins… Established new {"cartesia_request_i… + Cartesia TTS "224f16e5-8066-4a40-… + WebSocket connection + 00:33:48 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:53 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:33:58 DEBUG livekit.agents received user {"user_transcript": + transcript "if I am ready before + you begin.", + "language": "en-US", + "transcript_delay": + 0.5082473754882812} + INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:03 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:07 DEBUG livekit.agents received user {"user_transcript": + transcript "Yeah.", "language": + "en-US", + "transcript_delay": + 0.5147531032562256} + 00:34:08 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:13 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:18 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:23 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:24 DEBUG livekit.agents received user {"user_transcript": + transcript "Cut off.", + "language": "en-US", + "transcript_delay": + 0.9414539337158203} + 00:34:28 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:33 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:38 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:43 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:44 DEBUG livekit.agents received user {"user_transcript": + transcript "Explain recursion in + detail.", "language": + "en-US", + "transcript_delay": + 0.5036284923553467} + 00:34:48 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:53 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:34:58 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:03 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:04 DEBUG livekit.agents received user {"user_transcript": + transcript "Yes.", "language": + "en-US", + "transcript_delay": + 1.245689868927002} + 00:35:08 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:12 DEBUG livekit.agents backchannel in {"transcript": + interim transcript, "Yeah."} + ignoring + DEBUG livekit.agents backchannel detected {"transcript": " + while agent Yeah."} + speaking, ignoring + DEBUG livekit.agents received user {"user_transcript": + transcript "Yeah. But", + "language": "en-US", + "transcript_delay": + 1.0423028469085693} + 00:35:13 DEBUG livekit.plugins… Established new {"cartesia_request_i… + Cartesia TTS "279abbf8-ec3a-45c5-… + WebSocket connection + INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:18 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:23 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:28 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:33 INFO root ignoring byte stream with topic + 'lk.agent.session', no callback attached + 00:35:34 INFO livekit.agents closing agent {"participant": + session due to "identity-gK1w", + participant "reason": + disconnect (disable "CLIENT_INITIATED"} + via + `RoomInputOptions.cl + ose_on_disconnect=Fa + lse`) + DEBUG livekit.agents input stream {"participant": + detached "identity-gK1w", + "source": + "SOURCE_UNKNOWN", + "accepted_sources": + ["SOURCE_MICROPHONE"… + DEBUG livekit.agents stream closed {"participant": + "identity-gK1w", "source": + "SOURCE_MICROPHONE"} + DEBUG livekit.agents session closed {"reason": + "participant_disconnected", + "error": null} + 00:35:56 DEBUG livekit.agents shutting down job {"reason": "", + task "user_initiated": + false} + DEBUG livekit.agents job exiting {"reason": "", "tid": 9008, + "job_id": "AJ_rhbhqVnexstD", + "room_id": "RM_W8stbbJWr7sS"} + DEBUG livekit.agents http_session(): closing the httpclient + ctx