fix: MCP session not cleaned up gracefully on timeout

[yoavkatz]

Summary

When an agent session times out, the MCP sidecar process is killed before the harness has a chance to call MCP.DeleteSession, resulting in an error during cleanup and a non-graceful session teardown.

Observed Behavior

In trace tr-6635ef947049886b3b143e621b508cab (TAU2, tool_calling, Qwen3.6-35B-A3B), the MCP.DeleteSession span errors with:

RuntimeError: Session deletion failed: Session deletion failed:
  {'error': 'Error closing session 5788bd81-33f1-471b-9ec2-c75a9110bd02:
   Service at http://127.0.0.1:59685 is no longer alive'}

The session output is "Session completed (timeout)", meaning the MCP process was already dead when the harness tried to tear it down.

The Agent.Session span inherits this as its top-level error status, making the trace appear as ERROR even though the primary failure was the agent timeout — not a crash.

Impact

• Session cleanup leaves resources in an indeterminate state.
• The Agent.Session span is marked ERROR due to the cleanup failure, conflating two distinct failure modes (agent timeout vs. infrastructure error) in trace analysis.
• A second MCP.DeleteSession call is attempted and silently succeeds (status UNSET), indicating a retry path exists but the first attempt's error still propagates up.

Root Cause

The MCP sidecar at the local address (127.0.0.1:<port>) is terminated as part of the timeout handling before MCP.DeleteSession is invoked. The delete call then hits a dead socket.

Expected Behavior

• MCP.DeleteSession should check whether the session/process is still alive before attempting to close it, and handle the "already dead" case as a clean no-op (or at most a warning), not an error.
• Alternatively, the timeout handler should ensure MCP.DeleteSession is called (or skipped safely) before killing the sidecar process.
• The Agent.Session span's error status should reflect the agent-level outcome (timeout), not a cleanup side-effect.

Trace Reference

• Trace ID: tr-6635ef947049886b3b143e621b508cab
• Benchmark: tau2
• Agent: tool_calling
• Model: openai/Qwen3.6-35B-A3B
• Session ID: 5788bd81-33f1-471b-9ec2-c75a9110bd02

[mahirhir]

Here is a clean fix for MCP.DeleteSession to make teardown idempotent and prevent cleanup errors from masking root-cause timeout spans:

1. Idempotent Session Cleanup Guard

Wrap the HTTP/RPC teardown call in delete_session to treat dead sidecars (ConnectionRefusedError, is no longer alive) as benign cleanups:

import logging
import httpx

logger = logging.getLogger(__name__)

def delete_session(self, session_id: str, span=None) -> None:
    """Closes an active MCP session.
    If the sidecar process was already terminated (e.g. timeout / kill signal),
    this completes as a graceful no-op without raising a top-level span error.
    """
    try:
        response = self.client.post(
            f"/sessions/{session_id}/close",
            timeout=5.0,
        )
        if response.status_code not in (200, 204, 404):
            response.raise_for_status()
    except (httpx.ConnectError, httpx.RemoteProtocolError, httpx.TimeoutException) as e:
        logger.warning(
            f"MCP sidecar at {self.base_url} already unreachable during session deletion ({session_id}): {e}. "
            "Treating as already reclaimed."
        )
    except Exception as e:
        error_msg = str(e)
        if "no longer alive" in error_msg.lower() or "connection refused" in error_msg.lower():
            logger.warning(f"MCP session {session_id} sidecar already dead: {e}")
            return
        logger.error(f"Unexpected error during MCP.DeleteSession: {e}")
        # Only re-raise genuine unexpected infrastructure errors if not shutting down
        raise RuntimeError(f"Session deletion failed: {e}") from e

2. Guard Agent Session Span Outcome

In the runner/harness session lifecycle coordinator, ensure cleanup exceptions during finally: blocks do not overwrite an existing TIMEOUT status:

try:
    outcome = run_agent_turn(agent, session)
except TimeoutError:
    outcome = "TIMEOUT"
    span.set_status(Status(StatusCode.OK, description="Session completed (timeout)"))
finally:
    # Non-fatal cleanup: teardown failure must not flip a timeout span to ERROR
    try:
        mcp_client.delete_session(session_id)
    except Exception as cleanup_err:
        logger.warning(f"Secondary cleanup error ignored: {cleanup_err}")

This ensures trace analytics accurately categorize runs as timeouts without infrastructure noise.