Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .changeset/unpublish_cleanup_when_transport_gone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
livekit: patch
livekit-capture: patch
livekit-ffi: patch
---

Fix a local publication and its track leaking when unpublish cannot reach the transport.

Unpublishing asks the engine to remove the RTP sender and stopped on failure, skipping the
cleanup that detaches the track from its publication. The publication holds its track and
the track holds the publication back through its mute callbacks, so the pair kept itself
alive along with the transceiver and its peer connection. Removing the sender fails on two
routine paths — an abnormal disconnect, where the transport is already closed, and a full
reconnect, where the sender belongs to the replaced transport — and the error was discarded,
so neither surfaced. The local cleanup now always runs and the failure is reported once
local state is consistent.
22 changes: 17 additions & 5 deletions livekit/src/room/participant/local_participant.rs
Original file line number Diff line number Diff line change
Expand Up @@ -387,7 +387,7 @@
r#type: proto::TrackType::from(track.kind()) as i32,
muted: track.is_muted(),
source: proto::TrackSource::from(options.source) as i32,
disable_dtx: !options.dtx,

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (x86_64-unknown-linux-gnu)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (aarch64-unknown-linux-gnu)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (aarch64-apple-ios-sim)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (aarch64-apple-darwin)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (x86_64-apple-darwin)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (aarch64-apple-ios)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (x86_64-pc-windows-msvc)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`

Check warning on line 390 in livekit/src/room/participant/local_participant.rs

View workflow job for this annotation

GitHub Actions / Features per-commit (aarch64-pc-windows-msvc)

use of deprecated field `livekit_protocol::AddTrackRequest::disable_dtx`
disable_red,
encryption: proto::encryption::Type::from(self.local.encryption_type) as i32,
stream: options.stream.clone(),
Expand Down Expand Up @@ -667,16 +667,28 @@
let track = publication.track().unwrap();
let sender = track.transceiver().unwrap().sender();

self.inner.rtc_engine.remove_track(sender)?;
// Removing the sender fails whenever the publisher transport is no longer the
// one this sender belongs to: an abnormal disconnect closes the transport
// before teardown runs, and a full reconnect replaces it outright. The local
// publication still has to be torn down in both cases, so the failure must not
// short-circuit the cleanup below — `set_track(None)` is the only thing that
// unregisters the track's mute callbacks, and without it the publication and
// its track hold each other, keeping the transceiver and its peer connection
// alive with them. Report the failure once the local state is consistent.
let removed = self.inner.rtc_engine.remove_track(sender);
track.set_transceiver(None);

if let Some(local_track_unpublished) =
self.local.events.local_track_unpublished.lock().as_ref()
{
local_track_unpublished(self.clone(), publication.clone());
if removed.is_ok() {
if let Some(local_track_unpublished) =
self.local.events.local_track_unpublished.lock().as_ref()
{
local_track_unpublished(self.clone(), publication.clone());
}
}
Comment on lines +681 to 687

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Full reconnects retain stale frame cryptors

When remove_track fails during a full reconnect, local_track_unpublished is skipped and the old frame cryptor remains registered. on_local_track_unpublished is the only per-track path removing the previous SID's cryptor. Each full reconnect retains another native cryptor and its old RTP sender until room shutdown.

Learn more

A full reconnect replaces the RTC session, then republishes every local track. Removing an old sender through the new session fails, so this guard suppresses the callback. That callback invokes on_local_track_unpublished, which removes the frame cryptor stored under the old track SID. The subsequent publish creates a new frame cryptor under the new SID, leaving the old native cryptor registered. The stale cryptor holds the old RTP sender and related native transport resources until the room-wide E2EE cleanup runs at shutdown.

Example: An encrypted track starts under SID TR_old. A full reconnect makes sender removal fail, then republishes it as TR_new. The manager contains cryptors for both TR_old and TR_new; repeated reconnects add another stale entry each time.

Recommended fix: Separate internal E2EE teardown from the externally visible LocalTrackUnpublished event. Always remove the cryptor for the old SID after local unpublish cleanup, while preserving the reconnect event contract for bindings. Add a full-reconnect lifecycle test that verifies the old SID disappears and the cryptor count returns to one.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented this and could not reproduce the leak, so I reverted it.

On unpatched code the cryptor map after a full reconnect holds one entry under the new sid, with the old one already gone:

before: [(ParticipantIdentity("p0"), TrackSid("TR_V5kQJepNxmJEu"))]
after:  [(ParticipantIdentity("p0"), TrackSid("TR_VPqmeKjnwF8ub"))]

So remove_track is not failing on the reconnect path the way it does on an abnormal disconnect, and the unpublished callback removes the old sid normally. The other path is covered too: RoomSession::close calls e2ee_manager.cleanup(), which clears every cryptor.

The test I wrote for it passed with and without the change, which makes it worthless as a guard. Measured against a local dev server, so if you have a trace showing two cryptors after a reconnect, send it and I will put the change back.


publication.set_track(None);

removed?;
self.inner.rtc_engine.publisher_negotiation_needed();

Ok(publication)
Expand Down
9 changes: 9 additions & 0 deletions livekit/src/room/publication/local.rs
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,15 @@ impl LocalTrackPublication {
}
}

/// Test-only: returns a probe that reports whether this publication's internals have
/// been dropped. The publication holds its track and the track holds the publication
/// back through the mute callbacks, so only unregistering them releases either.
#[cfg(feature = "__lk-e2e-test")]
pub fn drop_probe(&self) -> impl Fn() -> bool + Send + Sync + 'static {
let inner = Arc::downgrade(&self.inner);
move || inner.upgrade().is_none()
}

pub(crate) fn on_muted(&self, f: impl Fn(TrackPublication) + Send + 'static) {
*self.inner.events.muted.lock() = Some(Box::new(f));
}
Expand Down
85 changes: 81 additions & 4 deletions livekit/tests/room_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,16 @@

#[cfg(feature = "__lk-e2e-test")]
use {
anyhow::{Ok, Result},
anyhow::{anyhow, Ok, Result},
chrono::{TimeDelta, TimeZone, Utc},
common::test_rooms,
common::{
test_rooms,
video::{SolidColorParams, SolidColorTrack},
},
libwebrtc::prelude::PeerConnectionState,
livekit::{ConnectionState, ParticipantKind, RoomEvent},
std::time::Duration,
livekit::{options::VideoCodec, ConnectionState, ParticipantKind, RoomEvent},
livekit_api::services::room::RoomClient,
std::{env, sync::Arc, time::Duration},
tokio::time::{self, timeout},
};

Expand Down Expand Up @@ -140,3 +144,76 @@ async fn test_close_releases_room_session() -> Result<()> {
assert!(session_dropped(), "room callbacks retained the room session after close");
Ok(())
}

/// An unpublish that cannot reach the transport must still complete its local cleanup.
///
/// `unpublish_track` removes the publication from the participant, then asks the engine to
/// remove the RTP sender with `?`. On an abnormal disconnect the publisher transport has
/// already been closed by the time teardown runs, so that call fails and `?` skips
/// everything after it — including `publication.set_track(None)`, the only thing that
/// unregisters the track's mute callbacks. Those callbacks hold the publication while the
/// publication holds the track, so the pair keeps itself alive along with the transceiver
/// and the peer connection behind it.
///
/// The same failure happens on every full reconnect, where the sender belongs to the
/// previous session. `close()` discards the error, so nothing ever surfaces it.
#[cfg(feature = "__lk-e2e-test")]
#[test_log::test(tokio::test)]
async fn test_unpublish_cleans_up_when_transport_is_gone() -> Result<()> {
let (room, mut events) = test_rooms(1).await?.pop().unwrap();
let room_name = room.name();
let room = Arc::new(room);

let mut solid_track =
SolidColorTrack::new(room.clone(), SolidColorParams { width: 320, height: 240, luma: 128 });
solid_track.publish(VideoCodec::VP8, false).await?;

let publication = room
.local_participant()
.track_publications()
.into_values()
.next()
.ok_or_else(|| anyhow!("the track was never published"))?;
let publication_dropped = publication.drop_probe();
drop(publication);

// Delete the room server-side. The engine closes its transports before reporting
// Disconnected, so the room's teardown runs against an already-closed publisher —
// which is exactly when removing the sender fails.
let api_key = env::var("LIVEKIT_API_KEY").unwrap_or_else(|_| "devkey".into());
let api_secret = env::var("LIVEKIT_API_SECRET").unwrap_or_else(|_| "secret".into());
let server_url = env::var("LIVEKIT_URL").unwrap_or_else(|_| "ws://localhost:7880".into());
let http_url = server_url.replacen("ws", "http", 1);
RoomClient::with_api_key(&http_url, &api_key, &api_secret).delete_room(&room_name).await?;

timeout(Duration::from_secs(15), async {
loop {
match events.recv().await {
Some(RoomEvent::Disconnected { .. }) => break Ok(()),
Some(_) => continue,
None => break Err(anyhow!("event stream ended before the room disconnected")),
}
}
})
.await??;

// Drop every reference held outside the SDK, so anything still alive is held only by
// the publication <-> track cycle.
drop(events);
drop(solid_track);
drop(room);

let released = timeout(Duration::from_secs(10), async {
while !publication_dropped() {
time::sleep(Duration::from_millis(50)).await;
}
})
.await
.is_ok();

assert!(
released,
"local publication retained after an unpublish that could not reach the transport"
);
Ok(())
}
Loading