Self-hosted MatrixRTC LiveKit call disconnects after ~15s on Android with UNKNOWN_ERROR

[MortezaJavadian]

Steps to reproduce

Hi, thanks for taking a look at this.

I am running a self-hosted MatrixRTC setup with LiveKit for Element Call / Element X, behind nginx. I am seeing a disconnect issue on Android that I have not been able to narrow down yet.

The call starts normally at first:

1. Element / MatrixRTC is configured to use a self-hosted LiveKit SFU through the JWT service.
2. I join a call from Element X on Android.
3. The participant joins the room successfully.
4. The publisher PeerConnection connects.
5. The microphone track is published.
6. RTP starts flowing.
7. After around 15 to 20 seconds, the call disconnects.

On Android, Element X only shows this user-facing error:

UNKNOWN_ERROR

From the LiveKit side, the interesting part is that the WebSocket signal connection appears to be closed by the client first:

finishing WS connection ... "closedByClient": true
signal stream closed ... "error": null

Shortly after that, the subscriber transport fails and LiveKit closes the signal connection with TRANSPORT_FAILURE:

ice connection state change ... "transport": "SUBSCRIBER", "state": "failed"
peer connection state change ... "transport": "SUBSCRIBER", "state": "failed"
ignoring prefer candidate check by ICE failure because signal connection interrupted
closing signal connection ... "reason": "TRANSPORT_FAILURE"

The participant is then closed with:

participant closing ... "reason": "PEER_CONNECTION_DISCONNECTED"

The RTP stats also show very high upstream audio packet loss before the participant is closed:

rtp stats ... "direction": "upstream", "packetsExpected": 264, "packetsSeenPrimary": 127, "packetsLost": 137, "packetLostPercentage": 51.893936, "rtt": 59

Before I enabled full debug logging in the LiveKit config, I had also seen errors like these in the logs:

dtls timeout
peerconnection disconnected

My setup is:

• LiveKit server: livekit/livekit-server:v1.10.1
• LiveKit JWT service: ghcr.io/element-hq/lk-jwt-service:0.4.4
• Synapse: ghcr.io/element-hq/synapse:v1.150.0
• Element Web: ghcr.io/element-hq/element-web:v1.12.6
• Deployment: Docker Compose
• Reverse proxy: nginx
• TLS is terminated at nginx for the LiveKit WebSocket/API endpoint
• MatrixRTC SFU URL: wss://<MATRIX_DOMAIN>/livekit/sfu
• MatrixRTC JWT URL: https://<MATRIX_DOMAIN>/livekit/jwt
• Client from the logs: Element X / LiveKit JS SDK 2.16.0, Android 13, Wi-Fi

LiveKit is started like this:

livekit:
  image: livekit/livekit-server:v1.10.1
  command: ["--config", "/etc/livekit/config.yaml", "--node-ip", "<PUBLIC_SERVER_IP>"]
  volumes:
    - ${LIVEKIT_DATA_PATH}:/etc/livekit:ro
    - ${LIVEKIT_TLS_CERT_DIR}:/etc/lk-certs:ro
  ports:
    - "7881:7881/tcp"
    - "50201-50501:50201-50501/udp"
    - "3478:3478/udp"
    - "5349:5349/tcp"
    - "50502-50601:50502-50601/udp"

The main RTC/TURN part of the LiveKit config is:

port: 7880

rtc:
  tcp_port: 7881
  node_ip: <PUBLIC_SERVER_IP>
  port_range_start: 50201
  port_range_end: 50501
  use_external_ip: false
  allow_tcp_fallback: true
  strict_acks: false
  reconnect_on_publication_error: true
  reconnect_on_subscription_error: true
  reconnect_on_data_channel_error: true

turn:
  enabled: true
  domain: <MATRIX_DOMAIN>
  udp_port: 3478
  tls_port: 5349
  relay_range_start: 50502
  relay_range_end: 50601
  external_tls: false
  cert_file: /etc/lk-certs/live/<MATRIX_DOMAIN>/fullchain.pem
  key_file: /etc/lk-certs/live/<MATRIX_DOMAIN>/privkey.pem

The JWT service points clients to:

LIVEKIT_URL: "wss://<MATRIX_DOMAIN>/livekit/sfu"

nginx proxies /livekit/sfu to LiveKit port 7880 with WebSocket upgrade enabled:

location = /livekit/sfu {
  proxy_pass http://livekit_up/;
  proxy_http_version 1.1;
  proxy_send_timeout 3600s;
  proxy_read_timeout 3600s;
  proxy_buffering off;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
}

Outcome

Expected:

The participant should remain connected after joining the call and publishing audio. Both publisher and subscriber PeerConnections should stay healthy.

Actual:

The call starts successfully, but after around 15 to 20 seconds it disconnects. On Android, Element X shows UNKNOWN_ERROR.

From the LiveKit logs, the sequence looks like this:

starting signal connection ... "Client":{"sdk":1,"version":"2.16.0","protocol":16,"os":"Android","os_version":"13","network":"wifi"}

sent signal response ... "iceServers": [{"urls": ["turn:<PUBLIC_SERVER_IP>:3478?transport=udp", "turns:<MATRIX_DOMAIN>:443?transport=tcp"]}]

ice connection state change ... "transport": "PUBLISHER", "state": "connected"

mediaTrack published ... "kind": "audio", "mime": "audio/opus"

quality drop ... "direction": "up", "reason": "packet", "packetLostPercentage": 39.86928

error reading data channel ... "label": "_lossy", "error": "abort chunk, with following errors: (User Initiated Abort: Close called)"
error reading data channel ... "label": "_reliable", "error": "abort chunk, with following errors: (User Initiated Abort: Close called)"

finishing WS connection ... "closedByClient": true

ice connection state change ... "transport": "SUBSCRIBER", "state": "failed"
peer connection state change ... "transport": "SUBSCRIBER", "state": "failed"

closing signal connection ... "reason": "TRANSPORT_FAILURE"

participant closing ... "reason": "PEER_CONNECTION_DISCONNECTED"

rtp stats ... "packetsExpected": 264, "packetsSeenPrimary": 127, "packetsLost": 137, "packetLostPercentage": 51.893936

I am not sure if this is just a network/NAT packet loss issue, a mistake in my LiveKit/TURN/nginx configuration, or an interoperability issue between Element X / MatrixRTC and this LiveKit setup.

The part I find most confusing is the order of events. The client seems to close the signal WebSocket first, then the subscriber ICE transport fails, and then LiveKit reports TRANSPORT_FAILURE / PEER_CONNECTION_DISCONNECTED.

Since Android only shows UNKNOWN_ERROR, it is difficult to tell from the client side what the real root cause is.

Any guidance on what I should check next would be appreciated, especially around TURN advertisement, nginx path setup, and whether this level of upstream packet loss would be enough to explain the Android UNKNOWN_ERROR.

Operating system

Android 13

Browser information

Not using a browser directly.

The client is Element X on Android. The LiveKit server logs identify the client as:

LiveKit JS SDK 2.16.0
OS: Android 13
Network: Wi-Fi

URL for webapp

Private self-hosted Matrix / Element deployment.

Element Web image: ghcr.io/element-hq/element-web:v1.12.6
LiveKit JWT URL: https://<MATRIX_DOMAIN>/livekit/jwt
LiveKit SFU URL: wss://<MATRIX_DOMAIN>/livekit/sfu

Will you send logs?

Yes.

I can provide redacted LiveKit debug logs from the server side. I have not submitted client debug logs yet, but I can try to reproduce the issue again and submit feedback with debug logs linked to this issue.

[BillCarsonFr]

@MortezaJavadian what element x android version are you using?

[MortezaJavadian]

Element X version 26.04.3
Android 13

[BillCarsonFr]

Right, so the ElementCall version is 0.19.0 (unless you use a custom URL with devtools).

So right, this was version ^2.13.0
but that was resolved to 2.16.0

[MortezaJavadian]

@BillCarsonFr Thanks for clarifying.

I am not using a custom Element Call URL through devtools. This is the regular Element X Android app, version 26.04.3.

So as you mentioned, this should be Element Call 0.19.0, and the LiveKit JS SDK showing up as 2.16.0 in the LiveKit server logs makes sense if the dependency was resolved from ^2.13.0.

One extra thing I tested yesterday: with a much simpler LiveKit RTC configuration, the call worked correctly from the Android phone. The Android client joined the call and stayed connected without the UNKNOWN_ERROR.

This was the LiveKit config that worked:

port: 7880

redis:
  address: elementx-redis:6379

rtc:
  udp_port: 7882
  tcp_port: 7881
  use_external_ip: false
  enable_loopback_candidate: false

keys:
  <LIVEKIT_API_KEY_REDACTED>: "<LIVEKIT_SECRET_REDACTED>"

And the Docker Compose service was only exposing the single UDP RTC port:

livekit:
  image: ${LIVEKIT_IMAGE}
  container_name: ${LIVEKIT_HOST}
  hostname: ${LIVEKIT_HOST}
  restart: ${RESTART_POLICY}
  depends_on:
    redis:
      condition: service_started
  command: ["--config", "/etc/livekit/config.yaml", "--node-ip", "${SERVER_IP}"]
  volumes:
    - ${LIVEKIT_DATA_PATH}:/etc/livekit:ro
    - ${LIVEKIT_TLS_CERT_DIR}:/etc/lk-certs:ro
  ports:
    - "7882:7882/udp"
  networks:
    matrix-internal:
      aliases:
        - ${LIVEKIT_HOST}
    nginx:
      aliases:
        - ${LIVEKIT_HOST}

So the issue does not seem to be a general Android / Element X inability to connect to my LiveKit server. With the exact config above, the Android call worked.

The problem is that I would prefer to use a UDP port range like before, for example:

rtc:
  tcp_port: 7881
  port_range_start: 50201
  port_range_end: 50501

instead of a single udp_port.

What makes this confusing is that LiveKit seems quite sensitive to small RTC/TURN config changes in this setup. When I slightly changed the working config above, the same disconnect pattern came back: Android shows UNKNOWN_ERROR, and the LiveKit logs show closedByClient: true, subscriber ICE failure, then TRANSPORT_FAILURE / PEER_CONNECTION_DISCONNECTED.

So at this point I suspect the issue may be related to how the RTC ports or ICE candidates are being advertised when using a port range, rather than Element X itself.

If there is a recommended LiveKit config for MatrixRTC / Element X when using a UDP port range behind nginx, I would really appreciate guidance.

[fkwp]

@MortezaJavadian can you please also provide your nginx config

counterintuitively

rtc:
  udp_port: 7882

udp_port is NOT signle-port as outlined here https://github.com/livekit/livekit/blob/27c2b149d7308d1f53f7995a4b9317c37a8787a0/config-sample.yaml#L72

The recommendation is to use a port range matching the number of (v)CPUs.

what do you mean with

...using a UDP port range behind nginx...

We recommend to do proper port forwarding rather than piping UDP traffic.

[MortezaJavadian]

@fkwp
Thanks, I understand now. By “behind nginx” I only meant the HTTP/WebSocket signalling endpoint is proxied by nginx. I am not trying to proxy UDP media through nginx. The UDP ports are exposed directly from Docker/host to the LiveKit container.

I tested both styles while debugging:

• rtc.udp_port
• rtc.port_range_start/rtc.port_range_end

In both cases, depending on the exact config, I could still reproduce the Android UNKNOWN_ERROR, with the same LiveKit-side pattern: closedByClient: true, then subscriber ICE failure, then TRANSPORT_FAILURE / PEER_CONNECTION_DISCONNECTED.

Here is my nginx config:

upstream synapse_up     { server elementx-synapse:8008;     keepalive 32; }
upstream mas_up         { server elementx-mas:8080;         keepalive 32; }
upstream livekit_up     { server elementx-livekit:7880;     keepalive 32; }
upstream livekit_jwt_up { server elementx-livekit-jwt:8080; keepalive 32; }
upstream element_web_up { server elementx-element-web:80;   keepalive 16; }
upstream mas_admin_up   { server elementx-mas:8081;         keepalive 16; }
upstream ketesa_up      { server elementx-ketesa:8080;      keepalive 16; }

server {
  server_name <MATRIX_DOMAIN>;

  listen 443 ssl;
  listen [::]:443 ssl;
  http2 on;

  ssl_certificate     /etc/letsencrypt/live/<MATRIX_DOMAIN>/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/<MATRIX_DOMAIN>/privkey.pem;

  client_max_body_size 1024M;

  location = /.well-known/matrix/client {
    default_type application/json;
    add_header Access-Control-Allow-Origin * always;
    add_header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" always;
    add_header Access-Control-Allow-Headers "X-Requested-With, Content-Type, Authorization" always;
    add_header Cache-Control "no-store, max-age=0" always;

    return 200 '{"m.homeserver":{"base_url":"https://<MATRIX_DOMAIN>"},"org.matrix.msc4143.rtc_foci":[{"type":"livekit","livekit_service_url":"https://<MATRIX_DOMAIN>/livekit/jwt"}]}';
  }

  location = /_matrix/client/unstable/org.matrix.msc4143/rtc/transports {
    default_type application/json;
    add_header Access-Control-Allow-Origin * always;
    add_header Access-Control-Allow-Methods "GET, OPTIONS" always;
    add_header Access-Control-Allow-Headers "X-Requested-With, Content-Type, Authorization" always;
    add_header Cache-Control "no-store, max-age=0" always;

    return 200 '{"rtc_transports":[{"type":"livekit","livekit_service_url":"https://<MATRIX_DOMAIN>/livekit/jwt"}]}';
  }

  location = /_matrix/client/v1/rtc/transports {
    default_type application/json;
    add_header Access-Control-Allow-Origin * always;
    add_header Access-Control-Allow-Methods "GET, OPTIONS" always;
    add_header Access-Control-Allow-Headers "X-Requested-With, Content-Type, Authorization" always;
    add_header Cache-Control "no-store, max-age=0" always;

    return 200 '{"rtc_transports":[{"type":"livekit","livekit_service_url":"https://<MATRIX_DOMAIN>/livekit/jwt"}]}';
  }

  location = /livekit/jwt {
    return 308 https://$host/livekit/jwt/;
  }

  location ^~ /livekit/jwt/ {
    proxy_pass http://livekit_jwt_up/;
    proxy_http_version 1.1;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  location = /livekit/sfu {
    proxy_pass http://livekit_up/;

    proxy_http_version 1.1;
    proxy_send_timeout 3600s;
    proxy_read_timeout 3600s;
    proxy_buffering off;

    proxy_set_header Accept-Encoding gzip;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  location ^~ /livekit/sfu/ {
    proxy_pass http://livekit_up/;

    proxy_http_version 1.1;
    proxy_send_timeout 3600s;
    proxy_read_timeout 3600s;
    proxy_buffering off;

    proxy_set_header Accept-Encoding gzip;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  location ~ ^(/_matrix|/_synapse/client|/_synapse/mas|/_synapse/federation) {
    proxy_pass http://synapse_up;
    proxy_http_version 1.1;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

And nginx only proxies /livekit/sfu to elementx-livekit:7880 with WebSocket upgrade headers. The RTC UDP ports are not going through nginx.

Could you please tell me the exact LiveKit config you recommend for this setup, especially for the RTC port range and Docker port mappings?
Also, let me know exactly which outputs would help you debug this.

[LBlaboon]

I would like to chime in to say I've been having this exact same issue with the latest version of Cinny that released last week, which ships with element-call 0.19.1 (the previous version shipped with 0.16.3). Everything was working perfectly fine before and no changes have been made to my synapse/livekit setup, but after the update I get automatically disconnected from any call I try to join after about 10-15 seconds.

I noticed that the console logs have a bunch of lines like "error updating delayed event: Error: Failed to override function" and "delayed event update timeout error, retrying in 3s" before ultimately timing out and disconnecting the call.

I have tried bumping the rc_delayed_event_mgmt values in my homeserver config but that hasn't helped at all. So far the only solution I have found that works is downgrading to element-call 0.16.3 (by way of downgrading Cinny to its previous version).

[fkwp]

I would like to chime in to say I've been having this exact same issue with the latest version of Cinny that released last week, which ships with element-call 0.19.1 (the previous version shipped with 0.16.3). Everything was working perfectly fine before and no changes have been made to my synapse/livekit setup, but after the update I get automatically disconnected from any call I try to join after about 10-15 seconds.

I noticed that the console logs have a bunch of lines like "error updating delayed event: Error: Failed to override function" and "delayed event update timeout error, retrying in 3s" before ultimately timing out and disconnecting the call.

I have tried bumping the rc_delayed_event_mgmt values in my homeserver config but that hasn't helped at all. So far the only solution I have found that works is downgrading to element-call 0.16.3 (by way of downgrading Cinny to its previous version).

what is your specific setting of rc_delayed_event_mgmt now?

[LBlaboon]

what is your specific setting of rc_delayed_event_mgmt now?

Previously:

rc_delayed_event_mgmt:
  per_second: 1
  burst_count: 20

Updated to:

rc_delayed_event_mgmt:
  per_second: 2
  burst_count: 50

Should I go even further?

[fkwp]

what is your specific setting of rc_delayed_event_mgmt now?

Previously:

rc_delayed_event_mgmt:
  per_second: 1
  burst_count: 20

Updated to:

rc_delayed_event_mgmt:
  per_second: 2
  burst_count: 50

Should I go even further?

that should be fine. Which homeserver are you using?

[LBlaboon]

that should be fine. Which homeserver are you using?

Synapse 1.150.0

[fkwp]

that should be fine. Which homeserver are you using?

Synapse 1.150.0

are you using synapse workers?

[LBlaboon]

are you using synapse workers?

I am not. More details in case they are relevant:

• Server OS: Arch linux
• Auth delegated to MAS v1.16.0
• Livekit v1.10.1
• lk-jwt-service v0.4.0
• coturn v4.9.0
• Apache 2.4.66 as reverse proxy and SSL termination
• All services running directly on the host OS (no docker, etc)

[BillCarsonFr]

@LBlaboon Your issue is different, can you open a separate ticket?
Do you have more info about the Failed to override function error? can you share rageshake?