Random call disconnections when call reach a certain size with people on "same network"

[BillCarsonFr]

Steps to reproduce

1. Where are you starting? What can you see?

Some calls will see random disconection when the number of participant reach a certain size (not particularly big ~10 people).

The key to reproduce it is to have all the users with the same public facing IP

Outcome

Clients start seeing a lot of quota errors 429:M_LIMIT_EXCEEDED on delayed_events/<id>/restart calls.
If this call fail too many time in a row, the user will get disconnected

The call is back stable when people get disconnected, but is back unstable as soon as a user re-connect, this will then cause a random disconnection of another user

Operating system

No response

Browser information

No response

URL for webapp

No response

Will you send logs?

Yes

[BillCarsonFr]

Root cause: delayed_events/<id>/restart is a non authenticated call and will be rate limited by synapse based on IP, so if users share the same public IP they will use the same quota bucket.
Even if each client respect the retry_after_ms, if there are too many of them, they will keep applying pressure and get rejected. If a client gets rejected maximumRateLimitRetryCount (default 10) times, then ElementCall will disconnect the user

MembershipManager stopped because: Error: The MembershipManager shut down because of the end condition: Error: Exceeded maximum retries for RestartDelayedEvent attempts (client.restartScheduledDelayedEvent)

Proposed Fix: Make delayed_events/<id>/restart optionally authenticated, like that clients would use the authenticated version and jwt service the non-authenticated one for delegation of delayed events.

Synapse: element-hq/synapse#19794
js-sdk: matrix-org/matrix-js-sdk#5340

[BillCarsonFr]

For synapse admins, a temporary workaround could be to update rc_delayed_event_mgmt config to a very high value in synapse rate limit config https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#rc_delayed_event_mgmt