RFC: Provider-Based Media Previews (Opt-In)

[GNUGradyn]

The issue

Hello. Many other chat platforms like Discord have embeds for embedding previews of links from various services such as YouTube. I understand why this is not already here - these embeds don't really work with the existing proxy system for link previews, which would mean leaking the users IP to these services. I did some thinking, and I think along with some other improvements to link previews in general, I have a reasonable way to implement this. I wanted to get some feedback before I get to work.

Threat model distinction

The main concern here, like with the opengraph-based link previews, is leaking the users info to the service provider. However, the specifics of the risk with embedding these services is not quite the same. The opengraph-based link previews work for any site, which means anyone can create a malicious link to grab the users info via the preview. However, the information at risk is limited, especially with the help of the proxy. This means by enabling "always show", you are accepting a broad risk of leaking low-sensitivity data. Site embeds would be much the opposite. You would be directly leaking your IP address to the service provider, but only an explicit list of providers. This means by enabling "always show" for this theoretical feature, you are accepting a narrow risk of leaking high-sensitivity data.

The solution

I think if this feature is added, it should be off by default and togglable for individual providers. A user might not care about leaking their IP to YouTube but want nothing to do with another provider. This should also be on a channel-by-channel basis like we have now with the opengraph previews.

This does create another issue, which is that it makes the timeline media configuration for a channel quite complex. This means for a new channel, you would have to configure quite a few settings. I think having somewhere to configure global defaults would be helpful, as I know having to change just the 1 setting we already have every time is already an annoyance for many.

This does not change how events are stored or federated. It only affects how the local client renders links.

Implementation

I did poke at the code a little bit already. I was thinking we could have a typescript interface for a link preview provider, which would contain a function to return a React component for a specific link, as well as information about what links can be handled by that provider, and metadata about the provider itself. These would be registered to a manager. MediaPreviewAccountSettings and the global defaults would use the manager to generate the configuration options. From there, TextualBody can use the manager instead of LinkPreviewGroup for supported and enabled providers. Alternatively, LinkPreviewGroup could be made into a GenericPreviewProvider/OpengraphPreviewProvider/whatever, but since the opengraph previews are pretty functionally distinct in that they are built by element using data from the homeserver, I prefer the former. The shape of a provider would be something like this

interface PreviewProvider {
  id: string;
  displayName: string;
  matches(url: URL): boolean;
  render(url: URL): React.ReactNode;
  privacyLevel: "proxied" | "direct";
}

The goal here is to provide better link previews while still respecting the privacy-first philosophy of the project and giving users fine-control, while improving the existing UX for link previews. Let me know what you think. Thanks