feat(plugins/servo): interactive web content — input forwarding and JS bridge

[staging-devin-ai-integration]

Summary

Enable two-way interaction between StreamKit pipelines and rendered web content in the Servo plugin. This unlocks use cases like interactive overlays (polls, chat, scoreboards), web-based control surfaces, and data visualization dashboards that respond to pipeline data.

Parent issue: #167
Depends on: #323 (PoC — merged)

Features

1. Input forwarding

Forward mouse/keyboard events from a UI control surface to the Servo WebView.

Servo's WebView supports input events via notify_keyboard_event(), notify_mouse_event(), notify_scroll_event(), etc. The challenge is building a control surface in the StreamKit UI that captures user input and forwards it to the pipeline.

Approach:

• Add a new ControlType::WebSurface that renders an interactive canvas in the Stream View
• Capture mouse/keyboard events on the canvas and forward them to the servo node via TuneNodeAsync
• The servo node translates these into Servo's input event types

2. JavaScript bridge

Bidirectional messaging between the StreamKit pipeline and the rendered web page.

Pipeline → page:

// Inject a global API into the page
window.streamkit = {
  data: {},  // Updated by pipeline
  on: function(event, callback) { ... },
};

Page → pipeline:

// Page sends data back to the pipeline
window.streamkit.send({ type: "button_click", id: "start" });

Implementation:

• Use WebView::evaluate_javascript() to inject the bridge API
• Poll for outbound messages from the page via a JS callback mechanism
• Expose received messages as a new output pin (e.g., OutputPin::WebEvent)

3. Custom protocol handler

streamkit:// URLs for loading local assets (fonts, images, data files) from the pipeline's working directory.

Implementation:

• Implement Servo's protocol handler trait for streamkit:// scheme
• Map streamkit://assets/logo.png to a configured local directory
• Security: restrict to a configured allow-list of directories

Use Cases

• Interactive poll overlay: Viewers click options in the Stream View, the web page updates in real-time
• Live scoreboard: Pipeline sends score data to the page, which renders an animated scoreboard
• Chat overlay: Pipeline feeds chat messages to a web page that renders them with animations
• Control surface: A web-based mixing console that controls other pipeline nodes

Acceptance Criteria

Input forwarding

• Mouse click/move/scroll events forwarded from UI to Servo WebView
• Keyboard events forwarded from UI to Servo WebView
• New ControlType::WebSurface in Stream View for interactive canvas
• Coordinate mapping from UI canvas to Servo viewport

JS bridge

• window.streamkit.send(data) API available in rendered pages
• Pipeline can send data to page via window.streamkit.on('data', callback)
• Messages delivered as pipeline packets (new WebEvent data type or JSON text)
• Security: JS bridge has configurable permissions (enable/disable per-node)

Custom protocol

• streamkit:// URL scheme handled by Servo
• Maps to configured local asset directory
• Security: restricted to allow-listed paths

General

• Sample pipeline demonstrating interactive web overlay
• Documentation for the JS bridge API
• Unit tests for event translation and message serialization

Complexity

Effort: XL (2–4 weeks)
Touches UI (new control type), plugin (input events, JS bridge), and potentially core (new data type for web events). Each sub-feature (input, bridge, protocol) could be a separate PR.

Notes

• Input forwarding and JS bridge are largely independent and can be implemented in either order.
• The custom protocol handler is lower priority and could be deferred.
• Security is critical: the JS bridge should be opt-in and sandboxed. Malicious web pages should not be able to access pipeline internals.
• Consider starting with just the JS bridge (pipeline → page direction) as it has the most immediate value for overlay use cases.