Skip to content

Support the kitty image protocol natively through rio-vt - #142

Open
raphamorim wants to merge 10 commits into
orhun:mainfrom
raphamorim:kitty-graphics-native
Open

Support the kitty image protocol natively through rio-vt#142
raphamorim wants to merge 10 commits into
orhun:mainfrom
raphamorim:kitty-graphics-native

Conversation

@raphamorim

@raphamorim raphamorim commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Now that the vt100 engine is gone (#140), we can lean on rio-vt for the kitty graphics protocol too. This deletes ratty's hand-rolled APC parser and enables rio-vt's graphics feature instead, so the whole protocol — chunked transmissions, PNG/RGB/RGBA payloads, placement ids, deletes, queries, quiet modes, the 320MB eviction budget, and per-screen state on alt-screen swaps — is handled by the engine, which already answers acks and queries through the same reply path we use for DA/DSR.

On ratty's side, kitty state is now derived from the engine each frame: decoded pixels arrive through RioEvent::UpdateGraphics and become Bevy textures, and placements are resolved from the engine's absolute, scrollback-aware positions. That fixes a few things the old parser couldn't do:

  • images scroll with the content they were placed next to, including into and out of history (the old row-text diffing heuristic is gone)
  • multiple placements of the same image share one texture instead of re-decoding
  • source crops (x=/y=/w=/h=) and sub-cell offsets (X=/Y=) are honored, both for sprites and for the warped 3D planes
  • kitty z-index ordering is respected between images
  • kitten icat --unicode-placeholder style virtual placements are matched against the engine's placement registry rather than guessed from cell colors alone

The PTY grid resize now passes pixel dimensions through to the engine, since placements are sized against the cell pixel metrics. Sixel stays unadvertised in DA1: the engine can decode it now, but ratty only renders kitty placements.

Tests cover direct and chunked transfers, RGB expansion, crops, scrollback tracking, deletes dropping their textures, placeholder-driven placements, and queries being acked without placing anything.

kitty-ratty

@raphamorim

Copy link
Copy Markdown
Contributor Author

Ran a double review over this branch (a general correctness pass with adversarial verification, plus an independent kitty-conformance/integration pass) and pushed fixes for everything that survived:

  • textures now come straight from the engine's image store, validated by transmission timestamp. The event-queue copy could desync when an alt-screen app reused an image id from the main screen, or when eviction dropped the inactive screen's copy of a shared id — both now just rebuild from whatever the engine holds.
  • direct placements use the engine's kitty_overlay_geometry + clip_overlay_to_rect instead of a hand-copied version of the same math. That deletes the duplication and fixes clipping: a placement scrolled half off the top now shows the bottom slice of the image instead of overhanging the grid.
  • virtual (U=1) placements render per placeholder row-run using the engine's IncompletePlacement decoding and compute_run_geometry, so the row/column diacritics, indexed-color image ids (38;5;n), the placement id in the underline color, and the id high byte all work now. A placeholder image half-scrolled into history shows the correct half instead of squashing the whole image into the visible rows.
  • negative z-indices layer behind the terminal surface per the spec (sprites below the text quad, planes biased behind the surface), and 3D planes get a per-placement depth bias so overlapping images stack deterministically instead of z-fighting.
  • resizes pass the renderer's fractional cell metrics to the engine instead of pw / cols, whose truncation mis-scaled native-size images by up to ~6% with fractional font metrics. Also covers the font-size-change resize path.
  • the per-frame refresh is gated on the engine's dirty flag, the pending-redraw flag, and a scroll-state snapshot, so idle frames (and all frames without graphics) cost a few comparisons; refresh_kitty_graphics is now explicitly ordered after window resizes.
  • APC scanning only intercepts RGP sequences and streams everything else to the engine as it arrives — a multi-megabyte single-APC kitty transfer is no longer buffered and rescanned per chunk, and a stray unterminated APC introducer can't freeze the display.

Three engine-side issues surfaced that belong in rio-vt rather than here, filing separately: a=p for a nonexistent image and every a=q reply OK unconditionally where kitty would report ENOENT; deletes never remove virtual (U=1) placements; and atlas (sixel/iTerm2) pixel data is tracked against the 320MB budget with no untrack path once the update queues are drained.

Now 116 tests, including per-run slicing, indexed-color placeholders, alt-screen id collisions, clipped scrolling, z ordering, and refresh gating.

@raphamorim

Copy link
Copy Markdown
Contributor Author

The three engine-side issues from the review are fixed upstream and released in rio-vt 0.5.20 (raphamorim/rio@390a66f477, 531b897293, e6772529be), so this PR now depends on it:

  • a=p referencing an image the engine doesn't hold answers ENOENT instead of OK, and a=q runs the real decode path before replying — clients get honest feature detection and retransmit evicted images. There's a ratty-side test pinning the ENOENT reply end-to-end through the PTY reply path.
  • deletes (d=a/d=i/d=n) now remove virtual (U=1) placements, so placeholder cells stop rendering deleted images.
  • atlas (sixel/iTerm2) graphics give their bytes back to the 320MB budget when released, so heavy imgcat use no longer starves kitty images into permanent eviction.

No ratty code changes were needed for the bump — 117 tests pass against 0.5.20.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant