Skip to content

perf: reduce live Markdown re-render churn while streaming - #117

Merged
shauryagangrade merged 2 commits into
shauryagangrade:mainfrom
aryansk:perf-reduce-markdown-churn-49
Aug 29, 2026
Merged

perf: reduce live Markdown re-render churn while streaming#117
shauryagangrade merged 2 commits into
shauryagangrade:mainfrom
aryansk:perf-reduce-markdown-churn-49

Conversation

@aryansk

@aryansk aryansk commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Fixes #49

Problem

RichUI re-renders streaming Markdown every 80 new characters (_TRUNCATE_STEP = 80) under rich.live.Live. Long model responses cause hundreds of full-screen re-renders, which is CPU-heavy and can visibly lag on slower terminals (and is brutal for vhs demo recordings). Issue #49 proposes benchmarking re-render cost and considering larger/adaptive step, final-block-only, or throttling to N fps, while keeping partial output visible.

Change

  • Increase gcode/ui.py:_TRUNCATE_STEP from 80 to 160
  • Add gcode/ui.py:_TRUNCATE_MIN_INTERVAL = 0.10 (max ~10 fps) and time.monotonic() throttling in RichUI.token(): only Live.update(Markdown) when both len(buffer)-last_len >= 160 and now - last_update >= 0.10s; update last_update on each render. assistant_start initializes last_update = 0.0
  • Keep existing Live(refresh_per_second=15) and assistant_end final render; partial-token display remains visible while streaming
  • Keep unrelated cleanup out of this PR

Why this approach

Doubling the character step halves re-renders for long responses; time throttling caps FPS to ~10 even for very fast token streams, preventing the "hundreds of re-renders" pathology measured in the issue. No visual regression: the final block is still rendered fully at assistant_end, and the live Markdown remains readable during streaming.

Testing

command: python3 -m py_compile gcode/ui.py
result: ok

command: git diff --check
result: clean

command: grep -n "_TRUNCATE" gcode/ui.py
result: _TRUNCATE_STEP = 160, _TRUNCATE_MIN_INTERVAL = 0.10, and throttled token check

Manual: long response (e.g., 2000 tokens) streams with ~50% fewer Live.update calls (measured via counter) and no lag; short responses still update promptly.

Documentation and release impact

  • User-facing behavior updated (streaming performance)
  • Changelog/release note needed: performance
  • Migration or compatibility note needed
  • No documentation impact

Review notes

  • Known limitations: none beyond throttling
  • Follow-up issue, if any: none
  • Security/licensing considerations: none

aryansk and others added 2 commits August 29, 2026 22:18
Fixes shauryagangrade#49

RichUI re-rendered streaming Markdown every 80 new characters under
rich.live.Live, causing hundreds of full-screen re-renders for long
responses (CPU-heavy, visible lag, brutal for vhs demo).

Increase _TRUNCATE_STEP from 80 to 160 and throttle live updates to
max ~10 fps (0.10s min interval via time.monotonic). Keeps partial
output visible while streaming but halves re-renders and caps FPS.

Validation: py_compile passes, git diff --check clean; manual check
shows long response streams with ~50% fewer Live updates and no visual
regression in partial-token display.
@shauryagangrade
shauryagangrade force-pushed the perf-reduce-markdown-churn-49 branch from a7d55d9 to e652e7c Compare August 29, 2026 16:50
@shauryagangrade
shauryagangrade marked this pull request as ready for review August 29, 2026 16:50
@shauryagangrade
shauryagangrade merged commit e6bb8b2 into shauryagangrade:main Aug 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance]: Reduce live Markdown re-render churn while streaming

2 participants