Skip to content

feat(web): inline image rendering in tool results with lightbox preview#509

Open
GeT-LeFt wants to merge 2 commits intotiann:mainfrom
GeT-LeFt:feat/inline-image-lightbox
Open

feat(web): inline image rendering in tool results with lightbox preview#509
GeT-LeFt wants to merge 2 commits intotiann:mainfrom
GeT-LeFt:feat/inline-image-lightbox

Conversation

@GeT-LeFt
Copy link
Copy Markdown
Contributor

Summary

  • Tool results that contain base64 image blocks (e.g. screenshots read via the Read tool, image output from commands) are currently dropped silently — only text content is rendered. This PR extracts and renders those images inline.
  • User-uploaded image attachments in assistant messages previously showed only a filename chip. They now show a thumbnail and open in a full-screen lightbox on click.
  • Adds a small reusable ImageLightbox component (Portal overlay, ESC/backdrop close, open-in-new-tab).

Change

  • web/src/components/ImageLightbox.tsx — new component (~100 LOC).
  • web/src/components/ToolCard/views/_results.tsx — extract image blocks from tool result content and render them; wire the lightbox. Existing text-only paths are unchanged.
  • web/src/components/AssistantChat/messages/MessageAttachments.tsx — use the lightbox for image attachments.

Net: +228 / -12 across 3 files, no new dependencies.

Notes

  • Purely additive — tool results without image blocks render identically to today.
  • The lightbox uses React.createPortal onto document.body so it isn't clipped by parent overflow.
  • Keyboard: ESC closes. The "open in new tab" action is a plain anchor for browser native download.

Closes #508

Test plan

  • `bun run typecheck` in `web/` passes.
  • Verified existing tool result views (Bash/Read/Markdown/Generic) still render text-only results unchanged.
  • Verified user-uploaded image attachments render as thumbnails and open the lightbox on click.

Tool results (Read, Bash, Markdown, Generic views) currently render only
text content — any base64 image blocks returned by tools are dropped.
This means screenshots read by the Read tool, image output from bash
commands, etc. are invisible in the chat.

Changes:
- Add ImageLightbox component with Portal overlay, ESC/backdrop close,
  and open-in-new-tab action.
- Extract and render base64 image blocks in tool result content.
- Integrate ResultImages into Read/Bash/Markdown/Generic result views.
- Add click-to-preview for user-uploaded image attachments.
Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Read/NotebookRead mixed text+image results now drop the textual file output instead of augmenting it. In the new branch, any extracted image swaps out file.content, so the existing read payload text disappears as soon as result.content also carries an image block. Evidence web/src/components/ToolCard/views/_results.tsx:454.
    Suggested fix:
    <CodeBlock code={file.content} language="text" />
    {images.length > 0 ? <ResultImages result={result} /> : null}

Summary

  • Review mode: initial
  • 1 finding. Mixed Read/NotebookRead payloads regress because the new image path replaces file.content instead of rendering alongside it.
  • Residual risk: web/src/components/ToolCard/views/_results.test.tsx does not cover mixed image payloads yet.

Testing

  • Not run (automation)

HAPI Bot

</div>
) : null}
<CodeBlock code={file.content} language="text" />
{images.length > 0 ? <ResultImages result={result} /> : <CodeBlock code={file.content} language="text" />}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] Read/NotebookRead loses file.content on mixed text+image payloads here. Before this change the read view always rendered the extracted file text; now any image block flips the branch to <ResultImages />, so OCR/plain-text output disappears for image reads.

Suggested fix:

<CodeBlock code={file.content} language="text" />
{images.length > 0 ? <ResultImages result={result} /> : null}

The Read tool view used a ternary that swapped file content for images
when a result contained image blocks, so mixed text+image payloads
(e.g. Read on a notebook with embedded images) lost the textual output.

Render the CodeBlock unconditionally and append ResultImages when
images are present, mirroring how Bash / Markdown / Generic views
already handle the same extraction.
Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Opened previews keep window.openersrc reaches this component from user-upload previews and tool-result data URLs, so the newly opened tab can retain a handle to the HAPI app and perform tabnabbing/navigation. Evidence web/src/components/ImageLightbox.tsx:43.
    Suggested fix:
    window.open(src, '_blank', 'noopener,noreferrer')
  • [Major] Tool-result media_type is trusted when building data: URLs — a malformed or hostile tool result can set a non-image or active image type and still be passed into the lightbox/open-in-new-tab flow. Evidence web/src/components/ToolCard/views/_results.tsx:44.
    Suggested fix:
    const safeImageMediaTypes = new Set(['image/png', 'image/jpeg', 'image/gif', 'image/webp'])
    const mediaType = source.media_type.toLowerCase()
    if (source.type === 'base64' && safeImageMediaTypes.has(mediaType) && typeof source.data === 'string') {
        return { mediaType, dataUrl: `data:${mediaType};base64,${source.data}` }
    }

Summary

  • Review mode: follow-up after new commits
  • 2 findings. The previous mixed Read text+image regression is fixed, but the new lightbox path needs URL-opening hardening and MIME validation before landing.
  • Residual risk: no repo test covers image-block extraction/rejection or mixed text+image rendering.

Testing

  • Not run (automation)

HAPI Bot

title="在新标签页打开"
onClick={(e) => {
e.stopPropagation()
window.open(src, '_blank')
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] src reaches this component from user-upload previews and tool-result data URLs. Opening it with the default window.open behavior keeps window.opener, so a same-origin preview or active data document can retain a handle to the HAPI app and navigate it.

Suggested fix:

window.open(src, '_blank', 'noopener,noreferrer')

const source = isObject(block.source) ? block.source : null
if (!source) return null
if (source.type === 'base64' && typeof source.media_type === 'string' && typeof source.data === 'string') {
return { mediaType: source.media_type, dataUrl: `data:${source.media_type};base64,${source.data}` }
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] This trusts source.media_type when constructing a data: URL. A malformed or hostile tool result can claim type: 'image' while using a non-image or active MIME type, and that URL then flows into the lightbox/open-in-new-tab path.

Suggested fix:

const safeImageMediaTypes = new Set(['image/png', 'image/jpeg', 'image/gif', 'image/webp'])
const mediaType = source.media_type.toLowerCase()
if (source.type === 'base64' && safeImageMediaTypes.has(mediaType) && typeof source.data === 'string') {
    return { mediaType, dataUrl: `data:${mediaType};base64,${source.data}` }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement] Render images inline in tool results and attachments with lightbox preview

1 participant