feat(web): inline image rendering in tool results with lightbox preview#509
feat(web): inline image rendering in tool results with lightbox preview#509GeT-LeFt wants to merge 2 commits intotiann:mainfrom
Conversation
Tool results (Read, Bash, Markdown, Generic views) currently render only text content — any base64 image blocks returned by tools are dropped. This means screenshots read by the Read tool, image output from bash commands, etc. are invisible in the chat. Changes: - Add ImageLightbox component with Portal overlay, ESC/backdrop close, and open-in-new-tab action. - Extract and render base64 image blocks in tool result content. - Integrate ResultImages into Read/Bash/Markdown/Generic result views. - Add click-to-preview for user-uploaded image attachments.
There was a problem hiding this comment.
Findings
- [Major]
Read/NotebookReadmixed text+image results now drop the textual file output instead of augmenting it. In the new branch, any extracted image swaps outfile.content, so the existing read payload text disappears as soon asresult.contentalso carries an image block. Evidenceweb/src/components/ToolCard/views/_results.tsx:454.
Suggested fix:<CodeBlock code={file.content} language="text" /> {images.length > 0 ? <ResultImages result={result} /> : null}
Summary
- Review mode: initial
- 1 finding. Mixed
Read/NotebookReadpayloads regress because the new image path replacesfile.contentinstead of rendering alongside it. - Residual risk:
web/src/components/ToolCard/views/_results.test.tsxdoes not cover mixed image payloads yet.
Testing
- Not run (automation)
HAPI Bot
| </div> | ||
| ) : null} | ||
| <CodeBlock code={file.content} language="text" /> | ||
| {images.length > 0 ? <ResultImages result={result} /> : <CodeBlock code={file.content} language="text" />} |
There was a problem hiding this comment.
[MAJOR] Read/NotebookRead loses file.content on mixed text+image payloads here. Before this change the read view always rendered the extracted file text; now any image block flips the branch to <ResultImages />, so OCR/plain-text output disappears for image reads.
Suggested fix:
<CodeBlock code={file.content} language="text" />
{images.length > 0 ? <ResultImages result={result} /> : null}The Read tool view used a ternary that swapped file content for images when a result contained image blocks, so mixed text+image payloads (e.g. Read on a notebook with embedded images) lost the textual output. Render the CodeBlock unconditionally and append ResultImages when images are present, mirroring how Bash / Markdown / Generic views already handle the same extraction.
There was a problem hiding this comment.
Findings
- [Major] Opened previews keep
window.opener—srcreaches this component from user-upload previews and tool-result data URLs, so the newly opened tab can retain a handle to the HAPI app and perform tabnabbing/navigation. Evidenceweb/src/components/ImageLightbox.tsx:43.
Suggested fix:window.open(src, '_blank', 'noopener,noreferrer')
- [Major] Tool-result
media_typeis trusted when buildingdata:URLs — a malformed or hostile tool result can set a non-image or active image type and still be passed into the lightbox/open-in-new-tab flow. Evidenceweb/src/components/ToolCard/views/_results.tsx:44.
Suggested fix:const safeImageMediaTypes = new Set(['image/png', 'image/jpeg', 'image/gif', 'image/webp']) const mediaType = source.media_type.toLowerCase() if (source.type === 'base64' && safeImageMediaTypes.has(mediaType) && typeof source.data === 'string') { return { mediaType, dataUrl: `data:${mediaType};base64,${source.data}` } }
Summary
- Review mode: follow-up after new commits
- 2 findings. The previous mixed
Readtext+image regression is fixed, but the new lightbox path needs URL-opening hardening and MIME validation before landing. - Residual risk: no repo test covers image-block extraction/rejection or mixed text+image rendering.
Testing
- Not run (automation)
HAPI Bot
| title="在新标签页打开" | ||
| onClick={(e) => { | ||
| e.stopPropagation() | ||
| window.open(src, '_blank') |
There was a problem hiding this comment.
[MAJOR] src reaches this component from user-upload previews and tool-result data URLs. Opening it with the default window.open behavior keeps window.opener, so a same-origin preview or active data document can retain a handle to the HAPI app and navigate it.
Suggested fix:
window.open(src, '_blank', 'noopener,noreferrer')| const source = isObject(block.source) ? block.source : null | ||
| if (!source) return null | ||
| if (source.type === 'base64' && typeof source.media_type === 'string' && typeof source.data === 'string') { | ||
| return { mediaType: source.media_type, dataUrl: `data:${source.media_type};base64,${source.data}` } |
There was a problem hiding this comment.
[MAJOR] This trusts source.media_type when constructing a data: URL. A malformed or hostile tool result can claim type: 'image' while using a non-image or active MIME type, and that URL then flows into the lightbox/open-in-new-tab path.
Suggested fix:
const safeImageMediaTypes = new Set(['image/png', 'image/jpeg', 'image/gif', 'image/webp'])
const mediaType = source.media_type.toLowerCase()
if (source.type === 'base64' && safeImageMediaTypes.has(mediaType) && typeof source.data === 'string') {
return { mediaType, dataUrl: `data:${mediaType};base64,${source.data}` }
}
Summary
Readtool, image output from commands) are currently dropped silently — only text content is rendered. This PR extracts and renders those images inline.ImageLightboxcomponent (Portal overlay, ESC/backdrop close, open-in-new-tab).Change
web/src/components/ImageLightbox.tsx— new component (~100 LOC).web/src/components/ToolCard/views/_results.tsx— extract image blocks from tool result content and render them; wire the lightbox. Existing text-only paths are unchanged.web/src/components/AssistantChat/messages/MessageAttachments.tsx— use the lightbox for image attachments.Net: +228 / -12 across 3 files, no new dependencies.
Notes
React.createPortalontodocument.bodyso it isn't clipped by parent overflow.Closes #508
Test plan