Skip to content

agent: add tool-driven media reply path #111

Description

@XnLemon

Parent

Part of #98. PR #106 verifies the protocol-neutral attachment/outbox/provider foundation, but natural-language agent execution still cannot call tools that produce structured media replies.

Problem

A user can ask the deployed WeCom agent to "send me an image", but the current runtime only streams model text and the durable materializer converts completed assistant text into text ReplyOutbox segments. Native media delivery works when an image ReplyOutbox is constructed directly, yet there is no tool/runtime bridge that lets an agent create or select an attachment and return a structured image/file reply from natural language.

This leaves a visible product gap:

  • Channel providers can send ReplyOutbox{kind=image, attachment_ref=...}.
  • Attachment storage can persist and verify media objects.
  • The LLM runner can answer text.
  • No registered tool can create or choose media and hand the result to Outbox as an immutable structured reply.

Scope

Add a minimal, protocol-neutral tool-driven media reply path for agents.

Recommended MVP:

  • Introduce an internal tool contract such as send_test_image or create_media_reply that returns a validated media reply intent rather than provider-specific URLs or tokens.
  • Persist generated or selected image/document bytes through the existing tenant-scoped attachment/object storage boundary.
  • Extend the Gateway/materialization path so a completed runner turn can produce structured ReplyOutbox media segments with deterministic text fallback.
  • Keep provider credentials and Telegram/WeCom media IDs outside runner/tool-visible payloads.
  • Gate tools through the existing agent revision allowlist and runtime policy limits.
  • Record audit/trace events for tool allowed/denied/executed and media reply materialization outcomes without logging raw tool arguments, generated media bytes, provider URLs, tokens, or secrets.

Acceptance Criteria

  • A published agent revision can explicitly allow a media reply tool through ToolAuthorization.
  • A natural-language request in the fake/E2E path can trigger the tool and produce a structured image or document ReplyOutbox segment.
  • The resulting outbox row includes reply_kind, attachment metadata, fallback text, reply target, and correlation fields.
  • WeCom and Telegram providers send supported media natively from the structured outbox row; unsupported destinations use persisted deterministic fallback.
  • Tool output cannot expose provider download URLs, access tokens, WeCom media_id download URLs, Telegram file URLs, or raw storage internals to the model.
  • Model/tool failures still materialize a deterministic fallback and preserve request_id/trace_id observability.
  • Contract tests cover allowed tool execution, denied tool execution, structured media materialization, fallback behavior, and idempotent retry/replay.
  • Documentation explains the distinction between native media transport support and agent/tool-driven media generation or selection.

Non-goals

  • Full image generation provider integration unless a fake/static tool is insufficient for the first slice.
  • Video understanding, OCR, ASR, or multimodal analysis.
  • Provider-specific media URLs or channel tokens in runner-visible state.
  • Making free-form model text magically become media without an explicit tool/result contract.

Notes From PR #106 Validation

  • Direct image outbox probes can be sent by WeCom through native media upload and delivery.
  • Asking the current LLM-only agent to send an image only exercises text generation; it does not create a structured media reply.
  • The Responses adapter should not require reasoning.encrypted_content for this path; that compatibility issue is handled separately in PR channels: add durable attachment storage #106.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions