One body, one format: body_format replaces body_html on the compose tools

.gitignore

@@ -63,3 +63,4 @@ squirrel_mcp/admin/
 # Competitor screenshots kept locally for reference only (which mail providers
 # and IMAP/SMTP settings to support). Never publish these -- this repo is public.
 imapmcp/
+uv.lock

CLAUDE.md

@@ -266,9 +266,10 @@ namespaces are reserved so they plug in later as sibling providers + tool mixins
   e2e proves a real server takes it and hands the same bytes back.
 - **Inline is the HTML compose path that unblocked it.** Inline was deferred
   above for a good reason -- it means nothing without an HTML body referencing
-  the part -- so `body_html` came first and the reason is now *enforced* rather
-  than avoided. `body_html` is added as an **alternative**: `body` stays the
-  text a plain-text client shows, and both halves are the same message. An
+  the part -- so the HTML body came first and the reason is now *enforced*
+  rather than avoided. The provider's `body_html` is added as an
+  **alternative**: `body` stays the text a plain-text client shows, and both
+  halves are the same message. An
   attachment marked `inline` (with a `content_id` the tool layer fills in when
   the caller omits one, since a part nothing can name is a part nothing can
   show) is only *placed* inline when there is HTML to point at it, and
@@ -285,6 +286,26 @@ namespaces are reserved so they plug in later as sibling providers + tool mixins
   `cid:logo`, `<logo>`). `tests/test_attachments_inline.py` pins the trees and
   the id handling; the GreenMail e2e reads the delivered `multipart/related`
   back off a real server.
+- **The tool layer asks which format, not for both bodies.** `body_html` is
+  what the *providers* take, and for a while it was also what the tools took:
+  a caller wanting HTML passed `body` **and** `body_html`, writing the same
+  message twice. That is two arguments for one decision, and the second copy
+  is the one that rots -- an edit to the HTML leaves yesterday's sentence in
+  the plain-text half, where nobody proofreading the mail will see it.
+  So the compose tools (`mail_send`, `mail_create_draft`, `mail_edit_draft`)
+  take one `body` plus `body_format` = `text` (the default) | `html`, and
+  `tools/_common.as_bodies` splits it into the pair `MailProvider` still
+  expects. The wire is unchanged. **The derived half is the trade being
+  made**: `html_text.html_to_text` flattens blocks to line breaks, list items
+  to dashes and entities to characters, which is worse than a plain-text
+  alternative written by hand and better than one that disagrees with the
+  HTML -- and a caller who wants to hand-write it can, one level down, in the
+  provider. The same function serves the *read* path, where a message with no
+  `text/plain` part has always been flattened; it used to be a private copy in
+  `imap.py` that collapsed every message to one line and left `&amp;` in the
+  text. An unrecognised format is **refused, not guessed**: `markdown` would
+  otherwise leave as literal asterisks. `tests/test_body_format.py` pins the
+  split, the flattening and the refusal.
 - `providers/soverin/contacts.py` **discovers** the address-book home rather than
   assuming a path. CardDAV standardises none, so `carddav_url` is a starting
   point: RFC 6764's `current-user-principal` → `addressbook-home-set` hops turn a
@@ -404,4 +425,5 @@ defaults -- a missing one is a config error, never a guess.
 | `error_handling.py` / `error_sanitizer.py` | Error hierarchy + message sanitizing |
 | `logging_config.py` | Structured logging to stderr |
 | `search_query.py` | Search grammar: parse, fold, verify, widen (both backends) |
+| `html_text.py` | HTML -> text: the read fallback and `body_format="html"`'s plain half |
 | `usage.py` | Usage-tracking stub (full version in admin package) |

squirrel_mcp/html_text.py

@@ -0,0 +1,44 @@
+"""HTML -> plain text, used in both directions of the mail path.
+
+*Reading*: a message whose only body is ``text/html`` still has to be readable,
+so the markup is stripped down to text rather than handed on as tags.
+
+*Writing*: ``body_format="html"`` composes the plain-text alternative out of
+the same HTML. That is what lets a caller choose **one** format and pass
+**one** body: the two halves of a ``multipart/alternative`` say the same thing
+by construction, instead of by the caller writing the message twice and the
+plain-text half quietly drifting.
+
+Deliberately not a renderer. Block elements become line breaks and list items
+become dashes, because a mail flattened into one paragraph is unreadable; a
+table stays a table's worth of words, and CSS is dropped on the floor.
+"""
+
+from __future__ import annotations
+
+import html as _html
+import re
+
+# script/style/head carry text that is not content -- stripped whole, because
+# removing only the tags would leave the CSS in the message body.
+_DROP_RE = re.compile(r"(?is)<(script|style|head)\b[^>]*>.*?</\1>")
+_ITEM_RE = re.compile(r"(?i)<li\b[^>]*>")
+_BREAK_RE = re.compile(
+    r"(?i)</?(br|p|div|tr|ul|ol|h[1-6]|blockquote|table|section|article)\b[^>]*>"
+)
+_TAG_RE = re.compile(r"<[^>]+>")
+
+
+def html_to_text(html: str) -> str:
+    """Flatten an HTML body to the text a plain-text client should show."""
+    text = _DROP_RE.sub(" ", html)
+    text = _ITEM_RE.sub("\n- ", text)
+    text = _BREAK_RE.sub("\n", text)
+    # Remaining tags are inline (<b>, <a>, <span>): replaced by a space rather
+    # than removed, so "<td>a</td><td>b</td>" does not read as one word.
+    text = _TAG_RE.sub(" ", text)
+    text = _html.unescape(text)
+    text = re.sub(r"[ \t\r\f\v]+", " ", text)
+    text = re.sub(r" *\n *", "\n", text)
+    text = re.sub(r"\n{3,}", "\n\n", text)
+    return text.strip()

squirrel_mcp/knowledge.py

@@ -65,13 +65,21 @@
   you ask the user to approve a send: a file leaving the mailbox is as much a
   decision as the recipient is. Editing a draft leaves its existing attachments
   alone unless you pass the argument.
-- EMBEDDING AN IMAGE in the message rather than hanging it off: pass `body_html`
-  alongside `body` (the plain text stays, and is what a client without HTML
-  shows), mark the attachment `"inline": true` with a `"content_id"`, and refer
-  to it from the HTML as `<img src="cid:that-id">`. The reference is what makes
-  it show; an inline file with no HTML pointing at it simply arrives as an
-  ordinary attachment. Use it for signatures and screenshots that belong in the
-  flow of the text, not to dress up a message the user asked to keep plain.
+- PLAIN TEXT OR HTML is one argument, `body_format`, on mail_send,
+  mail_create_draft and mail_edit_draft. It defaults to `text` and that is the
+  right default: write the message in `body` and leave it alone. Pass
+  `body_format="html"` and `body` is HTML instead -- the plain-text half of the
+  message is derived from it, so you never write the same mail twice and the
+  two halves cannot drift apart. Choose html when the content needs it (a link
+  the user should see as a link, a list, an embedded image), not to dress up a
+  message the user asked to keep plain. Editing a draft rewrites the format
+  along with the rest, so re-pass `"html"` for a draft that had an HTML body.
+- EMBEDDING AN IMAGE in the message rather than hanging it off needs
+  `body_format="html"`: mark the attachment `"inline": true` with a
+  `"content_id"` and refer to it from the HTML as `<img src="cid:that-id">`.
+  The reference is what makes it show; an inline file with a plain-text body
+  simply arrives as an ordinary attachment. Use it for signatures and
+  screenshots that belong in the flow of the text.
 - REPLYING is not the same as sending: whenever the user is answering a message
   they just read, pass that message's uid as `reply_to_uid` (plus the
   `reply_to_folder` it lives in) to mail_send or mail_create_draft. Only that puts the

squirrel_mcp/providers/soverin/imap.py

@@ -10,7 +10,6 @@
 
 import datetime
 import imaplib
-import re
 import ssl
 from email.message import Message
 from typing import Callable, List, Optional, Tuple, TypeVar
@@ -28,6 +27,7 @@
 from imap_tools.errors import MailboxFlagError, MailboxLoginError
 from imap_tools.utils import check_command_status, clean_uids
 
+from ...html_text import html_to_text
 from ...logging_config import get_logger
 from ...search_query import MailQuery, parse
 from ..protocol import (
@@ -51,21 +51,13 @@
 # Connection-level failures that warrant one transparent reconnect + retry.
 _CONNECTION_ERRORS = (imaplib.IMAP4.abort, OSError, ConnectionError, EOFError)
 
-_TAG_RE = re.compile(r"<[^>]+>")
 
 # The SPECIAL-USE names the Sent path used before the table moved to
 # ``providers.protocol`` -- kept so this module still reads as IMAP.
 SENT_SPECIAL_USE = SPECIAL_USE_FOLDERS["sent"][0]
 DEFAULT_SENT_FOLDER = SPECIAL_USE_FOLDERS["sent"][1]
 
 
-def _html_to_text(html: str) -> str:
-    """Very small HTML -> text fallback for messages with no text/plain part."""
-    text = re.sub(r"(?is)<(script|style).*?>.*?</\1>", "", html)
-    text = _TAG_RE.sub(" ", text)
-    return re.sub(r"\s+", " ", text).strip()
-
-
 class SoverinImapClient:
     """Persistent IMAP session with lazy login and reconnect-on-drop."""
 
@@ -655,7 +647,7 @@ def _from_display(msg) -> str:
 
     @classmethod
     def _to_summary(cls, msg, folder: str) -> MessageSummary:
-        body = msg.text or (_html_to_text(msg.html) if msg.html else "")
+        body = msg.text or (html_to_text(msg.html) if msg.html else "")
         return MessageSummary(
             uid=msg.uid or "",
             folder=folder,
@@ -671,7 +663,7 @@ def _to_summary(cls, msg, folder: str) -> MessageSummary:
 
     @classmethod
     def _to_detail(cls, msg, folder: str) -> MessageDetail:
-        body = msg.text or (_html_to_text(msg.html) if msg.html else "")
+        body = msg.text or (html_to_text(msg.html) if msg.html else "")
         attachments = [
             AttachmentInfo(
                 index=i,

squirrel_mcp/tools/_common.py

@@ -15,6 +15,7 @@
     SystemError,
     ValidationError,
 )
+from ..html_text import html_to_text
 from ..logging_config import get_logger
 from ..providers import ProviderAuthError, ProviderError, ProviderNotFoundError
 
@@ -108,6 +109,32 @@ def reply_subject(subject: str) -> str:
     return f"Re: {stripped}"
 
 
+BODY_FORMATS = ("text", "html")
+
+
+def as_bodies(body: str, body_format: Any) -> tuple[str, str | None]:
+    """Turn one body plus its format into the ``(text, html)`` pair providers take.
+
+    The tool surface asks for a body and *which of the two it is*, because a
+    caller composing a message writes it once. The wire still carries both --
+    ``multipart/alternative``, text first -- so an HTML body gets its plain-text
+    half derived here rather than retyped by the caller, where it would drift
+    from the HTML the moment either was edited.
+
+    Deriving is the deliberate trade: a hand-written plain-text alternative that
+    reads better than the stripped HTML is the case being dropped, and it is
+    worth one argument instead of two.
+    """
+    fmt = str(body_format or "text").strip().lower()
+    if fmt not in BODY_FORMATS:
+        raise ValidationError(
+            f"body_format must be 'text' or 'html', got {body_format!r}"
+        )
+    if fmt == "text":
+        return body, None
+    return html_to_text(body), body
+
+
 def bare_addresses(values: Iterable[str]) -> list[str]:
     """Header values like ``Anna <anna@x.eu>`` -> ``anna@x.eu``, de-duplicated.
 

squirrel_mcp/tools/mail/attachments.py

@@ -15,8 +15,8 @@
 
 Either form takes ``"inline": true`` (plus an optional ``"content_id"``) to
 embed an image in the body rather than hang it off the message. That only
-renders against a ``body_html`` referring to the part as ``cid:<id>``; without
-one it stays an ordinary attachment, which is the honest outcome rather than a
+renders against a ``body_format="html"`` body referring to the part as
+``cid:<id>``; with a plain-text body it stays an ordinary attachment, which is the honest outcome rather than a
 ``multipart/related`` nothing points into.
 
 There is deliberately **no file path**. In the local single-mailbox deployment

squirrel_mcp/tools/mail/compose.py

@@ -29,7 +29,14 @@
 
 from ...error_handling import ValidationError
 from ...schemas import DraftResult, SendResult
-from .._common import as_str_list, bare_addresses, reply_subject, require_confirm, run_blocking
+from .._common import (
+    as_bodies,
+    as_str_list,
+    bare_addresses,
+    reply_subject,
+    require_confirm,
+    run_blocking,
+)
 from .attachments import resolve_attachments
 
 
@@ -93,7 +100,7 @@ async def mail_create_draft(
             reply_to_folder: str = "INBOX",
             reply_all: bool = False,
             attachments: Optional[Any] = None,
-            body_html: Optional[str] = None,
+            body_format: str = "text",
             account: Optional[str] = None,
         ) -> DraftResult:
             """Save a new draft to the Drafts folder (nothing is sent).
@@ -116,16 +123,23 @@ async def mail_create_draft(
             bytes you supply. ALWAYS prefer the first form when forwarding
             something the user received -- it copies nothing through you.
 
-            EMBEDDING AN IMAGE: pass ``body_html`` (the rich version of the
-            same message -- ``body`` stays the plain text and is still what a
-            plain-text client shows) and mark the attachment ``"inline": true``
-            with a ``"content_id"``, then refer to it from the HTML as
-            ``<img src="cid:that-id">``. Without an HTML body pointing at it,
-            an inline file simply arrives as an ordinary attachment.
+            FORMAT: ``body`` is plain text unless ``body_format="html"``, in
+            which case ``body`` *is* the HTML and the plain-text half of the
+            message is derived from it -- so you write the message once, in
+            whichever form suits it. Default to text; reach for html when the
+            content needs it (a link the user should see as a link, a list, an
+            embedded image), not to decorate a message the user asked to keep
+            plain.
+
+            EMBEDDING AN IMAGE: needs ``body_format="html"``. Mark the
+            attachment ``"inline": true`` with a ``"content_id"`` and refer to
+            it from the HTML as ``<img src="cid:that-id">``. With a plain-text
+            body an inline file simply arrives as an ordinary attachment.
             """
             provider, sub = await self._get_provider(account, writes=True)
             if body is None:
                 raise ValidationError("'body' is required")
+            body, html_body = as_bodies(body, body_format)
             files = await resolve_attachments(
                 provider, attachments, default_folder=reply_to_folder
             )
@@ -158,7 +172,7 @@ async def mail_create_draft(
                 reply_to_uid=reply_to_uid,
                 reply_to_folder=reply_to_folder,
                 attachments=files,
-                body_html=body_html,
+                body_html=html_body,
             )
             self._track_usage(sub, "mail_create_draft")
             return DraftResult(
@@ -190,7 +204,7 @@ async def mail_edit_draft(
             bcc: Optional[Any] = None,
             folder: str = "Drafts",
             attachments: Optional[Any] = None,
-            body_html: Optional[str] = None,
+            body_format: str = "text",
             account: Optional[str] = None,
         ) -> DraftResult:
             """Replace an existing draft with new content. Returns the new uid.
@@ -203,13 +217,14 @@ async def mail_edit_draft(
             ATTACHMENTS: omit the argument and whatever the draft already
             carries is kept -- fixing a typo does not drop the file. Pass a list
             (same shape as mail_create_draft) to replace them, or [] to strip them.
-            ``body_html`` is rewritten like the rest of the draft, so re-pass
-            it when editing a message that had one.
+            ``body_format`` is rewritten like the rest of the draft, so
+            re-pass ``"html"`` when editing a message that had an HTML body.
             """
             provider, sub = await self._get_provider(account, writes=True)
             recipients = as_str_list(to)
             if not recipients:
                 raise ValidationError("'to' is required (at least one recipient)")
+            body, html_body = as_bodies(body, body_format)
             # None and [] mean different things here, so the resolve is
             # conditional: omitted keeps the draft's own files, [] clears them.
             files = (
@@ -228,7 +243,7 @@ async def mail_edit_draft(
                 cc=as_str_list(cc),
                 bcc=as_str_list(bcc),
                 attachments=files,
-                body_html=body_html,
+                body_html=html_body,
             )
             self._track_usage(sub, "mail_edit_draft")
             return DraftResult(
@@ -324,7 +339,7 @@ async def mail_send(
             reply_to_folder: str = "INBOX",
             reply_all: bool = False,
             attachments: Optional[Any] = None,
-            body_html: Optional[str] = None,
+            body_format: str = "text",
             confirm: bool = False,
             account: Optional[str] = None,
         ) -> SendResult:
@@ -354,12 +369,18 @@ async def mail_send(
             user received. Name every attachment in the approval you ask for --
             a file leaving the mailbox is as much a decision as the recipient.
 
-            EMBEDDING AN IMAGE: pass ``body_html`` (the rich version of the
-            same message -- ``body`` stays the plain text and is still what a
-            plain-text client shows) and mark the attachment ``"inline": true``
-            with a ``"content_id"``, then refer to it from the HTML as
-            ``<img src="cid:that-id">``. Without an HTML body pointing at it,
-            an inline file simply arrives as an ordinary attachment.
+            FORMAT: ``body`` is plain text unless ``body_format="html"``, in
+            which case ``body`` *is* the HTML and the plain-text half of the
+            message is derived from it -- so you write the message once, in
+            whichever form suits it. Default to text; reach for html when the
+            content needs it (a link the user should see as a link, a list, an
+            embedded image), not to decorate a message the user asked to keep
+            plain.
+
+            EMBEDDING AN IMAGE: needs ``body_format="html"``. Mark the
+            attachment ``"inline": true`` with a ``"content_id"`` and refer to
+            it from the HTML as ``<img src="cid:that-id">``. With a plain-text
+            body an inline file simply arrives as an ordinary attachment.
 
             THE COPY IN SENT: ``saved_to_sent`` says whether the message was
             also filed in the account's Sent folder (``sent_folder`` names it).
@@ -369,6 +390,7 @@ async def mail_send(
             provider, sub = await self._get_provider(account, writes=True)
             if body is None:
                 raise ValidationError("'body' is required")
+            body, html_body = as_bodies(body, body_format)
             # Resolved before the confirm gate: a wrong uid or bad base64 must
             # fail as a validation error while nothing has been sent, not after.
             files = await resolve_attachments(
@@ -403,7 +425,7 @@ async def mail_send(
                 reply_to_uid=reply_to_uid,
                 reply_to_folder=reply_to_folder,
                 attachments=files,
-                body_html=body_html,
+                body_html=html_body,
             )
             self._track_usage(sub, "mail_send")
             return SendResult(