Search for the words, not the exact string

CLAUDE.md

@@ -57,6 +57,38 @@ namespaces are reserved so they plug in later as sibling providers + tool mixins
   Flagging is advertised as non-destructive, so it does not get to delete
   anything. `tests/test_imap_flag.py` pins the command shape and the absent
   expunge; the GreenMail e2e proves both against a real server.
+- **A search query is parsed once, here, not interpreted by each backend.**
+  `search_query.py` owns the grammar (`from:`/`to:`/`cc:`/`subject:`/`body:`,
+  `"phrases"`, `-exclusions`, `OR`, `has:attachment`) and both backends compile
+  the same `MailQuery` -- IMAP into its search keys, the admin package's Graph
+  provider into KQL. It replaced handing the raw string to whoever was
+  answering, which failed twice over. **A literal is the wrong default**: IMAP's
+  `TEXT` is a substring of the raw message, so a contact's name missed a
+  signature reading `Iris  van 't Klooster` over the second space -- and would
+  equally have missed a header fold, a curly apostrophe or `Klooster, Iris`.
+  Unquoted words now AND as separate keys, which none of those defeat, and a
+  phrase is what you get by quoting. **And the fallback was undefined**: RFC
+  3501 explicitly lets a server "implement flexible matching" for `TEXT`, so
+  one host word-matched and ignored order while the next did strict substring
+  and Graph ran keywords -- the same tool meaning three things, which is the
+  divergence the attachment and meeting seams are held against.
+  Three rules make the backends agree. The compiled query is always **at least
+  as broad** as what was asked (`José` leaves as `Jos`, since no server will
+  fold accents for us and sending one spelling misses the other). Anything a
+  `MessageSummary` can prove -- sender, recipients, subject, attachment flag --
+  is then **verified exactly** here, taking that breadth back; anything it
+  cannot (the body, cc) stays the server's word, because a 200-character
+  preview refutes nothing. And when a query matches nothing the tool **widens
+  and says so**: `widen()` gives up one term at a time, least distinctive
+  first, and `matched`/`dropped_terms` carry which. That ladder tries *each*
+  term in turn rather than guessing once, because the absent word is usually
+  the rarest one, so a single "drop the weakest" keeps the word that is missing
+  and stays empty. ORing the terms together is deliberately not on it: it
+  always "works" and never informs. `parsed` is an **additive** keyword on
+  `MailProvider.search` offered via signature inspection, so a backend written
+  before it keeps its raw `query`. `tests/test_search_query.py` pins the
+  grammar and the ladder, `tests/test_mail_search.py` the IMAP keys and the
+  widening policy, and the GreenMail e2e proves a real server answers them.
 - **Replying is a provider concern, not a header the tool layer writes.**
   `mail_send` / `mail_draft` take a `reply_to_uid` (+ `reply_to_folder`) and
   hand it to the provider untouched, because *how* you join a thread is the
@@ -235,4 +267,5 @@ defaults -- a missing one is a config error, never a guess.
 | `knowledge.py` | Server instructions handed to the MCP client |
 | `error_handling.py` / `error_sanitizer.py` | Error hierarchy + message sanitizing |
 | `logging_config.py` | Structured logging to stderr |
+| `search_query.py` | Search grammar: parse, fold, verify, widen (both backends) |
 | `usage.py` | Usage-tracking stub (full version in admin package) |

squirrel_mcp/knowledge.py

@@ -20,6 +20,18 @@
   mailbox can be large, so never try to pull everything at once. `unseen_only`
   and `flagged_only` narrow it server-side -- use `flagged_only=true` to answer
   "what have I flagged", rather than paging the folder and sifting yourself.
+  The `query` takes the syntax any mail client takes: bare words must ALL
+  appear, in any order (`iris klooster`), `from:`/`to:`/`cc:`/`subject:`/`body:`
+  scope a word to one field, `"..."` demands a phrase, `-word` excludes, `OR`
+  offers a choice, `has:attachment` narrows to messages carrying a file.
+  Case, accents and apostrophes are ignored. Two habits worth having: search
+  for the distinctive WORDS of a name rather than quoting the whole thing --
+  a phrase has to survive the exact spacing and punctuation the sender used,
+  and separate words do not -- and use `from:` when you want mail FROM someone
+  rather than every mail mentioning them. If nothing matches everything asked
+  for, the search widens rather than coming back empty: `matched` says
+  "partial" and `dropped_terms` says which words were given up, so say so
+  instead of reporting the hits as exact.
 - mail_read: read one message by uid. Large bodies are truncated -- the response
   tells you the total length and how to page the rest.
 - mail_read_chunk: fetch the next slice of a large body (offset + length).

squirrel_mcp/providers/protocol.py

@@ -18,6 +18,8 @@
 from dataclasses import dataclass, field
 from typing import List, Optional, Protocol, Tuple, runtime_checkable
 
+from ..search_query import MailQuery
+
 
 # --------------------------------------------------------------------------- #
 # Errors raised by provider implementations. The tool layer catches these and
@@ -194,13 +196,34 @@ def search(
         since: Optional[str] = None,
         limit: int = 25,
         offset: int = 0,
+        parsed: Optional["MailQuery"] = None,
     ) -> Tuple[List[MessageSummary], int]:
         """Search a folder. Returns (page of summaries, total matching count).
 
         ``query`` is free text matched against the message (subject/body/headers).
         ``flagged_only`` narrows to messages carrying ``\\Flagged`` (see
         ``flag``). ``since`` is an ISO date (YYYY-MM-DD) lower bound. Newest
         first.
+
+        ``parsed`` is that same ``query`` already run through
+        ``search_query.parse`` -- the terms, the field scopes, the negations and
+        the quoted phrases, in the one grammar every backend compiles. Honour it
+        when it is given: it is what stops each backend inventing its own
+        meaning for a multi-word query, which is what ``TEXT "Iris van 't
+        Klooster"`` did on IMAP and ``$search`` did on Graph.
+
+        Two rules go with it, and the tool layer depends on both:
+
+        * **Compile it at least as broadly as it reads.** A term that cannot be
+          sent faithfully -- an accent an IMAP server will not fold, a phrase a
+          keyword index cannot keep together -- is widened, never narrowed. The
+          tool layer re-checks every field a summary can prove and takes the
+          breadth back; a match dropped down here is gone for good.
+        * **It is an addition, not a replacement.** ``query`` stays
+          authoritative, so a backend written before this argument existed (or
+          one that would rather use its own engine) keeps working by ignoring
+          it, and one that wants the structure without being handed it can call
+          ``parse(query)`` itself.
         """
         ...
 

squirrel_mcp/providers/soverin/imap.py

@@ -16,6 +16,8 @@
 
 from imap_tools import (
     AND,
+    NOT,
+    OR,
     BaseMailBox,
     MailBox,
     MailBoxStartTls,
@@ -26,6 +28,7 @@
 from imap_tools.utils import check_command_status, clean_uids
 
 from ...logging_config import get_logger
+from ...search_query import MailQuery, parse
 from ..protocol import (
     AttachmentInfo,
     AttachmentPayload,
@@ -157,8 +160,9 @@ def search(
         since: Optional[str] = None,
         limit: int = 25,
         offset: int = 0,
+        parsed: Optional[MailQuery] = None,
     ) -> Tuple[List[MessageSummary], int]:
-        criteria = self._build_criteria(query, unseen_only, flagged_only, since)
+        criteria = self._build_criteria(query, unseen_only, flagged_only, since, parsed)
         charset = "UTF-8" if query else "US-ASCII"
 
         def op(mb: BaseMailBox) -> Tuple[List[MessageSummary], int]:
@@ -395,12 +399,74 @@ def _select(mb: MailBox, folder: str) -> None:
             raise MailNotFoundError(f"Folder {folder!r} not found: {exc}") from exc
 
     @staticmethod
+    def _term_key(term) -> dict:
+        """One parsed term as the imap-tools kwarg for its IMAP search key.
+
+        ``from:``/``to:``/``cc:``/``subject:`` are real IMAP keys, so a scoped
+        term is answered by the server against that header alone instead of by
+        us against a page of it. ``body:`` is BODY (the body without the
+        headers) and the unscoped default is TEXT (both).
+        """
+        return {
+            "from": {"from_": term.wire},
+            "to": {"to": term.wire},
+            "cc": {"cc": term.wire},
+            "subject": {"subject": term.wire},
+            "body": {"body": term.wire},
+        }.get(term.field, {"text": term.wire})
+
+    @classmethod
+    def _query_criteria(cls, parsed: MailQuery) -> list:
+        """Compile a parsed query into IMAP keys -- one key per term, ANDed.
+
+        This is the fix for the literal phrase. IMAP ANDs the keys of a search
+        by definition, and each key is matched independently, so ``iris`` and
+        ``klooster`` as two TEXT keys find a signature reading ``Iris  van 't
+        Klooster`` that no single ``TEXT "Iris van 't Klooster"`` ever will --
+        the words need not be adjacent, in order, or on the same folded header
+        line.
+
+        A quoted phrase stays one key: the caller asked for those words
+        together, and the server's literal is the closest IMAP has to that.
+        """
+        keys: list = []
+        for clause in parsed.clauses:
+            if clause.negated:
+                # "Neither of these": one NOT around the alternatives, rather
+                # than one per term, so the clause excludes all of them.
+                inner = [AND(**cls._term_key(t)) for t in clause.terms]
+                keys.append(NOT(inner[0] if len(inner) == 1 else OR(*inner)))
+                continue
+            alternatives = [
+                NOT(AND(**cls._term_key(t))) if t.negated else AND(**cls._term_key(t))
+                for t in clause.terms
+            ]
+            # imap-tools' OR takes two or more keys and nests; a clause of one
+            # is just the key itself.
+            keys.append(alternatives[0] if len(alternatives) == 1 else OR(*alternatives))
+        return keys
+
+    @classmethod
     def _build_criteria(
-        query: Optional[str], unseen_only: bool, flagged_only: bool, since: Optional[str]
+        cls,
+        query: Optional[str],
+        unseen_only: bool,
+        flagged_only: bool,
+        since: Optional[str],
+        parsed: Optional[MailQuery] = None,
     ):
         kwargs: dict = {}
+        positional: list = []
         if query:
-            kwargs["text"] = query
+            # A provider is allowed to be handed the raw string on its own (the
+            # protocol keeps ``query`` authoritative), so parse it here rather
+            # than falling back to the literal this module used to send.
+            positional = cls._query_criteria(parsed or parse(query))
+            if not positional:
+                # Nothing survived tokenizing -- a query of pure punctuation.
+                # Search for it as written instead of silently listing the
+                # folder.
+                kwargs["text"] = query
         if unseen_only:
             kwargs["seen"] = False
         if flagged_only:
@@ -415,7 +481,9 @@ def _build_criteria(
                 raise MailProviderError(
                     f"'since' must be an ISO date (YYYY-MM-DD), got {since!r}"
                 ) from exc
-        return AND(**kwargs) if kwargs else "ALL"
+        if positional or kwargs:
+            return AND(*positional, **kwargs)
+        return "ALL"
 
     @staticmethod
     def _from_display(msg) -> str:

squirrel_mcp/providers/soverin/provider.py

@@ -11,6 +11,7 @@
 from typing import List, Optional, Tuple
 
 from ...config import SquirrelConfig
+from ...search_query import MailQuery
 from ..protocol import (
     AttachmentPayload,
     FolderInfo,
@@ -85,6 +86,7 @@ def search(
         since: Optional[str] = None,
         limit: int = 25,
         offset: int = 0,
+        parsed: Optional[MailQuery] = None,
     ) -> Tuple[List[MessageSummary], int]:
         return self._imap.search(
             folder,
@@ -94,6 +96,7 @@ def search(
             since=since,
             limit=limit,
             offset=offset,
+            parsed=parsed,
         )
 
     def fetch_message(self, folder: str, uid: str) -> MessageDetail:

squirrel_mcp/schemas.py

@@ -51,6 +51,28 @@ class SearchResult(BaseModel):
     limit: int = Field(description="Max messages returned per page")
     offset: int = Field(description="Messages skipped")
     folder: str
+    matched: str = Field(
+        default="all",
+        description=(
+            "How the query was matched. 'all' = every term was required and "
+            "found. 'partial' = nothing matched all of them, so the least "
+            "distinctive terms were dropped and these results match the rest -- "
+            "see dropped_terms and say so when reporting them. 'none' = nothing "
+            "matched even then."
+        ),
+    )
+    searched_terms: List[str] = Field(
+        default_factory=list,
+        description="The terms the query was understood as, after parsing",
+    )
+    dropped_terms: List[str] = Field(
+        default_factory=list,
+        description=(
+            "Terms given up to get any results at all. Non-empty means these "
+            "messages do NOT contain them -- tell the user which words were "
+            "ignored rather than presenting the hits as exact."
+        ),
+    )
 
 
 class AttachmentMeta(BaseModel):