search: recoverable payloads for Postgres-invalid regexes (#75)

README.md

@@ -220,8 +220,9 @@ Unsupported syntax. Most of these raise; the first is silent and therefore more
 
 Two different engines run in sequence, and neither is zoekt's RE2:
 
-1. Postgres POSIX ARE (`~` / `~*`) selects which files match. Invalid patterns surface as
-   a query-time database error, not a parse error.
+1. Postgres POSIX ARE (`~` / `~*`) selects which files match. An invalid pattern (any
+   polarity) surfaces as the recoverable `regex_invalid` payload field carrying the
+   Postgres message, not a parse error and not a failed tool call.
 2. Python `re` rescans those files to produce the highlighted line matches.
 
 The practical consequences: `^` and `$` are line anchors and `.` never crosses lines;
@@ -265,9 +266,12 @@ never the literal string `"HEAD"` unless that is genuinely the resolved branch (
 repo with no `default_branch` recorded).
 
 Recoverable conditions come back as payload fields —
-`query_parse_error`, `query_too_broad`, `truncated`, `regex_incompatible`,
+`query_parse_error`, `query_too_broad`, `truncated`, `regex_incompatible`, `regex_invalid`,
 `no_content_atom`, `zero_width_only_atoms`, `commit_not_indexed` — rather than errors, so
-an agent can react without a failed tool call. Pagination rides the same envelope as
+an agent can react without a failed tool call. `regex_invalid` is distinct from
+`regex_incompatible`: the latter means Python `re` (not Postgres) rejected an otherwise-valid
+pattern and only degrades highlighting; `regex_invalid` means Postgres rejected the pattern
+outright and the query did not run. Pagination rides the same envelope as
 `next_cursor`, and the semantic tool adds its own status fields (`semantic_enabled`,
 `semantic_schema_missing`).
 

app/AGENTS.md

@@ -30,7 +30,7 @@ The MCP server Databricks App: a FastMCP streamable-HTTP service exposing the in
 ### Working In This Directory
 - **The engine is a process-scoped module singleton, not lifespan-owned.** A FastMCP `lifespan` re-enters once per MCP session; building the engine there would re-pay Lakebase cold start and open N×5 pools. `get_engine()` in `main.py` is the only builder; the lifespan only references it and never disposes (that is `atexit`'s job).
 - **Blocking work runs off the event loop.** Every tool body goes through `_dispatch` → `anyio.to_thread.run_sync` under `_DB_LIMITER` (sized to the 5-conn pool). Never run SQL or the Python `re` rescan inline in an async handler.
-- **Recoverable conditions are payload fields, never exceptions**: `truncated`, `query_too_broad`, `query_parse_error`, `regex_incompatible`, `no_content_atom`, `zero_width_only_atoms`, `semantic_schema_missing`. Only genuinely unexpected faults reach `_dispatch`, which logs the traceback and re-raises. Envelope keys are additive and permanent — agents depend on them; never remove or reshape one.
+- **Recoverable conditions are payload fields, never exceptions**: `truncated`, `query_too_broad`, `query_parse_error`, `regex_incompatible`, `regex_invalid`, `no_content_atom`, `zero_width_only_atoms`, `semantic_schema_missing`. Only genuinely unexpected faults reach `_dispatch`, which logs the traceback and re-raises. Envelope keys are additive and permanent — agents depend on them; never remove or reshape one.
 - **`clamp_limit` gates every caller-supplied limit** (`<=0` → `row_limit`, `> max` → `max_row_limit`) before it reaches a builder.
 - `main.py` aliases `service.*` builders (`_search_code_payload = service.search_code_payload`); tests monkeypatching collaborators must patch `service.*`, since function globals resolve in the defining module.
 - Tools/routes are registered on a fresh `FastMCP` inside `create_app()` (a `streamable_http_app`'s session manager is single-use); do not decorate onto a module-global instance.

app/main.py

@@ -23,10 +23,10 @@
    shutdown via ``atexit``; the per-session lifespan only *references* it and never disposes.
 
 Recoverable conditions (``truncated``, ``query_too_broad``, ``query_parse_error``,
-``regex_incompatible``, ``no_content_atom``, ``zero_width_only_atoms``) are structured payload
-fields, never exceptions; only genuinely unexpected faults reach the ``_dispatch``
-choke-point, which logs a full traceback and re-raises (never swallows). Output shapes are
-pinned to the zoekt parity assertions in ``tests/unit/test_main.py``.
+``regex_incompatible``, ``regex_invalid``, ``no_content_atom``, ``zero_width_only_atoms``) are
+structured payload fields, never exceptions; only genuinely unexpected faults reach the
+``_dispatch`` choke-point, which logs a full traceback and re-raises (never swallows). Output
+shapes are pinned to the zoekt parity assertions in ``tests/unit/test_main.py``.
 """
 
 from __future__ import annotations
@@ -105,6 +105,9 @@ def _signals(payload: dict[str, Any]) -> dict[str, Any]:
         "truncated": payload.get("truncated"),
         "query_too_broad": payload.get("query_too_broad"),
         "query_parse_error": payload.get("query_parse_error"),
+        # Without this, a Postgres-rejected regex is log-indistinguishable from a genuine
+        # zero-result query.
+        "regex_invalid": payload.get("regex_invalid"),
         # Query-shape signals: a filter-only or all-zero-width query returns zero files
         # legitimately, so without these a shape problem is indistinguishable in the logs
         # from a genuine no-match.
@@ -243,15 +246,19 @@ async def search_code(
     ``branch``/``commit`` are convenience params equivalent to appending ``branch:"<value>"`` /
     ``commit:<value>`` to ``query``. ``limit`` caps the number of files scanned (clamped to a
     server maximum). Recoverable conditions surface as fields (``query_parse_error``,
-    ``query_too_broad``, ``truncated``, ``regex_incompatible``, ``no_content_atom``,
-    ``zero_width_only_atoms``). The last two explain an empty result that is NOT a true negative:
-    ``no_content_atom`` means the query carried no affirmative content atom to highlight --
-    either a filter-only query (e.g. ``lang:go`` alone) or one that is entirely negated (e.g.
-    ``-foo`` alone: an exclusion is never a highlight) -- and ``zero_width_only_atoms`` means
-    every atom it carried matches zero-width (e.g. ``/^/``). A query mixing content with an
-    exclusion (e.g. ``foo -bar``) highlights only the affirmative term; excluded terms never
-    appear as matches. ``no_content_atom`` does not distinguish "no atom at all" from "fully
-    negated" -- recover which one it was from the echoed ``query`` field, if it matters.
+    ``query_too_broad``, ``truncated``, ``regex_incompatible``, ``regex_invalid``,
+    ``no_content_atom``, ``zero_width_only_atoms``). The middle two explain an empty result
+    that is NOT a true negative: ``no_content_atom`` means the query carried no affirmative
+    content atom to highlight -- either a filter-only query (e.g. ``lang:go`` alone) or one
+    that is entirely negated (e.g. ``-foo`` alone: an exclusion is never a highlight) -- and
+    ``zero_width_only_atoms`` means every atom it carried matches zero-width (e.g. ``/^/``). A
+    query mixing content with an exclusion (e.g. ``foo -bar``) highlights only the affirmative
+    term; excluded terms never appear as matches. ``no_content_atom`` does not distinguish "no
+    atom at all" from "fully negated" -- recover which one it was from the echoed ``query``
+    field, if it matters. ``regex_invalid`` carries the Postgres error message when a
+    ``/regex/``, ``repo:``, ``file:``, or ``sym:`` pattern is not a valid Postgres POSIX ARE
+    (e.g. ``/[/``) -- distinct from ``regex_incompatible``, which means Python ``re`` (not
+    Postgres) rejected an otherwise-valid pattern and only degrades highlighting.
     """
     lc = ctx.request_context.lifespan_context
     engine, cfg = lc["engine"], lc["config"]
@@ -290,6 +297,11 @@ async def semantic_search(
     stripped or reinterpreted). A query that is only filters, or empty/whitespace-only, leaves
     nothing to embed and returns ``nothing_to_embed: true`` with no embedding call made.
 
+    ``repo:``/``file:`` filter values ARE matched as Postgres regular expressions (unlike a
+    bare token, which is only rejected if written ``/like/this/``): a Postgres-invalid pattern
+    there (e.g. ``repo:[``) returns ``regex_invalid`` set to the Postgres error message, with a
+    remedy in ``reason`` -- fix the pattern rather than resubmitting the query verbatim.
+
     ``branch`` is sugar for a ``branch:`` atom -- conjunctive with any ``branch:`` atom already
     in ``query`` (mirrors ``search_code``'s ``branch`` param) -- restricting to files whose
     indexed branches include the given name (exact match); with no branch given anywhere,

app/query/AGENTS.md

@@ -18,7 +18,7 @@ The pure halves of the query pipeline. `parser.py` turns a zoekt-style query str
 
 ### Working In This Directory
 - **Parser purity is a hard invariant.** `parser.py` (and this package `__init__`) must import nothing beyond stdlib; `test_parser_import_is_pure` runs `import app.query.parser` in a subprocess and fails on any sqlalchemy/SDK/tree-sitter leakage.
-- **Regex bodies are stored RAW, never `re.compile`d or escaped.** Postgres POSIX ARE ≠ Python `re`; validity is the DB's problem at execution time (`/[/` parses fine). The compiler binds patterns as parameters — never interpolate.
+- **Regex bodies are stored RAW, never `re.compile`d or escaped.** Postgres POSIX ARE ≠ Python `re`; validity is the DB's problem at execution time (`/[/` parses fine). The compiler binds patterns as parameters — never interpolate. `app/search/errors.py`'s `reraise_or_recoverable` maps a Postgres-invalid pattern to a typed `RegexInvalidError` -> the `regex_invalid` payload field (issue #75), never an uncaught fault.
 - **Case is query-global** (last `case:` wins) but stamped only on `Substring`/`Regex` leaves; the compiler derives it from any such leaf, and callers holding the raw query pass `resolve_case(query)` so a filter-only `case:yes file:x` still resolves exactly. `repo:` is ALWAYS case-insensitive (`~*`).
 - **The `coalesce(default_branch, 'HEAD')` expression must stay byte-identical** across its four sites: this compiler, the 0003 backfill, the semantic default leg, and `get_file_payload`.
 - `lang:` normalizes with `.strip().lower()` and unknown values match nothing (empty result, no error, no `indexer` import). Substring literals escape `\`, `%`, `_` (backslash first) with `escape="\\"`.

app/query/compiler.py

@@ -23,7 +23,9 @@
   case-flip set.
 * Regex is opaque. Regex/filter patterns bind RAW as parameters -- never escaped, never
   ``re.compile``-d. An invalid POSIX ARE surfaces as a DB execution error at query time,
-  not at compile time.
+  not at compile time -- which the search layer (:mod:`app.search.errors`) maps to a typed
+  ``RegexInvalidError`` -> the ``regex_invalid`` payload field, never an uncaught fault
+  (issue #75).
 * ``lang:`` normalization. ``File.lang == lang.strip().lower()``; unknown values match
   nothing (empty result) rather than raising. No ``indexer`` import.
 * Substring escaping. ``LIKE``/``ILIKE`` literals escape ``\\``, ``%``, ``_`` (backslash