Skip to content

search: recoverable payloads for Postgres-invalid regexes (#75) - #99

Merged
IceRhymers merged 1 commit into
integration/search-hardening-and-ci-integrityfrom
feat/75-regex-invalid-payload
Jul 24, 2026
Merged

search: recoverable payloads for Postgres-invalid regexes (#75)#99
IceRhymers merged 1 commit into
integration/search-hardening-and-ci-integrityfrom
feat/75-regex-invalid-payload

Conversation

@IceRhymers

Copy link
Copy Markdown
Owner

Summary

Refs #75

A Postgres-invalid POSIX regex (e.g. /[/), negated or not, reaching search_code's grep/symbol legs, repo:/file:/sym: filter patterns, or semantic_search's repo:/file: filters previously escaped as an uncaught sqlalchemy.exc.DataError (an unhandled 500-class fault at the MCP surface) instead of a recoverable payload field — violating this codebase's stated contract that "recoverable conditions are payload fields, never exceptions" (app/main.py's module docstring).

  • app/search/errors.py: new RegexInvalidError; reraise_or_query_too_broad(OperationalError) replaced by a wider reraise_or_recoverable(error: DBAPIError) -> NoReturn mapping psycopg.errors.QueryCanceledQueryTooBroadError (unchanged) and psycopg.errors.InvalidRegularExpressionRegexInvalidError (new); every other DBAPIError re-raises unchanged (proven by identity assertions in tests).
  • All raw-execution call sites (grep.py ×2, symbols.py ×2, references.py ×3, semantic.py's RRF execute ×1 — newly wrapped, previously uncaught) switched to except DBAPIError + the new mapper.
  • app/service.py: new additive, permanent regex_invalid: str | None envelope field. search_code_payload gains a grep-leg trip arm (empty files, truncated=False — nothing was attempted-and-cut) and a symbol-leg-only trip arm (keeps grep's already-fetched files, mirroring the existing QueryTooBroadError partial-result pattern).
  • app/search/semantic.py: new _semantic_regex_invalid_payload for repo:/file: filter patterns Postgres rejects.
  • webui/main.py: /api/search and /api/semantic map regex_invalid to a 400 with the actual Postgres message (previously /api/search 500'd; /api/semantic never had the field).
  • tests/integration/test_grep.py's xfail(raises=DataError, strict=False) pin (asserting the desired future behavior) is replaced with 3 gating tests.
  • Docs: README.md, app/AGENTS.md, app/search/AGENTS.md, app/query/AGENTS.md, app/query/compiler.py header updated for the new field/contract.

Out of scope (by design): issue #38 (Python-side regex CPU/ReDoS bounding, RE2, complexity gating) — grep.py's "Uncapped Python CPU" caveat bullet is untouched verbatim.

Test plan

  • make lint — ruff check + format + mypy: clean
  • make test — 1076 passed (unit + observability)
  • make test-integration locally against the codesearch-pg Postgres container (tests/integration/test_grep.py, test_symbols_search.py, test_service.py): 80/80 passed, including new RegexInvalidError coverage (bare/negated regex, repo:/sym: filter patterns, full-stack envelope). The repo's CI integration workflow (ci-lakebase.yml) is hard-gated on CI_LAKEBASE_ENABLED, unprovisioned repo-wide, so this local run is the only real integration evidence.
    • Remaining local-suite failures (test_migrations.py, test_reconcile.py, test_semantic_rrf.py, test_store_chunk_writer.py, test_webui_semantic.py, test_commit_search.py, test_mcp_server.py) are pre-existing environment gaps unrelated to this change (missing lakebase_tokenizer/lakebase_ann/lakebase_bm25 extensions on vanilla local Postgres, and local env config) — confirmed identical on the unmodified base commit.
  • Independent code-review pass (separate agent, adversarial): APPROVE, 0 CRITICAL/HIGH/MEDIUM findings, 2 LOW informational notes (no changes required).

A Postgres-invalid POSIX regex (e.g. /[/), negated or not, through search_code,
symbol_search, or semantic_search's repo:/file: filters previously reached the
database raw and escaped as an uncaught sqlalchemy.exc.DataError instead of a
recoverable payload field. Adds a typed RegexInvalidError mapped by a single
reraise_or_recoverable(error: DBAPIError) mapper shared across grep/symbols/
references/semantic, surfaced as a new additive `regex_invalid` field on the
search_code and semantic_search envelopes, and mapped to a 400 with the Postgres
message on the webui /api/search and /api/semantic routes.

Refs #75
@IceRhymers
IceRhymers merged commit bd51c16 into integration/search-hardening-and-ci-integrity Jul 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant