Skip to content

fix(search): align search URLs with upstream site shapes - #141

Open
Lxr-max wants to merge 1 commit into
aiming-lab:mainfrom
Lxr-max:cursor/fix-search-url-shapes-3228
Open

Lxr-max wants to merge 1 commit into
aiming-lab:mainfrom
Lxr-max:cursor/fix-search-url-shapes-3228

Conversation

@Lxr-max

@Lxr-max Lxr-max commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Reviewer takeover of stale author PR #15 for issue #14 (search URL shapes). Reimplemented against current main instead of rebasing the May patch.

Fixes #14.

Problem

Mirror search UIs emitted generic /search?q=... even when the upstream site uses a more specific path or query key. That still works for local tasks, but it is less realistic when an agent inspects, copies, or navigates search URLs.

What changed

Canonical search URLs are now what the site UI emits. Legacy /search?q=... (and /thesaurus?q=... on Cambridge) stay registered as aliases for existing tasks, health checks, and old trajectories.

Site Canonical Legacy alias
Amazon /s?k=<query> /search?q=<query>
Booking /searchresults.html?ss=<query> /search?q=<query>
Google Maps /maps/search/<query> /search?q=<query>
ESPN /search/_/q/<query> /search?q=<query>
Apple /search/<query> /search?q=<query>
Coursera /search?query=<query> /search?q=<query>
Hugging Face /search/full-text?q=<query> /search?q=<query>
Cambridge Dictionary /search/direct/?datasetsearch=english&q=<query> /search?q=<query>
Cambridge Thesaurus /search/english-thesaurus/direct/?datasetsearch=english-thesaurus&q=<query> /thesaurus?q=<query>

Path-based shapes (Maps, ESPN, Apple) use a small submit handler so the browser location becomes the upstream path. No-JS still posts the query string at the canonical prefix, and the old alias still resolves.

Documented as already matching (no code change): Google Search, GitHub, BBC, arXiv, WolframAlpha, Google Flights (/flights?... is the primary search).

Also adds:

  • docs/search-url-realism.md
  • scripts/check_search_url_realism.py

Coordination (files not rewritten here)

Lineage

Item Detail
Issue #14
Stale author PR #15 (hqhq1025, May 2026, 0 review comments, based on outdated main)
Fork branch https://github.com/Lxr-max/WebHarbor/tree/cursor/fix-search-url-shapes-3228

Salvaged from #15: the canonical URL table, alias policy, path-search JS pattern, docs, and regression script. Reapplied as a fresh diff on current main (Amazon/ESPN/Maps/Apple/Booking handlers have all moved since May).

Verification

python3 scripts/check_search_url_realism.py
python3 -m py_compile \
  sites/amazon/app.py sites/booking/app.py sites/google_map/app.py \
  sites/espn/app.py sites/apple/app.py sites/coursera/app.py \
  sites/huggingface/app.py sites/cambridge_dictionary/app.py \
  scripts/check_search_url_realism.py

Both passed.

Flask test_client against local apps (empty/fresh sqlite; no HF asset fetch):

Site Canonical Legacy UI emits canonical form
Amazon /s?k=xbox 200 /search?q=xbox 200 / has action="/s" name="k"
Booking /searchresults.html?ss=Paris 200 /search?q=Paris 200 / has action="/searchresults.html" name="ss"
Google Maps /maps/search/central%20park 200 /search?q=central+park 200 / has action="/maps/search/"
ESPN /search/_/q/lakers 200 /search?q=lakers 200 /search has action="/search/_/q/"
Apple /search/iphone 200 /search?q=iphone 200 /search has action="/search/"
Coursera /search?query=python 200 /search?q=python 200 / has name="query"
Cambridge /search/direct/?q=hello 200 /search?q=hello 200 / has /search/direct/ + datasetsearch
Hugging Face full-text route registered /search alias kept static regression; live render needs HF-managed images

Not re-run here: Docker image rebuild, /reset byte-identity (no seed DB or handler write-path change), Playwright. _health.py probes that hit /search?q=... remain valid aliases.

Please close or supersede #15 once this lands so the stale May PR is not merged on top of it.

Reviewer takeover of stale PR aiming-lab#15 for issue aiming-lab#14. Search forms now emit
known upstream URL shapes while keeping /search?q=... as compatibility
aliases for existing tasks and health checks.

Co-authored-by: Xuanrui Li <xuanrui.li@se24.qmul.ac.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Search URLs should match upstream site shapes

1 participant