Conversation
…gration Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX5fZLCKjxyASaMAxMxxum
…Turso)
Phase 0: swap the DB access mechanism only — business logic, schema, and the
OLX collection method are unchanged. libSQL is SQLite-compatible (single code
path: file: locally, Turso URL in prod), enabling Render + Turso deploy.
- db.ts: createClient + initDb (executeMultiple schema.sql) + thin async
helpers dbGet/dbAll/dbRun; drop the historical migration scaffold
(addColumnIfMissing/migrateListingsTable/backfill/PRAGMA/WAL) — schema.sql
already holds every column.
- env.ts: load server/.env before db.ts reads TURSO_* (imported first in db.ts).
- Whole DB layer is async: every .get/.all/.run -> await dbGet/dbAll/dbRun;
interactive db.transaction('write') for read->decide->write (upsert,
statusEngine, analysis/relevance/aiPicks commit); db.batch('write') for pure
write sets (cascade delete, sort swaps, filtered_out recompute, migratePostedAt).
- index.ts: host 0.0.0.0, CORS origin from WEB_ORIGIN, await initDb before listen.
- CLI entrypoints (scan.ts, migratePostedAt.ts) call initDb.
- .env.example: TURSO_DATABASE_URL/TURSO_AUTH_TOKEN/WEB_ORIGIN.
- docs: architecture.md/structure.md updated; render-turso-phase0.md marked done.
Verified locally against libSQL file:: build green; /health ok; live OLX scan
(GraphQL) -> upsert+dedup; coverage-window disable; manual PATCH override;
CLI scan; cascade-delete/recompute via batch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX5fZLCKjxyASaMAxMxxum
Beginner-friendly walkthrough: Turso DB setup, Render Web Service (backend) with build/start commands and env vars, Render Static Site (frontend) with /api/* rewrite (same-origin, no CORS), verification, free-tier cold-start notes, and troubleshooting. No code changes — frontend keeps relative /api fetch; the Render rewrite proxies it to the backend. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX5fZLCKjxyASaMAxMxxum
Gate access to the dashboard behind Google login (allowlist of one email) without turning the app multi-user — no users table, no user_id, no data partitioning. Auth is a lock on the front door plus a session cookie. Server (server/src/auth/): - config.ts: env (GOOGLE_CLIENT_ID/ALLOWED_EMAILS/SESSION_SECRET), cookie flags (prod cross-site Secure+SameSite=None, local http Lax), fail-fast assertAuthConfigured, AUTH_DISABLED dev bypass. - plugin.ts: fastify-plugin (non-encapsulated) registering @fastify/cookie and @fastify/jwt, verifyGoogleIdToken via google-auth-library, global onRequest lock on /api/* (skips /health, /api/auth/*, CORS preflight). - routes.ts: POST /api/auth/google, GET /api/auth/me, POST /api/auth/logout. - index.ts: import ./env first, assertAuthConfigured, CORS credentials, register auth before domain routes. Frontend (web/src/auth/): - useAuth.ts: useSession/useLogin/useLogout, reacts to global 401 event. - AuthGate.tsx: login screen with GoogleLogin, wraps the app. - base.ts: credentials: 'include', VITE_API_BASE prefix, 401 -> gate event. - main.tsx: GoogleOAuthProvider. Header: logout button. DB schema untouched. Docs + .env examples updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wZmcZ6mmoQa5rv9gCa1KX
… cache tuning
Optimize database reads/writes for the Turso (libSQL) backend, where every
execute is a network round-trip.
- normalizer.upsertListings: replace the per-listing loop (EXISTS -> upsert ->
read-back -> UPDATE filtered_out, ~4 round-trips/row) with one bulk SELECT of
existing fields, in-memory COALESCE merge + filtered_out, and a single
db.batch('write'). A scan of N listings now does ~2 round-trips instead of ~4N.
filtered_out is folded into the UPSERT (no separate read/write).
- schema: add idx_listings_search_refresh (coverage window) and
idx_listings_search_lastseen (verify P1).
- scanRunLifecycle: throttle scan-progress writes to >=1s (poll is 1.5s),
avoiding dozens of redundant UPDATEs during deep scans.
- web: QueryClient defaults staleTime 60s + refetchOnWindowFocus false to stop
refetching the full listings list on every window focus.
- docs/architecture.md updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHUDe59mPDD1miT5km3n3u
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
strict mode bug
Render cannot reach www.googleapis.com cert endpoints (v1 PEM → 403, v3 JWK → non-200), so google-auth-library/jose-JWKS verification fails in production. Verify the ID token primarily via the tokeninfo endpoint on oauth2.googleapis.com (a reachable host; Google checks the signature, we validate aud/iss/email_verified), falling back to local jose+JWKS verification where www.googleapis.com is reachable (e.g. local dev). Якщо на Render увімкнено auto-deploy з гілки — після пушу він сам передеплоїться. Підтверди — і я запушу (або скажи, чи хочеш разом із рештою змін цієї сесії: оптимізація БД + фронтенд-фікси).
…sses On a plain page reload these always-fetched (stats) / on-demand (filter-options) endpoints scanned the search's listings multiple times. On Turso each pass bills N rows read, so for ~400 listings a reload cost ~2600 reads. - /stats: replace 4 separate scans (in_db COUNT, stale COUNT, verify P1 COUNT, verify P2 COUNT) with one aggregate using conditional SUM(CASE...). Reuse P1_CONDITION/P2_CONDITION (now exported from verifyScan.ts) so the verify predicate stays single-sourced. ~1600 -> ~400 reads. - /filter-options: replace 4 scans (DISTINCT city, DISTINCT seller_name, pros, cons) with one SELECT and JS dedup. categories still from category_facet. - SearchStats / FilterOptions contracts unchanged; frontend untouched. Verified on a temp DB: aggregate stats and consolidated filter lists match the old per-query results exactly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UHUDe59mPDD1miT5km3n3u
…sses On a plain page reload these always-fetched (stats) / on-demand (filter-options) endpoints scanned the search's listings multiple times. On Turso each pass bills N rows read, so for ~400 listings a reload cost ~2600 reads. - /stats: replace 4 separate scans (in_db COUNT, stale COUNT, verify P1 COUNT, verify P2 COUNT) with one aggregate using conditional SUM(CASE...). Reuse P1_CONDITION/P2_CONDITION (now exported from verifyScan.ts) so the verify predicate stays single-sourced. ~1600 -> ~400 reads. - /filter-options: replace 4 scans (DISTINCT city, DISTINCT seller_name, pros, cons) with one SELECT and JS dedup. categories still from category_facet. - SearchStats / FilterOptions contracts unchanged; frontend untouched. Verified on a temp DB: aggregate stats and consolidated filter lists match the old per-query results exactly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UHUDe59mPDD1miT5km3n3u
…q4p' into claude/turso-db-optimization-dv0q4p
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Контекст
Застосунок працює на Turso (
@libsql/client), де коженexecute— окремий мережевий round-trip, а операції рядків тарифікуються. Код у стилі «синхронний SQLite» (запит-на-рядок у циклі) перетворював один скан на сотні-тисячі мережевих викликів. Користувач помітив надлишкові запити й «дублювання (читання → перезаписування)».Раунд 1 — оптимізація скану + клієнтський кеш
1. Згортання upsert-циклу скану (головний виграш) —
server/src/scraper/normalizer.tsЗамість циклу на кожне оголошення
EXISTS → upsert → read-back → UPDATE filtered_out(~4 round-trip/рядок усередині довгої інтерактивної транзакції):SELECTнаявних полів поolx_id(чанками по 500);new_count, злиття COALESCE-значень іfiltered_outрахуються в пам'яті;filtered_outвписано в самUPSERT(без окремого read-back/UPDATE);db.batch('write')(атомарно, як транзакція).Скан на N оголошень: ~2 round-trip замість ~4N. Семантика збережена 1:1.
2. Індекси —
server/src/db/schema.sql:idx_listings_search_refresh (search_id, last_refresh_at),idx_listings_search_lastseen (search_id, last_seen_at).3. Тротлінг прогресу скану —
server/src/scanner/scanRunLifecycle.ts:onProgressписавscan_runsна КОЖЕН запит; додано тротл ≥1с.4. Тюнінг TanStack Query —
web/src/main.tsx:staleTime: 60_000+refetchOnWindowFocus: false.Раунд 2 — зайві читання при звичайному перезавантаженні сторінки
Reload з ~400 оголошеннями давав ~2600 читань Turso (≈6.5 на рядок). Причина — кілька повних проходів по рядках пошуку за один reload.
5.
/stats— 4 проходи → 1 —server/src/routes/searches.tsЗамість 4 окремих сканів (in_db
COUNT, staleCOUNT, verify P1COUNT, verify P2COUNT) — один агрегат черезSUM(CASE…).P1_CONDITION/P2_CONDITIONтепер експортуються зverifyScan.tsі реюзаться (єдине джерело предикату). ~1600 → ~400 читань.6.
/filter-options— 4 проходи → 1 —server/src/routes/searches.tsЗамість 4 сканів (DISTINCT city, DISTINCT seller, pros, cons) — один
SELECTі дедуп у JS.categoriesлишається зcategory_facet.Очікуваний reload: ~2600 → ~850 читань. Контракти
SearchStats/FilterOptionsнезмінні — фронт не чіпали.Перевірка
npm run build(server + web, strict TS) — зелено.file:БД індекси уEXPLAIN QUERY PLAN;new_count/filtered_out/дедуп/olx_status-disable збігаються з попередньою поведінкою./statsі консолідований/filter-optionsдають ідентичні числа/списки, що й старі окремі запити (in_db/stale/verify; cities/sellers).Поза скоупом (свідомо): Turso embedded replica, persist кешу запитів у localStorage, серверна пагінація списку (підлога ~400 читань = сам список).
🤖 Generated with Claude Code
https://claude.ai/code/session_01UHUDe59mPDD1miT5km3n3u