Conversation
…gration Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX5fZLCKjxyASaMAxMxxum
…Turso)
Phase 0: swap the DB access mechanism only — business logic, schema, and the
OLX collection method are unchanged. libSQL is SQLite-compatible (single code
path: file: locally, Turso URL in prod), enabling Render + Turso deploy.
- db.ts: createClient + initDb (executeMultiple schema.sql) + thin async
helpers dbGet/dbAll/dbRun; drop the historical migration scaffold
(addColumnIfMissing/migrateListingsTable/backfill/PRAGMA/WAL) — schema.sql
already holds every column.
- env.ts: load server/.env before db.ts reads TURSO_* (imported first in db.ts).
- Whole DB layer is async: every .get/.all/.run -> await dbGet/dbAll/dbRun;
interactive db.transaction('write') for read->decide->write (upsert,
statusEngine, analysis/relevance/aiPicks commit); db.batch('write') for pure
write sets (cascade delete, sort swaps, filtered_out recompute, migratePostedAt).
- index.ts: host 0.0.0.0, CORS origin from WEB_ORIGIN, await initDb before listen.
- CLI entrypoints (scan.ts, migratePostedAt.ts) call initDb.
- .env.example: TURSO_DATABASE_URL/TURSO_AUTH_TOKEN/WEB_ORIGIN.
- docs: architecture.md/structure.md updated; render-turso-phase0.md marked done.
Verified locally against libSQL file:: build green; /health ok; live OLX scan
(GraphQL) -> upsert+dedup; coverage-window disable; manual PATCH override;
CLI scan; cascade-delete/recompute via batch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX5fZLCKjxyASaMAxMxxum
Beginner-friendly walkthrough: Turso DB setup, Render Web Service (backend) with build/start commands and env vars, Render Static Site (frontend) with /api/* rewrite (same-origin, no CORS), verification, free-tier cold-start notes, and troubleshooting. No code changes — frontend keeps relative /api fetch; the Render rewrite proxies it to the backend. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX5fZLCKjxyASaMAxMxxum
Gate access to the dashboard behind Google login (allowlist of one email) without turning the app multi-user — no users table, no user_id, no data partitioning. Auth is a lock on the front door plus a session cookie. Server (server/src/auth/): - config.ts: env (GOOGLE_CLIENT_ID/ALLOWED_EMAILS/SESSION_SECRET), cookie flags (prod cross-site Secure+SameSite=None, local http Lax), fail-fast assertAuthConfigured, AUTH_DISABLED dev bypass. - plugin.ts: fastify-plugin (non-encapsulated) registering @fastify/cookie and @fastify/jwt, verifyGoogleIdToken via google-auth-library, global onRequest lock on /api/* (skips /health, /api/auth/*, CORS preflight). - routes.ts: POST /api/auth/google, GET /api/auth/me, POST /api/auth/logout. - index.ts: import ./env first, assertAuthConfigured, CORS credentials, register auth before domain routes. Frontend (web/src/auth/): - useAuth.ts: useSession/useLogin/useLogout, reacts to global 401 event. - AuthGate.tsx: login screen with GoogleLogin, wraps the app. - base.ts: credentials: 'include', VITE_API_BASE prefix, 401 -> gate event. - main.tsx: GoogleOAuthProvider. Header: logout button. DB schema untouched. Docs + .env examples updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wZmcZ6mmoQa5rv9gCa1KX
… cache tuning
Optimize database reads/writes for the Turso (libSQL) backend, where every
execute is a network round-trip.
- normalizer.upsertListings: replace the per-listing loop (EXISTS -> upsert ->
read-back -> UPDATE filtered_out, ~4 round-trips/row) with one bulk SELECT of
existing fields, in-memory COALESCE merge + filtered_out, and a single
db.batch('write'). A scan of N listings now does ~2 round-trips instead of ~4N.
filtered_out is folded into the UPSERT (no separate read/write).
- schema: add idx_listings_search_refresh (coverage window) and
idx_listings_search_lastseen (verify P1).
- scanRunLifecycle: throttle scan-progress writes to >=1s (poll is 1.5s),
avoiding dozens of redundant UPDATEs during deep scans.
- web: QueryClient defaults staleTime 60s + refetchOnWindowFocus false to stop
refetching the full listings list on every window focus.
- docs/architecture.md updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHUDe59mPDD1miT5km3n3u
strict mode bug
Render cannot reach www.googleapis.com cert endpoints (v1 PEM → 403, v3 JWK → non-200), so google-auth-library/jose-JWKS verification fails in production. Verify the ID token primarily via the tokeninfo endpoint on oauth2.googleapis.com (a reachable host; Google checks the signature, we validate aud/iss/email_verified), falling back to local jose+JWKS verification where www.googleapis.com is reachable (e.g. local dev). Якщо на Render увімкнено auto-deploy з гілки — після пушу він сам передеплоїться. Підтверди — і я запушу (або скажи, чи хочеш разом із рештою змін цієї сесії: оптимізація БД + фронтенд-фікси).
…sses On a plain page reload these always-fetched (stats) / on-demand (filter-options) endpoints scanned the search's listings multiple times. On Turso each pass bills N rows read, so for ~400 listings a reload cost ~2600 reads. - /stats: replace 4 separate scans (in_db COUNT, stale COUNT, verify P1 COUNT, verify P2 COUNT) with one aggregate using conditional SUM(CASE...). Reuse P1_CONDITION/P2_CONDITION (now exported from verifyScan.ts) so the verify predicate stays single-sourced. ~1600 -> ~400 reads. - /filter-options: replace 4 scans (DISTINCT city, DISTINCT seller_name, pros, cons) with one SELECT and JS dedup. categories still from category_facet. - SearchStats / FilterOptions contracts unchanged; frontend untouched. Verified on a temp DB: aggregate stats and consolidated filter lists match the old per-query results exactly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UHUDe59mPDD1miT5km3n3u
…sses On a plain page reload these always-fetched (stats) / on-demand (filter-options) endpoints scanned the search's listings multiple times. On Turso each pass bills N rows read, so for ~400 listings a reload cost ~2600 reads. - /stats: replace 4 separate scans (in_db COUNT, stale COUNT, verify P1 COUNT, verify P2 COUNT) with one aggregate using conditional SUM(CASE...). Reuse P1_CONDITION/P2_CONDITION (now exported from verifyScan.ts) so the verify predicate stays single-sourced. ~1600 -> ~400 reads. - /filter-options: replace 4 scans (DISTINCT city, DISTINCT seller_name, pros, cons) with one SELECT and JS dedup. categories still from category_facet. - SearchStats / FilterOptions contracts unchanged; frontend untouched. Verified on a temp DB: aggregate stats and consolidated filter lists match the old per-query results exactly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UHUDe59mPDD1miT5km3n3u
…q4p' into claude/turso-db-optimization-dv0q4p
new optimization for render-turso
…pass
The /stats endpoint ran a separate SUM(CASE…) scan over all listings on every
search-select to derive in_db/stale_count/verify_candidates — all derivable from
the listings array the client already has. Now /stats returns only last_scan
(~1 read) and the client computes aggregates via computeListingStats().
- server: trim /api/searches/:id/stats to LastScanResponse (last_scan only)
- web: add utils/searchStats.ts; useSearchStats merges last_scan + derived counts
from the ['listings'] cache (deduped with the table — no extra request)
- bonus: verify_candidates now updates live on status edits (same cache)
- expected: /stats 410→~1 read, search-select 1228→~819 (−33%); verify after deploy
docs: plan docs/plans/turso-stats-clientside.md, structure.md
…nder 401)
The blob-download helpers for the AI workflows (analysis package, relevance
package, AI Picks package, and the xlsx/json preview export) used a bare
fetch('/api/...') without the API_BASE prefix or credentials: 'include'. On
Render the web app and API are separate origins, so the SameSite=None session
cookie was not sent and the global JWT lock returned 401. Locally the Vite proxy
made it same-origin, hiding the bug.
Add a shared apiBlob() helper in api/base.ts that mirrors api() (API_BASE prefix,
credentials: 'include', conditional Content-Type, 401 -> re-login event) but
resolves to a Blob, and route all four download functions through it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ECwRYEMkkETto1AWc6e7Jm
…ua2' into render-turso
… instead of aborting the scan
A single transient OLX GraphQL request failure (HTTP 429 after the analyze
burst, a 5xx gateway blip, or an anti-bot 200 with a non-JSON body) aborted
the entire deep scan, and the inline HTML fallback in runDeepScanFromPlan
then swallowed the GraphQL cause, surfacing only the misleading "page renders
via JS" error.
- client.fetchPage: retry up to 3 attempts with backoff for transient
conditions (network error, HTTP 429/5xx, non-JSON body); deterministic
errors (4xx, ListingError, schema errors) still throw immediately.
- runDeepScanFromPlan: wrap the HTML fallback in try/catch and throw a
combined "graphql failed: …; html fallback failed: …" error, mirroring
fetchWithFallback, so the real GraphQL cause is never masked.
- fetchSearch / scanSingleBucket: when retries are exhausted mid-pagination
and data is already collected, stop with a partial-success warning and
move on (next bucket/variant) instead of throwing; only throw when no data
was collected yet (offset=0) so the HTML fallback still gets a chance.
Partial results skip the coverage window, so no false auto-disables.
Introduce a single shared "scope" control (AiScope: all/tab/selected/ candidates) used by the relevance filter, pros/cons wizard and AI Picks: - add utils/aiScope.ts (getScopeIds/getScopeCounts/getDefaultScope) and hooks/analysis/useAiScope.ts, reusing listingVisibility predicates - add components/analysis/ScopeSelector.tsx: 4 always-visible toggles (disabled instead of hidden); "candidates" set apart in amber + star, default only in AI Picks - "Весь пошук" now means literally all rows (incl. filtered/irrelevant); "tab" label shows the active tab name with a live count - AI Picks scopes are functional: client sends scope ids; server loadPickCandidates(id, ids?) and prompt/package.zip (GET->POST) + rank/import accept an optional ids pool - update CLAUDE.md scope invariant, architecture.md, structure.md, ai-flow.md; add docs/plans/ai-scope-selector.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0167F8cHBaaX9GJ9ccxw6Zqq
A deep scan adding +876 listings wrote ~6,500 Turso rows: every seen row (new and unchanged) was re-upserted, and each write hit the table plus all 3 secondary indexes (last_seen_at/last_refresh_at/status are always in the UPSERT SET clause), so each row cost 4 rows_written. - normalizer: diff existing rows; only new/HTML/genuinely-changed GraphQL rows take the full UPSERT. Unchanged rows get a cheap batched last_seen_at/ miss_count touch, throttled to once per day in SQL (fresh rows write nothing). - schema/db: drop idx_listings_search_lastseen (rewritten on every scan since last_seen_at always changes); verify P1 falls back to scan+sort. DROP INDEX IF EXISTS migration in initDb for deployed DBs. - statusEngine: replace per-candidate tx.execute loop with one db.batch; the non-disable branch updates only miss_count (no status-index churn). Behavior (statuses, coverage window, verify, reactivation) unchanged; verified end-to-end against a local libSQL file DB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCxt8KYN4Tm9feNhsBRMdo
…ists Empty TURSO_DATABASE_URL in .env bypassed the ?? fallback (empty string is not nullish), passing url:'' to libSQL (URL_INVALID). Normalize blank env vars to undefined and mkdir the parent dir for file: databases so a clean clone / empty env starts against server/data/olx.db instead of crashing.
The AI Picks manual (ZIP) prompt told the agent to print the final JSON to chat, unlike steps 1-2 which write output.json and forbid console output. It also lacked the shared forbidden-list / fallback scaffolding. - New manualZip.ts holds the shared prompt fragments (mechanicalIntro, packageContents, forbidden + FORBID_* items, resultLine, fallbackBlock, OUTPUT_FILE) so all three manual builders compose instead of duplicating. - AI Picks: rewrite as scriptless map-reduce ON FILES — each chunk nominates into nominations/nominees-NNN.json (nothing to console), then the agent reads all nominee files and writes the final top-N to output.json itself. No .py: ranking is judgment, so no deterministic engine and no merge script needed; files still survive weak-model context loss. Adds forbidden-list + fallback. - relevance.ts and prompts.ts refactored to source the shared skeleton from manualZip.ts; wording converged (map/reduce labels, single output.json/paste convention). - docs/ai-flow.md rewritten as a high-level view of all three AI steps (common agent pattern, per-step differences, code pointers); structure/architecture refs updated. Server prompts rendered and eyeballed; typecheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCxt8KYN4Tm9feNhsBRMdo
Weak agentic models fill an existing file more reliably than they create a correctly-named one, and a set of empty slots doubles as a visible checklist. So the AI Picks ZIP now ships an empty nominations/nominees-NNN.json ([]) per chunk, and the prompt tells the agent to FILL each slot from its paired candidates chunk (same NNN) instead of creating files. Still scriptless. Since step 3 has no verify.py, the prompt adds a self-check before the reduce: no slot may stay [] by mistake (empty is allowed only when a chunk genuinely has no worthy candidate). Docs (ai-flow/structure/architecture) updated. ZIP contents verified to include the nominee stubs; typecheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCxt8KYN4Tm9feNhsBRMdo
Symmetry with the AI Picks change: the AI Filter ZIP now ships an empty classifications/result-NNN.json ([]) per chunk, and the prompt tells the agent to FILL each slot from its paired descriptions chunk (same NNN) instead of creating the folder and files. Weak agentic models fill an existing, correctly-named file more reliably, and the empty slots act as a checklist. No self-check needed here (unlike AI Picks): the shipped verify.py already flags any unfilled slot as a missing id and sends the agent back to re-classify. Docs (ai-flow/structure/architecture) updated. Typecheck clean; archiver stub-append mechanics already verified for AI Picks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCxt8KYN4Tm9feNhsBRMdo
… scans Large deep scans (hundreds of requests, ~1 hour) previously lived entirely in memory and were written to the DB once at the very end - any exception near the end (e.g. OLX anti-bot 403 on a late synonym variant after both GraphQL and HTML fallback failed, or a failure mid price-bisection) lost everything collected. - Add ScanPersister: listings are upserted iteratively during the scan (per deep-scan page, per price bucket, per query variant) via new FetchOptions.onListings; already-persisted olx_ids are deduped so found/new_count stay correct and Turso writes are not duplicated. Intermediate flush failures are swallowed (retried on next flush); the final flush in finalizeScanResult still throws honestly. - Salvage partial data instead of failing the whole scan: a late query variant crash (fetchAllQueries / runDeepScanFromPlan) or a mid-bisection crash (SplitScanner.bisectPriceRange -> SplitPlan.probeWarning) now ends the scan early as a partial success with a warning, keeping everything collected so far. - Fix runDeepScanFromPlan calling scanFromPlan without deep:true - noSplit variants were paginated in normal mode (3 pages) instead of full depth. - Chunk db.batch in upsertListings (<=500 statements) so a huge final batch cannot fail on payload size after a successful fetch. - Docs: plan docs/plans/scan-failure-recovery.md, architecture/structure updates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XtybAF4xeHxLbLyKoxN9ac
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…I journal Logging was fragmented: Fastify's built-in pino wrote JSON to stdout only (lost on restart), scan_runs.error captured just the final scan message without the failing stage, two console.error calls bypassed the logger, and ~28 intentional best-effort catches were completely invisible. - Add server/src/logger.ts as the single logging service: root pino (passed to Fastify via loggerInstance so HTTP and app logs share one stream; pino-pretty in dev, enabled only when resolvable), getLogger(scope) for info/debug, and logError/logWarn(scope, stage,...) that write to stdout AND persist into the new app_logs table. Every journal entry carries scope (module) and stage (data-flow step, e.g. "bisect UAH0-5000", "variant 2/4", "POST /api/..."). Only warn+error are persisted (info/debug stay stdout-only to protect Turso writes); journal writes are fire-and-forget and never recurse or break logic. Retention: entries older than 14 days are purged on startup. - Global catchers: app.setErrorHandler (unhandled 5xx -> journal with method+URL), unhandledRejection / uncaughtException handlers. - Instrument previously invisible failure points: central scan-run catch (withScanRun, with searchId/runId), GraphQL transient retries (early "OLX is blocking" signal), bisection/bucket/variant salvage paths, best-effort category facet, verify page probes, failed OpenRouter attempts; ScanPersister.flushSafe now logs via the service instead of console.error. - Viewing: GET /api/logs (level/scope filters) + DELETE /api/logs, and a "Journal" dialog in the web header (filters, expandable stack/details, 5s auto-refresh while open). - Deps: pino (same version already inside Fastify's tree) as a direct server dependency, pino-pretty as devDep. - Docs: plan docs/plans/logging-system.md, architecture/structure/CLAUDE updates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XtybAF4xeHxLbLyKoxN9ac
RenJeka
changed the title
…eport Post-review cleanup of the two prior features: - Extract variantFailureNote() into fetchOrchestrator (shared by fetchAllQueries and runDeepScanFromPlan) so the salvage warning text stays identical in both scan paths instead of being copy-pasted. - Extract reportLoggerFailure() in logger.ts for the two identical "journal write failed" self-log sites; bind the retention interval as a SQL parameter instead of string-interpolating it. No behavior change. Verified: server tsc + web build green; persister, logger, and in-process setErrorHandler journaling smoke tests all pass (uncaught 5xx -> app_logs with stage=method+URL and stack; 4xx skipped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XtybAF4xeHxLbLyKoxN9ac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Частина 1 — Стійкість великих сканів (
7faa734)Проблема
Великий глибокий скан (сотні запитів, ~година) інколи падав майже наприкінці, і все зібране втрачалося — у БД не потрапляло нічого. Причини:
upsertListingsвикликався один раз уfinalizeScanResult, після завершення всього фетчу — весь скан жив у пам'яті.fetchAllQueries,runDeepScanFromPlan).bisectPriceRangeвалив весь варіант.runDeepScanFromPlanвикликавscanFromPlanбезdeep: true→ noSplit-варіанти пагінувалися лише на 3 сторінки.Рішення
ScanPersister(server/src/scanner/scanPersister.ts): зібране upsert-иться в БД ітераціями по ходу скану через новий колбекFetchOptions.onListings— після кожної сторінки deep-скану, кожного цінового бакету і кожного варіанта запиту. Дедуп уже збереженихolx_id(чесніfound/new_count, без подвійних Turso-записів); проміжнийflushSafeковтає транзієнтні збої БД, фінальнийflushкидає чесно. Звичайний (≤3 запити) скан пише один раз, як і раніше.SplitPlan.probeWarning) завершує скан частковим успіхом (warning, вікно покриття пропускається), а не помилкою.deep: trueуrunDeepScanFromPlan→ повна допагінація noSplit-варіантів.db.batch(≤500 statements) вupsertListings.Частина 2 — Система логування помилок (
739d85d)Проблема
Логування було фрагментарне: вбудований pino Fastify писав JSON лише в stdout (зникає після рестарту),
scan_runs.errorзберігав тільки фінальний текст без етапу, 2console.errorйшли повз логер, ~28 «тихих» best-effortcatchбули повністю невидимі.Рішення
server/src/logger.tsна базі pino (той самий інстанс у Fastify черезloggerInstance; у dev —pino-pretty, вмикається лише якщо пакет резолвиться):logError/logWarn(scope, stage, …)→ stdout + таблицяapp_logs. Кожен запис несеscope(модуль) іstage(крок data flow:bisect ₴0–5000,variant «x» 2/4,POST /api/…). У БД — лише warn+error (бережемо Turso writes); запис fire-and-forget, без рекурсії. Retention 14 днів.app.setErrorHandler(5xx → журнал з методом+URL),unhandledRejection/uncaughtException.withScanRun, зsearchId/runId), транзієнтні ретраї GraphQL (ранній сигнал «OLX відбиває»), порятунок бісекції/бакетів/варіантів, best-effort facet, verify-проби, невдалі спроби OpenRouter.GET /api/logs(фільтри level/scope) +DELETE /api/logs+ діалог «Журнал» у хедері фронта (фільтри, розгортання stack/details, авто-оновлення 5с).pino(версія з дерева fastify) як пряма залежність server,pino-pretty(devDep).Перевірка
tsc(server) +npm run build(server+web) — зелені.ScanPersisterна реальній файловій libSQL-БД: дедуп між флашами, чесніfound/new_count, запис 1200 рядків чанками — PASS.GET /health,GET/DELETE /api/logsпрацюють.docs/plans/scan-failure-recovery.md,docs/plans/logging-system.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01XtybAF4xeHxLbLyKoxN9ac