Skip to content

refactor(backend): DB as source of truth — all iOS reads from DB #17

@Jing-yilin

Description

@Jing-yilin

Problem

Every iOS request to /api/campaigns and /api/campaigns/search hits ScrapingBee directly, consuming 10 credits per call. The DB is only used as a fallback or for sort=hot.

Current Data Flow

iOS → GET /api/campaigns         → ScrapingBee real-time fetch (10 credits/request)
iOS → GET /api/campaigns/search  → ScrapingBee real-time fetch (10 credits/request)
Cron 02:00 UTC                   → ScrapingBee batch crawl → write to RDS

Target Architecture

ScrapingBee ──(cron only)──→ RDS (source of truth)
                                     ↓
iOS → GET /api/campaigns         → query DB
iOS → GET /api/campaigns/search  → query DB (ILIKE)

Changes

handler/campaigns.go

ListCampaigns — remove ScrapingBee call, read from DB only. Sort mapping:

sort param DB ORDER BY
trending velocity_24h DESC, percent_funded DESC
newest first_seen_at DESC
ending deadline ASC (where deadline > now)
hot velocity_24h DESC

Pagination: cursor encoded as "offset:N", compatible with existing iOS String?.

SearchCampaigns — remove ScrapingBee call, replace with DB ILIKE:

WHERE (name ILIKE '%q%' OR blurb ILIKE '%q%')
  AND state = 'live'
  [AND category_id = ?]
ORDER BY percent_funded DESC

Both handlers no longer need *service.KickstarterScrapingService param.

cmd/api/main.go

Remove scrapingService param from ListCampaigns and SearchCampaigns handler registrations.

No changes

  • cron.go — continues nightly crawl via ScrapingBee → DB
  • scrapingbee_client.go / kickstarter_scraping.go — unchanged, cron-only
  • ListCategories — already DB-first, unchanged

Trade-offs

  • Search is limited to crawled campaigns (~15 categories × 10 pages). Campaigns not yet crawled will not appear in search results.
  • ScrapingBee credit usage drops to cron-only: ~750 credits/day instead of per iOS request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions