Skip to content

Collection on-ramp: a fourth ingest path for user-saved collections #2

Description

@oojBuffalo

Problem

Scrolls has three on-ramps and none of them can copy a user's saved collection out of the service where it lives:

On-ramp Shape Limit
scrolls add <url> one URL cli.py:185 takes exactly one URL — no batch, no file, no stdin
scrolls import <sub> a file someone already exported only 5 external paths exist (below)
scrolls follow / sync RSS/Atom deltas feeds only (feeds.py)

The five external importers are bookmarks (browser HTML export), pocket (CSV), google-takeout (YouTube watch history only), opml (feed subscriptions), and fieldtheory (X bookmarks, by reading another application's local store). Everything else under scrolls importitems, bundle, events, archive — is Scrolls-to-Scrolls round-trip, not an on-ramp.

Consequence: a user with 200 saved Wikipedia articles, 400 GitHub stars, or a Mastodon bookmark folder has no way in except running scrolls add once per URL. That directly undercuts the vision's premise — a personalized library of what you deliberately saved — because the place your saves already live is the one place Scrolls can't read.

This is a missing fourth on-ramp, not a missing per-source feature. Fixing it one adapter at a time would bake in a shape that doesn't generalize (see #3 for the per-source audit: ~12 adapters have a real user collection and no path to it).

Prior art in-repo: IDEAS.md:239-241 already sketched exactly this (scrolls sync x --bookmarks, scrolls sync youtube --playlist, scrolls sync wikipedia …). It was never built and sync collapsed to feeds-only. ADR 0009's Consequences section explicitly leaves "native scrolls sync x --bookmarks" open as a later decision. This issue is that decision.

Decisions already taken (2026-07-29, Elijah)

  • Per-source credentials are acceptable. Scrolls may authenticate as the user to the services where their saves live, reading a token from the environment or config.toml. Nothing leaves the machine and no Scrolls-side account exists — this does not conflict with vision §4's "no persistent remote accounts," which concerns Scrolls-the-service, not authenticating to yours.
  • The general contract comes before the second implementation. Wikipedia (Wikipedia reading-lists import: first collection implementation #4) is the first proof; X (Native X bookmarks as a collection source #6) is the second.

Scope

Write an ADR establishing the collection on-ramp contract. It must settle:

  1. Surface. A new verb, a new import subcommand family, or an extension of sync. sync already means "live deltas from a followed thing," which is arguably what a saved collection is — but it is currently hard-wired to RSS/Atom, so reusing it is a real refactor, not a free ride.
  2. Declaration. How a source module declares the collections it can enumerate (a Wikipedia adapter offers reading lists; GitHub offers stars and watched repos; a package registry offers nothing). The absence of a collection must be a first-class, honest answer, not an error.
  3. Credentials. Where a per-source token lives, how it is resolved, and what happens when it is missing or rejected. Follow the existing LLMAuthError posture in llm.py:44 — a whole-batch failure that says plainly what to set, rather than a silent partial result.
  4. Custody semantics. What stage collection-imported items enter at, what provenance.adapter / extraction_method record, and how saved_at is derived from the service's own "when you saved it" timestamp. Precedent is consistent across the existing importers: Field Theory bookmarkedAt, Pocket time_added, bookmarks ADD_DATE — all become saved_at, never published_at.
  5. Idempotency and re-runs. Existing importers are INSERT OR IGNORE skips (ADR 0009). A collection that the user removes from upstream is a custody question this ADR should at least name, even if it defers it.
  6. Rate limits and volume. Enumerating a large collection is many requests; the ADR should state the pagination/backoff posture rather than leaving it per-adapter.

Acceptance criteria

  • A new ADR under docs/adr/ settling items 1–6, referencing docs/vision.md and the existing importer precedents.
  • ADR 0009 marked superseded (see Remove the Field Theory dependency; supersede ADR 0009 #5).
  • docs/architecture.md adapter-contract section extended with the collection declaration.
  • docs/agents/progress/260722/plans.md gains this work as an explicit objective — its current list (CI → dogfood → issues → one product slice) has no entry for the on-ramp gap, and a dogfood pass would hit this first.
  • No implementation in this issue; Wikipedia reading-lists import: first collection implementation #4 is the first implementation.

Verification

uv run pytest -q
uv run scrolls import --help    # unchanged until #4 lands

Notes

docs/agents/progress/260722/report.md states HEAD 2a5344f / 523 commits ahead; the trunk is now at ac527cc / 538 ahead after the docs-consolidation run. Expected drift for a dated snapshot, recorded here so the next check-in corrects it rather than rediscovering it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-humanNeeds human implementation/review

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions