Add Google Play Books as a source - #94
Merged
Merged
Conversation
Establishes empirically what the API will give us before we build against it.
Reading position is unavailable: /mylibrary/readingpositions/{id} returns
HTTP 501 'not implemented' for every volume, including public-domain ones,
so it is disabled server-side rather than missing data. It is still in the
discovery document, which is why the docs imply it works.
What does work: /volumes/mybooks?acquireMethod=... returns real Play Books
content (bookshelves returns zero for everything and is a red herring), and
mylibrary/annotations returns bookmarks and highlights carrying pageIds plus
a created timestamp - enough to derive dated progress against pageCount.
Tracks reading progress from the bookmarks you leave in Play Books. Progress is not where the API documentation suggests. /mylibrary/readingpositions returns HTTP 501 for every volume including public-domain ones, so it is disabled server-side rather than empty; userInfo.readingPosition is always null and userInfo.updated tracks acquisition, not reading; and mylibrary/bookshelves reports zero volumes because Play Books content never lands there. Bookmarks are the only reading signal the API exposes, and they carry a page id and a timestamp. Two constraints worth knowing: Page ids are prefixed and the prefix decides whether the number means anything. PA25 is a printed body page and divides into pageCount. PR7 is Roman front matter and reads as 0%, not 7/76. PT64 is a position in reflowable text with no printed equivalent, so we return null rather than inventing a percentage. The prefix is not predictable from volume metadata - volumes with identical readingModes produced PA in one case and PT in another - so the parser handles whatever arrives. The annotations endpoint rejects any request without a volumeId, so there is no library-wide incremental sync and we poll per volume. Deleted bookmarks are kept: moving a bookmark forward deletes the old one, and those tombstones are what make a reading trail reconstructable instead of only ever seeing the latest position. Also adds scripts/probe-google-books.mjs, which is how the above was established.
The proxy only injected x-public-origin for two hardcoded provider paths, so /v1/integrations/google-books/ fell through to resolvePublicOrigin's dev default and sent http://localhost:3002 as the redirect_uri regardless of where the browser actually was. Off localhost that fails as redirect_uri_mismatch, or worse succeeds and bounces the browser to an address it cannot reach. Matching on the /v1/integrations/ prefix means the next provider does not repeat it. Also strips any inbound x-public-origin. Previously a client could set it on any path the proxy did not overwrite it for, and origin.ts trusts that header - which the OAuth callback then redirects to.
Both eventMeta and summary switch on event_type with no case for google_books.progress or google_calendar.event, so reading events showed as a bare 'progress' with an empty summary and all calendar events as 'event'. Reading rows now carry the book, the page and - only for print-anchored positions - a percentage; reflowable positions show a position instead of a fabricated percent, and front matter keeps its page number so consecutive bookmarks are distinguishable rather than looking like duplicates. Deleted bookmarks are marked rather than hidden. A removed bookmark is still a position that was read to, usually meaning the bookmark moved on, so the row says so instead of disappearing or implying it still exists. Adds Calendar and Reading to the source filter, which only offered All, Time entries and GitHub. The events endpoint matches source as a plain column value, so no API change was needed.
Diagnostic for the redirect_uri_mismatch during Play Books setup. OAuth connects now, so it goes: it was unauthenticated by intent and reported request headers, which has no place in a shipped API.
Connected sources now expose settings, backed by a settings jsonb column on integrations. Absent settings means everything, so adding a setting never silently narrows what an already-connected source collects. GitHub: pick repositories, public or private - the grant already asks for repo scope, so private ones were always visible, just not selectable. Filtering happens before the per-repo commit calls, each of which is its own REST request, so a narrow selection cuts real work rather than just rows. Search and public/private filters with a bulk toggle over what is on screen, since selecting a handful out of many is the common case. Play Books: pick books and see current position per book, taken from the latest non-deleted bookmark. Progress bars appear only where a percentage is real - reflowable books say why they have none instead of showing a misleading bar. Also removes the coming-soon placeholder sources. The migration is hand-written: drizzle-kit generate cannot run here because the meta snapshots stop at 0019 while migrations run to 0023, so it diffs against a stale snapshot and prompts. Recent migrations in this repo are hand-authored anyway.
PT positions were being suppressed on the grounds that they had no printed-page equivalent. That was wrong: PA and PT are both page indices into the volume and the only difference is whether the page carries a printed number, so both divide into pageCount. Observed values bear this out - PT 45 to 65 against a 461-page volume, comfortably inside the range. Clamped at 100 so an index on an unexpected scale saturates rather than producing nonsense. Labels distinguish the two: 'page 6 of 76' for a printed page, 'position 47 of 461' for an unnumbered one, so neither claims a page number the book does not have. Existing reflowable rows were backfilled, since percent was stored at write time.
Play Books never reports pages read, but consecutive bookmarks in a book do: PA5 followed by PA6 is one page, PT45 to PT49 is four. Each advance is credited to the later bookmark, since that is when the reading had happened by. Four metrics, following the calendar source's shape: pages advanced, progress gained in percentage points, books read, and reading days. What deliberately scores nothing: - the first bookmark in a book, because we do not know where you started and assuming page one would invent a chunk of reading - steps between different numbering schemes, since Roman front matter and Arabic body pages are separate scales - backwards steps, which clamp to zero - re-reading is not un-reading Steps are computed over the whole trail rather than the reporting window, or a week continuing a book already in progress would score zero for its first session. Deleted bookmarks still count: a removed one marks a position that was reached, and dropping them would erase most of the trail. Verified against real data: 12 bookmarks reduce to 6 steps, 8 pages, 2.3%. Also fixes bookmark deletions never propagating - onConflictDoNothing meant a re-synced annotation kept whatever we first saw, so the deleted flag never flipped - and matches the delete-account card to its siblings, keeping the red outline on the button rather than the whole panel.
The percentage was pages advanced over the book's page count - the same number scaled - so it added a tile without adding information, and summing percentages across books of different lengths made a 461-page study guide and a 76-page paper count the same. Pages are the honest aggregate. Removes the percent field from ProgressStep rather than leaving it computed and unused. Behaviour is unchanged: percent derives from the page number, so the two were always zero or non-zero together. Verified: still 6 steps, 8 pages.
Mirrors the GitHub tile: longest run of consecutive days with a bookmark, within the selected period. streakDaysV1 could not be reused - it filters on github.commit.pushed - so the run-length half is extracted as longestDayStreak, which takes day keys and lets each source decide what an active day is. streakDaysV1 now delegates to it, so GitHub's numbers are unchanged; verified across gaps, month and February boundaries, unsorted input and duplicate days.
The card picked its metric from a hardcoded per-source chain that fell through to GitHub's active days for anything unlisted. Play Books therefore resolved to a metric outside its own view and the card hid itself. The METRICS entries already carry a days flag meaning exactly 'this is the active-days metric', so the card now derives from that instead. A new integration gets the card by setting the flag rather than by editing a chain someone has to remember exists. Verified against every source: all, github, calendar and baseline resolve exactly as before; google_books goes from hidden to reading_days.
Play Books tiles and its consistency ring showed '-' because the overview fetch listed github, google_calendar and baseline by hand while the series fetch beside it already derived its list from the metric definitions. The result was charts that loaded and tile values that did not, which reads as missing data rather than missing wiring. Both now come off the same grouping, so a new integration is wired by adding its metrics and a SOURCE_BASE entry. Audited every tile against its routes: all seventeen have an overview and a timeseries entry, and each source has exactly one days-flagged metric driving its consistency card.
The Dockerfile enumerates workspace packages one COPY at a time, in both the deps and builder stages, and google-books was in neither - so the image installed and built without it and next build failed resolving the import from lib/ingestion. Typecheck and lint pass locally because they run against the full workspace, which is why this only surfaced in CI. Also wires google_books into revokeIntegrations, which still only knew about google_calendar and github: disconnecting Play Books left Baseline in the user's Google authorized-apps list with the books scope intact. That revoke needs care. Every Google source shares one OAuth client, and consent is extended rather than duplicated (include_granted_scopes), so they share a single grant - revoking either would silently break the other. Disconnect now skips the revoke while another Google source is still connected, and clears our own tokens regardless.
Replaces the pill row with the SourceDropdown already used on the metrics page, so both filters look and behave the same. Labels now come from SOURCE_META instead of a second hardcoded list, which had drifted: the same source read 'Calendar' here and 'Google Calendar' there. That meant adding entries for 'manual', a real value in events.source that had no name, and teaching the dropdown about 'all' - it documented an 'All sources' option it never actually offered, because 'all' is a filter rather than a source and has no registry entry. Also shortens the Play Books label, which is a chip in both places.
Names the origin rather than the one event type it currently produces, matching how the metrics filter already labels Baseline's own data. The registry now holds two entries reading 'Baseline' - 'baseline' is the metrics source id, 'manual' is the events.source value - so the comment says why. Metrics filters on the former and history on the latter, and neither list contains both, so no dropdown can show it twice.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third integration, following the GitHub and Google Calendar pattern. Tracks reading progress from the bookmarks you leave in Play Books.
What the API will and won't give you
Reading position is not where the docs suggest. Established empirically against a real library — see
scripts/probe-google-books.mjs:/mylibrary/readingpositions/{id}userInfo.readingPositionuserInfo.updated/mylibrary/bookshelves/volumes/mybooks?acquireMethod=acquireMethodreturns 0, which reads as an empty account)/mylibrary/annotations?volumeId=So bookmarks are the signal. One tap when you stop reading; no bookmark, no data point.
Two things that shaped the design
Page prefixes decide whether the number means anything.
PA2525 / pageCountPR7PT64The prefix is not predictable from metadata: volumes with identical
readingModesproducedPAin one case andPTin another. So the parser handles whatever arrives rather than deciding in advance, and refuses to invent a percentage it can't justify.volumeIdis mandatory on the annotations endpoint — every call without it 400s. No library-wide incremental sync; we poll per volume. Fine at small library sizes.Deleted bookmarks are kept. Moving a bookmark forward deletes the old one, and those tombstones are what make a reading trail reconstructable rather than just the latest position.
Verified
tsc --noEmitclean across api, web, events and the new package. Normaliser checked against the real observed data:Setup
Enable the Books API on the existing Google Cloud project, add
https://www.googleapis.com/auth/booksto the consent screen, and register the new redirect URI/v1/integrations/google-books/callbackon the existing OAuth client. SameGOOGLE_CLIENT_ID/GOOGLE_CLIENT_SECRET;include_granted_scopeskeeps any existing Calendar grant intact.🤖 Generated with Claude Code