Skip to content

Download paywalled PDFs with the browser sign-in - #449

Open
0thernet wants to merge 9 commits into
mainfrom
codex/pdf-browser-auth
Open

0thernet wants to merge 9 commits into
mainfrom
codex/pdf-browser-auth

Conversation

@0thernet

Copy link
Copy Markdown
Member

Summary

ghostget pdf <url> passed every link to Wordcell, which downloads anonymously, so papers behind a library or university sign-in failed even when Chrome was signed in.

With --cookie-source, --browser-profile, --cookie-profile, --auth, or --cookies-file, Ghostget now downloads the PDF itself:

  • HTTPS only, DNS-pinned public addresses (pinnedHttpsFetch), at most five redirects; credential-bearing, plain-HTTP, and private targets fail before any cookie is read.
  • Cookies come from the same reader and per-URL domain/path filter that read uses, looked up separately for each hop, so a site never receives another host's cookies.
  • Bounded by --max-pdf-bytes and --timeout-ms; the body must start with %PDF-.
  • A sign-in or landing page produces a plain-language no-access error that suggests opening the link in the browser first.
  • The PDF is written to an owner-only temporary file, passed to Wordcell's pdf import with the remaining options, then removed, including on failure.

Without these options the command is byte-for-byte the previous delegation. The help topic, golden fixture, Agent Skill, and SECURITY.md describe the new path. No browser profile is launched, so profile egress consent does not apply.

Validation

  • bun run typecheck, bun run check:effect, bun run check:cost-surfaces: pass
  • bun test src/pdf-auth.test.ts src/cli.test.ts src/usage.test.ts scripts/ci-pr-gate.test.ts: 72 pass, 0 fail, "Ran 72 tests across 4 files"
  • Live smoke with --cookie-source chrome (no cookie values printed): arXiv 1706.03762 imported end to end (15 pages); bioRxiv bytes downloaded and passed the signature check, then Wordcell's Poppler parse failed identically to the anonymous path ("Poppler XML contains an invalid text top"); PMC returned its HTML bot challenge and produced the new no-access message (the anonymous path reports only "does not have a valid PDF signature").

🤖 Generated with Claude Code

`ghostget pdf <url>` passed every link to Wordcell, which fetches
anonymously, so papers behind a university sign-in failed even when
Chrome was signed in. With --cookie-source, --browser-profile,
--cookie-profile, --auth or --cookies-file, Ghostget now downloads the
PDF over pinned HTTPS with the same cookie reader and per-URL filter
that `read` uses, checks the PDF signature, explains a sign-in page in
plain language, and hands Wordcell an owner-only temporary copy.
Without these options the command is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ghostget Ready Ready Preview Sep 29, 2026 1:18am UTC

Request Review

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Pack src/pdf-auth.ts and remeasure the reviewed package budget (619
  entries); the no-sign-in pdf path now decides in usage.ts and never
  imports pdf-auth.
- Refuse private IP literals and local names before reading that host's
  cookies; SECURITY.md states the enforced order.
- Configure the keychain notice for every run, so agents and non-TTY runs
  get the permission-notice line before a browser store is read.
- Treat a cookie file with no cookies for a hop as an empty set, so a DOI
  redirect reaches the publisher.
- Keep the paper's requested and final URLs as the note's source by
  handing Wordcell a remote source for the local copy.
- Drop --mode from the signed-in options; Wordcell rejects it as before.
- Plain messages for unknown accounts and cookie-file failures, sizes in
  MB, and exact-text tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Hold back SameSite=Strict cookies on redirect hops that leave the link's site.
- Count every cookie read against --timeout-ms.
- Stream the PDF into an owner-only file instead of buffering it twice.
- Report a missing sign-in (wrong browser or profile) separately from missing access.
- Resolve a browser data folder given to --browser-profile to its Default profile, and refuse folders with no sign-in.
- Tell users to switch an http:// link to https://.
- Accept --trust-profile-egress and --mode copied from read next to a sign-in option.
- Restore the ghostgetHelpRequest doc comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 681470fb Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant