doc-dl is a command-line downloader for documents that do not always behave like ordinary download links. Give it a public document or landing-page URL and it works through the sensible options: direct download first, then page discovery, browser downloads, and finally a verified PDF reconstruction when a viewer is the only available public representation.

It is built for the slightly annoying reality of modern document sites: pages that hide the file behind JavaScript, viewers that lazy-load pages, redirects, and downloads that fail halfway through. The goal is simple: get a usable, verified file or get a clear reason why that was not possible.

• INSTALLATION
  • Windows
  • macOS or Linux
  • Other installation methods
  • Uninstall
• USAGE
  • Try it with real sites
  • Which command should I use?
  • Everyday downloads
  • Download options
  • Archiving a page
  • Extracting a page range from a PDF
  • When a login is needed
  • Useful commands
• HOW IT WORKS
  • Managing the Chromium browser runtime
  • Provider support
• DEVELOPMENT
  • Building portable releases
  • Field notes: issues and fixes

What it can do

• Download PDFs, Office documents, EPUBs, OpenDocument files, RTF, CSV, and plain text when a site exposes a real document response.
• Find document URLs embedded in links, metadata, JSON-LD, and page source.
• Watch browser downloads and document responses on JavaScript-heavy sites.
• Reconstruct a complete, image-backed PDF from visible viewer pages when no original file is available.
• Resume interrupted HTTP downloads when the server provides safe validators.
• Verify files before saving them to their final name.
• Keep sign-in cookies in an isolated browser profile, never in command-line arguments.

Quickstart

The portable release includes Python and every required library, so you do not need to install Python yourself. There are two kinds of release:

• Slim (recommended): small download. The first time you open a site that needs a browser to extract its document, doc-dl automatically downloads Chromium once and reuses it after that. Direct file downloads never need Chromium at all.
• Full: a larger, offline-ready download with Chromium already included, for machines that are offline or on a locked-down network where an on-demand download would not work.

If you are not sure which one you need, use slim. The commands below install the slim release by default.

Windows

Open PowerShell and run:

irm https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/install.ps1 | iex

The installer downloads the latest slim Windows release, verifies its SHA-256 checksum, installs it under your local application-data folder, and adds doc-dl to your user PATH. To install the full offline build instead, set DOC_DL_VARIANT first:

$env:DOC_DL_VARIANT = "full"
irm https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/install.ps1 | iex

macOS or Linux

curl -fsSL https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/install.sh | sh

The installer selects the right slim release for Linux x64, Intel macOS, or Apple Silicon, verifies its checksum, and links doc-dl into ~/.local/bin. For the full offline build instead:

curl -fsSL https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/install.sh | DOC_DL_VARIANT=full sh

Uninstall

Remove the installed program and its PATH entry:

irm https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/uninstall.ps1 | iex

On macOS or Linux:

curl -fsSL https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/uninstall.sh | sh

Both commands preserve isolated sign-in profiles, any Chromium runtime doc-dl downloaded on demand, and other runtime state. To remove that state too (sign-in sessions and any downloaded Chromium runtime together), use one of these explicit commands:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/uninstall.ps1))) -PurgeData

curl -fsSL https://raw.githubusercontent.com/mkhlz/doc-dl/master/scripts/uninstall.sh | sh -s -- --purge-data

To remove only the downloaded Chromium runtime and keep your sign-in profiles, use doc-dl uninstall-browser instead; see Managing the Chromium browser runtime.

Download a document

Open a new terminal after installation and give doc-dl a URL:

doc-dl "https://www.scribd.com/document/1039114955/GMAT-Syllabus-PDF"

PowerShell users can also use the winget-style -Url form:

doc-dl -Url "https://www.scribd.com/document/1039114955/GMAT-Syllabus-PDF"

That is the whole everyday workflow. Run doc-dl doctor if you ever want to check the installation.

Try it with real sites

Try these in PowerShell:

• W3C direct PDF, quick smoke test (a plain file download, no browser needed):

  doc-dl "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"

• Internet Archive public-domain PDF:

  doc-dl "https://ia801907.us.archive.org/17/items/librivoxcdcoverart36/bestfriend_2006.pdf"

• arXiv research paper PDF:

  doc-dl "https://arxiv.org/pdf/2212.07286"

• Scribd multi-page viewer test: this is the case that needs Chromium, so it is also a good way to see the one-time browser download in action on a fresh install:

  doc-dl "https://www.scribd.com/document/959028055/Ten-Page-Sample" --verbose

  Scribd downloads depend on the access available through your normal Scribd account; the example above is a public Scribd sample that does not require signing in.

• SlideShare presentation: slide images are read straight from the page, so this never needs Chromium at all:

  doc-dl "https://www.slideshare.net/RachelElliottRigolino/sample-oral-report-powerpoint"

Use this pattern to keep files in your Downloads folder instead of the current directory:

doc-dl "PASTE-URL-HERE" --output "$HOME\Downloads" --verbose

Other installation methods

Download a release archive manually

Open the GitHub Releases page and download the archive for your computer. Extract the doc-dl folder somewhere permanent and add that folder to PATH.

Slim (recommended): Python and every required library, but no Chromium. The first browser-backed download triggers a one-time Chromium download; plain file downloads never need it.

Release file	System
doc-dl_win.zip	Windows 10 or newer, Intel or AMD 64-bit
doc-dl_linux.tar.gz	64-bit Linux
doc-dl_macos_x64.tar.gz	Intel Mac
doc-dl_macos_arm64.tar.gz	Apple Silicon Mac

Full: everything in slim, plus Chromium already included, for offline use or locked-down networks. Noticeably larger.

Release file	System
doc-dl_win_full.zip	Windows 10 or newer, Intel or AMD 64-bit
doc-dl_linux_full.tar.gz	64-bit Linux
doc-dl_macos_x64_full.tar.gz	Intel Mac
doc-dl_macos_arm64_full.tar.gz	Apple Silicon Mac

SHA2-256SUMS in the same release lists SHA-256 checksums for every archive above, slim and full alike. During a workflow run, GitHub also displays temporary build-bin-* artifacts that transfer these archives between jobs; the final GitHub Release shows the public filenames in the tables above.

Install from a Python wheel

Python users can keep using the smaller wheel:

python -m pip install .\dist\doc_dl-0.1.9-py3-none-any.whl
doc-dl doctor

This method requires Python 3.11 or newer. Chromium is not installed yet; doc-dl downloads it automatically the first time a browser-backed site needs it, or you can install it ahead of time with doc-dl install-browser.

Install for development

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
.\.venv\Scripts\doc-dl.exe doctor

On macOS and Linux, use .venv/bin/python and .venv/bin/doc-dl instead. To run the full test suite, including the browser-backed tests, also install Chromium once:

.\.venv\Scripts\python.exe -m playwright install --no-shell chromium

If the virtual environment is not activated, call its executable directly:

.\.venv\Scripts\doc-dl.exe -Url "https://www.scribd.com/document/1039114955/GMAT-Syllabus-PDF"

Which command should I use?

doc-dl has two commands with different jobs: download (the default, so just doc-dl URL) looks for an actual document file; archive snapshots a page as it renders, for pages that were never a document in the first place.

	doc-dl URL (download)	doc-dl archive URL
What it's for	A PDF, Office file, EPUB, or other document behind the link	A news article, blog post, or any web page you want a record of
What you get	The original file when one exists; a rebuilt PDF from a viewer's pages otherwise	One PDF made from the page's own rendered layout
Result quality	Byte-identical to the source when original; otherwise a faithful reconstruction	A screenshot of the page -- readable, but not the site's own typography
If nothing downloadable is found	Falls back to the same page-capture archive does, unless --original-only is set	N/A -- that's the whole point of the command
History log entry	Opt-in (--write-metadata)	On by default (--no-metadata to skip) -- capture provenance is the point
Distinguishing options	--original-only, --no-browser, --retries, --no-resume	--max-pages, --select-range, --wait-for
Shared options	--output, --filename, --profile, --timeout, --overwrite, --quiet, --verbose, --json, --batch, --concurrency

Rule of thumb: if you'd expect a "Download" or "Export as PDF" button somewhere on the page, use download. If you're saving a page the way a browser print-to-PDF would, use archive.

doc-dl "https://example.com/report.pdf"                 # a real document
doc-dl archive "https://example.com/news/some-article"   # a page, not a document

Not sure? Let it choose for you

--choose reads the link, tells you what it looks like and why, then offers only the routes that make sense for it, with the likeliest one marked:

doc-dl "https://www.slideshare.net/slideshow/some-deck/123456" --choose

  slideshare.net
  This looks like a document on slideshare.net, which doc-dl has a dedicated provider for.
  The slideshare provider fetches the document itself instead of screenshotting a
  browser, so it is both higher quality and less likely to trip a bot check.

  What would you like to do?
  1. Download the document  (recommended)
     doc-dl URL
  2. Download, but only the publisher's own file (never a rebuild)
     doc-dl URL --original-only
  3. Snapshot the page as a PDF
     doc-dl archive URL
  0. Cancel
  Choice:

Each option shows the plain command it stands for, so it doubles as a way to learn the flags rather than a thing you have to keep reaching for. The read is based on the link alone — no network call, so the menu is instant — which makes it a strong hint rather than a certainty. Every route stays on the list whatever the link looks like; a wrong guess costs a glance, not a second command.

--choose needs a real terminal and handles one URL, so it can't be combined with --json, --quiet, or --batch.

Rule of thumb by site

The short version: if the site is hosting a file, use download. If the site is the content, use archive. A few concrete cases:

Site / kind of link	Use	Why
SlideShare, Scribd	download	Both have a dedicated provider that pulls the real page images or file directly — no browser screenshot involved, so nothing to fool a bot check with, and the result is the actual document, not a picture of it.
Google Drive, Dropbox, OneDrive share links	download	Same idea — these resolve straight to the file's real download endpoint.
A direct link ending in .pdf, .docx, .pptx, etc.	download	It's already a file. archive would just screenshot whatever viewer the browser shows for it.
News articles, blog posts	archive	There's no file to find — the page itself is what you want a record of.
A site behind Cloudflare/Fastly-style bot checks (a CAPTCHA, "checking your browser")	download if it's a known document host (see above); otherwise expect archive to refuse with a clear message rather than save a screenshot of the challenge	Loading a full browser to screenshot a page is exactly the kind of behavior these checks are built to catch. A direct-file provider mostly avoids the browser step entirely.
A long news homepage or "feed" page that bunches several stories together	archive --select-range 1-3 or --max-pages N	There's no single "document" here either, and letting it capture the whole scroll length would sweep in unrelated stories below the one you wanted.
A page that's slow to load (heavy ad-laden news sites)	archive --wait-for "selector"	Gives the page a specific thing to wait for instead of guessing.

If you're not sure which applies, download is the safer first try — it falls back to the same page-capture archive does when nothing downloadable turns up (unless --original-only is set), so you're unlikely to walk away with nothing either way.

Why a "stuck" spinner might actually mean it already finished: if archive refuses a page (a bot challenge, for example), the spinner and the error can appear to overlap in some terminals — the spinner writes to standard output and the error to standard error, and depending on how the terminal buffers the two, the error can print before the spinner's own "stop" frame clears. It looks like:

⠸ Resolving link  ▲ This page is showing a bot-detection challenge instead of its content

That is cosmetic, not a hang — the command has already exited by the time that line appears. If it's sitting there for more than a few seconds with no error at all, that is worth interrupting and re-checking with --verbose.

Everyday downloads

Save into a downloads folder and record redacted provenance metadata:

doc-dl URL --output .\downloads --write-metadata --verbose

Choose one exact output path:

doc-dl URL --output .\downloads\my-document.pdf

Use a filename pattern. Available fields are {title}, {ext}, {provider} and {filename}:

doc-dl URL --output .\downloads --filename "{provider}-{title}.{ext}"

Process a list of URLs at once, several at a time:

doc-dl --batch urls.txt --concurrency 4 --output .\downloads

urls.txt is one URL per line (blank lines and # comments are skipped). Each URL goes through the exact same pipeline as a single doc-dl URL run — same retries, same reconstruction — just several running concurrently. One line prints per result as it completes, in whichever order that happens to be, followed by a succeeded / failed count; a failed URL doesn't stop the rest of the batch.

Download options

Option	What it does
--output, -o	Output directory or exact file path.
--filename	Filename template for downloads saved to a directory.
--original-only	Accept only original document bytes. Never use a rendered PDF.
--no-browser	Skip browser discovery and viewer reconstruction.
--profile NAME	Use an isolated signed-in browser profile.
--timeout 5m	Set the full operation timeout. Supports seconds, minutes, and hours.
--retries 5	Retry transient transfer failures.
--no-resume	Disable protected resume for interrupted HTTP transfers.
--overwrite	Replace an existing output file.
--write-metadata	Record a redacted provenance entry in the history log.
--choose	Inspect the link and pick a route interactively instead of guessing the flags.
--batch FILE	Process every URL in this file instead of one URL.
--concurrency N	How many --batch URLs to run at once (default 3, max 10).
--verbose	Show which strategies are being tried.
--json	Emit newline-delimited JSON events for automation.
--quiet	Print only errors and the final output path.

Archiving a page (news articles, blog posts, and other web pages)

doc-dl archive snapshots a page as a PDF instead of looking for a downloadable file. It expands any "Read more" / "Continue reading" toggle and dismisses cookie banners first, then takes one continuous full-page screenshot — the same idea as a scrolling-capture tool — and slices that into normal PDF pages in Python, rather than scrolling and re-screenshotting step by step (which could crop or duplicate content at page boundaries if anything shifted between shots). No attempt is made to isolate or re-style "the article part" of the page, since that varies too much from site to site to do reliably:

doc-dl archive "https://example.com/news/some-article"

A provenance entry is recorded by default (opt out with --no-metadata) — title, byline, publish date, site name, and whether paywall indicators were detected — but never as a file next to the PDF. Pasting a link and getting a document back should mean exactly that: the download folder holds just the file. Everything else goes into a single history log instead (run doc-dl doctor to see where):

{
  "title": "Harbor City Approves New Transit Line",
  "byline": "Jordan Rivera",
  "published": "2026-08-01T09:00:00Z",
  "site_name": "The Harbor Gazette",
  "provenance": "captured",
  "paywall_suspected": false
}

A page flagged paywall_suspected: true still captures whatever is visible (usually just the free teaser above the paywall) — the flag is a signal to check the result, not a guarantee the capture is incomplete.

The default filename is the title, site name, and capture date — Harbor City Approves New Transit Line - The Harbor Gazette - 2026-08-01.pdf — so a folder of archived pages sorts sensibly and generic titles from different sites ("Live Updates", "Home") don't collide. A --filename template still gets the plain article title as {title}.

A long feed-style page can bunch the linked article together with unrelated ones below it (a news homepage, or a listing that keeps loading more stories as you scroll). --max-pages stops the capture after a set number of pages; --select-range keeps a specific range instead of just the first N:

doc-dl archive "https://example.com/news/some-article" --max-pages 3
doc-dl archive "https://example.com/news/some-article" --select-range 2-4

Some sites need more time than the built-in settle heuristics guess — --wait-for waits for a CSS selector to appear before capturing starts:

doc-dl archive "https://example.com/slow-page" --wait-for "article"

Archiving supports the same --batch / --concurrency options as download, for saving a list of articles at once.

Option	What it does
--profile NAME	Use an isolated browser profile (useful for sites you're signed into).
--timeout 90s	Set the full operation timeout.
--max-pages N	Stop after N pages instead of capturing the whole scroll length.
--select-range 2-4	Keep only this page range (1-indexed, inclusive); mutually exclusive with --max-pages.
--wait-for SELECTOR	Wait for this CSS selector to become visible before capturing.
--batch FILE / --concurrency N	Same as download.
--no-metadata	Skip recording a history entry.
--overwrite	Replace an existing output file.

Extracting a page range from a PDF

doc-dl extract-pages copies a page range out of a local PDF into a new file — grabbing one chapter out of a long book without re-uploading the whole thing:

doc-dl extract-pages book.pdf --pages 37-75
doc-dl extract-pages book.pdf --pages 37-75 --output chapter-2.pdf

The default output filename is <source>-pages-<start>-<end>.pdf next to the source file.

When a login is needed

For documents your normal provider account can access, create an isolated profile and finish sign-in in the browser window:

doc-dl login scribd --profile personal
doc-dl URL --profile personal

Remove that profile when you are finished:

doc-dl logout scribd --profile personal --yes

How results are handled

doc-dl prefers an original file whenever one is available. If a site offers only a visible viewer, it can create a reconstructed PDF from the pages the viewer actually displays. Successful results identify their source as original, exported, reconstructed, or captured in the metadata.

The tool rejects empty files, HTML masquerading as a document, corrupt PDFs, invalid Office containers, and visibly blank reconstructed pages. A partial or unverified result is not promoted to the final output path.

If nothing downloadable turns up at all — a news article or blog post with no attached file, for example — download falls back to snapshotting the page itself, the same way doc-dl archive would, rather than failing outright. This is reported as captured provenance so it's never confused with the real document. Pass --original-only to disable this and get a hard failure instead.

Useful commands

doc-dl version
doc-dl providers
doc-dl doctor
doc-dl archive URL
doc-dl extract-pages book.pdf --pages 37-75
doc-dl install-browser
doc-dl uninstall-browser

doctor checks Python, the required libraries, whether Chromium is installed, and the writable application-state location. It reports Chromium's status but does not fail because of it: a missing Chromium is expected for a fresh slim install and is not a problem until a browser-backed site actually needs it.

Managing the Chromium browser runtime

You never have to think about Chromium for direct file downloads; those never touch a browser. The first time a browser-backed site does need one, doc-dl downloads Chromium automatically, shows progress, and reuses it for every download after that:

doc-dl "https://example.com/some-javascript-viewer"
# Chromium is not installed. Downloading the browser runtime now
# (one-time download into C:\Users\you\AppData\Local\doc-dl\browsers)...

To install it ahead of time instead of waiting for the first browser-backed download:

doc-dl install-browser

To remove the downloaded browser runtime later and reclaim the disk space, without touching your sign-in profiles:

doc-dl uninstall-browser

Chromium is stored in a stable per-user location (doc-dl doctor reports the exact path), never beside the installed program and never inside a release archive. --no-browser still skips browser escalation entirely, so it also skips this download. A full offline release already has Chromium bundled, so install-browser and uninstall-browser report it as already installed; uninstall-browser refuses to remove a bundled offline copy, since that would defeat the purpose of choosing the full build.

Development checks

.\.venv\Scripts\ruff.exe check src tests
.\.venv\Scripts\python.exe -m pytest
.\.venv\Scripts\python.exe -m pip wheel --no-deps . --wheel-dir dist

docs/ISSUES.md records real failures hit against live sites, their root causes, and how each was fixed. Worth reading before debugging a similar report: several of those symptoms turned out to be something other than what the error message said.

Building portable releases

The release workflow lives in .github/workflows/release.yml and follows the same build, collect, tag, and publish sequence for every entry point:

• Run Release manually with the version field empty to build and verify all packages without creating a tag or GitHub Release.
• Run Release manually with a bare version such as 0.1.3 to verify that it matches the package, build the packages, push the tag, and publish the release.
• Push a bare version tag such as 0.1.3 to run the same build and publish path.

Before tagging, update the version in pyproject.toml and src/doc_dl/__init__.py, run the development checks, and commit the release changes. The tag should be the exact version without a v prefix.

For a local slim build, install the build extras and run the builder for the current operating system. --variant slim is the default, so it can be omitted:

python -m pip install ".[build]"
python scripts\build_portable.py --target windows-x64 --variant slim

For a local full build, also place Chromium in a dedicated directory first:

python -m pip install ".[build]"
$env:PLAYWRIGHT_BROWSERS_PATH = "$PWD\build\playwright-browsers"
python -m playwright install --no-shell chromium
python scripts\build_portable.py --target windows-x64 --variant full

The resulting archive is written to release-assets/. The builder runs both doc-dl version and doc-dl doctor against the frozen executable before it creates the archive; doc-dl doctor passing without Chromium is expected and required for a slim build. The GitHub Actions release workflow builds both variants for every platform in the same run.

Provider support

• generic: direct files, redirects, page-source discovery, JavaScript downloads, document responses, and compatible viewer reconstruction.
• scribd: normalized document URLs, isolated login profiles, embedded viewer activation, lazy page loading, and complete-page PDF reconstruction.
• slideshare: presentation slide URLs parsed directly from the page, no browser required; every slide is fetched and merged into one PDF.
• googledrive: shared file links resolved to Drive's own download endpoint, preserving the resource key that older shared links require.
• dropbox: share links resolved to the file itself rather than the preview page, keeping the access key newer links carry.
• onedrive: personal share links resolved through OneDrive's public shares endpoint. Business and SharePoint links are not rewritten.

Websites can change their layouts or restrict access. In those cases, doc-dl returns a stable error and leaves no unverified final file behind.

License

This project is free for personal, educational, research, and other non-commercial use.

Commercial use requires prior written permission from the copyright holder.

See the LICENSE file for full terms.