Skip to content

fix(pdf): bundle PDF.js CMap/font assets so CJK documents render and index - #131

Merged
mrsibe merged 1 commit into
mainfrom
fix/pdfjs-cmap-assets
Sep 25, 2026
Merged

mrsibe merged 1 commit into
mainfrom
fix/pdfjs-cmap-assets

Conversation

@mrsibe

@mrsibe mrsibe commented Sep 25, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Makes CJK (Chinese / Japanese / Korean) PDFs work end to end by wiring up PDF.js's three external asset roots — predefined CMaps, standard fonts and the wasm image decoders — for both consumers:

  • main process (PdfLoader) reads them as filesystem paths, so getTextContent() returns the CJK text instead of dropping it;
  • renderer (PdfSourceReader) reads them through a new knownote-asset:// protocol, so the canvas paints CJK and the text layer is selectable.

Ships pdfjs-dist/{cmaps,standard_fonts,wasm} with the app as real files under resources/pdfjs/ (electron-builder extraResources, outside app.asar).

Builds on #129 (merged), which put the reader on the pdfjs legacy build. A correct renderer still needs these assets.

Why?

The v1.4 QA (#125) found CJK PDFs broken in a way that a getDocument({ data }) call hides:

Warning: loadFont - translateFont failed:
"UnknownErrorException: Ensure that the `cMapUrl` API parameter is provided."

PDF.js treats CMaps / standard fonts / wasm as external assets fetched by filename (${cMapUrl}${name}.bcmap). Nothing passed any of the three, and the failure is a warning, not an exception — so the reader painted blank and the loader indexed the document without its CJK text. Both are release blockers: the second is a plausible part of why notebook chat could not retrieve CJK sources.

Reproduced against main with noembed-jis7.pdf (non-embedded Japanese font):

extracted text CJK present
no cMapUrl "ABC 1231" no
cMapUrl + cMapPacked "ABCあいうえお123 1" yes

Fixes #130. Related to #125, #129, #82.

What changed?

  • electron-builder.yml: extraResources copies cmaps, standard_fonts, wasm to resources/pdfjs/. Real files, outside the asar — the main process reads them with fs, and Chromium cannot read inside app.asar.
  • src/shared/utils/pdfjsAssets.ts: the shared contract (directory allowlist, protocol URL shape with the trailing slash PDF.js requires, and a parser that rejects traversal).
  • src/main/protocol/pdfjsAssetProtocol.ts: knownote-asset://, mirroring knownote-doc://. The renderer names a directory; the main process resolves the file from an allowlisted root and serves it with CORS + application/wasm.
  • src/main/pdfjsAssetPaths.ts: resolves the asset root (resources/pdfjs packaged, node_modules/pdfjs-dist in dev) and the three Node-side directory paths.
  • PdfLoader.ts / PdfSourceReader.tsx: pass cMapUrl, cMapPacked: true, standardFontDataUrl, wasmUrl.
  • index.html: connect-src gains knownote-asset:; script-src gains 'wasm-unsafe-eval'. 'unsafe-eval' is still absent, so the security(pdf): remediate GHSA-hq66-cqwq-w95j before v1.4 (pdfjs-dist 5.7.284 in affected range) #121 scripting boundary is unchanged.
  • smokeTest.ts: packaged assertion that the three asset directories shipped and are non-empty.
  • test/pdfjsAssets.test.ts: URL shape, traversal rejection, both call sites, and the CSP directives.

How was this tested?

  • Real protocol handler + real pdfjs in headless Electron (39.8.10), against noembed-jis7.pdf, with the repo's actual registerPdfjsAssetScheme / registerPdfjsAssetProtocolHandler: text is "ABC あいうえお 1231" and page.render() resolves. CMaps fetched over the protocol: H.bcmap, Adobe-Japan1-UCS2.bcmap.
  • Real PdfLoader in Node on the same fixture returns the Japanese text; without cMapUrl it returns "ABC 1231" and emits the warning above.
  • npm test — 276 pass, 0 fail (5 new). The call-site guard was falsified by reverting cMapUrl in the reader, which fails it as intended.
  • npm run typecheck, npm run build — pass.
  • npm run build:unpack — ships cmaps 169, standard_fonts 16, wasm 13 under resources/pdfjs/.
  • smoke:packaged — 21/21 checks pass, including the new packaged-assets check (run headless; CI uses xvfb-run).

Not verified: a CJK document end to end in the packaged window (canvas paint, text-layer selection). That needs a real CJK fixture and a display, and is manual QA for #125.

Screenshots / recordings

Not applicable (no bundled CJK fixture to screenshot), see "How was this tested?".

Checklist

  • I have reviewed my own changes.
  • npm run typecheck passes.
  • npm run build passes.
  • I have tested the affected user workflow.
  • I have not included unrelated changes.
  • I have updated documentation when necessary.

Desktop / build changes

  • npm run build:unpack passes.
  • npm run smoke:packaged passes.

…index

PDF.js fetches predefined CMaps, standard fonts and the wasm image decoders as
external assets by filename (`${cMapUrl}${name}.bcmap`), and neither getDocument
call site passed any of the three. Nor did it fail loudly - PDF.js logs

    Warning: loadFont - translateFont failed:
    "UnknownErrorException: Ensure that the `cMapUrl` API parameter is provided."

and then extracts the document without its CJK runs. So a Chinese/Japanese PDF
rendered blank *and* indexed as if the CJK text did not exist, which is why chat
could not retrieve it (#125, #130).

- Ship pdfjs-dist/{cmaps,standard_fonts,wasm} as real files under
  resources/pdfjs/ via electron-builder extraResources. They stay outside
  app.asar deliberately: the main process reads them with fs, and Chromium
  cannot read inside the archive.
- Serve them to the renderer through a new `knownote-asset://` protocol that
  mirrors `knownote-doc://` - the renderer names a directory, the main process
  resolves the file from an allowlisted root. No filesystem path reaches the
  renderer.
- Pass cMapUrl + cMapPacked + standardFontDataUrl + wasmUrl in PdfLoader (fs
  paths) and PdfSourceReader (protocol URLs). PDF.js requires a trailing slash
  on each; `pdfjsAssetUrl()` owns that shape.
- CSP: allow `knownote-asset:` in connect-src, and 'wasm-unsafe-eval' in
  script-src so the wasm decoders can compile. 'unsafe-eval' is still not
  allowed, so the #121 scripting boundary is unchanged.

Verification (Linux, Electron 39.8.10):
- noembed-jis7.pdf through the real protocol handler in a headless Electron
  renderer: without cMapUrl the text is "ABC 1231"; with it the text is
  "ABC あいうえお 1231" and page.render() resolves. CMaps fetched over the
  protocol: H.bcmap, Adobe-Japan1-UCS2.bcmap.
- The same fixture through the real PdfLoader in Node returns the Japanese text.
- npm test 276 pass (5 new: URL shape, traversal rejection, both call sites,
  CSP), npm run typecheck, npm run build.
- npm run build:unpack ships cmaps 169 / standard_fonts 16 / wasm 13, and
  smoke:packaged passes 21/21 including the new packaged-assets check.

Not verified: a CJK document end to end in the packaged window (canvas paint,
text-layer selection). That needs a real CJK fixture and a display.
@mrsibe
mrsibe force-pushed the fix/pdfjs-cmap-assets branch from 97bff1c to b5286bb Compare September 25, 2026 17:22
@github-actions github-actions Bot added the bug Something isn't working label Sep 25, 2026
@mrsibe
mrsibe merged commit d42138a into main Sep 25, 2026
3 checks passed
@mrsibe
mrsibe deleted the fix/pdfjs-cmap-assets branch September 25, 2026 17:23
mrsibe added a commit that referenced this pull request Sep 25, 2026
`protocol.registerSchemesAsPrivileged()` replaces the whole privilege table
instead of appending to it. `registerDocumentScheme()` and
`registerPdfjsAssetScheme()` each called it once, so the second call silently
dropped `knownote-doc`'s `supportFetchAPI` privilege. The renderer's
`fetch('knownote-doc://docs/<id>')` then failed with

    Fetch API cannot load knownote-doc://docs/...: URL scheme "knownote-doc" is not supported

and every PDF failed to open. Introduced by #131.

Protocol modules now only declare their scheme
(`declareDocumentScheme()` / `declarePdfjsAssetScheme()`); the single call to
Electron lives in `privilegedSchemes.ts::applyPrivilegedSchemes()`, invoked once
from `index.ts` before the app is ready.

`smokeTest` gains a renderer-side fetch of `knownote-doc://`. The existing check
used the main process's `net.fetch`, which works even without the privilege -
that is why the packaged gate stayed green while the reader was broken. The
renderer check creates a hidden window and deliberately does not destroy it:
destroying the last window fires `window-all-closed`, and this app closes the
database on that event, which would abort the remaining checks.

Verification (Linux, Electron 39.8.10):
- headless Electron running the repo's real protocol modules: both
  `knownote-doc://` (15306 bytes) and `knownote-asset://` (553 bytes) fetch from a
  renderer. With two separate `registerSchemesAsPrivileged()` calls the same
  harness returns "ERR Failed to fetch" for `knownote-doc://` - the regression,
  reproduced.
- `npm test` 283 pass (2 new: one submission point, all declarations applied).
- `npm run typecheck`, `npm run build`.
- `npm run build:unpack` + `smoke:packaged` 22/22, including the new renderer
  check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] CJK PDFs render blank and index empty: PDF.js CMap/font assets are not bundled

1 participant