Skip to content

Import fails at section-tree build when a heading exceeds the 140-char title field #4

Description

@Rl0007

What happens

Importing Nephrology.pdf (398 pages, CMC Vellore) fails at the very last step of the
remediate pass. The import shows Failed at stage_label: "Building section tree",
stage_progress: 100, page_count: 0.

Live: IMP-2026-00001 on wikify.m.frappe.cloud — Helpdesk ticket
#0700.

File "apps/wikify/wikify/engine/store.py", line 342, in replace_sections
  doc.insert(ignore_permissions=True)
frappe.exceptions.CharacterLengthExceededError: Source Section 8t7tco15qn:
'Title' (5. The transplant coordinator also has the responsibility to maintain up to
date records and reports as required for the Renal Transplant license.)
will get truncated, as max characters allowed is 140

Why

Page 72 has a bold numbered clause. The cleanup remediation pass promoted it to a real
heading, so canonical markdown for that page starts with:

## 5. The transplant coordinator also has the responsibility to maintain up to date records and reports as required for the Renal Transplant license.

sectionize() believes it and opens a section titled with that whole 146-character
sentence. Source Section.title is a Data field — 140 chars — and insert() throws
rather than truncating. The existing _looks_like_list_item guard can't catch it: it
only fires on a stray * or double numbering, and only for level-1 headings.

Parsing itself is fine — the document is Parsed, 398 pages, mean score 0.961,
canonical mean 0.943, 357 pass / 33 review / 8 escalate. Only the tree write fails.
Because the rebuild rolls back, the site keeps the older parse-time tree (134 sections
built off baseline markdown), so the UI renders stale content rather than nothing.

Fix

  1. Length guard at the ORM seam. A store.fit() helper trims any content-derived
    string to what its column holds (mirroring _validate_length), applied to
    Source Section.title and Source Document.title. Content out of a PDF or an LLM is
    unbounded by definition; one runaway heading must not be able to fail a 400-page import.
  2. Stop the mis-detection. A _looks_like_prose() guard in the sectionizer: a heading
    over 120 chars that ends in terminal punctuation is body text, not a heading, and stays
    in the section it was found in. The embedded ToC still wins where it lists a title.
    Without this the guard above would just store a truncated junk section title.

Both plus regression tests (the exact page-72 line, a ToC-listed long title, and an
over-long title surviving a real insert).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions