Improve note citations - #6
Open
luisherrmann wants to merge 110 commits into
Open
Conversation
… id for more robustness.
… parallel requests to api endpoint.
…ath and citations.
A table continued on the next page arrived as two separate entries. Tables now merge across a page break the way listings already do, keeping the first or last fragment's caption. The shared grouping logic moves into match-captions.js so both extractors use one implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… merge. Extends the page-split table merge: a continuation whose caption carries an explicit "continued" keyword is merged into the table it continues. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the "continued" keyword check with a comparison of caption label
numbers, so two fragments whose captions start with the same label ("Tab. 1")
merge regardless of what sits between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The width cap assumed a two-column layout and rejected legitimate regions in single-column papers. Region plausibility is now decided without it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ole. A block was assigned to at most one region, so a listing spanning a column break lost half its text. A block may now be claimed by several regions, admitted only when it covers ground they do not already cover (see BLOCK_NEW_COVERAGE_RATIO), and regions sharing a claimed block are merged into one element. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Admits a contested block into a region on less new coverage than before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The lower threshold let a block enter a neighbouring region as well as its own, duplicating that text across two entries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The area floor was chosen for figures that are a single large image, and discarded the individual panels of a tiled figure -- which are much smaller -- before the grouper could assemble them into one figure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… apart. pairWithCaptions gains an absorbMargin option. Figures pass a larger value than the default, since a multi-panel figure prints a label line under every row and its panels sit further apart than a listing's fragments do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The aspect cap was added for a single wide image taken to be a decorative divider, which is in fact a display equation rendered as a raster. It also discarded rows of side-by-side panels, which are inherently wide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Figures sharing a block index came out in detection order rather than document order. Ties now break on page, then vertical position, then horizontal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… model. Answers "list all figures" and similar by reading the extraction caches directly and emitting a numbered list of ref: links, rather than leaving it to the model and whatever subset of the paper fit its context. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…labels. The pattern moves into match-captions.js and now accepts roman numerals and single letters, guarded by single-case, canonical-numeral and I/V/X/L-only tests so it cannot match an ordinary word. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
list_elements read the extraction caches through the raw accessors, but cross-index deduplication runs in the pipeline and is also what rewrites each disk cache -- so on a cold cache the tool reported the phantom entries deduplication exists to remove. Figures stay on the raw accessor, since they do not take part in deduplication. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds LLMIndexPipeline.ensureIndexed, wrapping buildIndexes plus the figure and reference indexes. ui/index-all.js, _handleNormalChat and the tool dispatch all call it, so a paper indexed in the background, mid-chat or by a tool goes through the same path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
figures.js wrote its disk cache and embedding rows eagerly, so it could not take part in cross-index deduplication, which can only delete an entry before it has been persisted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…wn text. Text found inside a figure's bounds is that image's own content -- UI labels, diagram annotations, chart axis text -- not a listing. Such regions are now removed in a pass after the caption contest, where the figure always wins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…miss. The pairwise tests ask only how much of a region sits inside one element, so a region straddling two of them belonged to neither and survived. A third pass removes a region whose own area is accounted for by the union of the surviving elements around it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The geometric passes condemned an entire preformatted_id when any one of its fragments overlapped, which deleted unrelated fragments of a listing split across pages. They now mark individual fragments. The caption contest stays group-level, where captioned-ness genuinely is inherited across fragments. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…misses. A region routinely absorbs the descriptive line printed above its body, so the separator between two listings ends up inside the later region rather than between the two, where the intervening-prose test cannot see it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
startsWithProse accepted only paragraph blocks, but SDT types the same kind of separator line as a caption in places, which left those listings merged into one entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…late. The candidate loop never declined a match, so once the nearest pairing was made each remaining body took whatever caption was still free, at any distance -- and since a match unions the caption's rect into the body's, a distant caption inflated the region and swallowed its neighbour. A pair now survives only if each side is the other's nearest on that page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SDT sometimes types a code comment as a paragraph, and startsWithProse read it as the start of a new listing, severing one listing in two. Lines opening with #, // or /* are excluded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each extractor's cache fingerprint covered only its own script, never scripts/match-captions.js, which all three import and where most of the churn lands -- so a change confined to that file left every cached index serving its pre-change contents. Dependency tracking stays shallow: that file's own imports are not stat'd. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…belled. A paper may label its listings by name rather than by number, which the enumerated-label pattern cannot match, leaving those labels invisible to the caption machinery. Such a line is now split off the region it opens and paired as a caption through the normal path, so the body begins at the code block and captionBlockIndex < blockIndex holds -- which is what lets the captions-precede-listings tally read the arrangement correctly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_captionLabel recognised only an enumerated caption, so a paper that titles its listings rather than numbering them fell through to "Preformatted N" even once its captions were extracted correctly. It now also reads the text before the colon as named label, leaving the description in `caption`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The line it inspected is now recognised as the region's own caption, so the caption test covers the case directly and the heuristic is redundant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…guously. A listing the paper never labelled was called "Preformatted N", which says nothing about where it is -- two prompt templates in different appendix subsections were indistinguishable except by number. It now carries its nearest preceding section, the way extract-figures-sdt.js and extract-tables-sdt.js already name an uncaptioned figure or table. The counter stays GLOBAL rather than restarting per section, which is where this departs from those two. A per-section counter makes the number ambiguous on its own -- OSWorld's unlabelled figures produce "Unlabelled Figure 1" twice, in different sections, so only the full label distinguishes them, and the full label is exactly what is lost when it is abbreviated in conversation or truncated for display, since the section prefix comes first and the number is what gets cut. A global counter keeps the trailing number unique paper-wide while the section still supplies context. The section is resolved in the extractor, since the outline lives on the structure and the plugin no longer has it when labelling a cached index, and carried through as a sectionTitle field. The group's first fragment decides it: a listing spanning a page break belongs to the section it starts in. A listing with no outline entry before it keeps the bare name. Labelling runs on the view rather than the raw index, so these names reach neither the caches nor deduplication -- unlike the figure and table equivalents, which copy the synthetic label into `caption` and need _SYNTHETIC_LABEL_RE to tell it apart from a real one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #1, #2, #3.