Skip to content

fix(arxiv): dedupe adjacent LaTeX metadata fragments - #119

Open
Lxr-max wants to merge 1 commit into
aiming-lab:mainfrom
Lxr-max:cursor/arxiv-dedupe-latex-a950
Open

Lxr-max wants to merge 1 commit into
aiming-lab:mainfrom
Lxr-max:cursor/arxiv-dedupe-latex-a950

Conversation

@Lxr-max

@Lxr-max Lxr-max commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Some arXiv mirror titles were scraped with mixed text/MathJax extraction, so the same math fragment was stored twice (\gtrsim 100\times\gtrsim 100\times, z=1.37z=1.37, \mathbb{P}^1\mathbb{P}^1, and similar).

This change adds a conservative cleanup helper that only collapses clear adjacent duplicates. It does not rewrite unrelated titles, does not convert TeX to Unicode, and does not touch display templates.

  • New sites/arxiv/metadata_cleaning.py shared by seed-time import and startup backfill
  • seed_database() cleans title/abstract/comments/journal-ref while importing papers.json
  • normalize_paper_metadata() backfills already-packaged DB rows on startup (no-op commit when nothing changed)
  • Regression tests for the issue fixtures live in sites/arxiv/tests/

Example (/abs/2604.07983):

A Natural \gtrsim 100\times\gtrsim 100\times Telescope ... at z=1.37z=1.37

becomes

A Natural $\gtrsim 100\times$ Telescope ... at $z=1.37$

Fixes #16.

Checklist

  • Cleanup is conservative: only adjacent duplicate math/LaTeX fragments
  • Shared helper used at seed time and startup backfill
  • Unrelated prose titles are left unchanged (K_{11,11}, “Cone Conjecture”, “Command and Control”, …)
  • Intentional TeX such as ^{\prime\prime} and \log \log is preserved
  • Startup backfill skips db.session.commit() when no fields change (byte-identical reset on an already-clean DB)
  • Tests added under sites/arxiv/tests/

Test notes

python3 -m unittest discover -s sites/arxiv/tests -v

Result: 10 tests, all OK (stdlib unittest; no extra deps).

Covered:

  • Issue arXiv metadata renders duplicated LaTeX/math fragments #16 fixtures (2604.07983, \mathbb{P}^1, \mathcal{N}=1, GL(d_1)\times GL(d_2), \mathrm{GL}_4, isotope/ion fragments)
  • Idempotence of the helper, including across all papers.json titles/abstracts/comments
  • Wiring: seed_database and normalize_paper_metadata() both call the helper; normalize runs before backfill_paper_gaps()

python3 -m py_compile sites/arxiv/app.py sites/arxiv/metadata_cleaning.py succeeded.

Flask/Docker render of /abs/2604.07983 was not exercised in this environment (Flask stack not installed; no running site container). The helper was checked against the live papers.json row for that id and produces the expected cleaned title.

Note: packaged instance_seed/arxiv.db rows stay dirty until the asset bundle is regenerated. Startup backfill rewrites those rows so the running site is clean; after a cleaned seed DB is shipped, the backfill becomes a no-op and reset byte-identity holds.

Collapse mixed text/MathJax scrape duplicates in arXiv titles and other
metadata at seed time and on startup backfill, without rewriting unrelated
prose. Adds regression tests for the issue aiming-lab#16 fixtures.

Co-authored-by: Lxr-max <Lxr-max@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arXiv metadata renders duplicated LaTeX/math fragments

1 participant