fix(arxiv): dedupe adjacent LaTeX metadata fragments

[Lxr-max]

Summary

Some arXiv mirror titles were scraped with mixed text/MathJax extraction, so the same math fragment was stored twice (\gtrsim 100\times\gtrsim 100\times, z=1.37z=1.37, \mathbb{P}^1\mathbb{P}^1, and similar).

This change adds a conservative cleanup helper that only collapses clear adjacent duplicates. It does not rewrite unrelated titles, does not convert TeX to Unicode, and does not touch display templates.

• New sites/arxiv/metadata_cleaning.py shared by seed-time import and startup backfill
• seed_database() cleans title/abstract/comments/journal-ref while importing papers.json
• normalize_paper_metadata() backfills already-packaged DB rows on startup (no-op commit when nothing changed)
• Regression tests for the issue fixtures live in sites/arxiv/tests/

Example (/abs/2604.07983):

A Natural \gtrsim 100\times\gtrsim 100\times Telescope ... at z=1.37z=1.37

becomes

A Natural $\gtrsim 100\times$ Telescope ... at $z=1.37$

Fixes #16

Lineage

Rebased onto current aiming-lab/WebHarbor main (cb9a9b0) after syncing the Lxr-max fork. Supersedes the stale fork PR #119 (same approach, old base 145b200). Distinct from #17, which also rewrites display templates to Unicode.

Checklist

• Cleanup is conservative: only adjacent duplicate math/LaTeX fragments
• Shared helper used at seed time and startup backfill
• Unrelated prose titles are left unchanged (K_{11,11}, “Cone Conjecture”, “Command and Control”, …)
• Intentional TeX such as ^{\prime\prime} and \log \log is preserved
• Startup backfill skips db.session.commit() when no fields change (byte-identical reset on an already-clean DB)
• Tests added under sites/arxiv/tests/

Test notes

python3 -m py_compile sites/arxiv/app.py sites/arxiv/metadata_cleaning.py
python3 -m unittest discover -s sites/arxiv/tests -v

Result: 10 tests, all OK (stdlib unittest; no extra deps).

Covered:

• Issue arXiv metadata renders duplicated LaTeX/math fragments #16 fixtures (2604.07983, \mathbb{P}^1, \mathcal{N}=1, GL(d_1)\times GL(d_2), \mathrm{GL}_4, isotope/ion fragments)
• Idempotence of the helper, including across all papers.json titles/abstracts/comments
• Wiring: seed_database and normalize_paper_metadata() both call the helper; normalize runs before backfill_paper_gaps()
• Helper also cleaned 22 titles in the current papers.json corpus (including \mathcal{F}\mathcal{F}, (D+1)(D+1), WSe_2_2, ^{173}^{173}Yb^+^+)

Note: packaged instance_seed/arxiv.db rows stay dirty until the asset bundle is regenerated. Startup backfill rewrites those rows so the running site is clean; after a cleaned seed DB is shipped, the backfill becomes a no-op and reset byte-identity holds.

Please close or supersede #119 once this lands so the stale September rebase is not merged on top of current main.

[Lxr-max]

Closing as a duplicate of the existing vehicle: #119.

This run confirmed #119 still cherry-picks cleanly onto current main (cb9a9b0) and the 10 arXiv metadata tests pass. Please review/merge #119 rather than this rebase copy.