Skip to content

Fix source-aware artifact detection and add benchmark tracks#1

Draft
kiriakostsourekas-wq wants to merge 4 commits into
Zangir:mainfrom
kiriakostsourekas-wq:feature/artifact-benchmark-repair
Draft

Fix source-aware artifact detection and add benchmark tracks#1
kiriakostsourekas-wq wants to merge 4 commits into
Zangir:mainfrom
kiriakostsourekas-wq:feature/artifact-benchmark-repair

Conversation

@kiriakostsourekas-wq

Copy link
Copy Markdown

What changed

  • Canonicalize curly/full-width apostrophes and nonbreaking spaces in a same-length matching view, preserving original source offsets and fixes.
  • Parse .bib, .bbl, and embedded thebibliography structure before artifact scanning.
  • Suppress chat-residue deletion suggestions inside bibliographic metadata while continuing normal citation verification.
  • Treat unstructured text outside BibTeX entries as review-only.
  • Detect standalone assistant offers such as “If you’d like, I can also shorten the abstract.”
  • Consolidate overlapping sentence findings into one UI action while retaining every matched rule ID.
  • Report “No clear residue found” without implying human authorship.

Root cause

The browser previously routed .bib content through the ordinary text scanner, so legitimate titles containing phrases such as “As an AI Language Model” were treated as manuscript prose. Chat rules also recognized ASCII contractions but missed typographic apostrophes copied from common editors and chat interfaces.

Benchmarking

This adds two deliberately separate tracks under benchmark/:

  • Track A: verifAI’s actual clear-artifact task, with exact/overlap span metrics, file recall, false positives per 1,000 clean files, destructive-fix metrics, context/format slices, before/after adapters, source-cluster bootstraps, and a fail-closed release gate.
  • Track B: a secondary broad AI-authorship comparison with a standardized prediction contract, train/dev/test leakage guards, dev-only thresholds, five-seed aggregation, low-FPR metrics, calibration, latency/memory reporting, robustness slices, and pinned literature baseline/dataset manifests.

The checked-in Track A data is a regression/smoke seed. It is intentionally marked releaseReady: false until the full licensed and independently adjudicated corpus is supplied. Track A and Track B must not be combined into one leaderboard.

No model, API, or runtime dependency was added to the browser product.

User impact

  • The reported AI-related bibliography title remains unchanged and produces no AI-text warning.
  • The five reported curly-apostrophe chat sentences produce five high-confidence findings and are removed when all fixes are accepted.
  • .bbl and embedded bibliography titles receive the same protection.
  • Straight and curly contraction variants produce equivalent rule IDs and severity.

Validation

  • node --test tests/*.test.mjs260 passed
  • python3 -m unittest benchmark.tests.test_track_b -v9 passed
  • JavaScript syntax, Python compilation, JSON, and TOML validation passed.
  • Track A smoke run: revised verifAI 16 TP / 0 FP / 0 FN over 32 test variants; pinned legacy revision 9 TP / 2 FP / 7 FN.
  • --release correctly fails because the publishable corpus requirements are not yet met.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant