fix: decode HTML entities in event ingestion text before storage

[chubes4]

Closes #844

Summary

Event ingestion stored HTML-entity-encoded source text verbatim in post_title: 5,392 published rows carry &/&, 180 carry ’, 126 carry – (verified at the byte level in the issue). Entities leak into JSON-LD (Extra-Chill/extrachill-seo#61), REST/JSON consumers, search, and every plain-text surface.

This closes both halves of the pipeline:

1. Decode at the ingestion boundary. EventUpsert::executeUpsert() — the single funnel every fetch handler passes through — now runs html_entity_decode( , ENT_QUOTES | ENT_HTML5, 'UTF-8' ) before sanitize_text_field(), and writes the decoded title back onto `` so every downstream consumer (validation gate, advisory-lock keys, duplicate detection, buildEventData) keys on the same canonical form. Decode-then-sanitize ordering means markup hidden behind entities (`<script>`) is exposed and stripped, not smuggled into storage. Decode, never strip.
2. Suspend the kses save filters for ingestion writes. This is the half that would have quietly undone fix fix(calendar): unique SVG borders & per-group badges for repeated weekday groups #1: wp_filter_kses on title_save_pre re-encodes every bare & to & whenever the writing context lacks unfiltered_html. On this multisite the ingestion author (user 32, author/extra_chill_team) has no unfiltered_html, and wp-cron runs with no user at all. Proven in production: wp_kses( 'Power Pilates & Matcha', … ) → 'Power Pilates & Matcha'. TextNormalization::with_kses_suspended() wraps the bounded persistence write and restores the exact prior filter state via kses_init(). Ingestion titles are already tag-free (sanitize_text_field runs first), so this removes a re-encoding side effect, not a sanitization layer.

Same treatment applied to the other affected free-text fields (all measured in production):

Surface	Affected (production)	Fixed at
post_title	5,392	EventUpsert title extraction
Venue term names	157 terms	VenueParameterProvider::resolveField() (single venue-field resolution point: locks, dedup, term creation, block attrs)
Artist term names	~2.3k terms	performer/artist decode in EventUpsert
Block content (descriptions, attrs)	55k rows	EventBlockContentBuilder (decode before wp_kses_post) + buildEventData field allowlist
Promoter/organizer names	0 (lucky)	Promoter_Taxonomy::find_or_create_promoter()

URLs, dates, enums, coordinates, and timezone identifiers are deliberately not decoded (is_decodable_text_field() allowlist).

Which shared code path was actually at fault, and how it was proved

Not one bad extractor — a two-stage pipeline on the shared path. Evidence chain:

• Affected rows span unrelated flows (307: 53, 94: 51, 369: 38, 9: 28, 670: 23, long tail), so the defect lives in shared code, not a source handler.
• Many extractors already decode (Eventbrite, JSON-LD, Bandzoogle, …) but API handlers (Dice — post 9312 came from flow 215/link.dice.fm — Ticketmaster) and some scrapers do not; that inconsistency produced the scatter. The shared upsert boundary is the correct backstop regardless of source.
• The save-path re-encode was proved directly on production: wp_kses( 'Move Wellness - Power Pilates & Matcha — A Night of R&B', array() ) returns 'Move Wellness - Power Pilates & Matcha — A Night of R&B'. The DB shows exactly this signature — & (kses re-encoded) dominating, plus raw ’/– rows that bypassed kses (CLI-context saves; 8,097 published titles with bare & prove such contexts exist).
• Ingestion author 32 lacks unfiltered_html (multisite: has_cap → 0, not a super admin), so every web/cron-context ingestion save re-encoded.

Dedup/identity hashes: verified unchanged, not assumed

The prompt asked whether existing hashes are invalidated. They are not, and this is proved at the byte level on production:

• EventDuplicateStrategy::computeTitleHash() = md5( EventIdentifierGenerator::normalizeBasic( ) ), and normalizeBasic() already entity-decodes before hashing (EventIdentifierGenerator.php:406). Therefore hash(encoded title) === hash(decoded title).
• Verified against the live identity index: for entity-titled posts, the stored datamachine_post_identity.title_hash equals the hash of the decoded title (5/5 sampled rows match, e.g. post 222428).
• Source identity (_datamachine_event_source_identity / DM processed-item claims) hashes through EventIdentifierGenerator::generate() → same normalizeBasic() → generate(encoded) === generate(decoded) verified in production.

So the repair backfill verifies hash stability per-row instead of rewriting hashes: EntityTitleRepairAbilities compares computeTitleHash(old) vs computeTitleHash(new) and the stored index row (DM core PostIdentityIndex when available), counts any drift in identity_hash_drift, and reports it loudly rather than silently assuming.

One related caveat worth knowing: DM core's upsert-post no-change path compares only the block-content hash, so re-ingesting an entity-titled event with identical content returns no_change and does not heal the stored title. Re-ingestion alone was never going to fix the existing rows — the backfill command is genuinely required, which is why it ships in this PR.

What this adds

1. TextNormalization (inc/Core/TextNormalization.php) — the ingestion text storage contract: decode_entities(), contains_entities() (shared ENTITY_PATTERN), with_kses_suspended().
2. Ingestion decode — title/performer/artist in EventUpsert; venue name + geography in VenueParameterProvider::resolveField(); organizer in Promoter_Taxonomy; description in EventBlockContentBuilder; free-text fields in buildEventData behind a decodable-field allowlist.
3. Quality rule — check quality gains entity_title (--issue entity_title filter): published events whose stored title matches the entity pattern, listed with flow attribution. Flows into check all automatically.
4. Backfill command — wp data-machine-events repair-entity-titles (thin CLI adapter over the data-machine-events/repair-entity-titles ability, mirroring repair-series-ends). Dry run by default; --execute applies. Repair writes run under the same kses suspension; decoded rows are left alone; per-row hash_stable is reported. No production data was mass-mutated by this PR — the command ships, it has not been run.

Verification

• New PHPUnit tests, all passing on a WordPress test harness (wordpress-develop + MariaDB):
  • TextNormalizationTest — named/decimal/hex/quote entity decoding, idempotence, no-strip guarantee, detection (bare & is not an entity), kses suspension removes/restores filters, survives a throwing callback, no-op when inactive.
  • EventUpsertEntityDecodeTest — end-to-end through executeUpsert: &/&/’/–/" titles stored decoded; entity-hidden markup stripped; real markup sanitized away; a decoded re-ingestion resolves to the existing entity-titled event (no duplicate); and with wp_filter_kses hooked (simulated no-unfiltered_html context) the stored title still comes out decoded.
  • EventQualityAuditEntityTitleTest — the new audit rule flags named and numeric entity titles, skips decoded titles and bare &.
  • EntityTitleRepairAbilitiesTest — dry run is a no-op; execute decodes stored titles (named, numeric, curly quotes, dashes) without re-encoding even with wp_filter_kses active; decoded rows untouched; per-row hash stability reported.
• Regression check: full Unit+Integration suite (1,322 tests) run on this branch and on clean origin/main with the identical harness — failure sets are byte-identical (71 pre-existing environment failures each: retained-venue lifecycle tests, VenueMergeHelper merge-command tests, multisite/lifecycle assumptions of the sandbox). Zero new failures.
• No lint config in the repo; all changed files pass php -l.
• The repair command was NOT executed against production. Pre-merge sanity, when it ships: wp --url=events.extrachill.com data-machine-events repair-entity-titles (dry run) then --execute.

Notes for review

• Scope guard: the 55k entity-bearing post_content rows and the 157 venue / ~2.3k artist entity-bearing term names are measured but deliberately not backfilled here — term repair needs merge semantics (two terms for the same venue, one encoded) and belongs with the VenueMergeHelper tooling; happy to file follow-ups. This PR closes the ingestion path (primary deliverable per the issue) and repairs the title rows the issue quantifies.
• post_name slugs are not entity-contaminated (sanitize_title() already strips entities — verified on production rows and directly), so the backfill does not touch slugs and no URLs change.
• chore(cli): retire one-off data-repair commands (EncodingFix, BatchTimeFix, TicketUrlResync, BatchActionRecovery) #779 proposes retiring the one-off EncodingFix repair: agreed, and this is why the fix had to include the ingestion path — without it this class recurs a third time.
• The kses suspension is deliberately narrow: only the datamachine/upsert-post execution inside the upsert lock, and the repair command's wp_update_post. Manual editorial saves keep WordPress-standard behavior.

---

This PR was authored by an AI coding agent (Extra Chill Bot) and reviewed by nobody yet.

[homeboy-ci]

Homeboy Results — data-machine-events

Review audit

✅ review audit — passed

Deep dive: homeboy review audit data-machine-events --changed-since e73088a

Artifacts and drill-down

• CI results artifact: homeboy-ci-results-data-machine-events-review-audit-homeboy-Linux-php8.2-node24 contains immediate command JSON for this action invocation.
• Observation artifact: homeboy-observations-data-machine-events-review-audit-homeboy-Linux-php8.2-node24 contains exported Homeboy run history for deeper queries.
• Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
• Artifacts are attached to the workflow run: https://github.com/Extra-Chill/data-machine-events/actions/runs/35453082817

Review lint

✅ review lint — passed

ℹ️ Full options: homeboy self docs commands/lint
Deep dive: homeboy review lint data-machine-events --changed-since e73088a

Artifacts and drill-down

• CI results artifact: homeboy-ci-results-data-machine-events-review-lint-homeboy-Linux-php8.2-node24 contains immediate command JSON for this action invocation.
• Observation artifact: homeboy-observations-data-machine-events-review-lint-homeboy-Linux-php8.2-node24 contains exported Homeboy run history for deeper queries.
• Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
• Artifacts are attached to the workflow run: https://github.com/Extra-Chill/data-machine-events/actions/runs/35453082817

Tooling versions

• Homeboy CLI: homeboy 0.377.0+5f3ae101fa4f63ef025e7b75de087d882411c42a
• Extension: wordpress from https://github.com/Extra-Chill/homeboy-extensions
• Extension revision: 28689748
• Action: Extra-Chill/homeboy-action@v2