Skip to content

feat: add ArtistContentConsistencyScorer transformer - #37

Open
jeffa-block wants to merge 1 commit into
tidal-music:mainfrom
jeffa-block:feat/artist-content-consistency-scorer
Open

feat: add ArtistContentConsistencyScorer transformer#37
jeffa-block wants to merge 1 commit into
tidal-music:mainfrom
jeffa-block:feat/artist-content-consistency-scorer

Conversation

@jeffa-block

Copy link
Copy Markdown

Summary

Adds a new transformer that scores how likely a track-artist pairing is correct, based on metadata consistency signals. This enables detection of content misattribution — tracks that appear on the wrong artist profile due to name collisions, test uploads, or distributor errors.

Motivation

TIDAL's content pipeline matches incoming tracks to artist profiles by name. When multiple artists share a name (e.g. Carlos Santana vs a rapper named Santana), or when test uploads land on real profiles, the mismatch is invisible to downstream systems. This scorer provides a quantitative signal that can feed into:

  • Recommendation filtering (don't recommend misattributed content)
  • Content moderation queues (surface high-confidence anomalies for review)
  • Artist profile quality metrics

This work is complementary to the ongoing content-artist-assigner fixes (PRs #525-#531) which address name assignment bugs at ingestion time. This scorer addresses the detection side: finding existing misattributed content.

Scoring Signals

Signal Weight Logic
Copyright match 0.35 Track copyright == artist's most common copyright holder
ISRC prefix match 0.25 First 5 chars of ISRC in artist's historical prefix set
Provider match 0.25 Track delivered by same provider as artist's majority content
Duration normality 0.15 Within 1 std dev = 1.0, linear decay to 0.0 at 3 std devs

Example Scores

Scenario Expected Score
Legitimate Sony release on Carlos Santana ~1.0
Legitimate reissue (different copyright, same ISRC/provider) ~0.65
Rapper "Santana" merged onto Carlos Santana (all mismatch except duration) ~0.15
22-second test upload via DistroKid (all mismatch) ~0.0

Design Decisions

  • Broadcast join on artist metadata (small table) for performance
  • Null-safe — missing fields score 0.0 for that signal, never crash
  • Read-only — adds a consistencyScore column, does not filter or modify data
  • No ML dependencies — pure PySpark column operations, deterministic

Test Plan

5 test cases in test_artist_content_consistency_scorer.py:

  1. Legitimate track (all match) → ~1.0
  2. Wrong artist (Santana rapper) → ~0.15
  3. Test upload (all mismatch, short duration) → 0.0
  4. Legitimate reissue (partial match) → ~0.65
  5. Null handling (missing fields) → scores 0.0 per signal, doesn't crash

Scores track-artist pairings by metadata consistency across four signals:
- Copyright holder match (weight 0.35)
- ISRC prefix match (weight 0.25)
- Provider/distributor match (weight 0.25)
- Duration normality (weight 0.15)

Outputs a consistencyScore column (0.0-1.0) that downstream systems can
use to detect content misattribution (e.g. different artists with the
same name merged onto one profile, test uploads on real artist pages).

This is a read-only scoring transformer with no side effects — it does
not filter, hide, or remove any content. It adds a column that other
transformers or services can optionally consume.

Includes 5 test cases covering: legitimate tracks, wrong-artist merges,
test uploads, legitimate reissues, and null handling.
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants