fix: Club Players endpoint - DOB_AGE XPath and season_id handling

[eskobar95]

Fixes

Club Players Endpoint Issues

1. Fixed DOB_AGE XPath selector - Updated to match zentriert td with date pattern, resolving issue where 0 players were returned despite finding URLs/names
2. Handle None season_id - Default to current year if season_id is None (prevents 'None' in URL, similar to Club Competitions)
3. Added missing response fields - Added club name and seasonId to Club Players response

Changes

• app/utils/xpath.py: Updated Clubs.Players.DOB_AGE XPath to use pattern matching for date columns
• app/services/clubs/players.py:
  • Default season_id to current year in __post_init__ if None
  • Improved __update_season_id to validate extracted season
  • Added name and seasonId to response in get_club_players

Testing

• ✅ Local: Returns 41 players for AC Milan (season 2024)
• ✅ XPath now correctly extracts DOB/Age data
• ✅ season_id defaults correctly when None

Related Issues

Fixes issue where Club Players endpoint returned 0 players despite finding player URLs and names. Root cause was DOB_AGE XPath returning 0 items, causing zip() to fail silently.

Summary by CodeRabbit

Release Notes

• New Features

  • Added debug endpoint for scraping diagnostics to test system health

• Improvements

  • Enhanced error handling and validation in search endpoints
  • Improved data extraction reliability with refined data selectors
  • Better handling of optional browser scraping with graceful fallbacks
  • Clearer error messages for easier troubleshooting

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

The changes integrate Playwright browser scraping with graceful fallback mechanisms across the codebase. Conditional Playwright installation is added to the Docker build, availability checks are implemented in services and endpoints, error handling and validation are enhanced throughout, and a debug scraping endpoint is introduced for testing multiple scraping methods.

Changes

Cohort / File(s)	Summary
Infrastructure & Dependencies Dockerfile, requirements.txt	Dockerfile adds conditional Playwright browser installation step guarded by import check; requirements.txt adds playwright==1.48.0 dependency
API Endpoints app/api/endpoints/clubs.py, app/api/endpoints/competitions.py	Both endpoints wrapped in try/except blocks with validation that results contain a "results" key; errors logged and re-raised
Main Application & Debug Endpoint app/main.py	Adds import guards for Playwright availability; introduces new /debug/scraping endpoint that performs sequential HTTP, browser fallback, and full-page scraping checks with aggregated results
Service Base Layer app/services/base.py	Introduces conditional Playwright imports with PLAYWRIGHT_AVAILABLE flag; adds error handling and validation for browser scraping, HTTP fallbacks, and content validation; enhances logging with URL context; adjusts browser scraper initialization and availability reporting
Club Services app/services/clubs/players.py, app/services/clubs/search.py	players.py defaults season_id to current year, adds guarded season_id extraction with fallback, augments response with seasonId and club name; search.py adds try/except with validation in post_init, debug logging, and content validation
Competition Services app/services/competitions/search.py	Replaces direct request_url_page() call with guarded block that validates result, raises ValueError on failure, and logs URL-aware errors
Utilities app/utils/xpath.py	Updates POSITIONS XPath to use contains() for class matching; updates DOB_AGE XPath to target cells with date patterns instead of positional index

Sequence Diagram

sequenceDiagram
    participant Client
    participant APIEndpoint as API Endpoint
    participant DebugEndpoint as /debug/scraping
    participant Base as Service Base
    participant HTTP as HTTP Request
    participant Browser as Playwright Browser
    
    Client->>DebugEndpoint: GET /debug/scraping?url=...
    DebugEndpoint->>Base: request_url_page(url)
    Base->>HTTP: HTTP GET request
    alt HTTP Success
        HTTP-->>Base: HTML content
        Base->>Base: Validate content
        Base-->>DebugEndpoint: success=true, content_length, error=null
    else HTTP Fails
        HTTP-->>Base: Error/Exception
        Base->>Base: Check PLAYWRIGHT_AVAILABLE
        alt Playwright Available
            Base->>Browser: scrape_with_browser(url)
            alt Browser Success
                Browser-->>Base: Page HTML
                Base->>Base: Validate content
                Base-->>DebugEndpoint: success=true, content_length, error=null
            else Browser Fails
                Browser-->>Base: Error
                Base-->>DebugEndpoint: success=false, content_length=0, error msg
            end
        else Playwright Unavailable
            Base-->>DebugEndpoint: success=false, content_length=0, error=ImportError
        end
    end
    DebugEndpoint->>Client: JSON: {http_success, browser_success, full_success, errors}

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

• app/services/base.py — Dense conditional logic with multiple error-handling paths, fallback mechanisms, and content validation; requires careful tracing of browser availability flags and error propagation
• Playwright availability consistency — Verify that conditional imports, PLAYWRIGHT_AVAILABLE flags, and guard clauses are consistent and correct across all affected files
• XPath expression changes — Validate that updated XPath expressions in app/utils/xpath.py still match the intended data and don't break existing queries
• Error handling flow — Ensure exception re-raising, logging, and error messages with URL context are consistent and don't mask underlying issues
• New /debug/scraping endpoint — Verify sequential scraping logic, aggregation of results, and response structure match intended behavior

Poem

🐰 Browsers and fallbacks, we now embrace,
Playwright joins the race with graceful grace,
When pages resist, our XPath runs deep,
Error guards watch—no secrets to keep,
Debug endpoints sparkle, robust and bright, ✨

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/anti-scraping-optimization

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 7f5da0e and 2ef4f62.

📒 Files selected for processing (10)

• Dockerfile (1 hunks)
• app/api/endpoints/clubs.py (1 hunks)
• app/api/endpoints/competitions.py (1 hunks)
• app/main.py (2 hunks)
• app/services/base.py (8 hunks)
• app/services/clubs/players.py (3 hunks)
• app/services/clubs/search.py (2 hunks)
• app/services/competitions/search.py (1 hunks)
• app/utils/xpath.py (1 hunks)
• requirements.txt (1 hunks)

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}