fix: Add name and seasonId fields to ClubPlayers schema

[eskobar95]

Fix

Club Players Schema Update

• Added missing fields - Added name and season_id fields to ClubPlayers schema
• These fields are set in get_club_players() but were missing from the Pydantic schema
• Resolves issue where name and seasonId were null in API response despite being set in service layer

Changes

• app/schemas/clubs/players.py: Added name: Optional[str] = None and season_id: Optional[str] = None to ClubPlayers class

Testing

• ✅ Schema validation passes with new fields
• ✅ Local: Returns name='AC Milan' and seasonId='2024' correctly
• ⚠️ Server: Currently returns null (will be fixed after deployment)

Related

Follow-up to PR #6 - adds schema fields that were missing for the response fields added in that PR.

Summary by CodeRabbit

• New Features

  • Added debug endpoint for troubleshooting scraping issues and diagnosing page retrieval performance
  • ClubPlayers responses now include club name and season information

• Bug Fixes

  • Improved robustness of player position and age data extraction with more flexible selectors
  • Enhanced error handling and logging for search endpoints with better diagnostics
  • Strengthened browser fallback mechanism with availability checks and explicit error reporting

• Chores

  • Added Playwright dependency for browser-based content retrieval

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

This PR adds Playwright browser scraping capability with graceful fallback handling, introduces a new diagnostic debug endpoint for multi-stage scraping metrics, enhances error handling and logging across search and competition endpoints, updates XPath selectors for robustness, expands the club player schema with season and name fields, and adds Playwright as a conditional dependency.

Changes

Cohort / File(s)	Summary
Infrastructure & Dependencies Dockerfile, requirements.txt	Adds conditional Playwright browser installation in Docker build and declares playwright==1.48.0 as a Python 3.9+ dependency.
Error Handling & Logging app/api/endpoints/clubs.py, app/api/endpoints/competitions.py	Wraps search handlers in try/except blocks; logs warnings when "results" are empty and re-raises exceptions for proper error propagation.
Search Service Robustness app/services/clubs/search.py, app/services/competitions/search.py	Adds try/except guards on page loading in __post_init__, explicit validation that page is not None, detailed debug logging for content inspection, and XPath extraction diagnostics.
New Debug Endpoint app/main.py	Introduces /debug/scraping endpoint that orchestrates HTTP request, optional Playwright browser fallback, and full page retrieval, returning structured diagnostic metrics; enhances test_browser_scraping with availability guard.
Playwright Integration app/services/base.py	Implements guarded Playwright import with PLAYWRIGHT_AVAILABLE flag, conditional _browser_scraper initialization, runtime ImportError raising when unavailable, enhanced monitoring stats, and expanded content validation.
Data Model & Schema app/schemas/clubs/players.py	Adds optional name and season_id fields to ClubPlayers schema with default None values.
Service Logic Updates app/services/clubs/players.py	Defaults season_id to current year in __post_init__, adds guarded extraction in __update_season_id, retrieves and includes club name and season ID in get_club_players response.
XPath Selector Improvements app/utils/xpath.py	Updates player position and date-of-birth selectors from index-based to pattern-based matching; POSITIONS now uses substring containment for class matching, DOB_AGE matches date patterns with slash validation.

Sequence Diagram

sequenceDiagram
    participant Client
    participant DebugEndpoint as /debug/scraping Endpoint
    participant HTTPScraper as HTTP Scraper
    participant BrowserScraper as Browser Scraper<br/>(Playwright)
    participant PageProcessor as Page Processor

    Client->>DebugEndpoint: GET /debug/scraping?url=...
    
    DebugEndpoint->>HTTPScraper: scrape_http(url)
    HTTPScraper-->>DebugEndpoint: html_content or error
    
    rect rgb(200, 220, 255)
        note over DebugEndpoint,BrowserScraper: Browser Fallback (if needed)
        alt Playwright Available & HTTP Failed
            DebugEndpoint->>BrowserScraper: scrape_with_browser(url)
            BrowserScraper->>BrowserScraper: Launch browser<br/>Navigate page
            BrowserScraper-->>DebugEndpoint: full_page_content
        else Playwright Unavailable
            DebugEndpoint-->>DebugEndpoint: Report<br/>browser_available=False
        end
    end

    DebugEndpoint->>PageProcessor: Parse & retrieve full content
    PageProcessor-->>DebugEndpoint: Parsed results
    
    DebugEndpoint-->>Client: {<br/>  http_success, http_content_length,<br/>  browser_success, browser_content_length,<br/>  full_success, total_content_length<br/>}

Loading

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• Areas requiring extra attention:
  • app/services/base.py: Conditional Playwright import and _browser_scraper initialization logic—verify graceful fallback doesn't mask underlying failures
  • app/services/clubs/search.py: Multiple debug logging blocks and page serialization logic—ensure performance impact is acceptable and logging doesn't leak sensitive data
  • app/utils/xpath.py: Updated XPath patterns for player position and DOB extraction—verify new substring/pattern matching produces consistent results across varied HTML structures
  • app/main.py: New debug endpoint and browser fallback orchestration—ensure error paths and metric reporting are exhaustive

Poem

🐰 Playwright now stands by, ready to catch what HTTP might miss,
With graceful fallbacks and logs that won't dismiss,
Debug endpoints divine, metrics precise and bright,
XPaths refined by pattern, not fragile by sight.
Robust and aware, we scrape through the night! 🌙✨

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/anti-scraping-optimization

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 057f689 and 58dda40.

📒 Files selected for processing (11)

• Dockerfile (1 hunks)
• app/api/endpoints/clubs.py (1 hunks)
• app/api/endpoints/competitions.py (1 hunks)
• app/main.py (2 hunks)
• app/schemas/clubs/players.py (1 hunks)
• app/services/base.py (8 hunks)
• app/services/clubs/players.py (3 hunks)
• app/services/clubs/search.py (2 hunks)
• app/services/competitions/search.py (1 hunks)
• app/utils/xpath.py (1 hunks)
• requirements.txt (1 hunks)

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}