Fix scraping - update for modern DownDetector HTML and improve bot detection bypass

[Copilot]

DownDetector changed their HTML structure and added bot detection, breaking the single-selector scraper. The CLI now fails silently or returns AttributeError.

Changes

Multi-strategy HTML parsing with backward compatibility:

• Strategy 1: Original div#company > div.h2.entry-title (preserves old structure)
• Strategy 2: Direct .entry-title lookup on h1/h2/h3/div (modern pages)
• Strategy 3: Class pattern matching for "status"/"entry-title" with text validation
• Strategy 4: Keyword-based h1 fallback (problem, outage, reports, etc.)

Bot detection bypass:

• Updated User-Agent to Chrome 120.0.0.0
• Added comprehensive browser headers (Accept-, Sec-Fetch-, DNT, Connection)
• Session-based requests for cookie persistence
• Extracted as module constants (USER_AGENT, DEFAULT_HEADERS)

Error handling improvements:

• New NetworkError exception separating network failures from invalid services
• 10-second timeout with raise_for_status() validation
• Exception handling in Main.py for better UX
• Improved error messages: Now includes the specific service name that failed, provides examples of valid service names (e.g., 'facebook', 'twitter', 'instagram'), and directs users to verify on downdetector.com

Code quality:

• Extracted MIN_STATUS_TEXT_LENGTH constant
• Generator-based iteration in Strategy 3 for early termination
• Cached .lower() calls to avoid redundant operations
• Removed obsolete AttributeError handling

Example

The scraper now handles multiple HTML structures:

# Modern structure (Strategy 2)
<h1 class="entry-title">User reports indicate problems at Facebook</h1>

# Original structure (Strategy 1) 
<div id="company">
  <div class="h2 entry-title">Problems at Facebook</div>
</div>

# Fallback (Strategy 4)
<h1>Major outage at Facebook</h1>

Error message example:

Unable to find status for 'invalid-service'. Please verify the service name is correct (e.g., 'facebook', 'twitter', 'instagram'). Visit https://downdetector.com to confirm the service exists.

All strategies tested with mock responses. CodeQL: 0 alerts.

Original prompt

this cli for down detector does not seem to work anymore. it is not able to scrape the correct status from the down detector website.

do the following things:

1. figure out why it is unable to scrape the website for fetching the status
2. make suitable changes to code in order to bypass any bot protections
3. modify the scraping technique to make it align with the new website structure

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[aaryanrr]

@copilot how about we improve this error message to better indicate the problem to the user?

[Copilot]

Improved the error message in commit e16e998. It now:

• Shows the specific service name that failed
• Provides examples of valid service names (facebook, twitter, instagram)
• Directs users to verify on downdetector.com

Example error message:

Unable to find status for 'invalid-service'. Please verify the service name is correct (e.g., 'facebook', 'twitter', 'instagram'). Visit https://downdetector.com to confirm the service exists.

[aaryanrr]

Fix the comments given in the code. And test the code in an environemnt first.