fix: NeathPortTalbotCouncil - user_agent and JS click for overlay bypass#1936
fix: NeathPortTalbotCouncil - user_agent and JS click for overlay bypass#1936InertiaUK wants to merge 1 commit intorobbrad:masterfrom
Conversation
📝 WalkthroughWalkthroughUpdated NeathPortTalbotCouncil WebDriver initialization to pass an explicit Chrome/138 Linux desktop user-agent string, and replaced three native Selenium clicks with JavaScript-invoked clicks, guarding the cookie banner click with error handling. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py`:
- Around line 38-44: Replace the bare except in the cookie banner handling (the
WebDriverWait block that finds and clicks "ccc-notify-accept" in
NeathPortTalbotCouncil.py) so it only catches
selenium.common.exceptions.TimeoutException; import TimeoutException if needed
and let any other exceptions propagate (do not swallow them). Ensure the
try/except wraps the WebDriverWait + execute_script call and only handles the
timeout case, re-raising or not catching other Selenium/driver errors so real
failures surface.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5479569a-8192-4f41-8d2d-2a5e0370d493
📒 Files selected for processing (1)
uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py
| try: | ||
| cookieAccept = WebDriverWait(driver, 10).until( | ||
| EC.presence_of_element_located((By.ID, "ccc-notify-accept")) | ||
| ) | ||
| driver.execute_script("arguments[0].click();", cookieAccept) | ||
| except Exception: | ||
| pass |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify broad exception swallowing around cookie handling in this file.
rg -n -C3 'ccc-notify-accept|except Exception|except [A-Za-z_]+Exception|pass' \
uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.pyRepository: robbrad/UKBinCollectionData
Length of output: 924
Catch only TimeoutException from the cookie banner click, not all exceptions.
Lines 43–44 swallow all exceptions with a bare except Exception: pass, which masks real Selenium/driver failures. The WebDriverWait timeout is the expected failure case; other errors should surface immediately to detect site or session changes.
🔧 Suggested change
+from selenium.common.exceptions import TimeoutException
...
- except Exception:
- pass
+ except TimeoutException:
+ # Cookie banner not shown; continue.
+ pass📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| try: | |
| cookieAccept = WebDriverWait(driver, 10).until( | |
| EC.presence_of_element_located((By.ID, "ccc-notify-accept")) | |
| ) | |
| driver.execute_script("arguments[0].click();", cookieAccept) | |
| except Exception: | |
| pass | |
| try: | |
| cookieAccept = WebDriverWait(driver, 10).until( | |
| EC.presence_of_element_located((By.ID, "ccc-notify-accept")) | |
| ) | |
| driver.execute_script("arguments[0].click();", cookieAccept) | |
| except TimeoutException: | |
| # Cookie banner not shown; continue. | |
| pass |
🧰 Tools
🪛 Ruff (0.15.9)
[error] 43-44: try-except-pass detected, consider logging the exception
(S110)
[warning] 43-43: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py`
around lines 38 - 44, Replace the bare except in the cookie banner handling (the
WebDriverWait block that finds and clicks "ccc-notify-accept" in
NeathPortTalbotCouncil.py) so it only catches
selenium.common.exceptions.TimeoutException; import TimeoutException if needed
and let any other exceptions propagate (do not swallow them). Ensure the
try/except wraps the WebDriverWait + execute_script call and only handles the
timeout case, re-raising or not catching other Selenium/driver errors so real
failures surface.
1 participant
The scraper was failing in headless Chrome because the default webdriver user agent gets challenged, and a modal overlay intercepts the submit click before the form can be posted.
Two small changes:
create_webdriver.driver.execute_script('arguments[0].click()')on the submit button so the overlay doesn't eat the click.Tested against a real address in Neath Port Talbot and the schedule now returns.
Summary by CodeRabbit
Release Notes