fix: add realistic user_agent to all Selenium scrapers missing one#1947
Open
InertiaUK wants to merge 1 commit intorobbrad:masterfrom
Open
fix: add realistic user_agent to all Selenium scrapers missing one#1947InertiaUK wants to merge 1 commit intorobbrad:masterfrom
InertiaUK wants to merge 1 commit intorobbrad:masterfrom
Conversation
Contributor
📝 WalkthroughWalkthroughThis PR systematically updates 65+ council scraper implementations across the UK bin collection module by adding explicit browser user-agent strings to webdriver initialization calls, replacing Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~30 minutes The PR applies a consistent, homogeneous pattern across 65+ files, reducing per-file review complexity. However, the large count of files necessitates spot-checking for consistency and careful attention to the two flagged indentation issues in Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (8)
uk_bin_collection/uk_bin_collection/councils/PowysCouncil.py (1)
36-37: Centralize the shared Selenium user-agent constant.This works, but the same literal is now duplicated across many scrapers. Moving it to a shared constant/helper (e.g., in
common) will reduce drift and make future updates one-touch.♻️ Suggested direction
- user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" - driver = create_webdriver(web_driver, headless, user_agent, __name__) + driver = create_webdriver( + web_driver, headless, DEFAULT_SELENIUM_USER_AGENT, __name__ + )# in common module (example) DEFAULT_SELENIUM_USER_AGENT = ( "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/PowysCouncil.py` around lines 36 - 37, Extract the duplicated user-agent literal into a shared constant (e.g., DEFAULT_SELENIUM_USER_AGENT) in the common module and replace the local assignment in PowysCouncil.py (the user_agent variable used when calling create_webdriver) with an import of that constant; update the create_webdriver(web_driver, headless, user_agent, __name__) call to pass the imported DEFAULT_SELENIUM_USER_AGENT and add the new constant export in common so other scrapers can reuse it.uk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.py (1)
39-40: Centralize the Selenium user-agent string in shared code.This change is correct, but duplicating the same hardcoded UA across many scrapers will be brittle to maintain. Please move it to a shared constant/helper (for example in
common) and reference that value here.♻️ Example refactor shape
# uk_bin_collection/uk_bin_collection/common.py +DEFAULT_SELENIUM_USER_AGENT = ( + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" +) # council scraper - user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" - driver = create_webdriver(web_driver, headless, user_agent, __name__) + driver = create_webdriver(web_driver, headless, DEFAULT_SELENIUM_USER_AGENT, __name__)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.py` around lines 39 - 40, Replace the hardcoded user_agent string in ForestOfDeanDistrictCouncil.py by importing a shared constant from the common module (e.g., DEFAULT_USER_AGENT) and pass that into create_webdriver instead of the inline user_agent variable; update any references to the local user_agent variable, add the constant (DEFAULT_USER_AGENT = "Mozilla/5.0 ...") to the common helper/module, and adjust the import statement in this file to use the shared constant so all scrapers reference the single source of truth.uk_bin_collection/uk_bin_collection/councils/CalderdaleCouncil.py (1)
42-43: Centralize the shared Selenium user-agent string.This works, but the same literal is now duplicated across many councils. Please move it to a single constant (e.g., in
uk_bin_collection/uk_bin_collection/common.py) to avoid version drift and bulk edits later.♻️ Proposed refactor
# uk_bin_collection/uk_bin_collection/common.py +DEFAULT_SELENIUM_USER_AGENT = ( + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" +) # uk_bin_collection/uk_bin_collection/councils/CalderdaleCouncil.py - user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" + user_agent = DEFAULT_SELENIUM_USER_AGENT driver = create_webdriver(web_driver, headless, user_agent, __name__)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/CalderdaleCouncil.py` around lines 42 - 43, The hard-coded Selenium user-agent literal used when calling create_webdriver (e.g., the user_agent variable in CalderdaleCouncil.py) is duplicated across multiple councils; centralize it by adding a single constant USER_AGENT (or SELENIUM_USER_AGENT) in uk_bin_collection.common and import that constant in CalderdaleCouncil.py, replacing the local user_agent assignment and passing USER_AGENT into create_webdriver(web_driver, headless, USER_AGENT, __name__); apply the same replacement to other council modules that define the same literal so all callers of create_webdriver use the shared constant.uk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.py (1)
27-28: Centralize the shared Selenium user-agent string to avoid drift across councils.Line 27 duplicates a long literal that is now repeated in many modules. Consider using a single constant (e.g., in
common.py) and reusing it here.♻️ Proposed refactor
- user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" - driver = create_webdriver(web_driver, headless, user_agent, __name__) + driver = create_webdriver( + web_driver, headless, DEFAULT_SELENIUM_USER_AGENT, __name__ + )+# uk_bin_collection/uk_bin_collection/common.py +DEFAULT_SELENIUM_USER_AGENT = ( + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" +)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.py` around lines 27 - 28, The duplicated Selenium user-agent literal assigned to user_agent before calling create_webdriver should be replaced with a shared constant: define a USER_AGENT (or SELENIUM_USER_AGENT) in the module that centralizes common values (e.g., common.py) and import it here; then remove the local literal and pass the imported USER_AGENT into create_webdriver in the ChichesterDistrictCouncil code (referencing the user_agent variable and create_webdriver function to locate the change).uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py (1)
34-35: Consider centralizing this shared user-agent value.This literal is now duplicated across many scrapers, which makes future updates error-prone. Prefer a single constant/helper in common code and reuse it here.
Refactor sketch
# uk_bin_collection/uk_bin_collection/common.py +DEFAULT_SELENIUM_USER_AGENT = ( + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" +) # uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py - user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" + user_agent = DEFAULT_SELENIUM_USER_AGENT driver = create_webdriver(web_driver, headless, user_agent, __name__)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py` around lines 34 - 35, The duplicated user-agent string used before calling create_webdriver in BroxtoweBoroughCouncil.py should be centralized: remove the local literal assignment to user_agent and instead import or reference a shared constant (e.g., DEFAULT_USER_AGENT) from a common module (e.g., a utils or settings module) and pass that into create_webdriver(web_driver, headless, DEFAULT_USER_AGENT, __name__); update any other scrapers to use the same constant to avoid duplication.uk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.py (1)
36-37: Centralize the user-agent constant to reduce maintenance churn.This literal is repeated across many council modules in the PR. Please consider moving it to a shared constant (e.g., in
uk_bin_collection/uk_bin_collection/common.py) and reusing it here, so future UA updates happen in one place.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.py` around lines 36 - 37, Replace the hard-coded user_agent string in AshfordBoroughCouncil.py by importing a shared constant from the common module: add a constant like DEFAULT_USER_AGENT (or USER_AGENT) in uk_bin_collection.common and then use that constant when calling create_webdriver (replace the local user_agent variable used before the create_webdriver(web_driver, headless, user_agent, __name__) call). Ensure the constant name is descriptive and update the import in AshfordBoroughCouncil.py so the create_webdriver call uses the shared value.uk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.py (1)
26-27: Consider moving the UA to one shared constant/default.This works, but copying the same literal across the blanket change will make the next UA rotation another multi-file edit. If this is now the common Selenium default, set it once in
create_webdriver()or a shared constant and only override councils that genuinely need a different value.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.py` around lines 26 - 27, The hard-coded user agent string near the call to create_webdriver(web_driver, headless, user_agent, __name__) should be centralized: remove the literal from BlaenauGwentCountyBoroughCouncil.py and instead provide a default/shared constant or default parameter inside the create_webdriver function (or a shared module constant like DEFAULT_USER_AGENT) so callers only pass a custom UA when needed; update create_webdriver to use that default and change this call to rely on the default (or reference the shared constant) so future UA rotations require a single-file change.uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py (1)
1-3: Minor: Redundant import shadows thedatetimemodule.The file imports
datetimeas a module on line 1, then imports thedatetimeclass from it on line 3, which shadows the module. While this is pre-existing code and works correctly, it could cause confusion.♻️ Suggested cleanup
-import datetime import time from datetime import datetime🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py` around lines 1 - 3, The file currently imports the datetime module and also imports the datetime class from it (imports at top: "import datetime" and "from datetime import datetime"), which causes the module name to be shadowed and is confusing; remove the redundant "from datetime import datetime" (or alternatively remove "import datetime" and qualify usages) and update any references to use either datetime.datetime or datetime (depending on which import you keep) so all usages (e.g., in functions/classes that call datetime.now or datetime.strptime) remain correct and unambiguous.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py`:
- Around line 29-30: There's an extra indentation before the driver assignment
causing a syntax/parsing error; align the driver = create_webdriver(...)
statement with the user_agent = ... line so both have the same indentation
level. Locate the driver assignment that calls create_webdriver(web_driver,
headless, user_agent, __name__) and remove the leading tab/spaces so it lines up
with the user_agent variable.
In `@uk_bin_collection/uk_bin_collection/councils/ThanetDistrictCouncil.py`:
- Around line 31-32: The indentation of the driver creation is incorrect: align
the `driver = create_webdriver(web_driver, headless, user_agent, __name__)`
statement to the same indentation level as the `user_agent = "..."` assignment
so both `user_agent` and `driver` are in the same scope; locate the `user_agent`
and `driver` lines in ThanetDistrictCouncil.py and un-indent or re-indent the
`driver` line accordingly so `create_webdriver(web_driver, headless, user_agent,
__name__)` executes without a syntax error.
---
Nitpick comments:
In `@uk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.py`:
- Around line 36-37: Replace the hard-coded user_agent string in
AshfordBoroughCouncil.py by importing a shared constant from the common module:
add a constant like DEFAULT_USER_AGENT (or USER_AGENT) in
uk_bin_collection.common and then use that constant when calling
create_webdriver (replace the local user_agent variable used before the
create_webdriver(web_driver, headless, user_agent, __name__) call). Ensure the
constant name is descriptive and update the import in AshfordBoroughCouncil.py
so the create_webdriver call uses the shared value.
In
`@uk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.py`:
- Around line 26-27: The hard-coded user agent string near the call to
create_webdriver(web_driver, headless, user_agent, __name__) should be
centralized: remove the literal from BlaenauGwentCountyBoroughCouncil.py and
instead provide a default/shared constant or default parameter inside the
create_webdriver function (or a shared module constant like DEFAULT_USER_AGENT)
so callers only pass a custom UA when needed; update create_webdriver to use
that default and change this call to rely on the default (or reference the
shared constant) so future UA rotations require a single-file change.
In `@uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py`:
- Around line 34-35: The duplicated user-agent string used before calling
create_webdriver in BroxtoweBoroughCouncil.py should be centralized: remove the
local literal assignment to user_agent and instead import or reference a shared
constant (e.g., DEFAULT_USER_AGENT) from a common module (e.g., a utils or
settings module) and pass that into create_webdriver(web_driver, headless,
DEFAULT_USER_AGENT, __name__); update any other scrapers to use the same
constant to avoid duplication.
In `@uk_bin_collection/uk_bin_collection/councils/CalderdaleCouncil.py`:
- Around line 42-43: The hard-coded Selenium user-agent literal used when
calling create_webdriver (e.g., the user_agent variable in CalderdaleCouncil.py)
is duplicated across multiple councils; centralize it by adding a single
constant USER_AGENT (or SELENIUM_USER_AGENT) in uk_bin_collection.common and
import that constant in CalderdaleCouncil.py, replacing the local user_agent
assignment and passing USER_AGENT into create_webdriver(web_driver, headless,
USER_AGENT, __name__); apply the same replacement to other council modules that
define the same literal so all callers of create_webdriver use the shared
constant.
In `@uk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.py`:
- Around line 27-28: The duplicated Selenium user-agent literal assigned to
user_agent before calling create_webdriver should be replaced with a shared
constant: define a USER_AGENT (or SELENIUM_USER_AGENT) in the module that
centralizes common values (e.g., common.py) and import it here; then remove the
local literal and pass the imported USER_AGENT into create_webdriver in the
ChichesterDistrictCouncil code (referencing the user_agent variable and
create_webdriver function to locate the change).
In `@uk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.py`:
- Around line 39-40: Replace the hardcoded user_agent string in
ForestOfDeanDistrictCouncil.py by importing a shared constant from the common
module (e.g., DEFAULT_USER_AGENT) and pass that into create_webdriver instead of
the inline user_agent variable; update any references to the local user_agent
variable, add the constant (DEFAULT_USER_AGENT = "Mozilla/5.0 ...") to the
common helper/module, and adjust the import statement in this file to use the
shared constant so all scrapers reference the single source of truth.
In `@uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py`:
- Around line 1-3: The file currently imports the datetime module and also
imports the datetime class from it (imports at top: "import datetime" and "from
datetime import datetime"), which causes the module name to be shadowed and is
confusing; remove the redundant "from datetime import datetime" (or
alternatively remove "import datetime" and qualify usages) and update any
references to use either datetime.datetime or datetime (depending on which
import you keep) so all usages (e.g., in functions/classes that call
datetime.now or datetime.strptime) remain correct and unambiguous.
In `@uk_bin_collection/uk_bin_collection/councils/PowysCouncil.py`:
- Around line 36-37: Extract the duplicated user-agent literal into a shared
constant (e.g., DEFAULT_SELENIUM_USER_AGENT) in the common module and replace
the local assignment in PowysCouncil.py (the user_agent variable used when
calling create_webdriver) with an import of that constant; update the
create_webdriver(web_driver, headless, user_agent, __name__) call to pass the
imported DEFAULT_SELENIUM_USER_AGENT and add the new constant export in common
so other scrapers can reuse it.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c69db34a-ba48-46d1-91fb-7540d71a6717
📒 Files selected for processing (63)
uk_bin_collection/uk_bin_collection/councils/ArgyllandButeCouncil.pyuk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/BaberghDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/BarkingDagenham.pyuk_bin_collection/uk_bin_collection/councils/BlackburnCouncil.pyuk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/BrightonandHoveCityCouncil.pyuk_bin_collection/uk_bin_collection/councils/BroadlandDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/CalderdaleCouncil.pyuk_bin_collection/uk_bin_collection/councils/CeredigionCountyCouncil.pyuk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/ColchesterCityCouncil.pyuk_bin_collection/uk_bin_collection/councils/CotswoldDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/CroydonCouncil.pyuk_bin_collection/uk_bin_collection/councils/DacorumBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/EastLindseyDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/EastRenfrewshireCouncil.pyuk_bin_collection/uk_bin_collection/councils/EastRidingCouncil.pyuk_bin_collection/uk_bin_collection/councils/EastSuffolkCouncil.pyuk_bin_collection/uk_bin_collection/councils/EpsomandEwellBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/GloucesterCityCouncil.pyuk_bin_collection/uk_bin_collection/councils/GreatYarmouthBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/GuildfordCouncil.pyuk_bin_collection/uk_bin_collection/councils/HertsmereBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/HighPeakCouncil.pyuk_bin_collection/uk_bin_collection/councils/Hillingdon.pyuk_bin_collection/uk_bin_collection/councils/HyndburnBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/KingstonUponThamesCouncil.pyuk_bin_collection/uk_bin_collection/councils/KnowsleyMBCouncil.pyuk_bin_collection/uk_bin_collection/councils/LondonBoroughRedbridge.pyuk_bin_collection/uk_bin_collection/councils/MaidstoneBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/MidAndEastAntrimBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/MidSuffolkDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/MidUlsterDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/NorthDevonCountyCouncil.pyuk_bin_collection/uk_bin_collection/councils/NorthEastDerbyshireDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/NorthWestLeicestershire.pyuk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.pyuk_bin_collection/uk_bin_collection/councils/PortsmouthCityCouncil.pyuk_bin_collection/uk_bin_collection/councils/PowysCouncil.pyuk_bin_collection/uk_bin_collection/councils/PrestonCityCouncil.pyuk_bin_collection/uk_bin_collection/councils/RugbyBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/RushcliffeBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/SevenoaksDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/StHelensBC.pyuk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/StocktonOnTeesCouncil.pyuk_bin_collection/uk_bin_collection/councils/SunderlandCityCouncil.pyuk_bin_collection/uk_bin_collection/councils/TeignbridgeCouncil.pyuk_bin_collection/uk_bin_collection/councils/TendringDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/ThanetDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/ThreeRiversDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/TorbayCouncil.pyuk_bin_collection/uk_bin_collection/councils/WalthamForest.pyuk_bin_collection/uk_bin_collection/councils/WestBerkshireCouncil.pyuk_bin_collection/uk_bin_collection/councils/WestLothianCouncil.pyuk_bin_collection/uk_bin_collection/councils/WestOxfordshireDistrictCouncil.pyuk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.pyuk_bin_collection/uk_bin_collection/councils/WirralCouncil.pyuk_bin_collection/uk_bin_collection/councils/WokinghamBoroughCouncil.pyuk_bin_collection/uk_bin_collection/councils/WrexhamCountyBoroughCouncil.py
Comment on lines
+29
to
+30
| user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" | ||
| driver = create_webdriver(web_driver, headless, user_agent, __name__) |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python - <<'PY'
from pathlib import Path
import ast
import sys
path = Path("uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py")
source = path.read_text(encoding="utf-8")
try:
ast.parse(source, filename=str(path))
print(f"{path}: syntax OK")
except SyntaxError as exc:
print(f"{path}:{exc.lineno}:{exc.offset}: {exc.msg}")
sys.exit(1)
PYRepository: robbrad/UKBinCollectionData
Length of output: 181
🏁 Script executed:
cat -n uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py | sed -n '25,35p'Repository: robbrad/UKBinCollectionData
Length of output: 666
Fix the extra indentation on line 30.
Line 30 is indented one level too deep, causing a syntax error that prevents the file from being parsed. The driver assignment must align with the user_agent assignment above it.
🐛 Minimal fix
# Create Selenium webdriver
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
- driver = create_webdriver(web_driver, headless, user_agent, __name__)
+ driver = create_webdriver(web_driver, headless, user_agent, __name__)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" | |
| driver = create_webdriver(web_driver, headless, user_agent, __name__) | |
| user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" | |
| driver = create_webdriver(web_driver, headless, user_agent, __name__) |
🧰 Tools
🪛 Ruff (0.15.9)
[warning] 30-30: Unexpected indentation
(invalid-syntax)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py`
around lines 29 - 30, There's an extra indentation before the driver assignment
causing a syntax/parsing error; align the driver = create_webdriver(...)
statement with the user_agent = ... line so both have the same indentation
level. Locate the driver assignment that calls create_webdriver(web_driver,
headless, user_agent, __name__) and remove the leading tab/spaces so it lines up
with the user_agent variable.
Comment on lines
+31
to
+32
| user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" | ||
| driver = create_webdriver(web_driver, headless, user_agent, __name__) |
Contributor
There was a problem hiding this comment.
Critical: Indentation error causes syntax failure.
Line 32 has incorrect indentation relative to line 31. The driver = create_webdriver(...) statement is indented further than the user_agent = ... assignment, which will cause a Python syntax error preventing this scraper from running.
🐛 Proposed fix
# Create the Selenium WebDriver
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
- driver = create_webdriver(web_driver, headless, user_agent, __name__)
+ driver = create_webdriver(web_driver, headless, user_agent, __name__)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" | |
| driver = create_webdriver(web_driver, headless, user_agent, __name__) | |
| # Create the Selenium WebDriver | |
| user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" | |
| driver = create_webdriver(web_driver, headless, user_agent, __name__) |
🧰 Tools
🪛 Ruff (0.15.9)
[warning] 32-32: Unexpected indentation
(invalid-syntax)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@uk_bin_collection/uk_bin_collection/councils/ThanetDistrictCouncil.py` around
lines 31 - 32, The indentation of the driver creation is incorrect: align the
`driver = create_webdriver(web_driver, headless, user_agent, __name__)`
statement to the same indentation level as the `user_agent = "..."` assignment
so both `user_agent` and `driver` are in the same scope; locate the `user_agent`
and `driver` lines in ThanetDistrictCouncil.py and un-indent or re-indent the
`driver` line accordingly so `create_webdriver(web_driver, headless, user_agent,
__name__)` executes without a syntax error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Several Selenium-based scrapers were running with Chrome's default headless user agent, which some council websites detect and block (returning 403s, Cloudflare challenges, or empty responses).
This adds a realistic desktop Chrome user agent string to the
create_webdrivercall in every Selenium scraper that was missing one. Thecreate_webdriverfunction already supports theuser_agentparameter — most scrapers were just passingNone.63 files changed, all with the same mechanical edit: define a
user_agentvariable and pass it instead ofNone. Scrapers that already set a user agent (15 councils) and those with separate open PRs are excluded.No functional changes beyond the user agent string.
Summary by CodeRabbit