Skip to content

feat: updated IMDB scraper to handle new HTML structure - #17

Open
kricha-code wants to merge 1 commit into
nishantsahoo:masterfrom
kricha-code:branch-B
Open

feat: updated IMDB scraper to handle new HTML structure#17
kricha-code wants to merge 1 commit into
nishantsahoo:masterfrom
kricha-code:branch-B

Conversation

@kricha-code

Copy link
Copy Markdown

The IMDB Top 50 scraper was failing to retrieve movie data. This was caused by two main factors:

IMDB actively blocking requests without standard browser headers.

IMDB completely updating their HTML structure, which broke the existing BeautifulSoup selectors and caused NoneType and IndexError crashes.

What Changed:

Added Bot Protection Bypass: Injected a User-Agent header into the requests.get() call to successfully retrieve the HTML instead of getting blocked.

   Updated HTML Selectors: Replaced outdated class names (e.g., lister-item mode-advanced) with the modern IMDB tags (e.g., ipc-metadata-list-summary-item and ipc-title__text).
   Removed deprecated BeautifulSoup findChildren() methods and replaced them with standard .find() and .find_all().

    Simplified the title text extraction to use the clean .text property instead of chaining multiple array indexes and string encodings.

    Updated the request URL string to use https and modern Python f-strings.

Important Notes for the Reviewer:

Limitation: The scraper currently extracts 25 movies per year instead of 50.Modifying the URL parameters (like &count=50) no longer overrides this limit in the initial HTML snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant