Welcome! This project is about scraping the top 100 YouTube channels from Social Blade. The HTML structure of the website was a bit complicated, but no worries, we managed to get through it!
-
Install necessary packages
Make sure you have all the necessary packages installed. You can find the list in the
requirements.txtfile.pip install -r requirements.txt
-
Run the Spider
Navigate to the project directory and run the
socialblade_spider.scrapy crawl socialblade_spider -O data.csv
The scraped data will be saved in a CSV file. You can easily open it using spreadsheet software like Microsoft Excel or Google Sheets to analyze the data.
The scraper extracts the following columns:
- SocialBlade Url
- Channel Name
- Channel ID
- Channel Link
- Uploads
- Subscribers
- Views
- Country
- Channel Type
- Created Date
- Social Blade Rank
- Subscriber Rank
- Video View Rank
- Country Rank
- People Rank
- Estimated Monthly Income
- Subscribers Last 30 Days
- Views Last 30 Days
- Yearly Income
- Daily Subscribers Average
- Daily Views Average
-
Unclear HTML Structure: The website structure wasn’t very friendly, but we rolled up our sleeves and decoded the necessary parts to scrape the data accurately.
-
HTTP Errors: We encountered some HTTP 403 and 302 errors during the process. To navigate around these, we utilized headers and cache strategies.
- Scrapy Documentation: If you want to dig deeper or face any issues, Scrapy’s official documentation is a great resource: Scrapy Documentation
Feel free to dive in! Open an issue, submit enhancements, or fix bugs. Your contributions are always welcome!
This project is open-source and available to everyone. You can use, modify, and distribute it as you like. Enjoy your scraping journey!