Skip to content

Bug fix and list replaced with deque#115

Open
Bchass wants to merge 2 commits into
neokd:mainfrom
Bchass:Gutenberg-improvements
Open

Bug fix and list replaced with deque#115
Bchass wants to merge 2 commits into
neokd:mainfrom
Bchass:Gutenberg-improvements

Conversation

@Bchass

@Bchass Bchass commented Oct 3, 2023

Copy link
Copy Markdown
Contributor

Description

Fixed a bug and switched over to a deque instead of a list

Related Issues

#112

Changes Made

load_proxy_list() doesn't rely on an absolute path anymore, this wasn't working as intended.
rotate_proxy() Switched over to a deque for better efficiency when rotating proxies

Screenshots (if applicable)

N/A

Checklist

Please review and check the following before submitting your pull request:

  • I have followed the project's coding conventions and guidelines
  • I have tested my changes thoroughly
  • I have added/updated relevant documentation
  • My code follows best practices and is easy to understand
  • I have added necessary test cases (if applicable)
  • All existing tests are passing
  • My changes do not introduce any new warnings or errors

Additional Notes

I would like to add test cases for this

@vercel

vercel Bot commented Oct 3, 2023

Copy link
Copy Markdown

Someone is attempting to deploy a commit to a Personal Account owned by @neokd on Vercel.

@neokd first needs to authorize it.

@neokd

neokd commented Oct 3, 2023

Copy link
Copy Markdown
Owner

Can the proxy be made like an parameter? Like if user set as True then proxy works or else it tries to scrape without proxy.?

@Bchass

Bchass commented Oct 3, 2023

Copy link
Copy Markdown
Contributor Author

Can the proxy be made like an parameter? Like if user set as True then proxy works or else it tries to scrape without proxy.?

Are you referring to a user-agent a browser would use?

@neokd

neokd commented Oct 3, 2023

Copy link
Copy Markdown
Owner

No, for now we were thinking that proxy is getting complicated so if we allow user to pass an argument that they want to use scrapper or not.

@Bchass

Bchass commented Oct 3, 2023

Copy link
Copy Markdown
Contributor Author

No, for now we were thinking that proxy is getting complicated so if we allow user to pass an argument that they want to use scrapper or not.

I see. I'll look into adding that option for the scraper.

@neokd neokd marked this pull request as ready for review October 3, 2023 14:37

@Bchass Bchass left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neokd thoughts on this?

@neokd

neokd commented Oct 20, 2023

Copy link
Copy Markdown
Owner

@Bchass looks good I'll review it fully and merge it. Thinking to publish the scrapper as package on Pypi?

@Bchass

Bchass commented Dec 1, 2023

Copy link
Copy Markdown
Contributor Author

Is this project being maintained anymore?

@neokd

neokd commented Dec 4, 2023

Copy link
Copy Markdown
Owner

@Bchass yeah the project is maintained. There was merge conflict so i didn't merge this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants