Add unit tests for the crawler

[marco-c]

So far, we have no tests.

[Sush0512]

Yes, I would like to work on it. Shall we create a test folder and add all the unit tests?

[marco-c]

Yes! Let's start by changing the project structure to put a test folder with an empty test and a directory "coverage_crawler" where we put the code.
We can use pytest for testing.

[Sush0512]

Hi @marco-c
Dividing the issue into subtasks, if anything else needs to be added please let me know.

• Change the project structure
• Add unit tests in test folder

[marco-c]

That sounds good, I would file a separate issue for changing the project structure.

[nnrepos]

@marco-c This repo looks pretty inactive, but if you're still here, could you please suggest some tests we could implement?

[marco-c]

@nnrepos it is inactive at the moment, but the plan is to work on this when time will allow.
Regarding tests, we have basically 0 coverage for the crawler, so you could start by adding tests for the simplest functions in crawler.py.

[nnrepos]

hey @marco-c, i created a unittest to see if i understand what you need. please let me know what you think 😄

[nnrepos]

thanks for the support @marco-c :)
i would like to know - what should the next steps be? would you like me to add a unittest for every crawler method? perhaps some integration tests? something specific you would like to be tested?
a task list would be appreciated

[marco-c]

@nnrepos we should add a unit test for each function in the crawler (when it is possible, of course). Feel free to open a PR for each test/function.

[nnrepos]

@marco-c alright.
since we don't want to create calls to google.com, is there any other website i am allowed to test on? something that is expected to rarely (hopefully never) change, and is reliable? perhaps the page of this repository in github?

[marco-c]

@nnrepos ideally we should run a local server so we can fully control the contents.