Crash with v0.2.0

[bulvara]

I have been running suckit https://cigeography.blogspot.com/ -i '.*\.png' -o .:

thread '' panicked at 'Failed to parse url: http://The POLISH AIR FORCE, ARMY AVIATION and NAVAL AVIATION BRIGADE | Error: invalid domain character', src/scraper.rs:193:31
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
thread 'main' panicked at 'called Result::unwrap() on an Err value: Any { .. }', src/scraper.rs:328:10

[Skallwar]

Thanks for the report. We will investigate

[red0888]

same error for me on windows

[CamoCatX]

@red0888 This script does not support windows... That's your error! For more information, read. The README.md

[dream-1ab]

the problem is not related to windows platform. it is a bug in this program.

The problem is caused by invalid url from some html files.
here is the original code of src/scraper.rs line:193:

suckit/src/scraper.rs

Lines 187 to 220 in f513cc5

	dom.find_urls_as_strings()
	.into_iter()
	.filter(|candidate| Scraper::should_visit(scraper, candidate))
	.for_each(|next_url| {
	let url_to_parse = Scraper::normalize_url(next_url.clone());
	let next_full_url = match url.join(url_to_parse.as_str()) {
	Ok(url) => url,
	Err(e) => panic!("Failed to parse url: {} | Error: {}", next_url, e),
	};
	let path = url_helper::to_path(&next_full_url, true);
	let path_no_fragments = url_helper::to_path(&next_full_url, false);
	// We only add urls without fragments to avoid duplication
	if scraper.map_url_path(&next_full_url, path_no_fragments.clone()) {
	if !Scraper::is_on_another_domain(next_url, url) {
	// If we are determining for a local domain
	if scraper.args.depth == INFINITE_DEPTH || depth < scraper.args.depth {
	Scraper::push(transmitter, next_full_url, depth + 1, ext_depth);
	}
	} else {
	// If we are determining for an external domain
	if scraper.args.ext_depth == INFINITE_DEPTH
	|| ext_depth < scraper.args.ext_depth
	{
	Scraper::push(transmitter, next_full_url, depth, ext_depth + 1);
	}
	}
	}
	scraper.fix_domtree(next_url, &source_path, &path);
	});

look at line 195. the program is panic if url parse is failed. this is not necessary I think and skipping for the broken url is totally acceptable solution, so I had modified it something like this:

        dom.find_urls_as_strings()
            .into_iter()
            .filter(|candidate| Scraper::should_visit(scraper, candidate))
            .for_each(|next_url| {
                let url_to_parse = Scraper::normalize_url(next_url.clone());

                let next_full_url = match url.join(url_to_parse.as_str()) {
                    Ok(url) => url,
                    Err(e) => {
/***********************OVER HERE********************/
                        warn!("Skipping failed url to parse: {} | Error: {}", next_url, e);
                        return;
/***********************OVER HERE********************/
                    },
                };

                let path = url_helper::to_path(&next_full_url, true);
                let path_no_fragments = url_helper::to_path(&next_full_url, false);

I have just skipped out the broken url to process.

[Skallwar]

@dream-1ab Could you make an PR out of your fix?

[dream-1ab]

this fix already PR by someone.
from here: #240