Skip to content

Propagate recursive collection errors - #3

Closed
sen-ye wants to merge 1 commit into
lrt950:feat/recursive-link-checkingfrom
sen-ye:fix/recursive-collection-errors
Closed

sen-ye wants to merge 1 commit into
lrt950:feat/recursive-link-checkingfrom
sen-ye:fix/recursive-collection-errors

Conversation

@sen-ye

@sen-ye sen-ye commented Sep 8, 2026

Copy link
Copy Markdown

Companion fix for lycheeverse#2283, related to the recursive-feature bounty in lycheeverse#78.

When a discovered page passes its link check but its subsequent content fetch fails, recursive collection currently logs a warning, discards the failure, and can still exit successfully. This patch routes recursive Result<Request, RequestError> values through the existing error handling alongside initial inputs. Fatal content errors propagate to the CLI; malformed extracted URLs become normal error responses while valid links continue to be checked. Wait guards and traversal depth are preserved through both paths.

Two localhost HTTP regressions reproduce the problem before the fix:

  • A child responds 200 for the link check and 503 for content collection. The unchanged branch exits 0 with 1 OK / 0 Errors; the fix exits 1 with the 503 diagnostic. The test verifies exactly two child requests.
  • A child contains 17 malformed URLs and one valid link, exceeding the partition buffer with concurrency 1. The unchanged branch exits 0 with 2 OK / 0 Errors; the fix exits 2, reports all 17 parse errors, and checks the valid link exactly once.

Validation on Rust 1.98.1:

  • cargo test --locked -p lychee --features check_example_domains --test cli test_recursive_link_checking -- --nocapture: 8 passed.
  • cargo fmt --all --check: passed.
  • cargo clippy --locked --all-targets --all-features -- -D warnings: passed.
  • cargo test --locked --no-fail-fast -- --test-threads=4: 600 passed, 1 failed, 1 ignored, including all 9 doctests passing. The sole failure is the unchanged lychee-lib::client::tests::test_youtube; its real thumbnail endpoint also timed out in a separate bounded connectivity check. Library and test-utils source files match the original feature commit byte-for-byte. The complete workspace suite is therefore not claimed to pass.

Public CI on this exact commit: run 34235850791. Lint, feature-flag checks, publish-check and typos passed. All eight recursive CLI tests passed there too. The test job stopped at the unchanged test_large_file_lazy_download, which received HTTP 429 from its external download host; that command does not enable recursion. Nextest reported 210 passed and 1 failed, with 566 tests not run after fail-fast. The rate-limited request has not been retried.

Integration note: this is based on 9623a95776f87151c319f47b99c5553adfeeb7e1. When combining it with the request-deduplication companion in #1, deduplicate Ok(Request) values and pass Err(RequestError) values to the shared error handler. Combined-stack execution has not been performed.

AI disclosure: OpenAI Codex authored and tested this patch on behalf of @sen-ye. The supervising agent reviewed the complete diff, and a separate agent reviewed asynchronous stream and termination behavior. Authorship and code review were performed by agents; @sen-ye is the account owner.

Route recursive results through the existing error handling and add local HTTP regressions for content-fetch failures and malformed links.

AI-authored and tested by OpenAI Codex on behalf of sen-ye.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant