Conversation
Route recursive results through the existing error handling and add local HTTP regressions for content-fetch failures and malformed links. AI-authored and tested by OpenAI Codex on behalf of sen-ye.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion fix for lycheeverse#2283, related to the recursive-feature bounty in lycheeverse#78.
When a discovered page passes its link check but its subsequent content fetch fails, recursive collection currently logs a warning, discards the failure, and can still exit successfully. This patch routes recursive
Result<Request, RequestError>values through the existing error handling alongside initial inputs. Fatal content errors propagate to the CLI; malformed extracted URLs become normal error responses while valid links continue to be checked. Wait guards and traversal depth are preserved through both paths.Two localhost HTTP regressions reproduce the problem before the fix:
1 OK / 0 Errors; the fix exits 1 with the 503 diagnostic. The test verifies exactly two child requests.2 OK / 0 Errors; the fix exits 2, reports all 17 parse errors, and checks the valid link exactly once.Validation on Rust 1.98.1:
cargo test --locked -p lychee --features check_example_domains --test cli test_recursive_link_checking -- --nocapture: 8 passed.cargo fmt --all --check: passed.cargo clippy --locked --all-targets --all-features -- -D warnings: passed.cargo test --locked --no-fail-fast -- --test-threads=4: 600 passed, 1 failed, 1 ignored, including all 9 doctests passing. The sole failure is the unchangedlychee-lib::client::tests::test_youtube; its real thumbnail endpoint also timed out in a separate bounded connectivity check. Library and test-utils source files match the original feature commit byte-for-byte. The complete workspace suite is therefore not claimed to pass.Public CI on this exact commit: run 34235850791. Lint, feature-flag checks, publish-check and typos passed. All eight recursive CLI tests passed there too. The test job stopped at the unchanged
test_large_file_lazy_download, which received HTTP 429 from its external download host; that command does not enable recursion. Nextest reported 210 passed and 1 failed, with 566 tests not run after fail-fast. The rate-limited request has not been retried.Integration note: this is based on
9623a95776f87151c319f47b99c5553adfeeb7e1. When combining it with the request-deduplication companion in #1, deduplicateOk(Request)values and passErr(RequestError)values to the shared error handler. Combined-stack execution has not been performed.AI disclosure: OpenAI Codex authored and tested this patch on behalf of @sen-ye. The supervising agent reviewed the complete diff, and a separate agent reviewed asynchronous stream and termination behavior. Authorship and code review were performed by agents; @sen-ye is the account owner.