refactor: reuse boilerplate text filter#31
Open
jordansilly77-stack wants to merge 1 commit into
Open
Conversation
Rekin226
approved these changes
Jun 26, 2026
Rekin226
left a comment
Owner
There was a problem hiding this comment.
Approving — clean refactor that kills a genuine duplication.
What's right
build_vector_storenow routes through the canonical_is_boilerplate_textinstead of its own private copy, so ingest-time and retrieval-time filtering finally agree.- The two things that actually matter are preserved: the
getattr(d, "page_content", "")guard and theif filtered: docs = filteredfallback (don't filter every chunk away). - Forward reference is fine —
_is_boilerplate_textis only called at runtime.py_compile+ full suite green (100 passed).
One note (not blocking): this isn't byte-for-byte equivalent, so 'the same filtering step' slightly understates it:
- The old closure matched the bare substring
"terms"; the canonical set uses"terms of use"/"terms and conditions", so generic 'terms' chunks now survive (fewer false positives — good). _is_boilerplate_textalso adds the short-line-ratio heuristic the old closure lacked, so ingest filtering is stricter in that dimension.
Net effect is exactly the point — both paths now use one stricter helper. Worth a one-line test pinning that a 'terms'-only chunk now survives ingest, but that can be a follow-up. LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_is_boilerplate_texthelper for the same filtering stepVerification
.venv/bin/python -m py_compile srcs/chatbot.pygit diff --check.venv/bin/python -m pytest -q(100 passed, 2 skipped)Closes #19