Skip to content

refactor: reuse boilerplate text filter#31

Open
jordansilly77-stack wants to merge 1 commit into
Rekin226:mainfrom
jordansilly77-stack:refactor/rag-boilerplate-filter
Open

refactor: reuse boilerplate text filter#31
jordansilly77-stack wants to merge 1 commit into
Rekin226:mainfrom
jordansilly77-stack:refactor/rag-boilerplate-filter

Conversation

@jordansilly77-stack

Copy link
Copy Markdown

Summary

  • remove the duplicated nested boilerplate-text filter in the RAG vector store path
  • reuse the existing module-level _is_boilerplate_text helper for the same filtering step

Verification

  • .venv/bin/python -m py_compile srcs/chatbot.py
  • git diff --check
  • .venv/bin/python -m pytest -q (100 passed, 2 skipped)

Closes #19

@Rekin226 Rekin226 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — clean refactor that kills a genuine duplication.

What's right

  • build_vector_store now routes through the canonical _is_boilerplate_text instead of its own private copy, so ingest-time and retrieval-time filtering finally agree.
  • The two things that actually matter are preserved: the getattr(d, "page_content", "") guard and the if filtered: docs = filtered fallback (don't filter every chunk away).
  • Forward reference is fine — _is_boilerplate_text is only called at runtime. py_compile + full suite green (100 passed).

One note (not blocking): this isn't byte-for-byte equivalent, so 'the same filtering step' slightly understates it:

  • The old closure matched the bare substring "terms"; the canonical set uses "terms of use"/"terms and conditions", so generic 'terms' chunks now survive (fewer false positives — good).
  • _is_boilerplate_text also adds the short-line-ratio heuristic the old closure lacked, so ingest filtering is stricter in that dimension.

Net effect is exactly the point — both paths now use one stricter helper. Worth a one-line test pinning that a 'terms'-only chunk now survives ingest, but that can be a follow-up. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collapse the duplicated boilerplate-text filter in the RAG layer

2 participants