Skip to content

docs: rewrite README to match shipped + in-flight behavior - #2

Open
arazmj wants to merge 1 commit into
mainfrom
docs/rewrite-readme
Open

docs: rewrite README to match shipped + in-flight behavior#2
arazmj wants to merge 1 commit into
mainfrom
docs/rewrite-readme

Conversation

@arazmj

@arazmj arazmj commented May 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • Rewrite README around the intended crawler + BM25 search workflow.
  • Document JSONL persistence, default store paths, same-origin crawling, crawl/search flags, and Rust 1.74+.
  • Include project structure entries for in-flight store and tokenizer modules.

Notes

This README is written to be consistent with the parallel PRs for search persistence, crawler termination, same-origin URL normalization, HTTP hardening, and tokenizer improvements. It may need a quick rebase or wording tweak if any of those land in a different shape.

Validation

  • cargo build

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@arazmj
arazmj force-pushed the docs/rewrite-readme branch from d5622c2 to 9a0a61d Compare May 22, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant