fair-scan

A two-phase fair scan over a grouped document corpus, where group order never decides results.

No dependencies. Python 3.9+. MIT.

---

The bug this was written to fix

I was building a retrieval layer over a local corpus of documentation, organised into groups — one group per subject family. Search worked. It returned plausible results, cited its sources, and never threw an error.

Then I asked it a question about Unreal Engine's FBX import, and it answered me confidently using Unity documentation.

The cause was one optimisation that looked entirely reasonable:

for group in groups:                    # groups walked IN ORDER
    for source in group:
        snippets.extend(search(source))
    if len(snippets) >= top_k * 2:
        break                           # "we have enough, stop early"

The first group held roughly 3,200 documents. Almost any query matched something in it. So the early break almost always fired before reaching group two — and every group after the first was effectively unsearchable. Not erroring. Not returning empty. Just silently invisible, while the system reported success.

That is the failure mode worth caring about: it doesn't look like a bug, it looks like an answer. Adding a new group to the corpus made it permanently unreachable, and nothing anywhere said so.

The fix

Split the scan into two phases and never let ordering carry weight.

Phase A — score everything, cheaply. Score every source in every group using only its manifest and README. No content files are opened. This is fast enough to run exhaustively, so it does.

Phase B — deep-read a bounded selection, fairly. Open only the highest-scoring sources, capped by a fixed budget. Then the guarantee that matters:

# Every group's best positive-scoring source is always deep-read,
# even when stronger groups would otherwise consume the whole budget.
best_per_group = {}
for candidate in positive:
    best_per_group.setdefault(candidate.group, candidate)
chosen.update(best_per_group.values())

A large group can no longer crowd out a small one. A group added tomorrow is reachable today.

Cost: none. The naive version deep-read every source in the first group on nearly every query. The fair version reads at most budget sources total — it is faster on the common case while being correct on the case that mattered.

See it fail, then see it pass

python demo.py

BEFORE  ordered scan with an early break
  sources consulted : ['UNITY_000', 'UNITY_001', 'UNITY_002', 'UNITY_003']
  found the answer  : NO
  -> answered confidently from the wrong engine's documentation.
     No error. No empty result. Just wrong.

AFTER   two-phase fair scan
  sources consulted : ['UNREAL_001', 'UNITY_000', ...]
  found the answer  : yes
  top result        : UNREAL_001 (UNREAL_DOCS)

Group order swapped -> identical results: yes

demo.py keeps the original buggy implementation next to the fixed one and runs both against the same corpus, so the difference is reproducible rather than asserted.

Usage

from fairscan import scan

result = scan(
    "unreal interchange fbx import options",
    group_roots=["corpus/UNITY_DOCS", "corpus/UNREAL_DOCS", "corpus/BLENDER_DOCS"],
    top_k=4,
)

for snippet in result.snippets:
    print(f"{snippet.source_id} ({snippet.group}): {snippet.text[:80]}")

Expected corpus layout — each source is a directory with a manifest.json and/or README.md describing it, plus any text content:

corpus/
  UNREAL_DOCS/            <- a group
    UNREAL_001/           <- a source
      manifest.json       <- Phase A reads this
      README.md           <- ...and this
      content/notes.md    <- Phase B reads this, only if selected

Access control

Pass access_filter to exclude sources. It is enforced in both phases — a denied source is never opened, never scored, never returned. Fail-closed, and there is a test that says so.

scan(query, roots, access_filter=lambda source_id: user.may_read(source_id))

Tuning and escape hatch

budget=12	max sources Phase B may open
top_k=4	returns up to 2*top_k, so callers can re-rank with a costlier model
FAIRSCAN_LEGACY=1	deep-read every candidate — exhaustive, slower, useful for confirming the budget isn't hiding a result

The escape hatch drops the budget but never restores the ordered early break. That behaviour is gone permanently, because it was the defect.

Tests

pip install -e ".[dev]"
pytest

Ten tests, all hermetic — corpus built in tmp_path, no network, no home directory, no ambient config. They run identically anywhere.

Test	Locks in
test_last_group_is_not_shadowed_by_a_noisy_first_group	the original live defect
test_group_order_does_not_change_results	the core property
test_every_group_root_is_visited	Phase A is exhaustive
test_deep_scan_budget_is_bounded	Phase B stays cheap
test_zero_scoring_sources_are_never_opened	non-matches cost zero reads
test_access_filter_is_enforced_in_both_phases	fail-closed access gate
test_legacy_escape_hatch_scans_exhaustively	hatch works, break stays dead
test_explicit_source_id_in_query_always_qualifies	naming a source surfaces it
test_empty_query_returns_nothing	no query, no guessing
test_missing_group_root_is_skipped_not_fatal	a bad path degrades, not crashes

Where this came from

Extracted from the retrieval layer of a larger personal project and generalised. The defect, the fix, and the tests are the real ones, found and repaired in production use on 27 July 2026.

Licence

MIT — see LICENSE.