feat(source): return only new companies, in a stable order

[anusbutt]

Two changes to prospector source, both about what happens when you run it again.

1. Only new companies come back

Sourcing had no memory. Running it twice returned the same businesses both times — you'd spend Places quota and a homepage fetch on every company, to produce rows you'd then throw away by hand.

Companies already in the vault are now dropped immediately after dedupe and before the fetch loop. A repeat sweep costs Places queries and nothing else.

Verified against the live vault: replaying the 108 previously sourced companies drops all 108, with no false passes.

Why the vault, and not the send ledger

The ask was "a company I already emailed shouldn't show up again". Checking the real data first changed the design:

Gate	Suppresses (of 108 re-sourced)
Send ledger (already emailed)	51
Vault (already known)	108

Only 51 of the 108 known companies had actually been emailed. Gating on the ledger alone would have let 57 already-researched, already-drafted companies back through. The vault is a superset of the ledger, so the stated requirement still holds — you just also stop re-finding the ones sitting in your review queue.

Why company slug, and not email or domain

Key	Catches (of 62 emailed)	Available before fetch?
company slug	51	✅ yes
scraped email	51	❌ no — needs the fetch first
website domain	33	✅ yes

Slug matches as well as email does and is the only strong key available before the homepage fetch — which is what lets the gate run early, and the whole point of the change.

Domain alone is a trap: 17 of the 62 addresses in the live ledger are on free providers, where the email domain says nothing about the business. It's kept as a second key, though, because it catches what slug can't — a business whose Places listing name has drifted since it was first sourced.

Reporting

Dropped companies are counted, never silently discarded — the same lesson as the recent send-skip fix, since a silent drop is exactly what reads as a bug:

  already in vault (skipped): 40   new companies: 7

A sweep that finds nothing new says so outright rather than printing an empty table. --include-known bypasses the gate, --vault points elsewhere, and a first run with no vault suppresses nothing.

2. Output is deterministic

Places can return the same businesses in a different order on different days. That used to change two things: the row order in the CSV, and — through first-seen-wins dedupe — which of two duplicates survived.

• Rows are now sorted by company name.
• Dedupe ranks candidates by metro position, then place_id, before collapsing. Metro precedence is unchanged (bigger metros still win ties, as documented in research.md R6); what changes is that the winner no longer depends on the order the API answered in.

An unchanged result set now produces an unchanged file, so two candidate CSVs can be diffed meaningfully.

How it was verified

• pytest -q → 607 passed (+10).
• The gate replayed against the real 115-note vault: 108/108 dropped, 0 false passes.

The determinism tests shuffle the Places response and assert a byte-identical CSV, rather than simply running twice — a run-twice test passed before this change and proved nothing. One test covers the duplicate-domain case specifically, since that's where reordering used to swap the winner.

Gate tests cover: known company dropped; its homepage never fetched (the actual point); renamed company still caught by domain; every-company-known returns a header-only file; --include-known bypasses; missing vault suppresses nothing; _Dashboard.md isn't mistaken for a company.

Notes for reviewers

source now reads the vault, which it previously did not touch at all. That's a new (mild) coupling between the two stages — it takes --vault and defaults to settings.vault_dir.

The gate is a heuristic and can miss. A renamed business with a new domain slips through and appears again. That fails safely: worst case is a duplicate row in a CSV you review before feeding it to run. It cannot cause a duplicate email — the send ledger still owns that guarantee and is untouched here.

🤖 Generated with Claude Code