Overnight/test harness 2026 04 20

AGENTS.md

@@ -65,12 +65,24 @@ CI quirks (`.github/workflows/test.yml`) that are easy to miss:
   `*_branches_test.go`. When fixing a bug or adding a branch, add a
   matching test — the iterate loop treats coverage growth as the default
   improvement signal.
+- **Hermetic test configs.** Tests that build a `config.Default()`
+  for `engine.New` / `daemon.New` MUST clear the user-level XDG paths
+  (`IdentityPath`, `ReputationPath`, `SeedListPath`, `BloomPath`,
+  `TrustPath`) — leaving them at the defaults points the test at
+  `~/.local/share/swartznet/*` and parallel tests then race on
+  shared on-disk state. The pattern is to set each to `""` (the
+  engine treats empty paths as "skip this subsystem"). Look at
+  `internal/engine/create_test.go`'s `newTestEngine` for the
+  canonical setup. The `TrustPath` / `BloomPath` are sometimes set
+  to a path inside `t.TempDir()` instead of `""` when the test
+  needs that subsystem live.
 
 ## Repository layout
 
 - `cmd/swartznet` — CLI. `main.go` dispatches to `cmd_*.go` per
   subcommand (`add`, `search`, `create`, `index`, `files`, `flag`,
-  `status`, `trust`, `confirm`).
+  `status`, `trust`, `confirm`, `aggregate {inspect,find,build}`,
+  `crawl-probe`).
 - `cmd/swartznet-gui` — Fyne-based GUI; thin shim over `internal/gui`.
 - `cmd/dht-smoke` — standalone DHT probe for ops.
 - `internal/daemon` — **the wiring point.** `daemon.New(ctx, Options)`

CHANGELOG.md

@@ -15,8 +15,131 @@ one second client implementing `sn_search` (the BEP-1
 requirement to take a draft to Final). Both require
 engagement from actual users of the v0.x prereleases.
 
+### Added — "Aggregate" distributed-layer redesign
+
+A multi-commit series inverts Layer-D from per-keyword BEP-44
+items into per-publisher pointers, moves the real keyword index
+into a piece-aligned B-tree inside a regular BitTorrent
+"companion index torrent", and reconciles updates between peers
+via a new Rateless IBLT set-sync session layered on `sn_search`.
+Design and rationale in
+[`docs/research/PROPOSAL.md`](docs/research/PROPOSAL.md);
+byte-level spec in [`docs/research/SPEC.md`](docs/research/SPEC.md);
+phase-by-phase tracking in
+[`docs/research/ROADMAP.md`](docs/research/ROADMAP.md).
+
+Shipped components:
+
+  - `internal/companion/btree.go` — B-tree page layout
+    (magic `SNAGG\0`, interior/leaf/trailer kinds) with
+    deterministic build, BFS piece assignment, ed25519-signed
+    trailer binding the tree to a publisher.
+  - `internal/companion/read_btree.go` — prefix-query walker
+    with trailer-sig verification and per-record sig re-check
+    on every returned hit.
+  - `internal/companion/pow.go` — hashcash mint +
+    `SignAndMineRecord` convenience (D=20 default).
+  - `internal/dhtindex/ppmi.go` — Publisher Pointer Mutable
+    Item. One BEP-44 item per publisher at the fixed
+    double-hashed salt `SHA256("snet.index")`. Collapses DHT
+    cost from O(publishers × keywords) to O(publishers).
+  - `internal/dhtindex/lookup.go` — `Lookup.Query` resolves
+    PPMIs first, falls back to legacy per-keyword for
+    publishers that haven't migrated. Dual-read migration.
+  - `internal/swarmsearch/riblt.go` — rateless IBLT encoder/
+    decoder (SIGCOMM 2024) with graduated-degree cycle.
+    Converges in ~3d symbols for symmetric difference d.
+  - `internal/swarmsearch/sync_{wire,session}.go` — msg_types
+    4-8 wire format and in-process session state machine.
+  - `internal/swarmsearch/handler.go` — LTEP dispatch for
+    sync msg_types behind the `BitSetReconciliation` (services
+    bit 9) capability gate. Peers without the bit receive
+    `reject code 2`.
+  - `internal/daemon/bootstrap.go` + `bootstrap_https.go` —
+    three-channel cold-start (anchors + BEP-51 candidates +
+    `peer_announce` endorsements) plus last-ditch HTTPS anchor
+    fallback.
+
+Test coverage: ~130 new test functions across the four
+packages, including the capstone `TestAggregateEndToEnd` that
+exercises publisher → PPMI → subscriber → prefix-query in one
+pass.
+
+Status of the BEP-51 crawler track: the primitives now land
+incrementally — `dhtindex.SampleInfohashes` (raw BEP-51 query),
+`dhtindex.PublisherFromMetainfo` (signature classifier), and
+`dhtindex.CrawlOnce` (one-tick glue: sample + fetch + classify
++ sink). A `swartznet crawl-probe --addr <host:port>` ops
+command exercises the primitive against a live peer.
+
+The remaining gap is a *production* MetainfoFetcher: BEP-9
+ut_metadata only transports the info dict, while
+`snet.pubkey` / `snet.sig` are top-level metainfo fields, so
+a BEP-9-only fetcher can't promote crawled infohashes to the
+"observed publishers" set without an additional signature-
+exchange channel. Closing the gap needs either a new LTEP
+extension (e.g. `ut_signature`) or a tracker / HTTP mirror
+convention. Documented in
+[`docs/research/MILESTONE-v0.5.0.md`](docs/research/MILESTONE-v0.5.0.md).
+LocalRecord sync was wired up in earlier commits so nodes do
+share records over the responder path; the engine attaches a
+RecordCache as both source and sink in `engine.New`.
+
+### Added — `swartznet crawl-probe` ops command
+
+One-shot CLI that issues a single BEP-51 `sample_infohashes`
+query against a DHT address and prints the response (samples,
+interval, num, closest nodes). Pure ops tooling — no running
+daemon needed. Useful for validating that a peer supports
+BEP-51 and for hand-inspecting the samples it volunteers
+during Channel-B crawler development. Text + JSON output.
+
+### Added — Engine periodic RecordCache prune
+
+`engine.DefaultRecordCacheMaxAge` (30 days) +
+`DefaultRecordCachePruneInterval` (1 hour). On startup
+`engine.New` launches a goroutine that calls
+`RecordCache.PruneOlderThan` on a ticker so the Aggregate
+cache's TTL is enforced automatically without operator
+intervention. Regtest mode swaps in 200 ms / 500 ms so
+scenario tests can observe the prune cycle.
+
 ### Fixed
 
+  - **`upgradeMagnetSession` race in `Engine.AddTorrentFile`**. The
+    metainfo-arrival upgrade goroutine was spawned for every
+    `registerLocked` caller, including `AddTorrentFile` and
+    `AddTorrentMetaInfo`. For those paths, the goroutine raced
+    `writeTorrentCopy` for the same `<hex>.torrent.tmp` target
+    file: when the goroutine won the rename, the saved bytes
+    contained a re-marshalled metainfo (anacrolix's
+    `Torrent.Metainfo()` synthesises CreationDate / CreatedBy
+    fields) that no longer byte-matched the original. On the
+    next `RestoreSession`, `metainfo.Load` rejected the file
+    with "error after decoding metainfo: expected EOF". Symptom:
+    intermittent flake under heavy parallel race testing where
+    `eng2.TorrentSnapshots()` returned 1 torrent instead of 2.
+    Fix moves the goroutine spawn out of `registerLocked` and
+    into `AddMagnet` / `AddInfoHash` only — the two paths that
+    genuinely benefit from a post-fetch metainfo upgrade. Same
+    gate added in `restoreEntry` so file/metainfo restores skip
+    the upgrade. Regression-gated by
+    `TestAddTorrentFileDoesNotRemarshalCopy` (deterministic:
+    fails reproducibly with original buggy code).
+
+  - **Anchor-source admit no-op**. `daemon.Bootstrap.admit`
+    contained a documented placeholder that called
+    `tracker.RecordReturned(pub, 0)`, which `RecordReturned`
+    silently ignores (`if n <= 0 { return }`). Anchor pubkeys
+    therefore never landed on the tracker, breaking the
+    intent of `opts.AnchorReputation` and leaving the lookup's
+    heavy-tail rule unable to identify trusted publishers.
+    Replaced with `tracker.MarkSeeded(pub, label)` for
+    `source == "anchor"` only — gives anchors the high
+    starting score the seeded-bonus branch of `scoreOf` is
+    designed to express. Candidate sources stay un-seeded
+    (default 0.5; earn or lose reputation organically).
+
   - **Layer-D BEP-44 put/get round-trip**. anacrolix/torrent's
     `NewAnacrolixDhtServer` builds a `dht.ServerConfig` from
     scratch and does NOT copy `dht.NewDefaultServerConfig`'s

cmd/swartznet/cmd_aggregate.go

@@ -0,0 +1,160 @@
+package main
+
+import (
+	"encoding/hex"
+	"flag"
+	"fmt"
+	"io"
+	"os"
+
+	"github.com/swartznet/swartznet/internal/companion"
+)
+
+// cmdAggregate dispatches the `swartznet aggregate <subcommand>`
+// tree. These subcommands are ops tooling for the v0.5.0
+// "Aggregate" redesign — they exercise the companion-index
+// B-tree path without needing a running daemon. Useful for
+// operators validating a locally-built index before publishing
+// and for subscribers inspecting index torrents they've
+// downloaded.
+func cmdAggregate(args []string, stdout, stderr io.Writer) int {
+	if len(args) == 0 {
+		fmt.Fprintln(stderr, "swartznet aggregate: missing subcommand")
+		printAggregateUsage(stderr)
+		return exitUsage
+	}
+	sub, rest := args[0], args[1:]
+	switch sub {
+	case "inspect":
+		return cmdAggregateInspect(rest, stdout, stderr)
+	case "find":
+		return cmdAggregateFind(rest, stdout, stderr)
+	case "build":
+		return cmdAggregateBuild(rest, stdout, stderr)
+	case "help", "-h", "--help":
+		printAggregateUsage(stdout)
+		return exitOK
+	default:
+		fmt.Fprintf(stderr, "swartznet aggregate: unknown subcommand %q\n\n", sub)
+		printAggregateUsage(stderr)
+		return exitUsage
+	}
+}
+
+func printAggregateUsage(w io.Writer) {
+	fmt.Fprintln(w, `swartznet aggregate — Aggregate (PPMI + B-tree) ops tooling
+
+Usage:
+  swartznet aggregate <subcommand> [args]
+
+Subcommands:
+  build  --out=FILE [flags]         Sign + pack JSONL records into a signed B-tree index.
+  inspect <index-file>              Print trailer metadata for an Aggregate index.
+  find <index-file> <prefix>        List records matching a keyword prefix.
+  help                              Print this message.
+
+The <index-file> path points at the single-file payload inside an
+Aggregate companion torrent (i.e. the bytes BuildBTree produces).
+Operators get this file by either building it locally via engine
+APIs or extracting it from a downloaded companion torrent via
+the engine's "files" listing.`)
+}
+
+// cmdAggregateInspect prints the trailer + high-level stats for
+// an Aggregate index file. Fails if the file isn't a well-formed
+// B-tree (bad magic, wrong version, bad trailer signature, etc.).
+func cmdAggregateInspect(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("aggregate inspect", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	var pieceSize int
+	fs.IntVar(&pieceSize, "piece-size", companion.MinPieceSize,
+		"piece size in bytes; must match the torrent's metainfo")
+	if err := fs.Parse(args); err != nil {
+		return exitUsage
+	}
+	if fs.NArg() != 1 {
+		fmt.Fprintln(stderr, "usage: swartznet aggregate inspect <index-file>")
+		return exitUsage
+	}
+	path := fs.Arg(0)
+
+	data, err := os.ReadFile(path)
+	if err != nil {
+		fmt.Fprintf(stderr, "swartznet: read %s: %v\n", path, err)
+		return exitRuntime
+	}
+	src := &companion.BytesPageSource{Data: data, PieceSize: pieceSize}
+	reader, err := companion.OpenBTree(src)
+	if err != nil {
+		fmt.Fprintf(stderr, "swartznet: open b-tree: %v\n", err)
+		return exitRuntime
+	}
+	tr := reader.Trailer()
+
+	fmt.Fprintln(stdout, "Aggregate index inspection")
+	fmt.Fprintf(stdout, "  file:           %s\n", path)
+	fmt.Fprintf(stdout, "  file size:      %d bytes\n", len(data))
+	fmt.Fprintf(stdout, "  piece size:     %d bytes\n", pieceSize)
+	fmt.Fprintf(stdout, "  pages:          %d\n", tr.NumPages)
+	fmt.Fprintf(stdout, "  records:        %d\n", tr.NumRecords)
+	fmt.Fprintf(stdout, "  publisher pk:   %s\n", hex.EncodeToString(tr.PubKey[:]))
+	fmt.Fprintf(stdout, "  sequence:       %d\n", tr.Seq)
+	fmt.Fprintf(stdout, "  created:        %d (unix)\n", tr.CreatedTs)
+	fmt.Fprintf(stdout, "  min PoW bits:   %d\n", tr.MinPoWBits)
+	fmt.Fprintf(stdout, "  fingerprint:    %s\n", hex.EncodeToString(tr.TreeFingerprint[:]))
+	return exitOK
+}
+
+// cmdAggregateFind runs a prefix-query and prints the matching
+// records' infohash + keyword. Mirrors what a subscriber would
+// get back from BTreeReader.Find.
+func cmdAggregateFind(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("aggregate find", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	var pieceSize int
+	var verify bool
+	fs.IntVar(&pieceSize, "piece-size", companion.MinPieceSize,
+		"piece size in bytes; must match the torrent's metainfo")
+	fs.BoolVar(&verify, "verify", false,
+		"also run VerifyFingerprint (scans every leaf; slower)")
+	if err := fs.Parse(args); err != nil {
+		return exitUsage
+	}
+	if fs.NArg() != 2 {
+		fmt.Fprintln(stderr, "usage: swartznet aggregate find [--piece-size=N] [--verify] <index-file> <prefix>")
+		return exitUsage
+	}
+	path, prefix := fs.Arg(0), fs.Arg(1)
+
+	data, err := os.ReadFile(path)
+	if err != nil {
+		fmt.Fprintf(stderr, "swartznet: read %s: %v\n", path, err)
+		return exitRuntime
+	}
+	src := &companion.BytesPageSource{Data: data, PieceSize: pieceSize}
+	reader, err := companion.OpenBTree(src)
+	if err != nil {
+		fmt.Fprintf(stderr, "swartznet: open b-tree: %v\n", err)
+		return exitRuntime
+	}
+	if verify {
+		if err := reader.VerifyFingerprint(); err != nil {
+			fmt.Fprintf(stderr, "swartznet: fingerprint verification failed: %v\n", err)
+			return exitRuntime
+		}
+	}
+	hits, err := reader.Find(prefix)
+	if err != nil {
+		fmt.Fprintf(stderr, "swartznet: find %q: %v\n", prefix, err)
+		return exitRuntime
+	}
+
+	fmt.Fprintf(stdout, "Matches for prefix %q: %d records\n", prefix, len(hits))
+	for _, h := range hits {
+		fmt.Fprintf(stdout, "  %s  %-40s  t=%d\n",
+			hex.EncodeToString(h.Ih[:]),
+			h.Kw,
+			h.T)
+	}
+	return exitOK
+}