feat: geny-memory-adaptor 1.8.0~1.11.0 이식

CHANGELOG.md

@@ -1,5 +1,161 @@
 # Changelog
 
+> 아래 1.8.0 ~ 1.11.0 은 `geny-memory-adaptor` 에서 이식했다. XGEN 은 Geny 에서
+> 갈라져 독자적으로 가는 프로젝트이지만, **같은 뿌리에서 온 결함은 여기에도
+> 그대로 있다.** 검증 방법(테스트)까지 함께 가져왔고, 이 저장소의 기존 코드와
+> 충돌하는 부분은 없었다 — 갈라진 이후의 차이는 전부 포매팅이었다.
+
+## [1.11.0] — 2026-08-11
+
+### Changed (one embedder table per process, not one per session)
+Measured on a production host: an engine opened over an **empty** vault still
+cost 68 MB of RSS, and six live vaults held only **two** distinct embedder
+tables between them. The table is `vocab_size × dim × 4B` — 64 MB at the
+defaults — so a host keeping ten sessions resident spent 640 MB on ten copies
+of identical numbers. That, not the stored memories, was what made "keep every
+session awake" expensive.
+
+The table is now shared process-wide, keyed on what determines its contents:
+the generator arguments for a fresh table, the digest of the persisted blob for
+a restored one.
+
+Sharing is safe because the table is never written in place — it is read, and
+distillation *replaces* the whole embedder with a scratch instance holding its
+own array. `setflags(write=False)` makes that structural rather than
+conventional: an in-place write now raises instead of silently corrupting every
+other session that shares the array.
+
+- `HashEmbedder(..., table=)` lets `loads()` skip generating a 64 MB table it
+  was about to throw away.
+- `shared_table_stats()` reports what the process holds, for host health
+  endpoints.
+- Cache is bounded (8 tables) so a content-keyed dict cannot leak.
+
+Measured against four real vaults in one process: **330.4 MB → 137.2 MB**, and
+the marginal cost of one more resident session falls from 64–87 MB to
+0–23 MB — now purely proportional to what that session actually remembers.
+
+## [1.10.0] — 2026-08-10
+
+### Added (a catalogue, so a browser can ask small questions)
+A vault browser asks in order: how much is there, what days, what is on this
+day, what does this note say. The host answered the first three by
+materialising the whole vault — 3.2 s and 4.8 MB of bodies held in memory to
+produce one count, because its only listing primitive walks every note. The
+index already holds exactly the metadata those questions need.
+
+- `catalog_counts(by="kind"|"day", kind=...)` — one GROUP BY, no bodies.
+- `catalog_page(day=, kind=, limit=, offset=)` — one page of note metadata,
+  filtered and paged in SQL rather than sliced after the fact.
+- `neighbourhood(node_ids, depth=, max_nodes=, max_edges=)` — the subgraph
+  around a selection, both bounds enforced and `truncated` reported. The
+  whole-vault snapshot it replaces was 5,384 nodes and 4.3 MB of JSON for a
+  single screen.
+- `Store.edges_touching(node_ids, limit=)` underneath it.
+
+## [1.9.3] — 2026-08-10
+
+### Fixed (the vault could no longer notice its own conversion)
+1.9.0 recorded the new geometry against rows that had been derived by the
+OLD tokenizer. From that moment the fingerprints matched, so 1.9.1 and 1.9.2
+— both correct in themselves — had nothing to compare against, and the
+production vault stayed on the old tokenization with every check reporting
+agreement.
+
+- `_GEOMETRY_VERSION` joins the fingerprint. Config fields cannot express
+  "the stemmer changed"; an explicit version can, and it is the lever for
+  every future change of meaning. Bumped to 2 for the Porter/trigram move,
+  so every vault re-derives exactly once.
+
+## [1.9.2] — 2026-08-10
+
+### Fixed (an unrecorded geometry counted as a matching one)
+1.9.1 recorded the geometry on open and cleared the digests when it changed
+— but skipped the clear when there was NO previous record, to avoid a mass
+re-index on upgrade. That is precisely the upgrade case: a vault written
+before the geometry was tracked was derived by some tokenization nobody can
+name. Production upgraded, recorded the new geometry against untouched rows,
+and re-indexed nothing.
+
+Unknown provenance now counts as stale. Derived data is rebuildable; a wrong
+assumption of freshness is not.
+
+## [1.9.1] — 2026-08-10
+
+### Fixed (a geometry change never reached the host)
+1.9.0 put the tokenizer's geometry into `content_sha`, which was necessary
+and not sufficient: the digest is only consulted for notes a HOST decides to
+offer, and a host that diffs on timestamps never offers an untouched note.
+The production upgrade re-indexed **nothing** — "0 indexed" — and the vault
+sat half-converted, its postings derived one way while queries were analysed
+another, with every signal saying it was in sync.
+
+- The geometry is now recorded in `params`. On open, a change clears every
+  row's `content_sha` (one UPDATE) so `manifest()` reports them as
+  "indexed, derived state unknown" — the signal a host can actually act on.
+- Reopening with the SAME geometry leaves the digests alone, so an ordinary
+  restart still costs nothing.
+
+## [1.9.0] — 2026-08-10
+
+### Added (Latin morphology — the stream that never had any)
+Korean got a guarded 조사/어미 stripper early. The Latin side never did:
+character trigrams over every non-Hangul word ≥4 chars stood in for it, so
+"browsing" and "browse" met because they share *brow, row, ows*, not because
+anything understood them.
+
+- `latin.py` — Porter's algorithm, deterministic and dictionary-free, on the
+  same ADDITIVE contract as the Korean stripper: the surface form is indexed
+  too, so an over-eager conflation costs one posting and never an exact
+  match.
+
+### Changed (trigrams move to the stream they belong in)
+Measured on a 5,347-note production vault, Latin trigrams were **72.8% of
+all postings** — 59.6% of that from fifty boilerplate types (`execution`,
+`screen`, `title`, `tags`, …) whose IDF is ~0, so they cost storage and
+contributed nothing to ranking.
+
+- `latin_ngram_min_len` defaults to `0`: out of BM25, exactly where jamo
+  already sits. `embed_tokens` asks for them explicitly, so fuzzy matching
+  stays in the recall stream at no posting cost.
+- Paired A/B on one frozen snapshot, old defaults vs new:
+  postings 1,221,979 → **442,609 (−64%)**; English known-item MRR
+  verbatim 0.583 → 0.579, **re-inflected 0.372 → 0.461 (+24%)**,
+  typo 0.411 → 0.419; Korean flat. MIRACL-ko floor unchanged and passing.
+- `eval/known_item.py` — known-item retrieval over a real vault with
+  verbatim / re-inflected / typo query flavours, reported per language.
+  Gold is known by construction, so no judgements are needed and the actual
+  corpus can be used.
+
+### Fixed (a tokenizer change left a half-converted index)
+`content_sha` covered the embedding geometry but not the tokenizer's. Flip
+`latin_stemming` or `char_ngrams` and the digest said "unchanged": nothing
+re-indexed, and the stored postings quietly stopped agreeing with how
+queries were analysed. Tokenizer settings are now part of the digest, so a
+change invalidates exactly the rows it should.
+
+## [1.8.0] — 2026-08-09
+
+### Changed (read/write lock, and a bound on waiting for it)
+One mutex guarded everything, so every search queued behind every index —
+15.9 ms idle vs 63.6 ms during indexing on a production vault.
+
+- Readers now run together; writers stay exclusive. FAIR (arrival order),
+  not writer-preferring: the first cut preferred writers, and against a
+  continuous indexing load the writer re-queued the instant it released, so
+  searches never got a turn and the benchmark never finished. Both
+  starvation directions are now covered by tests.
+- `search(timeout=)`, default `SEARCH_TIMEOUT_S = 20 s`, raising the new
+  `MemoryBusy`. This — not the lock split — is what bounds a WEDGED write:
+  an exclusive writer that never returns still blocks readers, so the only
+  remedy is to stop waiting. A caller that can answer without memory should;
+  one that cannot passes `timeout=None`.
+- The lock is NOT reentrant, unlike the `RLock` it replaces. Two call paths
+  self-deadlocked the moment it was swapped (`feedback`/`learn` →
+  `trust_feedback`, and `contradictions` → `get_text`); both are now split
+  into a locked entry point and an unlocked internal. Lazy cache builders
+  nest, so their build mutex stays reentrant on purpose.
+
 ## [1.6.0] — 2026-07-30
 
 ### Changed (postings storage: integer keys — 6.4× smaller vaults)

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "xgen-agent-memory"
-version = "1.7.0"
+version = "1.11.0"
 description = "Synapse — a learnable, lightweight graph-traversal memory engine: BM25 + local static embeddings + typed-edge PageRank + an online-learned ranker, in one SQLite file with zero API calls."
 readme = "README.md"
 license = "Apache-2.0"

src/xgen_agent_memory/__init__.py

@@ -14,13 +14,15 @@
 """
 
 from .config import SynapseConfig, load_dotenv
+from ._rwlock import MemoryBusy
 from .engine import SearchHit, SynapseMemory
 from .executor_adapter import SynapseRetriever, SynapseVectorHandle
 from .ranker import FEATURES
 
-__version__ = "1.7.0"
+__version__ = "1.6.0"
 
 __all__ = [
+    "MemoryBusy",
     "SynapseMemory",
     "SynapseConfig",
     "SearchHit",

src/xgen_agent_memory/_rwlock.py

@@ -0,0 +1,148 @@
+"""A many-readers/one-writer lock for the engine's in-memory state.
+
+One mutex over everything is correct and, for a small vault, invisible. It
+stops being invisible when a write takes real time: every search queues
+behind it. Measured on a production vault, a search cost 15.9 ms idle and
+63.6 ms while indexing ran — and when a write wedged entirely (a matmul
+spinning inside a broken BLAS thread pool), every read behind it wedged too.
+
+Readers don't conflict with each other, so letting them run together removes
+the queue between searches.
+
+What this does NOT do, and it matters: an exclusive writer that never
+finishes still blocks every reader. Splitting the lock does not shrink the
+blast radius of a wedged write — only ``timeout=`` does. Retrieval paths
+should pass one and degrade to "no memory" rather than inherit someone
+else's hang; that is what turned a single stuck matmul into a 27-hour
+outage with every health signal green.
+
+FAIR (first-come, first-served), which is not the obvious choice and is worth
+saying why. The first cut was writer-preferring — any waiting writer held new
+readers back — on the reasoning that a read-mostly engine would otherwise
+postpone indexing forever. Measured against a continuous indexing load, that
+inverted the problem completely: the writer re-queued the instant it released,
+so searches never got a turn at all and the benchmark never finished. Neither
+side may starve here, so arrival order decides. Readers that queued together
+still run together; a reader waits only for writers that arrived *before* it.
+
+NOT reentrant. The single ``RLock`` it replaces tolerated a locked method
+calling another locked method; here that self-deadlocks, so every such pair
+must be split into a public locked entry point and an unlocked internal
+(``_get_text_unlocked``, ``_trust_feedback_unlocked``).
+"""
+
+from __future__ import annotations
+
+import threading
+import time
+from contextlib import contextmanager
+from typing import Iterator, List, Optional
+
+
+class MemoryBusy(RuntimeError):
+    """Raised when a bounded acquisition ran out of patience.
+
+    A caller that can proceed without memory (retrieval on a turn) should
+    catch this and do so. A caller that cannot should let it propagate — an
+    error beats a hang, because a hang has no signal at all.
+    """
+
+
+class RWLock:
+    """Many concurrent readers, or one exclusive writer, in arrival order."""
+
+    __slots__ = ("_cond", "_readers", "_writer", "_ticket", "_waiting_readers", "_waiting_writers")
+
+    def __init__(self) -> None:
+        self._cond = threading.Condition(threading.Lock())
+        self._readers = 0
+        self._writer = False
+        self._ticket = 0
+        # Arrival-ordered tickets of everyone still waiting. Comparing a
+        # waiter's ticket against these is what makes the order fair.
+        self._waiting_readers: List[int] = []
+        self._waiting_writers: List[int] = []
+
+    # ── acquisition ──────────────────────────────────────────────────
+    @contextmanager
+    def read(self, timeout: Optional[float] = None) -> Iterator[None]:
+        deadline = None if timeout is None else time.monotonic() + timeout
+        with self._cond:
+            mine = self._ticket
+            self._ticket += 1
+            self._waiting_readers.append(mine)
+            try:
+                # Wait out any writer already running, and any writer that got
+                # in line first. Writers queued *behind* this reader do not
+                # hold it back — otherwise a steady write load starves reads.
+                while self._writer or (self._waiting_writers and self._waiting_writers[0] < mine):
+                    if deadline is None:
+                        self._cond.wait()
+                        continue
+                    left = deadline - time.monotonic()
+                    if left <= 0 or not self._cond.wait(left):
+                        if self._writer or (
+                            self._waiting_writers and self._waiting_writers[0] < mine
+                        ):
+                            raise MemoryBusy(
+                                f"memory is busy writing; gave up after {timeout:.1f}s"
+                            )
+            finally:
+                self._waiting_readers.remove(mine)
+                # A writer may have been waiting solely on this reader's
+                # place in line; leaving the queue changes its answer.
+                self._cond.notify_all()
+            self._readers += 1
+        try:
+            yield
+        finally:
+            with self._cond:
+                self._readers -= 1
+                if self._readers == 0:
+                    self._cond.notify_all()
+
+    @contextmanager
+    def write(self, timeout: Optional[float] = None) -> Iterator[None]:
+        deadline = None if timeout is None else time.monotonic() + timeout
+        with self._cond:
+            mine = self._ticket
+            self._ticket += 1
+            self._waiting_writers.append(mine)
+            try:
+                # Exclusive: no other writer, no active reader, and nobody
+                # ahead in line — including readers, so a busy read load
+                # cannot postpone indexing indefinitely.
+                def _blocked() -> bool:
+                    return bool(
+                        self._writer
+                        or self._readers
+                        or (self._waiting_writers and self._waiting_writers[0] != mine)
+                        or (self._waiting_readers and self._waiting_readers[0] < mine)
+                    )
+
+                while _blocked():
+                    if deadline is None:
+                        self._cond.wait()
+                        continue
+                    left = deadline - time.monotonic()
+                    if (left <= 0 or not self._cond.wait(left)) and _blocked():
+                        raise MemoryBusy(f"memory is busy; gave up after {timeout:.1f}s")
+            finally:
+                self._waiting_writers.remove(mine)
+                self._cond.notify_all()
+            self._writer = True
+        try:
+            yield
+        finally:
+            with self._cond:
+                self._writer = False
+                self._cond.notify_all()
+
+    # ── introspection (tests, diagnostics) ───────────────────────────
+    @property
+    def readers(self) -> int:
+        return self._readers
+
+    @property
+    def writing(self) -> bool:
+        return self._writer

src/xgen_agent_memory/config.py

@@ -87,6 +87,21 @@ class SynapseConfig:
     suffix_strip: bool = True
     #: Cross-space syllable bigrams (붙여쓰기 robustness) in the BM25 stream.
     cross_space: bool = True
+    #: Minimum length for a NON-Hangul word to also emit character 3-grams
+    #: into the BM25 stream. ``0`` (default) keeps them out of it entirely.
+    #:
+    #: They used to be unconditional and were 72.8% of a production vault's
+    #: postings — 59.6% of that from fifty boilerplate types (`execution`,
+    #: `screen`, `title`, `tags`, …) whose IDF is ~0, so they cost storage
+    #: and contributed nothing to ranking. With a real stemmer in place the
+    #: measured trade is: postings −64%, known-item MRR +4% verbatim /
+    #: +2% re-inflected, −12.5% on transposed-character typos. Typo
+    #: tolerance still lives in the EMBEDDING stream, which keeps them
+    #: (``embed_tokens`` does not take this knob) — the same split that
+    #: already keeps jamo out of BM25.
+    latin_ngram_min_len: int = 0
+    #: Porter stemming for Latin-script words. Additive (surface AND stem).
+    latin_stemming: bool = True
     #: BM25F-lite: title terms count this many times in the postings.
     title_boost: float = 2.0
     #: Per-document token cap (indexing) / per-query cap.