Read-path scaling and observability: bounded candidates, tenant-filter pushdown, segment reranking, live metrics

.gitignore

@@ -42,3 +42,7 @@ paper/*.out
 # Keep curated benchmark logs (committable evidence)
 !results/
 results/*.tmp
+
+# 第三方评测数据集：按 eval/locomo.py 的文档下载到此路径，2.8MB，不入库。
+# .gitignore 里原本只有 data/，挡住的是备选路径 eval/data/locomo10.json，主路径漏网。
+eval/locomo10.json

API.md

@@ -38,6 +38,50 @@ Authorization: Bearer <你的key>
 Content-Type: application/json
 ```
 
+密钥有两种来源，解析顺序为「运行时签发的密钥 → `ENGRAM_API_KEYS` 静态映射 → `ENGRAM_OPEN` 开放模式」。
+运行时密钥优先，所以吊销不会被残留的环境变量条目复活。
+
+### 1.1 运行时签发密钥（不用重启）
+
+静态映射要改环境变量再重启。托管部署可以用管理端点在运行中签发和吊销。**管理面默认不存在**——
+不设 `ENGRAM_ADMIN_TOKEN` 时这几个端点一律 403，所以开放模式部署不会被路人签发租户。
+
+```bash
+export ENGRAM_ADMIN_TOKEN=<强随机串>   # 与租户 key 不同，是独立的管理凭据
+
+# 签发（明文密钥只在这一次响应里出现，服务端只存 SHA-256 摘要，丢了只能重签不能找回）
+curl -s -X POST $B/v1/admin/keys -H "Authorization: Bearer $ENGRAM_ADMIN_TOKEN" \
+  -H 'Content-Type: application/json' -d '{"user":"alice","label":"laptop"}'
+# -> {"id":"key_...","user":"alice","label":"laptop","created_at":...,"revoked":false,"key":"sk-engram-..."}
+
+curl -s $B/v1/admin/keys -H "Authorization: Bearer $ENGRAM_ADMIN_TOKEN"          # 列出（不含密钥与摘要）
+curl -s -X DELETE $B/v1/admin/keys/key_xxx -H "Authorization: Bearer $ENGRAM_ADMIN_TOKEN"  # 立即吊销
+```
+
+密钥文件（数据目录下的 `api_keys.json`）仅属主可读写，且只保存摘要。若该文件损坏，服务**拒绝启动
+密钥库并对请求返回 503**，而不是以空状态继续——后者会先让所有已签发密钥失效，再在下次签发时覆盖掉
+那些只是读不出来的记录。
+
+### 1.2 重试安全（Idempotency-Key）
+
+`/v1/remember` 与 `/v1/import` 支持 `Idempotency-Key` 请求头。客户端超时重试时，服务端按
+「租户 + key」返回首次的响应而不重跑：首次请求其实成功了，丢的只是响应。
+
+```bash
+curl -s -X POST $B/v1/remember -H "Authorization: Bearer $KEY" \
+  -H 'Idempotency-Key: 2026-08-16-abc' -H 'Content-Type: application/json' \
+  -d '{"content":"..."}'
+```
+
+缓存是进程内的，默认保留 `ENGRAM_IDEMPOTENCY_TTL_S`（86400）秒。多副本部署下，重试若路由到另一个
+副本仍会重跑——需要跨副本幂等就换成共享存储。
+
+### 1.3 限流
+
+`ENGRAM_RATE_LIMIT_PER_MIN`（默认 0 = 关闭）按租户做滑动窗口限流，超限返回 `429` 并带
+`Retry-After`。`/health`、`/ready`、`/metrics` 不鉴权也不限流——租户被限流时探针必须还能用。
+同样是进程内的：多副本下有效限额是 `每分钟配额 × 副本数`。
+
 健康检查不需要鉴权：
 ```bash
 curl -s $B/health
@@ -164,9 +208,51 @@ agent session。
 | POST | `/v1/conflicts/{id}/resolve` | `{"keep":"newer\|older\|both"}` 让冲突由人确认 |
 | PATCH | `/v1/facts/{id}` | 改事实 `{"object":"...","sensitive":true}` |
 | DELETE | `/v1/facts/{id}` | 删一条 |
-| GET | `/v1/export?include_sensitive=false` | 安全结构化导出：仅非敏感 facts + graph；不含画像、摘要、原始对话 |
+| GET | `/v1/export?include_sensitive=false` | 安全结构化导出：仅非敏感 facts + graph；`include_sensitive=true` 为完整可迁移导出（双时间戳、supersedes 链、provenance、原始对话、摘要、focus） |
+| POST | `/v1/import` | 批量导入：`{"data":..., "format":"chatgpt\|messages\|records\|jsonl\|transcript\|engram\|auto"}`；`engram` 格式（自动嗅探）直接还原一份 `/v1/export` 导出——跨实例迁移路径 |
 | POST | `/v1/forget` | 需 `{"confirm":true}`；清空该 key 的全部记忆（不可逆） |
 
+### 运行指标（`GET /metrics`，不鉴权）
+
+与 `/health` 一样开放，因为载荷**按构造只含聚合量**：没有命名空间名、没有查询、没有正文，所以它
+无法暴露某个租户是否存在。用于回答「写路径 <50ms、读路径 <100ms 这两个目标现在还成立吗」——
+没有实时分位数的话，那两个数字只是断言而非测量。
+
+```bash
+curl -s $B/metrics
+```
+
+```json
+{
+  "uptime_s": 1234.5,
+  "ops": {"remember": {"n": 42, "p50_ms": 8.1, "p95_ms": 22.4, "avg_ms": 9.7, "max_ms": 31.0, "window": 42}},
+  "counts": {"rate_limited": 3, "idempotent_replays": 1, "auth_rejected": 0, "remember_degraded": 0},
+  "tokens": {"context_total": 9600, "calls_with_baseline": 12, "savings_ratio": 8.2}
+}
+```
+
+- `ops` 是每个操作的滑动窗口分位数，反映**当前**表现，不会被历史均值稀释。
+- `counts` 里 `rate_limited` / `idempotent_replays` / `auth_rejected` 让三层防护可见——否则无法判断
+  它们是否在生效。`auth_misconfigured`（503）与 `auth_rejected`（401）分开计：前者是配置坏了，
+  后者是调用方拿错了密钥。
+- `savings_ratio` **只用两侧都测量过的调用**计算（即带 `answer=true` 的召回），没有配对样本时为
+  `null` 而不是编一个数。
+
+### 跨实例迁移（换服务器 / 换账号）
+
+记忆属于用户，不属于某一个部署。把一个 namespace 从实例 A 搬到实例 B：
+
+```bash
+# 1. 从 A 完整导出（含敏感事实与原始对话）
+curl -s "$A/v1/export?include_sensitive=true" -H "Authorization: Bearer $KEY_A" > export.json
+# 2. 导入 B（事实保留原 id/双时间戳/supersedes 链；目标端用自己的 embedder 重新向量化）
+curl -s -X POST "$B/v1/import" -H "Authorization: Bearer $KEY_B" -H "Content-Type: application/json" \
+  -d "{\"data\": $(cat export.json), \"format\": \"engram\"}"
+```
+
+导入按 id 幂等：已存在的条目跳过、绝不覆盖，重复导入不会产生重复记忆。两端 embedder 可以不同——
+这也是更换 embedder 的官方迁移路径。CLI 等价：`python -m engram.connectors -f export.json --api-url $B --key $KEY_B`。
+
 ## 5. 控制台（可视化）
 浏览器开 **`<Base URL>/ui/`** → 输入你的 key → 看「画像 / 事实管理 / 时间线 / 关系图谱 / 记忆问答 / 冲突待确认」。
 

CHANGELOG.md

@@ -3,6 +3,30 @@
 All notable changes to Engram are documented here. The project uses semantic versioning for distributed
 artifacts; storage schema compatibility is documented separately when it changes.
 
+## [Unreleased]
+
+### Added
+
+- Native export→import roundtrip (`format="engram"`, auto-sniffed): a `/v1/export` payload now restores
+  directly into another instance via `POST /v1/import`, `Memory.import_export()`, the `engram_import`
+  MCP tool, or `python -m engram.connectors` — preserving fact ids, bi-temporal stamps, supersession
+  chains, and provenance. Idempotent by id; the target re-embeds with its own embedder, which is also
+  the supported embedder-migration path.
+- `ENGRAM_STORAGE` environment variable selects the vector backend (`memory` default, `lancedb`
+  opt-in) for the server/MCP surfaces; unknown values fail closed at startup.
+- Optional Bearer authentication for the MCP streamable-HTTP transport (`--http-token` /
+  `ENGRAM_MCP_HTTP_TOKEN`). Non-loopback `--http` binds without a token are refused at startup unless
+  `ENGRAM_MCP_HTTP_OPEN=1` explicitly delegates access control to an external layer.
+
+### Fixed
+
+- `POST /v1/import` returns 400 with the parser's reason on malformed payloads instead of an
+  unhandled 500.
+- The import CLI's local mode writes through `MemoryService`, so it uses the same digest-backed
+  namespace directories and locks as the HTTP/MCP surfaces (previously a namespace like `a/b` landed
+  in a different directory than the one the server reads).
+- `/v1/stats` now filters by the canonical linked identity, matching every other read path.
+
 ## [0.1.0] - 2026-07-14
 
 ### Added

README.md

@@ -236,7 +236,32 @@ For a user-facing "my memory" page, the SDK also exposes paged inspection:
 The standalone graph endpoint is share-safe by default; pass `/v1/graph?include_sensitive=true` only for
 an explicit private graph inspection.
 
-**3. Batch import** — bring your whole history (ChatGPT export, OpenAI messages, JSONL, transcript;
+**3. Python SDK** — the same API from Python, with no runtime dependencies (it speaks over stdlib
+`urllib`, so installing Engram still pulls in nothing):
+
+```python
+from engram.client import EngramClient
+
+engram = EngramClient(base_url="http://localhost:8000", api_key="sk-engram-...")
+engram.remember("I live in Shenzhen and work on retrieval.")
+print(engram.recall("where do I live?")["context"])
+
+# retry-safe writes: the same key replays the first response instead of storing twice
+engram.remember("...", idempotency_key="2026-08-16-abc")
+```
+
+`EngramError` carries `status` so callers can branch without parsing prose — `401` wrong key, `429` back
+off (`err.retry_after` is the server's `Retry-After`), `503` server misconfigured rather than a bad
+request. The `transport` hook swaps the HTTP layer for `httpx`/`requests` or an in-process test client.
+
+**4. Runtime keys, rate limits and metrics** — for self-hosting: mint and revoke tenant keys without a
+restart (`POST /v1/admin/keys`, gated by its own `ENGRAM_ADMIN_TOKEN` and absent unless you set it),
+per-tenant rate limiting (`ENGRAM_RATE_LIMIT_PER_MIN`), retry-safe writes (`Idempotency-Key`), and
+`GET /metrics` for live latency percentiles, token totals and defence counters. `/metrics` is
+aggregate-only by construction — no namespace names, no queries, no content — so it can stay as open as
+`/health`. See [`API.md`](API.md).
+
+**5. Batch import** — bring your whole history (ChatGPT export, OpenAI messages, JSONL, transcript;
 auto-detected):
 
 ```bash

README.zh-CN.md

@@ -237,6 +237,37 @@ TypeScript SDK 的 `engram.export()` 默认走这个安全导出；只有用户
 `engram.memories({ factsLimit, factsOffset, episodesLimit, status, query, includeSensitive })`。
 独立关系图接口默认也是安全视图；只有明确要完整私有图谱时才传 `/v1/graph?include_sensitive=true`。
 
+### Python SDK（零运行时依赖）
+
+Agent 生态是 Python 优先的，所以 SDK 直接装在核心包里，走 stdlib `urllib`，装 Engram 依然不会额外
+拉进任何依赖：
+
+```python
+from engram.client import EngramClient
+
+engram = EngramClient(base_url="http://localhost:8000", api_key="sk-engram-...")
+engram.remember("我住在深圳，做检索相关的工作。")
+print(engram.recall("我住在哪儿？")["context"])
+
+# 重试安全：同一个 key 会重放首次响应，而不是写第二遍
+engram.remember("...", idempotency_key="2026-08-16-abc")
+```
+
+`EngramError` 带 `status`，调用方不用解析文案就能分支——`401` 密钥错、`429` 该退避
+（`err.retry_after` 就是服务端的 `Retry-After`）、`503` 是服务端配置有问题而不是请求有问题。
+`transport` 钩子可以把 HTTP 层换成 `httpx`/`requests` 或进程内测试客户端。
+
+### 运行时密钥、限流与运行指标（自托管用）
+
+- **运行时签发密钥**：`POST /v1/admin/keys` 不重启就能签发/吊销租户密钥；只落盘 SHA-256 摘要，
+  明文只在签发响应里出现一次。管理面由独立的 `ENGRAM_ADMIN_TOKEN` 把守，**不设置就完全不存在**。
+- **按租户限流**：`ENGRAM_RATE_LIMIT_PER_MIN`（默认 0 = 关闭），超限返回 `429` + `Retry-After`。
+- **重试安全**：`/v1/remember` 与 `/v1/import` 支持 `Idempotency-Key` 头。
+- **运行指标**：`GET /metrics` 给出实时延迟分位数、token 总量和防护计数。载荷按构造只含聚合量——
+  无命名空间名、无查询、无正文——所以可以和 `/health` 一样开放。
+
+细节见 [`API.md`](API.md)。
+
 ### 自部署（数据完全在你自己机器上）
 
 ```bash

clients/typescript/src/client.ts

@@ -15,7 +15,6 @@ import type {
   ChatCompletionCreateParams,
   CloseSessionResult,
   AgentStatus,
-  Fact,
   FactInput,
   FactPatch,
   ForgetOptions,
@@ -46,12 +45,35 @@ export class EngramError extends Error {
     public readonly status: number,
     message: string,
     public readonly detail?: unknown,
+    /**
+     * Seconds to wait before retrying, from a 429's `Retry-After`. Undefined when the response did
+     * not carry the header — which is every status except a rate-limit rejection.
+     */
+    public readonly retryAfter?: number,
   ) {
     super(message)
     this.name = 'EngramError'
   }
 }
 
+/**
+ * Seconds to wait, read off the response headers.
+ *
+ * It has to come from the headers because that is the only place the server puts it — the 429 body
+ * says "rate limit exceeded" and nothing more. A transport that hands back only status + body makes
+ * this permanently undefined, which is an error object nobody can act on; the optional calls below
+ * are for that shape of injected fetch, not for a real `Response`.
+ *
+ * Only delta-seconds are parsed: the server always sends an integer (`str(int(retry_after) + 1)`),
+ * so an HTTP-date branch here would be a claim about the wire that nothing can reach.
+ */
+function retryAfterOf(res: Response): number | undefined {
+  const raw = res.headers?.get?.('Retry-After')
+  if (raw == null) return undefined
+  const seconds = Number.parseInt(raw.trim(), 10)
+  return Number.isFinite(seconds) ? seconds : undefined
+}
+
 export interface EngramClientOptions {
   /** Server base URL. Default 'http://localhost:8000'. A trailing '/v1' is allowed and normalized away. */
   baseUrl?: string
@@ -115,7 +137,7 @@ export class EngramClient {
     } catch {
       /* non-JSON error body */
     }
-    throw new EngramError(res.status, message, detail)
+    throw new EngramError(res.status, message, detail, retryAfterOf(res))
   }
 
   private async request<T>(path: string, init: RequestInit = {}): Promise<T> {