Fix packed private Git backup restore and verify installed recovery under load

PROGRESS.md

@@ -0,0 +1,62 @@
+# 交付进度 · 2026-09-30
+
+目标：逐项处理全部开放 issue/PR，交付经验证的安装包和部署步骤。
+进度以实际提交、测试运行和现场材料为准，不把旧记录或测试数量当完成率。
+
+## 本轮基线与完成事项
+
+- 已核对 16 个开放 issue（#64、#65、#68、#69、#70、#72、#76–#85）与唯一开放 PR #186。
+- 已读取当前权威设计及有效归档；现有功能和历史协议保持原义。
+- #186 已审查并合并为 `dfb89fe989eaef66c99f14ae6a953b07204b7170`。
+  主线 tree `e1d65b624266527bef5f66d2843fa1a40c4ddc5f` 与实际通过测试的 tree 相同。
+- [核心验收 36632579472](https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36632579472)：
+  八份 ZIP 摘要、manifest 和 JUnit 真实 node-ID 已核验，2541 core + 8 conformance，
+  无失败、错误、跳过、重复或遗漏。
+- [质量 36632578929](https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36632578929)：
+  Ruff JSON 零诊断、stderr 为空、585 文件格式、Python 3.15.0rc2 编译通过。
+- [部署演练 36632578807](https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36632578807)：
+  同一 wheel 的锁定 server/client 安装、真实启动/重启、四传输、只读 doctor、
+  禁外发 worker、真实 OpenSSH/Git 撤权及正式隔离 selftest 42 项均通过。
+  wheel/sdist 与核心构建产物逐字节一致；精确摘要见 PR #186。
+
+## 当前推进
+
+- 增加安装包的有界并发写入/读取测量、真实 SIGKILL 重启与全表/sequence 不变量检查。
+- 增加安装包的真实 PostgreSQL/Git/CAS 备份恢复计时和全表值比较；只允许恢复流程
+  明确规定的 runtime_config/quarantine 变化，验证删 marker 后仍隔离业务与幂等重放。
+- 首次新演练 [36643217304](https://github.com/TokenNotIncluded/msg.lmm.best/actions/runs/36643217304)
+  在真实恢复时失败：私有 Git 打包引用后，ZIP 丢失空 refs 目录，Git 无法识别仓库。
+  本轮修复私有/用户仓库统一结构目录重建，保持只读 proof 校验不修改磁盘，
+  并新增 packed-ref 恢复、只读拒绝和符号链接拒绝回归。新最终 tree 仍须重新完整验收。
+- 逐条核对 issue 中尚未刷新到当前源码的描述，保留原验收范围和历史证据。
+
+## issue 交接
+
+| issue | 已有代码/隔离证据入口 | 本轮需要继续完成的范围 |
+| --- | --- | --- |
+| #64 | URL/Host/真实隔离 Nginx 日志回归；#186 主体读 effect 栅栏 | 实际 listener/CDN/proxy/APM/日志链材料 |
+| #65 | 固定旧源码 fixture、LedgerAccount 迁移/回滚 | 真实旧快照的合法来源、冻结点及快照演练 |
+| #68 | 历史密文迁移、逐项 ACK、备份退役证明校验 | 真实保留密文及独立备份范围退役证据 |
+| #69 | 完整 proof/current-policy reconcile/promotion、持久隔离 | rollback 集之外的真实当前 pin 与本机受控恢复 |
+| #70 | Root/CA/控制台拒绝与 bank fund 源码回归 | 真实 VT/串口及有限存量 CA 审核 |
+| #72 | Bounty/官方 market_e2e/邮件/恢复/版本政策回归 | 对照原清单逐断言验收，更新过时的 #103 描述 |
+| #76 | 多来源授权及跨协议/恢复矩阵 | 完整投影、缓存、当前权限条款核对 |
+| #77 | 私有内容、独立签名、Legacy、荣誉挑战与展示 | 全条款/客户端/当前失效事实核对 |
+| #78 | ReadQuery/Search/Sync/游标及 #186 | 完整成本/表示/当前授权矩阵核对 |
+| #79 | 文件、patch/rebase、Revision、原子发布/物理失败 | 完整内容/文件条款核对 |
+| #80 | Transfer/Git/LFS/hosting/一致存储及真实 SSH | 共享卷/多实例目标拓扑容量与持久性 |
+| #81 | 通知/EffectJob/协作/SMTP socket/Webhook 回归 | 全事件契约与受控部署接收端 |
+| #82 | CLI/TUI/工具真实隔离/SSH/安装包 | 目标机路径/PAM/systemd 与全部客户端条款 |
+| #83 | 配置、Registry、规则及有限设计分母 | 逐条断言映射、当前文档与真实现场闸门 |
+| #84 | 锁定安装包、隔离服务/SSH/selftest 演练 | 本轮补测量；真实主机/旧数据/切流/回滚仍需材料 |
+| #85 | 唯一市场 owner、受保护账本/版本兼容/严格质量 | 对当前热点/重复实现核对，不按旧目录诊断 |
+
+## 完成条件与外部阻塞
+
+新改动须在同一最终 tree 上通过质量、完整八分片/协议、安装包演练与适用专项；
+合并后再核对独立 main push。问题解决后才更新状态，不能删断言或把取消当通过。
+
+现场尚缺：目标主机只读入口与配置/日志链、受保护真实旧备份及来源、真实本机
+Root/CA 操作者、独立当前检查点、密文/备份退役材料、目标容量/停止/回滚条件。
+所需材料及操作步骤见 `docs/RELEASE_ACCEPTANCE.md` 和 `docs/DEPLOYMENT.md`。
+当前没有生产部署、资金操作、真实外发、备份删除或恢复放行。

docs/RELEASE_ACCEPTANCE.md

@@ -1,7 +1,8 @@
 # 发布验收与现场交接
 
-本页是 PR #185 的当前入口，取代把历史进度数字当作当前部署结论的做法。
-源码基线为已合并 #176 的 `25cf518a`；#182/#183/#184 均已在该历史。
+本页是发布验收的入口，取代把历史进度数字当作当前部署结论的做法。
+本轮基线为已合并 #186 的 `dfb89fe9`，tree `e1d65b62`；#185 及其前置修复已在历史。
+本轮事项与未完成范围见仓库根目录 `PROGRESS.md`。
 每次最终验证以产物 `source.json` 中的 commit/tree、对应 workflow 和完整 JUnit
 node-ID 为准。PR 的模拟 merge commit 与分支 head 可不同，必须核对实际 tree。
 本页索引测试，不单凭测试文件名声称原设计所有分支或生产已经验收。
@@ -23,7 +24,15 @@ server 包在 checkout 外运行，Root 目录确实不可由网络账号读取
 client-only 包不得安装或导入 server runtime；其传输检查使用 MockTransport，
 不冒称真实远程服务器连接。安装包正式 selftest 的 Test Root 也不代表真实物理控制台。
 
-通过结果与 ZIP/wheel/锁文件摘要附在 PR #185 和 #83，不为写入成功数字再改代码树。
+通过结果与 ZIP/wheel/锁文件摘要附在对应 PR 和 #83，不为写入成功数字再改代码树。
+
+安装包演练还执行固定四个独立签名客户端的 32 次写入与 32 次读取，输出实际
+耗时/吞吐/p50/p95、daemon RSS/峰值 RSS、内容文件字节和剩余磁盘。真实 SIGKILL
+后重启须保留所有表值、sequence 和同请求幂等；真实 PostgreSQL/Git/CAS 备份
+恢复须逐表逐值比较，仅允许确切的 runtime_config 暂停及持久 quarantine 改动。
+删除 marker 后重新创建应用仍须隔离业务与幂等重放。失败直接使部署演练失败。
+这些数值仅代表该次有界 CI 工作负载，不是目标主机容量或负载上限，也不取代
+共享卷、多实例、断电、旧生产数据及真实恢复 pin 的现场验收。
 
 ## 开放 issue 的源码与验收入口
 

scripts/check_installed_server.py

@@ -11,24 +11,29 @@
 import hashlib
 import importlib.metadata
 import json
+import math
 import os
+import platform
+import shutil
 import socket
 import subprocess
 import sys
 import tempfile
+import time
 from pathlib import Path
 
 import httpx
 import psycopg
 from psycopg import sql
 
 import msg
+from msg.admin.backups import backup, restore
 from msg.admin.diagnostics import temporary_postgres
 from msg.admin.root import _approve_csr, _provision
 from msg.application import Application
 from msg.client import ClientState, MsgClient
-from msg.config import write_example
-from msg.core.codec import canonical
+from msg.config import load_settings, write_example
+from msg.core.codec import canonical, loads
 from msg.transports.client import (
     GraphQLTransport,
     HTTPTransport,
@@ -41,36 +46,171 @@ def run(*command):
     return subprocess.run(command, capture_output=True, text=True, timeout=60, check=True)
 
 
-def database_digest(dsn):
+def database_facts(dsn):
     """Read complete row/sequence values without advancing PostgreSQL sequences."""
-    hasher = hashlib.sha256()
+    facts = {'tables': {}, 'sequences': {}}
     with psycopg.connect(dsn) as connection:
         connection.execute('SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY')
         tables = connection.execute(
             "SELECT tablename FROM pg_tables WHERE schemaname='public' ORDER BY tablename"
         ).fetchall()
         for (table,) in tables:
-            hasher.update(canonical(table))
             rows = connection.execute(sql.SQL('SELECT * FROM {}').format(sql.Identifier(table)))
-            for row in sorted(canonical(row) for row in rows):
-                hasher.update(len(row).to_bytes(8, 'big'))
-                hasher.update(row)
+            facts['tables'][table] = sorted(canonical(row) for row in rows)
         sequences = connection.execute(
             "SELECT sequencename FROM pg_sequences WHERE schemaname='public' ORDER BY sequencename"
         ).fetchall()
         for (sequence,) in sequences:
             value = connection.execute(
                 sql.SQL('SELECT last_value,is_called FROM {}').format(sql.Identifier(sequence))
             ).fetchone()
-            hasher.update(canonical((sequence, value)))
-    return hasher.hexdigest()
+            facts['sequences'][sequence] = canonical(value)
+    return facts
+
+
+def database_digest(dsn):
+    return hashlib.sha256(canonical(database_facts(dsn))).hexdigest()
+
+
+def latency_report(samples, elapsed):
+    ordered = sorted(samples)
+    assert ordered and elapsed > 0
+    return {
+        'completed': len(ordered),
+        'elapsed_seconds': elapsed,
+        'operations_per_second': len(ordered) / elapsed,
+        'latency_p50_seconds': ordered[math.ceil(len(ordered) * 0.5) - 1],
+        'latency_p95_seconds': ordered[math.ceil(len(ordered) * 0.95) - 1],
+        'latency_max_seconds': ordered[-1],
+    }
+
+
+async def concurrent_workload(folder, origin, http, dsn):
+    """Bounded workload on independently signed clients, not production load."""
+    writers = []
+    for index in range(4):
+        client = MsgClient(
+            ClientState(folder / f'capacity-client-{index}', server=origin),
+            HTTPTransport(origin, http=http),
+        )
+        assert (await client.register(f'capacity-client-{index}')).status == 'ok'
+        writers.append(client)
+
+    async def write(client, index):
+        measured = []
+        for offset in range(8):
+            body = f'capacity-{index}-{offset}\n' + 'bounded content\n' * 64
+            packet = client.prepare(
+                'content.post_create',
+                {'parent': '/main', 'body': body},
+                request_id=f'capacity-{index}-{offset}',
+            )
+            started = time.perf_counter()
+            posted = await client.send(packet)
+            measured.append((time.perf_counter() - started, posted.resources, body))
+            assert posted.status == 'ok', 'Concurrent signed write failed'
+            assert len(posted.resources) == 1
+        return measured
+
+    started = time.perf_counter()
+    written = await asyncio.gather(*(write(client, i) for i, client in enumerate(writers)))
+    writes = latency_report(
+        [row[0] for group in written for row in group], time.perf_counter() - started
+    )
+    assert len({row[1][0].id for group in written for row in group}) == 32
+    before = database_digest(dsn)
+
+    async def read(client, group):
+        measured = []
+        for _, references, body in group:
+            started = time.perf_counter()
+            result = await client.call('discovery.get', {'id': references[0].id})
+            measured.append(time.perf_counter() - started)
+            assert result.status == 'ok' and result.data['content'] == body
+        return measured
+
+    started = time.perf_counter()
+    reads = await asyncio.gather(
+        *(read(client, group) for client, group in zip(writers, written, strict=True))
+    )
+    reading = latency_report(
+        [sample for group in reads for sample in group], time.perf_counter() - started
+    )
+    assert database_digest(dsn) == before, 'Concurrent reads changed authoritative facts'
+    return {'concurrent_clients': 4, 'writes': writes, 'reads': reading}
+
+
+async def restore_drill(settings, folder, client, packet, source_dsn):
+    """Compare every row and sequence, allowing only exact quarantine changes."""
+    app = Application(settings)
+    try:
+        await app.load()
+        original = database_facts(source_dsn)
+        started = time.perf_counter()
+        archive = folder / 'recovery.zip'
+        result = await backup(app, archive)
+        backup_seconds = time.perf_counter() - started
+        assert result['root_private_key_included'] is False
+        assert database_facts(source_dsn) == original
+    finally:
+        await app.close()
+    with temporary_postgres() as destination_dsn:
+        started = time.perf_counter()
+        restored = restore(
+            archive, folder / 'restore-etc', folder / 'restore-data', postgres_dsn=destination_dsn
+        )
+        restore_seconds = time.perf_counter() - started
+        assert restored['promotion'] == 'blocked'
+        current = database_facts(destination_dsn)
+        expected_settings = dict(loads(row) for row in original['tables']['settings'])
+        current_settings = dict(loads(row) for row in current['tables']['settings'])
+        quarantine = loads(current_settings['recovery_quarantine'])
+        assert quarantine == {
+            'format': 'msg-recovery-quarantine-v1',
+            'outbound_enabled': False,
+            'source_backup_sha256': result['sha256'],
+            'revocation_replay': 'required',
+            'authority': 'health_only',
+        }
+        runtime = loads(expected_settings.get('runtime_config', '{}'))
+        expected_settings['runtime_config'] = canonical({
+            **runtime,
+            'accept_writes': False,
+            'cleanup_enabled': False,
+        }).decode()
+        expected_settings['recovery_quarantine'] = current_settings['recovery_quarantine']
+        original['tables']['settings'] = sorted(canonical(row) for row in expected_settings.items())
+        assert current == original, 'Restore changed other table values or sequence state'
+        for marker_present in (True, False):
+            if not marker_present:
+                (folder / 'restore-etc/recovery-drill.json').unlink()
+            recovered = Application(load_settings(folder / 'restore-etc'))
+            try:
+                await recovered.load()
+                replay = await recovered.executor.execute(packet)
+                assert replay.error.code == 'writes_paused'
+                read = client.prepare('discovery.get', {'id': client.state.subject})
+                denied = await recovered.executor.execute(read)
+                assert denied.error.code == 'recovery_quarantined'
+                assert database_facts(destination_dsn) == current
+            finally:
+                await recovered.close()
+    return {
+        'backup_seconds': backup_seconds,
+        'restore_seconds': restore_seconds,
+        'archive_bytes': archive.stat().st_size,
+        'all_table_values_and_sequences_preserved': True,
+        'persistent_quarantine_without_marker': True,
+        'promotion_performed': False,
+    }
 
 
 async def acceptance():
     assert os.geteuid() != 0
     assert 'site-packages' in Path(msg.__file__).parts, 'Use the installed wheel with -I'
     assert msg.__version__ == importlib.metadata.version('msgctl')
     checks = ['installed_wheel_and_version']
+    measurements = {'platform': platform.platform(), 'cpu_count': os.cpu_count()}
     with (
         tempfile.TemporaryDirectory(prefix='msg-installed-rehearsal-') as temporary,
         temporary_postgres() as dsn,
@@ -128,7 +268,9 @@ async def start():
                     child.wait(timeout=15)
                     raise
 
+            started = time.perf_counter()
             process = await start()
+            measurements['startup_seconds'] = time.perf_counter() - started
             checks.append('real_daemon_startup')
             state = ClientState(folder / 'client', server=origin)
             async with httpx.AsyncClient(timeout=10, trust_env=False) as http:
@@ -162,6 +304,27 @@ async def start():
                     assert read.status == 'ok' and read.data['content'] == 'installed release\r\n'
                 assert database_digest(dsn) == before
                 checks.append('four_real_transports_preserve_all_database_facts')
+                measurements['workload'] = await concurrent_workload(folder, origin, http, dsn)
+                checks.append('concurrent_signed_writes_and_read_conservation')
+                status = (Path('/proc') / str(process.pid) / 'status').read_text()
+                measurements['daemon_memory_kib'] = {
+                    name: int(line.split()[1])
+                    for line in status.splitlines()
+                    for name in ('VmRSS:', 'VmHWM:')
+                    if line.startswith(name)
+                }
+                before = database_digest(dsn)
+                process.kill()
+                process.wait(timeout=15)
+                started = time.perf_counter()
+                process = await start()
+                measurements['sigkill_restart_seconds'] = time.perf_counter() - started
+                assert database_digest(dsn) == before
+                repeated = await client.send(packet)
+                assert repeated.status == 'ok' and repeated.replayed
+                assert repeated.resources == posted.resources
+                assert database_digest(dsn) == before
+                checks.append('sigkill_restart_preserves_all_facts_and_idempotency')
             before = database_digest(dsn)
             diagnosed = json.loads(
                 run(str(executable), '--config-dir', str(settings.config_dir), 'doctor').stdout
@@ -175,6 +338,20 @@ async def start():
             process.wait(timeout=15)
             process = None
             checks.append('clean_daemon_shutdown')
+            measurements['recovery'] = await restore_drill(settings, folder, client, packet, dsn)
+            checks.append('installed_backup_restore_conservation_and_persistent_quarantine')
+            measurements['content_file_bytes'] = sum(
+                entry.stat().st_size
+                for directory in (
+                    settings.server.content_dir,
+                    settings.server.repositories_dir,
+                    settings.server.blob_dir,
+                    settings.server.staging_dir,
+                )
+                for entry in directory.rglob('*')
+                if entry.is_file()
+            )
+            measurements['disk_free_bytes'] = shutil.disk_usage(folder).free
         finally:
             if process is not None and process.poll() is None:
                 process.terminate()
@@ -190,6 +367,8 @@ async def start():
         'version': msg.__version__,
         'temporary_installation_removed': True,
         'external_recipients': 0,
+        'measurements': measurements,
+        'measurement_scope': 'bounded disposable CI workload; not target-host capacity',
     }