Fix mixed cumulative overcounting and release v2.6.3

README.en.md

@@ -29,7 +29,7 @@ All statistics stay on the current computer. codex-usage never stores prompts, r
 
 ## Install directly
 
-This README covers stable **[v2.6.2](https://github.com/zJay26/codex-usage/releases/tag/v2.6.2)**; see the [release notes](docs/releases/v2.6.2.md) for changes and upgrade boundaries. Download links below always resolve to the latest stable release.
+This README covers stable **[v2.6.3](https://github.com/zJay26/codex-usage/releases/tag/v2.6.3)**; see the [release notes](docs/releases/v2.6.3.md) for changes and upgrade boundaries. Download links below always resolve to the latest stable release.
 
 | System | amd64 / x64 | arm64 |
 |---|---|---|
@@ -92,7 +92,7 @@ Starting with **v2.5.0**, the application checks GitHub for the latest stable re
 
 **Software updates** lets you set the update download directory, open the folder, or copy its path. The default is `codex-usage` inside your user Downloads directory. Version folders retain the release binaries, and the UI shows the full path of the last download. Directory changes only affect future downloads; existing files stay in place and database backups remain in the local state directory.
 
-Upgrading to v2.6 preserves existing statistics and applies accounting fixes to newly read records. To correct earlier undercounts, back up the state and verify retained JSONL coverage before explicitly running `scan --rebuild`; rebuilding cannot recover history whose source files have been deleted.
+**v2.6.3 fixes repeated accounting of old usage in mixed cumulative records processed by v2.6.0–v2.6.2.** After upgrading from those versions, existing statistics are retained and flagged for review; incremental scans pause. Back up the state and verify source JSONL coverage, then select **Rescan → Approve and rebuild** or explicitly run `codex-usage scan --rebuild` to correct stored history. Rebuilding cannot recover deleted source files, and upgrading the binary alone does not correct old totals.
 
 ## What you can see
 
@@ -165,7 +165,7 @@ flowchart LR
 
 The tool reads the current machine's `CODEX_HOME`. It first discovers canonical session metadata from the Codex state database, then streams JSONL files under `sessions/` and `archived_sessions/`.
 
-Token records can be cumulative per session or per turn. Persisted scope and the last token turn distinguish them: total equal to last at a new turn establishes turn scope whether it is smaller than, equal to, or larger than the preceding total. Legacy session counters retain their cross-turn baseline. At **each** `token_count` record, the scanner computes the delta against the corresponding session or turn baseline and assigns it to the record timestamp's accounting calendar day. It never moves an entire multi-day session to the session's latest update date. Stable event IDs and cursors keep repeated scans idempotent. Large prompt, response, reasoning, and tool-output records are skipped without loading the entire line into memory or writing content to the database.
+Token counters can continue across turns or reset at a turn boundary. Each boundary is evaluated separately: `total_token_usage` equal to `last_token_usage` at a new turn starts a fresh baseline, whether smaller than, equal to, or larger than the preceding total. A later continuation or repeated snapshot retains its baseline; one reset never classifies all future turns. When missing snapshots prevent distinguishing continuation from another reset, the scanner conservatively differences the counter and flags potentially incomplete usage. At **each** `token_count` record, the delta is assigned to the record timestamp's accounting calendar day. It never moves an entire multi-day session to the session's latest update date. Stable event IDs and cursors keep repeated scans idempotent. Large prompt, response, reasoning, and tool-output records are skipped without loading the entire line into memory or writing content to the database.
 
 The Codex state database is used only to discover rollout paths and enrich titles, projects, and other metadata. Its `tokens_used` value never changes token totals. OpenAI's [`account/usage/read`](https://learn.chatgpt.com/docs/app-server#7-token-usage-chatgpt) is service-backed account activity; this tool counts only current-machine local JSONL, so the scopes differ.
 

README.md

@@ -29,7 +29,7 @@
 
 ## 直接安装
 
-本文对应稳定版 **[v2.6.2](https://github.com/zJay26/codex-usage/releases/tag/v2.6.2)**；变更和升级边界见 [发布说明](docs/releases/v2.6.2.md)。以下下载链接始终指向最新稳定版。
+本文对应稳定版 **[v2.6.3](https://github.com/zJay26/codex-usage/releases/tag/v2.6.3)**；变更和升级边界见 [发布说明](docs/releases/v2.6.3.md)。以下下载链接始终指向最新稳定版。
 
 | 系统 | amd64 / x64 | arm64 |
 |---|---|---|
@@ -92,7 +92,7 @@ chmod +x codex-usage
 
 “软件更新”中可以设置**更新包下载目录**，并打开文件夹或复制路径。默认保存到用户下载目录下的 `codex-usage` 文件夹；版本文件夹内保留正式安装包，界面显示最近下载的完整路径。保存新目录只影响后续下载，已有文件留在原处；数据库备份仍保留在本机状态目录。
 
-v2.6 系列升级会保留现有统计，计量修复适用于新读入的记录。若要修正旧版本已经漏计的历史，需先备份并核对仍保留的 JSONL，再显式执行 `scan --rebuild`；源文件已删除的历史无法由重建恢复。
+**v2.6.3 修复了 v2.6.0–v2.6.2 在混合累计记录中重复计算旧用量的问题。** 从这些版本升级后，已有统计会保留并标记为需要核对，增量扫描暂停。先备份并确认源 JSONL 仍齐全，再点击“重新扫描”并选择“同意并重建”，或显式执行 `codex-usage scan --rebuild`，修正已入库的错误历史。源文件已删除的历史无法由重建恢复；只升级程序不会自动修正旧账。
 
 ## 你能看到什么
 
@@ -165,7 +165,7 @@ flowchart LR
 
 程序只读当前电脑的 `CODEX_HOME`。它优先从 Codex 状态库取得 session 路径、项目和 Thread 信息，再流式读取 `sessions/` 与 `archived_sessions/` 中的 JSONL。
 
-Token 记录可能按 Session 累计，也可能按 Turn 累计。扫描器保存计量范围与上次 Token 所属 Turn；新 Turn 的 `total_token_usage` 与 `last_token_usage` 相等时建立 Turn 范围，数值小于、等于或大于上一 Turn 均重新起算。旧版 Session 累计仍保留跨 Turn 基线。程序在**每一条** `token_count` 记录处按对应 Session 或 Turn 的累计基线计算增量，归到该条记录时间戳对应的计量自然日；不会按 session 的最后更新时间把整段历史塞到同一天。重复扫描仍由稳定事件 ID 与游标去重。超大的 prompt、回复和工具输出记录会被跳过，不会整行载入内存，也不会写进数据库。
+Token 记录可能跨 Turn 连续累计，也可能在某个 Turn 重置。扫描器逐个边界核对：新 Turn 的 `total_token_usage` 与 `last_token_usage` 相等时重新起算，支持小于、等于或大于上一累计值；后续 Turn 若继续累计或重发旧快照，则保留原基线。一次重置不会决定后续所有 Turn 的计量方式。缺少快照而无法区分连续累计与再次重置时，会保守按差值计量并提示可能缺少用量。程序在**每一条** `token_count` 记录处计算增量，归到该条记录时间戳对应的计量自然日；不会按 session 的最后更新时间把整段历史塞到同一天。重复扫描仍由稳定事件 ID 与游标去重。超大的 prompt、回复和工具输出记录会被跳过，不会整行载入内存，也不会写进数据库。
 
 Codex 状态库只用于发现 rollout 路径并补充标题、项目等 metadata；其中的 `tokens_used` 不参与 Token 总量。OpenAI 的 [`account/usage/read`](https://learn.chatgpt.com/docs/app-server#7-token-usage-chatgpt) 是服务端账号 Token 活动；本工具只统计当前电脑的本地 JSONL，两者范围不同。
 

docs/accounting-v2.6.md

@@ -2,7 +2,9 @@
 
 ## Counter scope and safe ingestion
 
-The persisted accounting state distinguishes an unknown/session counter from a turn counter. A changed `turn_id` alone does not reset legacy session totals. A new turn whose complete `total_token_usage` equals `last_token_usage` establishes turn scope, including totals smaller than, equal to, or larger than the previous turn. Subsequent snapshots of that turn contribute only their delta. Missing subset fields retain the existing classification only within the same counter scope.
+As of v2.6.3, every new turn is evaluated independently. A changed `turn_id` alone never resets cumulative totals, even after an earlier reset. A new turn whose complete `total_token_usage` equals `last_token_usage` establishes a fresh baseline, including totals smaller than, equal to, or larger than the preceding total. A later turn that continues the same series, or repeats an unchanged snapshot, retains its baseline. The duplicate path persists the new interpretation so the next incremental scan cannot inherit stale turn scope. Same-total classification corrections remain attached to the original usage.
+
+After a reset, a later boundary may lack enough snapshots to prove whether it continued or reset again. Positive monotonic differences are retained conservatively and marked `gap_fallback` with `cumulative_boundary_unverified`; the previous scope is never used as proof of another reset. Missing subset fields retain the existing classification only within the same counter scope.
 
 Turn-aware event identities also deduplicate copied rollouts. For proven turn counters created after migration, each file transaction checks the turn's existing ledger before reading its tail; a restored file that starts midway through a turn adds only the previously unseen increment. Session-cumulative writers keep the existing high-water behavior. Physical ownership remains the first `session_meta`; fork replay is excluded before accounting. Classification corrections for turn-scoped counters stay inside that turn, including fragments restored to another physical file.
 
@@ -12,9 +14,13 @@ This transaction is per physical file, so a large initial import can delay anoth
 
 ## Upgrade boundary
 
-Schema v9 is additive. Opening a v2.5.0 database preserves token amounts, timestamps, pricing classification, original local date labels and existing cursors. It adds accounting state, parent relationships, a persisted time zone and UTC hour identities. The Dashboard shows that pre-upgrade history has been retained. Metadata can be backfilled from retained sources without rebuilding token history.
+**v2.6.0–v2.6.2 contain a mixed-counter regression.** After one reset, later turns could count the full previous cumulative amount again, including unchanged snapshots. Zero warnings and idempotent repeated scans did not establish accuracy for those versions. v2.6.3 corrects this interpretation and upgrades the schema to v10.
+
+Opening a schema-v9 database with existing JSONL events preserves the ledger and cursors, immediately records a rebuild warning, and pauses incremental ingestion until an explicit rebuild. The Dashboard's **Rescan → Approve and rebuild** flow and CLI `scan --rebuild` recalculate retained sources with the corrected parser. Merely updating the program does not repair existing event deltas. Empty databases need no historical repair.
+
+Upgrades directly from v2.5.0 retain the earlier additive migration: token amounts, timestamps, pricing classification, original local date labels and cursors remain intact. Accounting state, parent relationships, a persisted time zone and UTC hour identities are added. Metadata can be backfilled from retained sources without rebuilding token history.
 
-The new parser applies to newly read records. It does not silently recalculate previously undercounted turns. Explicit `codex-usage scan --rebuild` recalculates only retained JSONL files and may lose entries whose original files have been deleted. Back up the state and verify source coverage first. A pre-upgrade session restored into a new physical file requests a rebuild when its old event identities cannot safely prove replay ownership.
+Explicit `codex-usage scan --rebuild` recalculates only retained JSONL files and may lose entries whose original files have been deleted. Back up the state and verify source coverage first. A pre-upgrade session restored into a new physical file also requests a rebuild when its old event identities cannot safely prove replay ownership.
 
 ## Query consistency and performance
 

docs/releases/v2.6.3.md

@@ -0,0 +1,29 @@
+# codex-usage v2.6.3
+
+修复 v2.6.0–v2.6.2 在历史重扫中可能大幅多计 Token 的累计范围回归。
+
+- **逐个 Turn 核对累计边界**：一次重置不再使后续所有 Turn 都从零计数；连续累计和新 Turn 重发的旧快照均保留正确基线。`100/100 → 20/20 → 50/30`（累计值／本次增量）现在合计 150，修复旧版算成 170 的问题。
+- **跨扫描一致性**：即使记录没有新增 Token，也保存其累计解释，避免下一次扫描重新继承旧范围；分类修正仍回到原有事件。
+- **明确不确定边界**：重置后若缺少快照，无法区分连续累计与再次重置，保守按差值计量并提示可能缺少用量，不再仅凭旧范围推断。
+- **旧账修复提示**：schema v9 的现有统计升级到 v10 后保留事件和游标、立即显示核对提示并暂停增量扫描，等待显式重建。空数据库无需修复；从 v2.5.0 直接升级保留原有迁移行为。
+- 更新中英文 README、重建对话框和计量技术说明，保留 v2.6.2 的更新包下载目录设置。
+
+**升级后修正历史：** 先备份并确认源 JSONL 仍齐全，再在 Dashboard 点击“重新扫描 → 同意并重建”，或运行 `codex-usage scan --rebuild`。仅更新程序不会自动消除已入库的多计记录；已删除源文件的历史无法通过重建恢复。
+
+**验证：** 固定同一时点的 766 个 rollout，提取计量所需的数值、metadata 和 Turn 边界后在隔离目录重建；742 个有用量会话与逐会话参考值完全匹配，总量由 19,646,071,899 修正到 14,941,760,325，消除 23 个会话的 4,704,311,574 重复 Token。独立核对覆盖了变化会话的原始记录，未变化会话沿用快照参考值；这不是所有历史的独立审计或账单验证。再次扫描新增事件为 0，总量不变。
+
+回归覆盖真实数值向量、混合累计、重复快照、分类修正、扫描重启、缺失快照、旧库保留和 API 确认重建。完整平台及 Dashboard 检查通过后发布六个平台程序与 `SHA256SUMS`。
+
+---
+
+Fix a counter-scope regression in v2.6.0–v2.6.2 that could substantially overcount tokens during historical rescans.
+
+- Evaluate each turn boundary independently. One reset no longer causes every later turn to count the entire cumulative total again. Continuations and repeated old snapshots retain their baseline. The `total/last` sequence `100/100 → 20/20 → 50/30` now correctly totals 150 instead of 170.
+- Persist the interpretation of duplicate snapshots across scans and keep classification corrections attached to the original usage.
+- Mark incomplete boundary evidence conservatively instead of inferring another reset from an earlier one.
+- Migrate schema v9 to v10 without deleting history. Existing JSONL ledgers immediately show a rebuild notice and pause incremental ingestion until an explicit rebuild. Empty stores need no repair; direct v2.5.0 upgrades retain the earlier migration behavior.
+- Update both READMEs, the rebuild dialog, and accounting documentation. Retain v2.6.2 download-directory settings.
+
+**Repair stored history after upgrading:** back up and verify retained JSONL coverage, then choose **Rescan → Approve and rebuild** or run `codex-usage scan --rebuild`. Updating the binary alone does not fix stored overcounts, and rebuilding cannot recover deleted source files.
+
+**Validation:** an isolated rebuild of accounting-only projections from 766 frozen rollouts matched reference totals for all 742 sessions with usage. The total changed from 19,646,071,899 to 14,941,760,325, removing 4,704,311,574 repeated tokens across 23 sessions. Changed sessions were independently reconciled against their source records; unchanged sessions retained snapshot reference totals. This is not an independent audit of all history or a billing reconciliation. A repeated scan inserted zero events and left totals unchanged. Regression coverage includes mixed counters, duplicate snapshots, corrections, restarts, missing evidence, migration preservation, and the API approval flow.

internal/app/app.go

@@ -31,7 +31,7 @@ import (
 )
 
 var (
-	Version   = "2.6.2"
+	Version   = "2.6.3"
 	Commit    = "dev"
 	BuildDate = "unknown"
 )

internal/server/accounting_migration_test.go

@@ -0,0 +1,83 @@
+package server
+
+import (
+	"context"
+	"database/sql"
+	"fmt"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/zJay26/codex-usage/internal/model"
+	"github.com/zJay26/codex-usage/internal/store"
+	"github.com/zJay26/codex-usage/internal/usage"
+)
+
+func TestCounterScopeUpgradeCanOnlyRecalculateThroughApprovedRescan(t *testing.T) {
+	ctx := context.Background()
+	root := t.TempDir()
+	home := filepath.Join(root, "codex")
+	if err := os.MkdirAll(filepath.Join(home, "sessions"), 0700); err != nil {
+		t.Fatal(err)
+	}
+	content := `{"type":"session_meta","payload":{"id":"mixed-counter"}}` + "\n"
+	for index, counters := range [][2]int{{100, 100}, {20, 20}, {50, 30}} {
+		content += fmt.Sprintf(`{"type":"turn_context","payload":{"turn_id":"turn-%d","model":"gpt-5.4"}}`+"\n", index)
+		content += fmt.Sprintf(`{"timestamp":"2026-09-08T01:00:0%dZ","type":"event_msg","payload":{"type":"token_count","info":{"total_token_usage":{"input_tokens":%d,"total_tokens":%d},"last_token_usage":{"input_tokens":%d,"total_tokens":%d}}}}`+"\n",
+			index, counters[0], counters[0], counters[1], counters[1])
+	}
+	if err := os.WriteFile(filepath.Join(home, "sessions", "mixed.jsonl"), []byte(content), 0600); err != nil {
+		t.Fatal(err)
+	}
+	path := filepath.Join(root, "usage.sqlite")
+	st, err := store.Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := st.InsertEvent(ctx, model.UsageEvent{ID: "old-overcount", SessionID: "mixed-counter", Timestamp: time.Now(),
+		Usage: model.TokenUsage{Input: 170, Total: 170}, Provenance: model.ProvenanceSessionJSONL, Confidence: model.ConfidenceExact,
+	}, "mixed.jsonl"); err != nil {
+		t.Fatal(err)
+	}
+	st.Close()
+	db, err := sql.Open("sqlite", path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := db.Exec(`UPDATE meta SET value='9' WHERE key='schema_version'`); err != nil {
+		t.Fatal(err)
+	}
+	db.Close()
+	st, err = store.Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer st.Close()
+	srv := &Server{Store: st, Scanner: &usage.Scanner{Store: st}, Homes: func() ([]string, error) { return []string{home}, nil }}
+	for _, step := range []struct {
+		rebuild bool
+		status  int
+		total   int64
+	}{
+		{false, 409, 170}, // Preserve retained history until the user confirms.
+		{true, 200, 150},  // Rebuild removes the repeated baseline.
+		{false, 200, 150}, // Normal incremental operation resumes idempotently.
+	} {
+		request := httptest.NewRequest("POST", "http://127.0.0.1/api/v1/rescan", strings.NewReader(fmt.Sprintf(`{"rebuild":%v}`, step.rebuild)))
+		response := httptest.NewRecorder()
+		srv.Handler().ServeHTTP(response, request)
+		if response.Code != step.status {
+			t.Fatalf("rescan status=%d want=%d: %s", response.Code, step.status, response.Body.String())
+		}
+		if step.status == 409 && !strings.Contains(response.Body.String(), `"rebuild_required":true`) {
+			t.Fatalf("Dashboard was not offered a rebuild: %s", response.Body.String())
+		}
+		summary, err := st.Summary(ctx, model.Filter{})
+		if err != nil || summary.GrandTotal != step.total || summary.CoverageIncomplete != (step.status == 409) {
+			t.Fatalf("history/quality after rescan: %+v, %v", summary, err)
+		}
+	}
+}