fix(storage): large Codex rollouts listed for import fail at the 64 MiB read cap

[sunrioa]

Summary

Maka lists a supported Codex Session in Settings → Import Tasks, but importing it fails because the rollout JSONL is larger than the adapter's fixed 64 MiB whole-file read limit.

The affected rollout is valid JSONL and belongs to a supported Codex source. Its metadata is available through Codex's SQLite catalog, so Maka presents it as importable. The failure occurs only after the user clicks Import, when CodexSessionAdapter.readSession() attempts to load the complete rollout into one string.

The Desktop then replaces the specific size-limit failure with:

This conversation could not be converted or saved. Check the source and try again.

#4226 tracks a different pre-read failure caused by a missing default model and explicitly identifies oversized rollouts as a separate, lower-priority case. Its linked PR #4227 improves the error shown for an unreadable source but does not remove the 64 MiB whole-file limit.

Environment

• Maka main: 148f8eb297c86aa3045c75e87e19cacd4967c2dc
• Platform: macOS
• Source adapter: Codex
• Codex Session source: vscode, which Maka supports
• Source rollout size: 438,450,309 bytes, approximately 418.1 MiB
• JSONL validation: all records parse successfully

Steps to reproduce

1. Have a supported Codex Session whose rollout JSONL exceeds 64 MiB.
2. Open Maka Desktop.
3. Go to Settings → Import Tasks.
4. Select the Codex source.
5. Locate the Session in the catalog.
6. Click Import.

The concrete Session used for verification appears normally in the import catalog because its metadata is present in Codex's state_5.sqlite.

Actual behavior

The import fails before conversion or persistence.

Directly invoking the current adapter against the affected Session consistently produces:

Error: Codex rollout exceeds 67108864 bytes

The Desktop displays only the generic conversion/save failure and does not tell the user that the source exceeded the import limit.

No partial Maka Session is committed.

Expected behavior

A supported and valid Codex Session should not appear importable and then fail only because its source rollout exceeds an undisclosed whole-file read limit.

Maka should either support importing the Session within bounded resource limits or clearly identify why the Session cannot be imported.

The Desktop should display an actionable, sanitized error instead of the generic conversion/save failure.

Root cause

CodexSessionAdapter defines a fixed 64 MiB source-file limit:

maka/packages/storage/src/codex-session-adapter.ts

Line 35 in 148f8eb

export const CODEX_ROLLOUT_MAX_BYTES = 64 * 1024 * 1024;

readSession() resolves the rollout and reads the complete file before conversion:

maka/packages/storage/src/codex-session-adapter.ts

Lines 110 to 118 in 148f8eb

	async readSession(sessionId: string): Promise<ExternalMakaSession> {
	assertSafeCodexSessionId(sessionId);
	const catalogEntry = await this.findCatalogEntry(sessionId);
	if (!catalogEntry) throw new Error(`Codex Session not found: ${sessionId}`);
	const rolloutPath = await this.resolveRolloutPath(catalogEntry.rolloutPath, sessionId);
	if (!rolloutPath) throw new Error(`Codex rollout is unavailable: ${sessionId}`);
	const text = await readBoundedUtf8File(rolloutPath, this.maxRolloutBytes);
	const converted = convertCodexRollout(text, sessionId, catalogEntry.name, catalogEntry.cwd);

readBoundedUtf8File() rejects the file using its stat() size before parsing any records:

maka/packages/storage/src/codex-session-adapter.ts

Lines 722 to 741 in 148f8eb

	async function readBoundedUtf8File(path: string, maxBytes: number): Promise<string> {
	const handle = await open(path, 'r');
	try {
	const metadata = await handle.stat();
	if (!metadata.isFile()) throw new Error('Codex rollout is not a regular file');
	if (metadata.size > maxBytes) throw new Error(`Codex rollout exceeds ${maxBytes} bytes`);
	const chunks: Buffer[] = [];
	let total = 0;
	for (;;) {
	const buffer = Buffer.allocUnsafe(Math.min(64 * 1024, maxBytes + 1 - total));
	const { bytesRead } = await handle.read(buffer, 0, buffer.length, total);
	if (bytesRead === 0) break;
	total += bytesRead;
	if (total > maxBytes) throw new Error(`Codex rollout exceeds ${maxBytes} bytes`);
	chunks.push(buffer.subarray(0, bytesRead));
	if (total === maxBytes) {
	const probe = Buffer.allocUnsafe(1);
	if ((await handle.read(probe, 0, 1, total)).bytesRead > 0) {
	throw new Error(`Codex rollout exceeds ${maxBytes} bytes`);
	}

Catalog listing does not expose this incompatibility because it reads lightweight metadata instead of the complete rollout.

Suggested fix

Use bounded streaming JSONL parsing instead of reading the complete Codex rollout into one string. Parse and convert records incrementally while preserving resource limits and atomic Session persistence. The fix should not simply raise the 64 MiB limit. If a Session still cannot be imported safely, return a specific sanitized error and expose that reason in the Desktop.

AI assistance

OpenAI Codex assisted with source inspection, local reproduction, and drafting this issue. I reviewed and approved the final report.

简体中文

概述

Maka 会在设置 → 导入任务中列出一个受支持的 Codex Session，但由于该会话的 rollout JSONL 超过适配器固定的 64 MiB 整文件读取上限，导入会失败。

受影响的 rollout 是有效的 JSONL，并且属于受支持的 Codex 来源。它的元数据存在于 Codex 的 SQLite 目录中，因此 Maka 会将其显示为可导入。只有在用户点击导入后，CodexSessionAdapter.readSession() 尝试把完整 rollout 一次性读取为一个字符串时，操作才会失败。

Desktop 随后把具体的大小限制错误替换为：

该对话无法转换或保存。请检查来源后重试。

#4226 跟踪的是未配置默认模型导致的另一项读取前失败，并明确将超大 rollout 标为独立、较低优先级的问题。其关联 PR #4227 改善了无法读取来源时显示的错误，但不会移除 64 MiB 整文件读取上限。

环境

• Maka main：148f8eb297c86aa3045c75e87e19cacd4967c2dc
• 平台：macOS
• 来源适配器：Codex
• Codex Session 来源：vscode，属于 Maka 支持的来源
• 来源 rollout 大小：438,450,309 字节，约 418.1 MiB
• JSONL 校验：全部记录均可成功解析

复现步骤

1. 准备一个 rollout JSONL 超过 64 MiB 的受支持 Codex Session。
2. 打开 Maka Desktop。
3. 进入设置 → 导入任务。
4. 选择 Codex 来源。
5. 在目录中找到该 Session。
6. 点击导入。

用于验证的实际 Session 会正常出现在导入目录中，因为它的元数据存在于 Codex 的 state_5.sqlite 中。

实际行为

导入在转换或持久化之前失败。

直接使用当前适配器读取受影响 Session 时，可以稳定得到：

Error: Codex rollout exceeds 67108864 bytes

Desktop 只显示通用的转换/保存失败提示，没有告诉用户来源超过了导入上限。

不会留下部分创建的 Maka Session。

预期行为

一个受支持且有效的 Codex Session 不应先显示为可导入，然后仅因为其来源 rollout 超过未公开的整文件读取上限而失败。

Maka 应在受限资源范围内支持导入该 Session，或者明确说明它无法导入的原因。

Desktop 应显示可操作且经过脱敏的错误，而不是通用的转换/保存失败提示。

根因

CodexSessionAdapter 定义了固定的 64 MiB 来源文件上限：

maka/packages/storage/src/codex-session-adapter.ts

Line 35 in 148f8eb

export const CODEX_ROLLOUT_MAX_BYTES = 64 * 1024 * 1024;

readSession() 在转换前解析 rollout 路径并读取完整文件：

maka/packages/storage/src/codex-session-adapter.ts

Lines 110 to 118 in 148f8eb

	async readSession(sessionId: string): Promise<ExternalMakaSession> {
	assertSafeCodexSessionId(sessionId);
	const catalogEntry = await this.findCatalogEntry(sessionId);
	if (!catalogEntry) throw new Error(`Codex Session not found: ${sessionId}`);
	const rolloutPath = await this.resolveRolloutPath(catalogEntry.rolloutPath, sessionId);
	if (!rolloutPath) throw new Error(`Codex rollout is unavailable: ${sessionId}`);
	const text = await readBoundedUtf8File(rolloutPath, this.maxRolloutBytes);
	const converted = convertCodexRollout(text, sessionId, catalogEntry.name, catalogEntry.cwd);

readBoundedUtf8File() 在解析任何记录之前，就根据 stat() 返回的文件大小拒绝该文件：

maka/packages/storage/src/codex-session-adapter.ts

Lines 722 to 741 in 148f8eb

	async function readBoundedUtf8File(path: string, maxBytes: number): Promise<string> {
	const handle = await open(path, 'r');
	try {
	const metadata = await handle.stat();
	if (!metadata.isFile()) throw new Error('Codex rollout is not a regular file');
	if (metadata.size > maxBytes) throw new Error(`Codex rollout exceeds ${maxBytes} bytes`);
	const chunks: Buffer[] = [];
	let total = 0;
	for (;;) {
	const buffer = Buffer.allocUnsafe(Math.min(64 * 1024, maxBytes + 1 - total));
	const { bytesRead } = await handle.read(buffer, 0, buffer.length, total);
	if (bytesRead === 0) break;
	total += bytesRead;
	if (total > maxBytes) throw new Error(`Codex rollout exceeds ${maxBytes} bytes`);
	chunks.push(buffer.subarray(0, bytesRead));
	if (total === maxBytes) {
	const probe = Buffer.allocUnsafe(1);
	if ((await handle.read(probe, 0, 1, total)).bytesRead > 0) {
	throw new Error(`Codex rollout exceeds ${maxBytes} bytes`);
	}

目录列表只读取轻量元数据，不会暴露这种不兼容状态。

建议修复

使用受限的流式 JSONL 解析，替代将完整 Codex rollout 一次性读取为字符串。逐条解析并转换记录，同时保留资源限制和 Session 原子持久化。不应仅提高 64 MiB 上限。如果某个 Session 仍无法安全导入，应返回明确且经过脱敏的错误，并在 Desktop 中展示具体原因。

AI 辅助说明

OpenAI Codex 协助进行了源码检查、本地复现和 Issue 起草；最终报告已经由我审核并确认。

[liuxiaocs7]

Hi, @sunrioa.
Thanks for file this issue, if no one work on this, I'll try to fix!

[sunrioa]

Thanks! As far as I know, no one is working on this yet and there is no linked PR, so please feel free to take it.

[liuxiaocs7]

take