Skip to content

fix: cache corruption 时自动 fallback 重新生成,上传失败不阻断 job - #31

Closed
moxt-ai[bot] wants to merge 2 commits into
mainfrom
fix/cache-corruption-resilience
Closed

fix: cache corruption 时自动 fallback 重新生成,上传失败不阻断 job#31
moxt-ai[bot] wants to merge 2 commits into
mainfrom
fix/cache-corruption-resilience

Conversation

@moxt-ai

@moxt-ai moxt-ai Bot commented Jul 9, 2026

Copy link
Copy Markdown

解决什么问题

代理链路(sg-squid-test / runner 基础代理)不稳定,导致 S3 HTTP body 被截断,出现两类故障:

  1. ZstdError(下载截断):其他 job 从 S3 下载 .tar.zst 时拿到残缺文件,Archiver.extract()ZstdError,当前代码没有捕获,job 直接 crash
  2. IncompleteBody(上传截断)boto3 上传时实际 body 少于 Content-Length,S3 拒收,CloudStorageErrortarget.py 捕获后 sys.exit(1),构建失败

怎么解决的

cache.py TargetCacheBackendTypeCloud.load()

  • 扩展捕获范围:(ChecksumError, zstd.ZstdError) — 本地缓存损坏时删除本地目录,fall through 到云端重新下载
  • 云端重新下载后若仍然 ZstdError(即上传时就已截断),清理本地并 re-raise,让 target.py 走重建路径

target.py Target.execute() NORMAL 模式

  • cache.load() 外包 try/except zstd.ZstdError:捕获后 warn + 触发 prepare_deps + build 重建,再走正常保存流程
  • CloudStorageErrorsys.exit(1) 改为 logger.warning,缓存上传失败不阻断构建本身

为什么能解决

  • 截断的根因在代理,属于偶发故障,quack 能自愈就不应该让 job 整体失败
  • 本地缓存损坏 → 重新从 S3 拉;S3 也损坏(上传就烂了)→ 重新 build;上传失败 → 下次再缓存,本次产物已生成
  • 新增 2 个 unit test 覆盖两种 corrupt 场景

🤖 Created via Moxt | Workspace: Moxt | Triggered by: lisw@paraflow.com

@tigerBeA tigerBeA self-assigned this Jul 9, 2026
@tigerBeA
tigerBeA force-pushed the fix/cache-corruption-resilience branch from a858e33 to c18c054 Compare July 9, 2026 07:17
tigerBeA and others added 2 commits July 9, 2026 15:20
代理链路不稳定导致 S3 body 截断,出现两类故障:
1. ZstdError:从 S3 下载的 .tar.zst 残缺,本地解压失败
2. IncompleteBody:上传阶段 body 与 Content-Length 不一致

修复:
- cache.py load():本地缓存 ZstdError 时删除本地并从云端重新下载;
  云端缓存也损坏时清理本地并 re-raise,让 target.py 走重建路径
- target.py execute():NORMAL 模式下 load() 抛出 ZstdError 时
  fallback 到重新构建并保存新缓存,不再 crash job
- 上传失败(CloudStorageError)从 sys.exit(1) 改为 warning,
  缓存上传失败不影响构建结果
- 新增对应测试:corrupt local cache fallback、corrupt cloud cache re-raise

Co-Authored-By:  via Moxt <noreply@moxt.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant