fix: cache corruption 时自动 fallback 重新生成,上传失败不阻断 job - #31
Closed
moxt-ai[bot] wants to merge 2 commits into
Closed
Conversation
tigerBeA
force-pushed
the
fix/cache-corruption-resilience
branch
from
July 9, 2026 07:17
a858e33 to
c18c054
Compare
代理链路不稳定导致 S3 body 截断,出现两类故障: 1. ZstdError:从 S3 下载的 .tar.zst 残缺,本地解压失败 2. IncompleteBody:上传阶段 body 与 Content-Length 不一致 修复: - cache.py load():本地缓存 ZstdError 时删除本地并从云端重新下载; 云端缓存也损坏时清理本地并 re-raise,让 target.py 走重建路径 - target.py execute():NORMAL 模式下 load() 抛出 ZstdError 时 fallback 到重新构建并保存新缓存,不再 crash job - 上传失败(CloudStorageError)从 sys.exit(1) 改为 warning, 缓存上传失败不影响构建结果 - 新增对应测试:corrupt local cache fallback、corrupt cloud cache re-raise Co-Authored-By: via Moxt <noreply@moxt.ai>
tigerBeA
force-pushed
the
fix/cache-corruption-resilience
branch
from
July 9, 2026 07:20
c18c054 to
f995028
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
解决什么问题
代理链路(sg-squid-test / runner 基础代理)不稳定,导致 S3 HTTP body 被截断,出现两类故障:
.tar.zst时拿到残缺文件,Archiver.extract()抛ZstdError,当前代码没有捕获,job 直接 crashboto3上传时实际 body 少于Content-Length,S3 拒收,CloudStorageError被target.py捕获后sys.exit(1),构建失败怎么解决的
cache.pyTargetCacheBackendTypeCloud.load()(ChecksumError, zstd.ZstdError)— 本地缓存损坏时删除本地目录,fall through 到云端重新下载target.py走重建路径target.pyTarget.execute()NORMAL 模式cache.load()外包try/except zstd.ZstdError:捕获后 warn + 触发prepare_deps + build重建,再走正常保存流程CloudStorageError由sys.exit(1)改为logger.warning,缓存上传失败不阻断构建本身为什么能解决