Skip to content

fix(search): improve async rebuild observability - #644

Closed
XiaoSeS wants to merge 1 commit into
mainfrom
fix/issue-597-search-rebuild-observability
Closed

fix(search): improve async rebuild observability#644
XiaoSeS wants to merge 1 commit into
mainfrom
fix/issue-597-search-rebuild-observability

Conversation

@XiaoSeS

@XiaoSeS XiaoSeS commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

修改内容

  • skillhubEventExecutor 增加 MDC 上下文传播与执行后恢复。
  • 搜索索引重建失败时增加 skillhub.search.rebuild.failure 计数。
  • 覆盖单个重建、批量部分失败、MDC 传播和 Registry 记录行为。

修复原因

标签搜索索引重建已是异步任务,但执行线程读取不到原始请求 MDC;重建异常也只有 ERROR 日志,无法通过指标发现。

Closes #597

实现方式

  • 任务提交时复制调用线程 MDC,执行结束后恢复工作线程原有上下文,避免线程复用串号。
  • 每个重建失败的 Skill 计数一次,正常重建不计数。
  • 不在线程池任务内增加重试,避免阻塞共享事件执行器;持久化补偿作为独立扩展。

验证

  • 当前 main 基线:MDC 断言稳定得到 null,两个指标断言均报告指标不存在。
  • Java 21 定向测试:6/6 通过。
  • 后端完整 reactor:662 项,失败 0、错误 0、跳过 1;已知顺序污染用例隔离运行 9/9 通过。
  • 前端:TypeScript、ESLint、181 个 Vitest 文件共 618 项测试通过。
  • 隔离 staging:15/15 smoke 通过,Web 健康检查通过。
  • 精确合入 big-main 的镜像远端隔离验证:83/83 通过。
  • 远端覆盖正常/失败/恢复、两个并发 request ID 的 MDC 隔离、单项与批量逐 Skill 计数、主事务、审计、幂等、权限、数据库级联、搜索结果和异常扫描。
  • 测试结束后隔离资源全部清理,既有服务镜像未变且保持健康。
  • 公开内容隐私扫描通过。

影响

  • 无 API、数据库、前端或依赖变更。
  • MDC 传播适用于共享执行器上的全部异步任务。
  • 新增一个无标签、低基数 Micrometer Counter。

Signed-off-by: XiaoSeS <87064762+XiaoSeS@users.noreply.github.com>
@XiaoSeS
XiaoSeS marked this pull request as ready for review July 30, 2026 13:10

@FenjuFu FenjuFu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — this closes exactly the observability gap I flagged on the #592 line. The MDC handling in the TaskDecorator is textbook-correct: it captures the caller's context at submit time, saves the worker thread's existing context before running, restores the caller context for the task, and in finally restores the worker's original context. That save-and-restore-the-worker step is the part people usually miss, and it's what prevents context bleed across pooled-thread reuse — nicely done.

The skillhub.search.rebuild.failure counter (one increment per failed skill, none on success) gives the metric-level visibility that was missing when rebuilds only produced ERROR logs. Not adding retry inside the shared executor is the right call — that would block the event pool; leaving persistent compensation as a separate extension keeps this focused. Tests cover single/batch-partial-failure/MDC-propagation/registry paths, and the reactor run is green.

@XiaoSeS

XiaoSeS commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

维护者复核结论:本 PR 的 MDC TaskDecorator 和低基数失败计数本身实现正确,但范围仍把通用日志关联基础设施与搜索索引交付绑定在一起,且失败计数不能解决索引永久陈旧的问题。

因此采用以下替代方案:

#664 已在精确合入 big-main 的镜像上完成隔离矩阵验证,68/68 断言通过。感谢本 PR 对 MDC 传播和指标边界的验证;这些思路已被吸收到通用方案和后续可靠交付设计中。由于实现已被更完整的通用方案替代,本 PR 关闭,不再合并。

@XiaoSeS

XiaoSeS commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

#663 / #664 的通用可观测性方案替代;搜索索引可靠交付继续在 #597 跟踪。

@XiaoSeS XiaoSeS closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Label search rebuild: async path loses MDC and has no retry/observability

2 participants