Skip to content

fix(cosmos): per-request TLS, secret-only token, complete service registration#183

Open
kurokoleung wants to merge 9 commits into
chaitin:mainfrom
kurokoleung:issue-31
Open

fix(cosmos): per-request TLS, secret-only token, complete service registration#183
kurokoleung wants to merge 9 commits into
chaitin:mainfrom
kurokoleung:issue-31

Conversation

@kurokoleung

@kurokoleung kurokoleung commented Jun 24, 2026

Copy link
Copy Markdown

Chaitin Cosmos (万象) OctoBus Service

OctoBus service package for Chaitin Cosmos (万象) platform log APIs via Pedestal JSON-RPC.

Features

  • SearchLogInfo: Get log details by IDs
  • SearchLogList: Search log list with keyword, time range, condition query, filters, and pagination
  • SearchAggregationStatistics: Get log aggregation statistics with time-series data

Configuration

Field Type Required Description
endpoint string yes Cosmos Pedestal RPC base URL, e.g. https://cosmos.example.com
headers object no Extra HTTP headers sent to Cosmos
timeoutMs integer no HTTP timeout in ms, default 5000
skipTlsVerify boolean no Skip TLS cert verification, default false

Secret

Field Type Required Description
api_token string yes Cosmos JWT token for Authorization bearer header

Quick Start

# Import service
octobus service import cosmos ./services/chaitin__cosmos

# Create instance
octobus instance create cosmos-test --service cosmos --config-json '{"endpoint":"https://cosmos.demo.chaitin.cn","skipTlsVerify":false}' --secret-json '{"api_token":"xxx"}'

# create capset

octobus capset create cosmos --name cosmos
octobus capset add-instance cosmos cosmos-test

# Call via gRPC (e.g. grpcurl)
grpcurl -plaintext \
    -H "x-octobus-capset: cosmos" \
    -H "x-octobus-service: cosmos" \
    -H "x-octobus-instance: cosmos-test" \
    -d '{"time_range_start": "1750550400", "time_range_end": "1750723200", "count": "1", "offset": "0"}' \
    octobus_addr:9000 \
    Chaitin_COSMOS.Chaitin_COSMOS/SearchLogList

# Call via MCP 

curl -X POST \
     -H "Content-Type: application/json" \
     "http://octobus_addr:9000/capsets/cosmos/mcp" \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

# Call via connect-rpc
curl -s -X POST \
    "http://octobus_addr:9000/capsets/cosmos/connect/cosmos-test/Chaitin_COSMOS.Chaitin_COSMOS/SearchLogList" \
    -H "Content-Type: application/json" \
    -d '{"time_range_start": "1750550400", "time_range_end": "1750723200", "count": "1", "offset": "0"}'
image image image

@kurokoleung kurokoleung changed the title 添加对接万象获取日志功能 Add: Chaitin Cosmos OctoBus Service Jun 25, 2026
@innomentats

Copy link
Copy Markdown
Member

Review 阻塞:这个服务包缺少自动化测试。diff 里没有 services/chaitin__cosmos/test/*.test.js,当前只有运行截图,不能充分证明实现的协议映射、错误处理和请求构造是正确的。请补充基于 mock upstream 的服务包测试,并保留/补充能证明 OctoBus 调用跑通的截图。

@monkeyscan

monkeyscan Bot commented Jun 26, 2026

Copy link
Copy Markdown

本次变更在 services/chaitin__cosmos 包中新增了约 1139 行的自动化测试文件 test/cosmos.test.js,并在 package.json 中添加了对应的 test 脚本。

测试覆盖了以下方面:

  • 输入验证:验证 api_tokenendpoint 缺失或格式不正确时抛出 INVALID_ARGUMENT
  • 请求构造:验证 SearchLogInfoSearchLogListSearchAggregationStatistics 三个 RPC 方法向上游发送的请求体、请求头、URL 拼接等是否正确。
  • 响应映射:验证上游返回的 flat array 和 nested object 两种数据格式都能正确映射到 gRPC 响应结构。
  • 错误处理:验证 HTTP 错误(401/403/500/502)、JSON-RPC 错误码、网络错误等都能正确映射为对应的 gRPC 状态码。
  • 配置解析:验证 skipTlsVerifytimeoutMsendpoint 别名(restBaseUrlbaseUrl)、请求头 JSON 字符串解析等配置项的行为。
  • 兼容性:测试了旧版 handlers 导出和常量定义的兼容性。

测试使用 node:test + node:assert/strict,通过 mock 全局 fetch 实现零外部依赖。整体测试结构清晰,覆盖率较高,没有发现严重的正确性、安全性或数据完整性缺陷。

@monkeyscan

monkeyscan Bot commented Jun 26, 2026

Copy link
Copy Markdown

该 PR 将 skipTlsVerify 的实现方式从向 fetch() 传入不支持的 TLS 选项,改为在请求前设置全局环境变量 NODE_TLS_REJECT_UNAUTHORIZED=0,并在 finally 块中恢复。这修正了 TLS 跳过无效的问题,但引入了全局状态竞争风险:在异步 fetch() 期间修改 process.env 会影响同一进程内所有并发的 TLS 连接。此外,timeoutMs 参数在重构后不再被使用,成为死代码。测试已更新为验证环境变量的设置与恢复行为。

Comment thread services/chaitin__cosmos/src/cosmos.js Outdated
@innomentats

Copy link
Copy Markdown
Member

google/protobuf/*.proto 是不是不要提交到仓库里来?

@innomentats
innomentats marked this pull request as draft June 26, 2026 11:43
@monkeyscan

monkeyscan Bot commented Jun 26, 2026

Copy link
Copy Markdown

PR Title: Add: Chaitin Cosmos OctoBus Service

Commit: 572f256

本次PR移除了两个vendored的Google Protocol Buffers标准库文件(struct.protowrappers.proto),它们都是protobuf的Well-Known Types(WKT)。这是一个清理性质的改动,目的是消除不必要的vendored依赖。

从diff来看,这两个文件都是Google官方的标准proto定义,protobuf编译器(protoc)本身会自带这些文件。因此,在正常的protobuf编译环境下,移除本地vendored版本不会影响编译。

需要确认的是:项目中其他proto文件若通过import "google/protobuf/struct.proto"等方式引用了这些WKT,只要编译时protoc的标准include路径可用,就能正常解析。建议在合并前确认CI/proto编译流程是否正常通过。

@kurokoleung
kurokoleung marked this pull request as ready for review June 26, 2026 13:12
@innomentats

Copy link
Copy Markdown
Member

这个 PR 需要修改后再合并。

目前有两个阻塞问题:

  1. service package 没有完成仓库级注册。PR 没有更新 services/package.json,没有增加 services/bin/cosmos.js,也没有更新 services/bin/octobus-tentacles.js。

  2. 实现里通过修改 process.env.NODE_TLS_REJECT_UNAUTHORIZED 来跳过 TLS 校验。这是进程级安全降级,会影响同一 Node 进程里的其它请求。请改为 per-request 的 HTTP client/agent 配置,或者移除运行时跳过 TLS 的能力,并在文档中说明需要自定义 CA 或有效证书。

另外,proto request message 中包含 api_token 字段。OctoBus service package 里凭据通常应来自 ctx.secret,而不是放在 request message 中,除非有非常明确的逐次调用覆盖需求。请调整 token 处理方式,并在补齐注册后增加 OctoBus Connect/gRPC/MCP 调用证据。

@innomentats
innomentats marked this pull request as draft June 30, 2026 16:11
@monkeyscan

monkeyscan Bot commented Jul 1, 2026

Copy link
Copy Markdown

PR Title: Add: Chaitin Cosmos OctoBus Service

Commit: 41df539

本 PR 对 Chaitin COSMOS 服务进行了三项核心变更:

  1. 移除 TLS 绕过选项:删除了 skipTlsVerify 配置及相关的 process.env.NODE_TLS_REJECT_UNAUTHORIZED 全局篡改逻辑,改用显式的 undici.Agent({ connect: { rejectUnauthorized: true } }) 作为 fetch dispatcher。这彻底修复了此前并发请求间的 TLS 设置竞争漏洞(historical finding)。

  2. 凭证来源限制为 secret:从 protobuf request 消息中删除 api_token 字段,并修改服务端逻辑使 token 只能从 ctx.secret(binding)获取,不再接受请求体中的 token。这符合 OctoBus 安全惯例。

  3. 完成服务注册:在根级 services/package.jsonbin/ 中补充了 cosmos 的入口注册,使服务可被 octobus-tentacles 加载。

整体评估:安全设计方向正确,代码结构清晰,测试覆盖了凭证来源与 TLS 无副作用场景。但 protobuf 字段重编号存在 wire-format 兼容性与数据完整性风险,建议处理。

@kurokoleung

Copy link
Copy Markdown
Author
image image image

lsl and others added 4 commits July 1, 2026 11:46
Add mock-upstream-based test suite covering protocol mapping,
error handling, and request construction for all 3 RPC methods
(SearchLogInfo, SearchLogList, SearchAggregationStatistics).

73 test cases across 12 suites:
- Input validation (api_token, endpoint)
- SearchLogInfo: request construction, headers, response mapping, proto3 defaults
- SearchLogList: keyword/time/condition/filter/pagination/organization mapping
- SearchAggregationStatistics: aggregation key/asc/response mapping
- Error handling: HTTP errors (401/403/500), JSON-RPC errors (-32600/-32601/-32000/1), network errors
- skipTlsVerify: tlsOptions passed to fetch
- timeoutMs: passed to fetch options
- Endpoint normalization and headers parsing
- handlers export verification

Uses node:test + node:assert (zero external test deps).

Co-Authored-By: Claude <noreply@anthropic.com>
The previous tlsOptions approach (insecureSkipVerify/tlsInsecureSkipVerify)
does not work with standard Node.js fetch (undici) — these are not valid
fetch options. Switch to setting NODE_TLS_REJECT_UNAUTHORIZED=0 before the
request and restoring it in a finally block, which is the correct way to
skip TLS verification with Node.js built-in fetch.

Also update tests: replace tlsOptions-based assertions with env var
manipulation verification, remove timeoutMs fetch option tests (no longer
passed to fetch).

Co-Authored-By: Claude <noreply@anthropic.com>
google/protobuf/struct.proto and wrappers.proto are standard well-known
types that proto compilers provide by default — they should not be vendored
in the service package.

Co-Authored-By: Claude <noreply@anthropic.com>
Comment thread services/chaitin__cosmos/proto/cosmos.proto Outdated
…istration

- Replace process.env.NODE_TLS_REJECT_UNAUTHORIZED with undici Agent
  dispatcher for per-request TLS enforcement (no process-wide downgrade)
- Remove skipTlsVerify from config.schema.json and runtime code
- Move api_token from proto request messages to ctx.secret only
- Re-number proto fields after removing api_token
- Add undici as explicit dependency in sub-package
- Complete service package registration: add cosmos entry to
  services/package.json bin/files, services/bin/cosmos.js wrapper,
  and services/bin/octobus-tentacles.js dispatch table
- Update README with TLS requirements and secret-only token docs
- Update tests: remove skipTlsVerify tests, add per-request dispatcher
  tests, update token tests to verify secret-only source (71 pass)

Co-Authored-By: Claude <noreply@anthropic.com>
@monkeyscan

monkeyscan Bot commented Jul 1, 2026

Copy link
Copy Markdown

PR Title: Add: Chaitin Cosmos OctoBus Service

Commit: 9b8eadb

本次 PR 是一次大规模的服务包扩容与重构,新增 20+ 个 OctoBus 服务适配器(Cosmos、DBAudit、Huorong ESM、Imperva WAF、Qingteng HIDS v5、Sangfor XDR 等),并将多个现有服务的 webhook/secret 从 config 迁移到 secret schema 以提升密钥安全性。SDK 侧新增 HTTP 工具模块(fetchWithTimeout、createTlsDispatcher)与更完善的错误处理/敏感信息脱敏逻辑。大量现有服务统一接入 SDK 新接口,标准化了超时、TLS 与响应处理。

整体架构趋于统一,但在具体实现中仍发现以下关键问题:

  1. Cosmos 服务中 callPedestalRpc 接收 timeoutMs 却未使用,导致请求无超时保护;
  2. Hermes Gateway CLI 的 sshOptions 未做白名单校验,存在任意 SSH 选项注入风险(可导致远程命令执行);
  3. Qingteng HIDS v5 新增实现仍沿用旧的 process.env.NODE_TLS_REJECT_UNAUTHORIZED 全局修改模式,在并发场景下会造成 TLS 校验竞争绕过,与 cosmos 已修复的历史缺陷同根同源。

建议优先修复上述安全与可靠性缺陷,并继续推动所有新服务统一使用 SDK 提供的 createTlsDispatcher + fetchWithTimeout 模式,避免重复引入全局 env 修改或超时失效问题。

@kurokoleung kurokoleung changed the title Add: Chaitin Cosmos OctoBus Service fix(cosmos): per-request TLS, secret-only token, complete service registration Jul 1, 2026
Comment thread services/chaitin__cosmos/src/cosmos.js
lsl and others added 2 commits July 1, 2026 11:59
Prevents wire-format misinterpretation by old clients — field 1 is
reserved instead of re-numbered, so legacy api_token payloads are
safely discarded rather than silently mapped to new fields.

Co-Authored-By: Claude <noreply@anthropic.com>
Previously timeoutMs was accepted as a parameter but never passed to
fetch, allowing requests to hang indefinitely. Now uses
AbortSignal.timeout(timeoutMs) on each fetch call, and maps
TimeoutError/AbortError to DEADLINE_EXCEEDED (gRPC code 4).

Co-Authored-By: Claude <noreply@anthropic.com>
@monkeyscan

monkeyscan Bot commented Jul 1, 2026

Copy link
Copy Markdown

PR Title: fix(cosmos): per-request TLS, secret-only token, c...

Commit: fbb8148

本次 PR 将之前未使用的 timeoutMs 参数通过 AbortSignal.timeout() 实际接入 fetch 调用,解决了请求可能无限挂起的问题。变更包括:在 callPedestalRpc 的 fetch options 中传入 AbortSignal.timeout(timeoutMs);在 catch 块中将 TimeoutErrorAbortError 统一映射为 gRPC DEADLINE_EXCEEDED;并补充了 4 个单元测试验证信号传递、默认值回退及超时错误码。整体逻辑正确,测试覆盖基本场景。但存在一处配置校验缺口:ctx.limits?.timeoutMs 来自外部配置,未对负值、非数字字符串或 Infinity 等非法值进行校验,这些值会导致 AbortSignal.timeout() 同步抛出 RangeError/TypeError,最终被错误地包装为 UNAVAILABLE 而非 INVALID_ARGUMENT

Comment thread services/chaitin__cosmos/src/cosmos.js
Prevents RangeError/TypeError from AbortSignal.timeout() when given
invalid values. Negative, zero, NaN, Infinity, and non-numeric
timeoutMs now throw INVALID_ARGUMENT (gRPC code 3) instead of being
silently passed through and wrapped as UNAVAILABLE.

Co-Authored-By: Claude <noreply@anthropic.com>
@monkeyscan

monkeyscan Bot commented Jul 1, 2026

Copy link
Copy Markdown

PR Title: fix(cosmos): per-request TLS, secret-only token, c...

Commit: 679b993

本次 PR 为 timeoutMs 参数引入了严格的输入校验函数 validateTimeoutMs,并在 rpcdef 中替换原先简单的 || 回退逻辑。

改动要点:

  1. 新增 validateTimeoutMs:对 ctx.limits?.timeoutMs 进行校验,要求必须是有限正整数;undefined/null 时回退到 DEFAULT_TIMEOUT_MS(5000ms)。
  2. 拒绝的非法值:负数、0、NaN、Infinity、非整数以及非数字字符串均会同步抛出 INVALID_ARGUMENT(gRPC code 3),避免此前将这些值直接传入 AbortSignal.timeout() 导致同步抛出的 RangeError/TypeError 被外层 catch 错误映射为 UNAVAILABLE 的问题。
  3. 错误信息可读性:通过 Number.isNaN / Number.isFinite 等分支生成清晰的 got xxx 提示。
  4. 测试覆盖:在 cosmos.test.js 中补充了 5 个断言,分别覆盖负值、Infinity、非数字字符串、0 和 NaN 五种非法输入场景。

总体评估:该改动精准修复了历史发现的配置注入风险,校验逻辑严谨,测试充分,无额外副作用。代码和测试均符合预期,未发现新的问题。

@kurokoleung

Copy link
Copy Markdown
Author
image image image

@kurokoleung
kurokoleung marked this pull request as ready for review July 1, 2026 04:59
@eetoc

eetoc commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

这个 PR 目前还不能合并。

阻塞问题:services/bin/octobus-tentacles.js 当前语法是坏的。cosmos 注册处多了一个 },,并且重复插入了 cloudatlas。我本地用当前 HEAD 内容执行语法检查会报错:

node --check services/bin/octobus-tentacles.js
SyntaxError: Unexpected string

这会导致 octobus-tentacles 入口不可用。请修复根级 dispatcher 注册,并建议补一个能覆盖 services/bin/octobus-tentacles.js 加载/语法的检查,避免 service package validate 通过但总入口损坏。

另外,PR 删除了 .github/workflows/ci.ymlpublish-sdk job 的 protobuf-compiler 安装步骤,这和 Cosmos 服务包本身无关,请从本 PR 移除,避免影响 SDK 发布流程。

截图方面已有补充,但建议在 PR 评论里明确说明最新截图分别对应当前 HEAD 的 gRPC / MCP / Connect 调用,并贴出脱敏后的 request/response 文本,方便确认真实 OctoBus 调用证据和当前代码一致。

@eetoc
eetoc marked this pull request as draft July 7, 2026 14:37
Co-Authored-By: Claude <noreply@anthropic.com>
@kurokoleung
kurokoleung marked this pull request as ready for review July 10, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants