v0.0.7 darwin-arm64: unsigned binary SIGKILLs; benchmark can't grade litellm-proxy endpoints (blocks custom-model benchmarking)

[adhd-diary]

Environment: macOS (Apple Silicon M1), role-model v0.0.7 (darwin-arm64 release asset), installed via scripts/install.sh.

Two independent problems — one blocking install, one blocking the benchmark workflow.

1. Release binary is unsigned → instant SIGKILL on Apple Silicon

role-model exits immediately with killed (exit 137). Diagnosis:

• codesign -dv → "code object is not signed at all"; spctl -a -vv → "rejected, source=no usable signature".
• On arm64, unsigned Mach-O binaries are SIGKILL'd by the kernel before main(). Intel tolerates this; Apple Silicon does not.

Workaround: codesign --sign - --force <binary>. Request: sign the darwin-arm64 release asset (at minimum ad-hoc).

Related: the SEA binary resolves its taxonomy data relative to process.cwd() instead of the executable location, so it only starts when launched from the install dir ("Unable to read taxonomy data file manifest.json"). It honors ROLE_MODEL_TAXONOMY_DATA_ROOT, so I wrapped the launcher — but resolving from the executable path would be the real fix.

2. Benchmark cannot grade models registered via litellm_proxy (circular blocker)

The static catalog for moonshotai-cn ships only 7 older models; kimi-k2.7-code, kimi-k3, kimi-k2.7-code-highspeed are absent. The documented way to add models is the litellm_proxy.providers runtime config. But endpoints registered that way get serving_source: vendor-litellm, and the Benchmark UI marks all of them "Excluded by current execution mode" (remote_only) — it appears to only grade direct remote-service endpoints.

Result: the exact models that require the litellm path to exist cannot be benchmarked through it. The benchmark is the step that feeds observed quality into routing, so this blocks the entire evidence-based-routing flow for any non-static-catalog model.

Smaller issues hit along the way (each worked around locally):

• litellm[proxy] 1.95.0 vendored proxy is incompatible with fastapi>=0.140 (ImportError: cannot import name 'get_flat_dependant'; removed in fastapi 0.140). Had to pin fastapi<0.140 in the vendored env.
• litellm does not expand shell ${VAR} in api_key — it forwards the literal text ("api key: ****KEY} is invalid"). Requires os.environ/VAR.
• The runtime strips the provider prefix before calling the vendor (resolveProviderLocalModelId), sending the bare id (kimi-k3) which the proxy rejects unless a bare-id model_name alias is also registered.
• For api.moonshot.cn, litellm_params.model must be openai/<id> pass-through; moonshot/<id> strips to a bare id the CN chat endpoint rejects.

Happy to provide logs or test a fix.

[adhd-diary]

Resolved — config-only path exists in v0.0.7: use the provider-account path with local-file credentials (no litellm needed).

Both requirements (benchmark-eligible + executable) can be met by direct remote-service endpoints; the missing piece is the credential backend. Execution gate in apps/runtime-host-bridge/src/index.ts:

• executeProviderRequest only requires the litellm vendor when the account's credential backend is not file-backed: useDirectExecution = backend === "local-file" || backend === "local-encrypted-file" (line ~20045). File-backed accounts fall through to direct HTTP execution via the ai-sdk-openai-compatible adapter even when currentUnifiedRuntimeConfig is non-null (config file always exists; version: "1.0" parses to a non-null config with decision_only).
• shouldUseLiveProviderExecution() (line ~12422) is the fallback gate for the no-config path; the file-backed path bypasses it entirely.

So the recipe is: register the accounts with the actual API key in credentialRef.ref. upsertProviderAccount (line ~23723) auto-converts {backend:"env", ref:"<sk-...>"} to {backend:"local-file", ref:"api-key/<provider>/<account>"} and persists the key via persistStaticCredentialFile — but only when looksLikeInlineApiKey(ref) matches (/^sk-[A-Za-z0-9_-]{8,}/, credential-ref-env.ts). Using the env var name ("MOONSHOT_API_KEY") skips the conversion, which is what produced the 503 VENDOR_NOT_CONFIGURED.

Verified live on the v0.0.7 SEA binary:

1. POST /api/role-model/accounts with credentialRef: {backend:"env", ref:"<sk-...>"} (full payload: providerAccountId, providerId, providerKind, authMode api-key-static, regionPolicy, baseUrlOverride, allowedModels, budgetPolicyRef/quotaPolicyRef default, status/healthStatus, rotationState).
2. Accounts flip to credentialBackendCanonical: local-file, lifecycle execution-ready.
3. Endpoints stay remote-service / remote_api, benchmarkEligible: true, healthy — no re-registration needed.
4. POST /v1/chat/completions → real upstream completions (deepseek: HTTP 200, content:"PONG"; moonshot: request reaches api.moonshot.cn, 429 quota_exhausted — user account needs recharge, not a config issue).

The earlier "Excluded by current execution mode" for vendor-litellm endpoints is the benchmarkEligible flag = routable-inventory membership; it flips false while an account is auth-broken/unroutable, and true once healthy (a litellm-configured run in this thread's history did execute, returning invalid-model 400s from upstream). The direct path sidesteps litellm entirely (also avoids the fastapi pin, os.environ/ syntax, openai/<id> pass-through and bare-id alias requirements).

Suggested upstream follow-ups (nice-to-have, not blockers):

• Accept inline keys for the env backend is already there; consider a docs note that pasting the key (not the env name) is what enables benchmark execution for API-key accounts.
• benchmarkEligible could additionally require "executable under current mode" so the UI stops presenting non-executable endpoints as runnable.

QA precedent in-repo: .recursive/run/69-benchmark-scoring-integrity/evidence/runtime/packaged-runtime-snapshot/accounts.json shows the packaged reruns used exactly this shape (deepseek.personal.deepseek-api-key / moonshot.personal.kimi-code with local-file refs) and produced real scores (0.75–0.98) for the same endpoints.

[try-works]

thanks for reporting this issue, looking into this and hope to provide a fix shortly