Skip to content
This repository was archived by the owner on Feb 25, 2026. It is now read-only.

feat(cli): add roll-call command for batch-testing model connectivity#463

Closed
cbrunnkvist wants to merge 1 commit intoKilo-Org:devfrom
cbrunnkvist:feat/roll-call-command
Closed

feat(cli): add roll-call command for batch-testing model connectivity#463
cbrunnkvist wants to merge 1 commit intoKilo-Org:devfrom
cbrunnkvist:feat/roll-call-command

Conversation

@cbrunnkvist
Copy link

@cbrunnkvist cbrunnkvist commented Feb 19, 2026

Summary

This PR adds a new roll-call subcommand that allows users to batch-test multiple models for connectivity and latency. This helps users discover which provider/model pairs are available and working.

Features

  • Filter models by regex pattern (required for safety - prevents accidental testing of all 300+ models)
  • Configurable prompt, timeout, and parallelization
  • JSON or table output formats
  • Proper provider options handling for reasoning models

Technical Details

The implementation:

  • Applies ProviderTransform options (maxOutputTokens, temperature, topP, topK, providerOptions)
  • Enables reasoning for dedicated thinking models via Kilo Gateway (models with "thinking" in their ID)
  • Uses proper session-based provider option generation

Testing

Tested with various model filters:

% bun run dev -- roll-call "free"                                                                                                                                                   [1]
$ bun run --cwd packages/opencode --conditions=browser src/index.ts roll-call free
Starting roll call for models with prompt: "Hello"
Timeout per model: 25000ms, Parallel calls: 5
Prompting 6 models...
Model                                    | Access | Snippet                                                                                      | Latency
----------------------------------------------------------------------------------------------------------------------------------------------------------
kilo/arcee-ai/trinity-large-preview:free | YES    | Hey there! I'm AFM, your multipurpose autonomous software agent. How can I assist you today? | 1215ms
kilo/openrouter/free                     | YES    | Hello! How can I assist you today?                                                           | 1681ms
kilo/minimax/minimax-m2.5:free           | YES    | Hello! How can I help you today?                                                             | 3171ms
kilo/stepfun/step-3.5-flash:free         | YES    | Hello! 👋 How can I assist you today?                                                        | 6340ms
kilo/z-ai/glm-5:free                     | YES    | Hello! How can I help you today?                                                             | 17428ms
kilo/corethink:free                      | NO     | (Invalid JSON response)                                                                      | 19473ms

5 accessible, 1 failed

Demonstrating truncation, error handling, color coding:

Screenshot 2026-02-20 at 11 22 15

Related Issue

Closes Kilo-Org/kilocode#6304

@vercel
Copy link

vercel bot commented Feb 19, 2026

@cbrunnkvist is attempting to deploy a commit to the Kilo Code Team on Vercel.

A member of the Team first needs to authorize it.

@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch from a8337fd to adb9f5d Compare February 19, 2026 10:08
@cbrunnkvist cbrunnkvist marked this pull request as draft February 19, 2026 10:58
@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch 2 times, most recently from 26183e0 to 21f6738 Compare February 19, 2026 14:49
@cbrunnkvist cbrunnkvist changed the title feat: add roll-call command for batch-testing model connectivity feat(cli): add roll-call command for batch-testing model connectivity Feb 20, 2026
@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch from 21f6738 to edafafb Compare February 20, 2026 02:41
@cbrunnkvist cbrunnkvist force-pushed the feat/roll-call-command branch from 733b6c7 to 886c3bb Compare February 20, 2026 04:06
@cbrunnkvist cbrunnkvist marked this pull request as ready for review February 20, 2026 04:27
.option("retries", {
type: "number",
default: 0,
describe: "Number of additional retries for each model call",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: The retries option is defined here but never used in the handler.

On line 129, the destructuring const { prompt, timeout, filter, parallel, output, verbose, quiet } = args omits retries, and no retry logic exists in processModel. Users passing --retries 3 would expect failed model calls to be retried, but the flag silently does nothing.

Either implement retry logic in processModel or remove this option to avoid misleading users.

@@ -0,0 +1,298 @@
import type { Argv } from "yargs"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Missing kilocode_change - new file marker.

Per AGENTS.md, new files outside of kilocode-named directories should have a // kilocode_change - new file comment at the top to help identify Kilo-specific additions during upstream merges. The test file has this marker but the source file does not.

Suggested change
import type { Argv } from "yargs"
// kilocode_change - new file
import type { Argv } from "yargs"

import { UpgradeCommand } from "./cli/cmd/upgrade"
import { UninstallCommand } from "./cli/cmd/uninstall"
import { ModelsCommand } from "./cli/cmd/models"
import { RollCallCommand } from "./cli/cmd/roll-call"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Missing kilocode_change marker on this new import.

Per AGENTS.md, modifications to shared upstream files should be marked with kilocode_change comments so they can be easily identified during merges.

Suggested change
import { RollCallCommand } from "./cli/cmd/roll-call"
import { RollCallCommand } from "./cli/cmd/roll-call" // kilocode_change

.command(ServeCommand)
.command(WebCommand)
.command(ModelsCommand)
.command(RollCallCommand)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Missing kilocode_change marker on this new command registration.

Suggested change
.command(RollCallCommand)
.command(RollCallCommand) // kilocode_change

@kilo-code-bot
Copy link
Contributor

kilo-code-bot bot commented Feb 20, 2026

Code Review Summary

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/cli/cmd/roll-call.ts 95 --retries CLI option is defined but never used — users passing --retries N get silently ignored

SUGGESTION

File Line Issue
packages/opencode/src/cli/cmd/roll-call.ts 1 Missing // kilocode_change - new file marker (test file has it, source file does not)
packages/opencode/src/index.ts 11 Missing kilocode_change marker on new import
packages/opencode/src/index.ts 135 Missing kilocode_change marker on new command registration
Files Reviewed (3 files)
  • packages/opencode/src/cli/cmd/roll-call.ts - 2 issues (1 WARNING, 1 SUGGESTION)
  • packages/opencode/src/index.ts - 2 issues (2 SUGGESTION)
  • packages/opencode/test/cli/roll-call.test.ts - 0 issues

Fix these issues in Kilo Cloud

@markijbema
Copy link
Contributor

Hi! Thank you for taking the time to contribute to this project—we really appreciate it. 🙏

We are currently working on re-platforming the core of our VS Code and JetBrains extensions to be based on our new Kilo CLI, with a complete rebuild based on OpenCode as our new foundation, and the moment has come to promote this repository to become the main repository. To do that, we moved the code from this repository to the kilocode repository.

This unfortunately means we cannot merge this branch here anymore. Please add https://github.com/Kilo-Org/kilocode.git as a remote, and push your branch there and create a new PR in https://github.com/Kilo-Org/kilocode . We unfortunately cannot do this for you as then the PR would not be in your name anymore. If you need any help, feel free to ask on our Discord in #kilo-dev-contributors

Sorry for the inconvenience and thank you for contributing to Kilo!

@markijbema markijbema closed this Feb 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Help user discover their provider/model pair availability

2 participants