test(btw): evaluate a separate model task classifier

[YUZHEthefool]

Author note

• This issue has exactly one author note, matching the actual author.

Goal

A separate model could make better routing decisions than keywords, but its extra call may add latency, cost, and another failure point. Those trade-offs need isolated evidence.

Proposal

Implement the separate model-classifier candidate in its own experimental PR: classify the request before conversation execution using the agreed request/capability fixtures, then measure it independently.

Design: PR #28 and its English proposal. Parent: P / #122. PR #28 is documentation only; this issue tracks follow-up implementation or evaluation.

Associated PR: PR #147 (draft; base split/btw-skill-routes).

Acceptance criteria

• Start from the same baseline as R1 / #133/R3 / #135 and use the same versioned cases, capability information, labels, and integration scenarios.
• Record model/version, prompts, parameters, per-case results, repeated-run variation, false/missed handoffs, added model calls, latency, and token cost.
• Define and test malformed/uncertain output, timeout, provider failure, unavailable capabilities, and requests neither loop can fulfill; route decisions do not authorize tools.
• Include deterministic contract tests and a reproducible model-evaluation command; report those forms of evidence separately and keep holdout cases separate from prompt tuning.
• The PR contains this candidate and its tests only, remains independent of the other classifier branches, and records its trade-offs for the parent comparison.

Dependencies and delivery

Blocked by: None. The common experiment protocol is agreed in the parent issue before classifier evaluation.

A parallel alternative to R1 / #133/R3 / #135. Establish the shared evaluation protocol first; product integration remains deferred until comparison.

Implementation anchors

Historical prototype 33ee103a62937db3e930c89ba47a648b75cc7772 provides provenance, not code to copy unchanged:

• astrbot/core/agent/btw/task_classifier.py: historical classification boundary, not an existing model implementation.
• astrbot/core/agent/conversation_loop.py: pre-conversation dispatch reference.
• astrbot/core/astr_main_agent.py: existing model/request ownership.

Integrate with the current tool catalog, Skill snapshot assembly, and authorization service, rechecking current master when implementation begins.

Validation

Commands below are required for the implementation PR, not results claimed by this issue.

The experimental PR must supply a reproducible evaluation command for the shared versioned dataset and deterministic contract tests. Real-model outcomes and repeated-run variation are reported separately; no candidate evaluation has been run by filing this issue.

For changed documentation, run pnpm run docs:build from docs/ and make check-md from the repository root. Update OpenAPI and generated outputs together only if the HTTP contract changes.

Out of scope

Selecting this architecture in advance, adding providers, and introducing a production framework for switching all candidate classifiers are outside scope.

Repository requirements

Follow AGENTS.md, GOVERNANCE.md, and AI_POLICY.md. This issue uses the task template. Keep Python 3.14+, current configuration shapes, existing authorization boundaries, and bilingual feature documentation. A follow-up PR requires human maintainer review and a separate AI-assisted review. Use the docs PR template for the corresponding work.

Author

Agent

Agent note

Codex prepared this development issue at the user's request from PR #28, its discussion, the referenced source, existing tests, and current-master integration points. AI assisted the research and English drafting. Filing this issue changes no runtime behavior and does not claim feature tests or model trials have passed. Remaining decisions concern implementation details and experimental evidence within the stated scope.