test(btw): evaluate rule-based task classification

[YUZHEthefool]

Author note

• This issue has exactly one author note, matching the actual author.

Goal

A deterministic classifier is a useful comparison baseline, but keyword matches alone do not establish correct routing or sensitivity to capability changes.

Proposal

Implement and evaluate only the rule-based candidate in its own experimental PR. Start with the prototype's frozen rules and run the shared dataset and capability fixtures before considering targeted changes.

Design: PR #28 and its English proposal. Parent: P / #122. PR #28 is documentation only; this issue tracks follow-up implementation or evaluation.

Associated PR: PR #146 (draft; base split/btw-skill-routes).

Acceptance criteria

• Use the same baseline commit, versioned cases, capability fixtures, expected decisions, and metrics as R2 / #134/R3 / #135; explicit /work remains the common control baseline.
• Cover Latin word boundaries, CJK substrings, everyday queries, negated/quoted instructions, mixed requests, and identical text paired with different capability sets.
• Publish per-case decisions, false handoffs, missed handoffs, task outcomes where integration is available, and rule latency/cost; state where the rules cannot use capability information.
• Keep evaluation cases separate from tuning cases and document rule revisions rather than silently optimizing against the reported evaluation set.
• Provide reproducible offline and deterministic regression commands in this candidate PR; product adoption follows a separate comparison decision.

Dependencies and delivery

Blocked by: None. The common experiment protocol is agreed in the parent issue before classifier evaluation.

A parallel alternative to R2 / #134/R3 / #135, not a prerequisite or a stacked branch. The parent issue owns the shared evaluation protocol; integration trials use the same runnable work baseline.

Implementation anchors

Historical prototype 33ee103a62937db3e930c89ba47a648b75cc7772 provides provenance, not code to copy unchanged:

• astrbot/core/agent/btw/task_classifier.py: TaskClassifier and keyword matching.
• astrbot/core/agent/conversation_loop.py: prototype pre-model classification.

Integrate with the current tool catalog, Skill snapshot assembly, and authorization service, rechecking current master when implementation begins.

Validation

Commands below are required for the implementation PR, not results claimed by this issue.

Historical prototype tests to adapt with the feature (they are not shipped by documentation-only PR #28):

• tests/unit/test_btw.py
• tests/unit/test_conversation_loop.py

The experimental PR must supply a reproducible evaluation command for the shared versioned dataset and deterministic contract tests. Real-model outcomes and repeated-run variation are reported separately; no candidate evaluation has been run by filing this issue.

For changed documentation, run pnpm run docs:build from docs/ and make check-md from the repository root. Update OpenAPI and generated outputs together only if the HTTP contract changes.

Out of scope

A separate model call, conversation-owned model routing, production default changes, and a multi-classifier switch framework are outside this candidate PR.

Repository requirements

Follow AGENTS.md, GOVERNANCE.md, and AI_POLICY.md. This issue uses the task template. Keep Python 3.14+, current configuration shapes, existing authorization boundaries, and bilingual feature documentation. A follow-up PR requires human maintainer review and a separate AI-assisted review. Use the feat PR template for the corresponding work.

Author

Agent

Agent note

Codex prepared this development issue at the user's request from PR #28, its discussion, the referenced source, existing tests, and current-master integration points. AI assisted the research and English drafting. Filing this issue changes no runtime behavior and does not claim feature tests or model trials have passed. Remaining decisions concern implementation details and experimental evidence within the stated scope.