Skip to content

fix(metrics): clarify chat-template fallback logging - #509

Draft
YibaiMeng wants to merge 1 commit into
mlcommons:mainfrom
YibaiMeng:codex/tokenizer-fallback-reporting
Draft

YibaiMeng wants to merge 1 commit into
mlcommons:mainfrom
YibaiMeng:codex/tokenizer-fallback-reporting

Conversation

@YibaiMeng

@YibaiMeng YibaiMeng commented Sep 18, 2026

Copy link
Copy Markdown

What does this PR do?

When a chat template cannot be applied, say that the client tokenizes the text directly. Log this handled fallback at INFO without a traceback. Token counting, n_errors, and reports are unchanged.

Type of change

  • Bug fix

Testing

  • Existing token-metrics unit tests: 62 passed.
  • uv run pre-commit run --all-files passed.

@YibaiMeng
YibaiMeng requested a review from a team September 18, 2026 02:24
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Sep 18, 2026
@YibaiMeng
YibaiMeng marked this pull request as draft September 18, 2026 02:29
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.34328% with 25 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@ad85144). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...utils/services/metrics_aggregator/token_metrics.py 80.95% 8 Missing ⚠️
..._utils/services/metrics_aggregator/tokenization.py 79.48% 8 Missing ⚠️
...sync_utils/services/metrics_aggregator/registry.py 61.11% 7 Missing ⚠️
...utils/services/metrics_aggregator/metrics_table.py 83.33% 1 Missing ⚠️
src/inference_endpoint/metrics/report.py 87.50% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #509   +/-   ##
=======================================
  Coverage        ?   80.89%           
=======================================
  Files           ?      154           
  Lines           ?    21323           
  Branches        ?        0           
=======================================
  Hits            ?    17250           
  Misses          ?     4073           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@YibaiMeng
YibaiMeng force-pushed the codex/tokenizer-fallback-reporting branch from cc2f1fe to 2938029 Compare September 18, 2026 02:36
@YibaiMeng YibaiMeng changed the title fix(metrics): report tokenization fallbacks for malformed responses fix(metrics): clarify chat-template fallback logging Sep 18, 2026
@YibaiMeng
YibaiMeng force-pushed the codex/tokenizer-fallback-reporting branch from 2938029 to 896cac8 Compare September 18, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants