Did evabyte support other evlaution architecture?

[EthanLI24]

Hi, wonderful and insightful job!

I attempted to reproduce your job and evaluate your Hugging Face checkpoint using OpenCompass. I set the batch size to 1 and enabled trust_remote_code, along with other settings you recommended. However, I found it challenging to achieve your results. I suspect there might be a discrepancy between my settings and yours. Have you evaluated your checkpoint on OpenCompass or other benchmarks? or any other experience can share with us?

Thanks!

[LZhengisme]

Thanks for your interest!

We haven't evaluated EvaByte using OpenCompass; instead, we've mainly relied on our fork of OLMES and bigcode-eval-harness for evals.

Evaluating byte-level models typically needs some tweaks compared to tokenized models. Here are a couple of things we've noticed during our evaluations that might help:

• Use >4x longer max_seqlen than tokenized models. Byte sequences are generally much longer (often >4x longer for natural text), so a longer context length is necessary to avoid truncation.
• For choice-based tasks (like MMLU), pay attention to the spacing between prompts and choices. Byte models typically perform better when whitespace is appended right after the prompt, rather than prepending spaces before each choice. For reference, check this implementation: https://github.com/allenai/olmes/blob/38af8b61741b01faeb60e90899addd7e6fa503a0/oe_eval/tasks/oe_eval_tasks/mmlu.py#L129.

Could you share a bit more details about the specific tasks you're evaluating and the discrepancies you're seeing? In particular, does your task involve generation (like coding or math problems), or is it a likelihood-based choice task (like MMLU)? That way we can help pinpoint the issue. Thanks!

[LZhengisme]

Hi we've updated the codebase to include our evaluation scripts for EvaByte. Feel free to check them out, and we hope they help xD

[EthanLI24]

Thanks for your interest!

We haven't evaluated EvaByte using OpenCompass; instead, we've mainly relied on our fork of OLMES and bigcode-eval-harness for evals.

Evaluating byte-level models typically needs some tweaks compared to tokenized models. Here are a couple of things we've noticed during our evaluations that might help:

• Use >4x longer max_seqlen than tokenized models. Byte sequences are generally much longer (often >4x longer for natural text), so a longer context length is necessary to avoid truncation.
• For choice-based tasks (like MMLU), pay attention to the spacing between prompts and choices. Byte models typically perform better when whitespace is appended right after the prompt, rather than prepending spaces before each choice. For reference, check this implementation: https://github.com/allenai/olmes/blob/38af8b61741b01faeb60e90899addd7e6fa503a0/oe_eval/tasks/oe_eval_tasks/mmlu.py#L129.

Could you share a bit more details about the specific tasks you're evaluating and the discrepancies you're seeing? In particular, does your task involve generation (like coding or math problems), or is it a likelihood-based choice task (like MMLU)? That way we can help pinpoint the issue. Thanks!

Thanks！！！！It's really tricky but useful hhh,
and I also noticed the length issue because when I first tested it, I used the same sequence length as the token-base model, and then found that many of the model’s responses were forcibly truncated by the evaluation framework before they were finished hhh

I will try to follow your scripts, many thanks!