Inquiry about evaluation

[EasonFasten]

Thank you for the good work!

I have some questions regarding the evaluation of MOSS-VL-Realtime on streaming benchmarks, particularly on the proactive tasks of StreamingBench and OVO-Bench. Since the two benchmarks both evaluate a model's proactivity in a pseudo-streaming manner (OVO-Bench queries the model at several preset time points, while StreamingBench polls the model every second and terminates upon the first positive response. Both of them do not actually let the model make the decision to response or remain silent by itself.), so I am wondering if you have followed the established evaluation protocals of these two benchmarks, or make some alterations to align them with real streaming settings.

If possible, could you please open-source your evaluation code? That would be a great help to me! Thank you again for your impressive work!

[ShacklesLay]

Thanks for the careful question!

StreamingBench – Proactive Output. Polling isn't the only way StreamingBench evaluates PO. As the paper explains (§4.1), polling is a fallback for models that "lack the ability to autonomously provide output without prompts"; for models that accept streaming input, PO is evaluated in streaming fashion (Appendix C.1, e.g. VideoLLM-online). The same procedure appears in the official repo's src/model/VideollmOnline.py: the query is inserted once at its timestamp, frames are fed one by one, and the model's first response is returned as its output. MOSS-VL-Realtime is a streaming model, so our evaluation follows that procedure. The query is given once at its query timestamp, frames then arrive at 1 fps, and at every frame the model itself decides whether to stay silent (<|silence|>) or to respond. No "Is it the right time to output …?" prompt is used. The first response is taken as the output time and scored with the official rule (correct if within ±2 s of the ground-truth time). The PO score therefore reflects the model's own decision about when to speak.

OVO-Bench – Forward Active Responding. Here we used the official protocol: questions are inserted into one continuous stream at the official probe timestamps and scored with the official calculate_score_forward. As you note, this measures whether the model answers correctly when probed, rather than whether it decides on its own when to respond.

Evaluation code. Our evaluation harness is still tied to internal infrastructure, and we're cleaning it up. The released realtime_inference/ code already covers the streaming setup (JSONL mode or model.create_realtime_session: fixed-fps frame feeding, an initial user turn, and user turns inserted mid-stream), so you can reproduce the protocol described above with it.

[EasonFasten]

Thank you for your swift response!