Evaluation

[Germany321]

Thank you for your valuable work. Do you support an evaluation pipeline based on the lmms_eval protocol? I believe this would provide a more standardized and efficient interface compared to API-based evaluation.

[Zhoues]

Sorry for the delayed response. Currently, we do not support the lmms_eval protocol. However, our framework is similar to VILA, and since VILA supports the lmms_eval protocol, we should be able to support it quickly as well — although we have not implemented it yet.