The training process detail

[lucasjinreal]

Hi, did u first train the projector, and then train projector + LLM, what's the detail of them.

[ByungKwanLee]

Oh!, before using external cv models, we slightly trained the projector only with 1 percent of total batch, and then we freeze it. I think it does not depend on the performance quietly, which is also evaluated in the paper MM1, apple paper.

We will add its minor training procedure to the manuscript. Thanks a lot!

[lucasjinreal]

so first trainng proj only , then free proj, and train vit and llm? Why freeze proj at second time?

[ByungKwanLee]

This is because I did not capture its effectiveness of training proj at second stage, at least on our model. The effectiveness of training MoAI-Mixer modules seem more better.

Actually, there is, either, no reason to not train vision encoder of MoAI, while LLaVA1.6 (Microsoft) and MM1 (Apple) trained vision encoder, but MoAI did not adopt vision encoder training and proj training.

[lucasjinreal]

Most of the time open ivison encoder training got worse result.
Do u think you can get better results if train them all?

[ByungKwanLee]

I cannot convince that training vision encoder always got worse result. I think it depends on the model setup or training setup. Recently, I read more papers training vision encoder leads to performance gain.

I would recommend reading Figure 10(c) of MM1 paper [link].

In addition, LLaVA1.6-blog [link] described full model training at second stage.

Thanks for great discussion of training detail!

[lucasjinreal]

The MM1 paper claims that open vit only works (better) then un free when imaga tokens are too many.

It might depends on the LLM and proj design

[ByungKwanLee]

Yes, recent papers have had lots of image tokens by using dynamic looking image. As I said, it depends on design of models. It makes sense many image feature can represent more rich onformation for VL tasks.

[cassiaaaaaa]

Sorry to ask a maybe naive question. What does the "projector" you mentioned mean? Is this MLP the projector? "Two linear layers with GELU activation function serve as the bridge connector between vision and language components, denoted by 'MLP'"

[ByungKwanLee]

The projector means a few layers with MLP, of which role is to make a bride connector from vision encoder to backbone multimodal LLM

[cassiaaaaaa]

The projector means a few layers with MLP, of which role is to make a bride connector from vision encoder to backbone multimodal LLM

Thanks for your reply. I noticed that you used qlora in training. Did you use qlora in training the projector? or you just use qlora in training the following steps?

[ByungKwanLee]

No reason to quatize projector. Thus, used backbone llm only

[cassiaaaaaa]

Thanks very much. Sorry to bother you again. Another question, did you use InternLM-7B or InternLM2-7B or InternLM2-base-7b as base model? And did you base "InternLM/lmdeploy" to write the training model?

[ByungKwanLee]

We used https://huggingface.co/internlm/internlm2-7b getting the highest likes

[cassiaaaaaa]

We used https://huggingface.co/internlm/internlm2-7b getting the highest likes

thanks very much.

[cassiaaaaaa]

Sorry to bother again, I noticed that you wrote all training is based on Llava-instruct-655k filtered by ShareGPT-4v. I want to ask that, when you train the vision projector, your mentioned 1/10 of the dataset, are they randomly selected from Llava-instruct-655k? Why didn't you use the LLAVA pretraining dataset of LCS-558K?