attempt to replicate LLaVa 1.0 paper using gpt2-xl as the base LLM. unfortunately the performance is quite bad with the fine-tuning testing i've done so im still working on that. you can find the checkpoints for the projection layer here: https://huggingface.co/snagasuri/multimodal-gpt-2/tree/main
i've also uploaded a bunch of visualizations that pretty clearly demonstrate the projection layer works fine, it's just the instruction following that's difficult, if anyone would like to collaborate feel free to reach out!