Is the inference done using only one <image> token?

[gowreesh-mago]

I have run the demo.sh and the prompt seems to digest all the images but has only 1 <image> token. The results seem to get ambiguous as the video gets longer. Is it doing inference based on the first frame it sees after segmenting?

[Stevetich]

I also found this problem, have you fixed it? Is this a bug, or the normal design?

[dipta007]

same issue

[ellemcfarlane]

I gave the demo with 7b model a video (16 frames only) in the demo that had a scene transition e.g. of a cat walking on the road, then someone biking and it was able to caption both, so this would be a counter example to what you're seeing, no?

It does look like they may only use 4-frames though, see here:
#12 (comment)
And here:
https://github.com/magic-research/PLLaVA/blob/main/tasks/eval/demo/pllava_demo.py#L122

Edit: ah but in the bash script that actually runs the demo, frames is set to 16:
https://github.com/magic-research/PLLaVA/blob/main/scripts/demo.sh#L3