A Multi-modal BLIP-2 Approach for Video Captioning

Recent advances in video captioning have been largely driven by the integration of vision–language models (VLMs). However, jointly aligning audio, visual, and textual information, while fully leveraging rich audio–visual interactions for natural video understanding, remains challenging. Bridging this gap is essential for developing video captioning systems that are both semantically rich and human-aligned from a perceptual point of view. In this work, we introduce a novel multi-modal adapter inspired by the Q-Former architecture in Bootstrapping Language-Image Pre-training 2 (BLIP-2), explicitly designed to perform early-stage cross-modal feature extraction. By promoting fine-grained audio–visual interactions at early layers, our method enables deeper multi-modal integration and more effective cross-modal reasoning. Following the paradigm that leverages Pretrained Image-Text Models are Secretly Video Captioners (PIT-VC), our approach avoids video–text pre-training by transferring knowledge from BLIP-2’s image–text pre-training. As a result, we achieve state-of-the-art performance on two widely used vision-centric datasets, Microsoft Research Video to Text (MSR-VTT) and the latest version of the Video And Text Explanations dataset (Latest-VATEX), and attain competitive results on the audio-centric AudioCaps benchmark under a minimal training regime. These results demonstrate the effectiveness of early audio–visual fusion and highlight the potential of our architecture for scalable and robust multi-modal video captioning.