A Lightweight Audio-Visual Speaker Detection System for Assistive Video Captioning

In this paper we introduce a novel dynamic subtitling system aimed at improving media accessibility for individuals with hearing impairments. The system is built upon an end-to-end active speaker detection framework that leverages joint audio and visual cues through multimodal feature integration. The architecture comprises two lightweight convolutional neural networks, each dedicated to a separate modality, with a cross-modal attention mechanism enhancing the interaction between them. This enables precise localization of the active speaker, allowing subtitle segments to be dynamically placed near the speaker's face to enhance readability and contextual alignment. The proposed approach is evaluated both quantitatively and qualitatively. On a benchmark dataset of 3 0 video samples, it achieves a gain of over 0.7 % in mean Average Precision compared to state-of-the-art ASD techniques. To evaluate the perceptual effectiveness of the system, a structured user study was conducted comparing three subtitle presentation strategies: traditional fixed-position captions, speaker-aligned speech-bubble overlays, and the proposed dynamic placement method based on region-ofinterest selection. Participants rated each version based on viewing comfort and visual effort, with the results showing a clear preference for the proposed system in terms of usability and reduced eye strain.