AI-Based semantic video indexing: a unified survey from segmentation to retrieval

AI-based semantic video indexing transforms audiovisual streams into searchable and interpretable semantic representations. These representations support retrieval, temporal localization, summarization, question answering, and knowledge-oriented access. Recent advances in video representation learning, video-language pretraining, audio and speech modeling, dense captioning, and large multimodal models have expanded video understanding capabilities. However, the literature remains fragmented across task-specific areas such as segmentation, captioning, retrieval, and multimodal reasoning. This survey addresses this fragmentation by proposing an indexing-oriented taxonomy that connects five functions: defining temporal units, extracting multimodal evidence, aligning and fusing heterogeneous signals, constructing searchable semantic indices, and supporting end-user search and exploration. Representative methods, datasets, and evaluation protocols are reviewed according to their roles in this pipeline and their relevance to temporal granularity, modality coverage, grounding quality, retrieval effectiveness, interpretability, scalability, and robustness. The analysis highlights persistent challenges related to temporal ambiguity, weak cross-modal grounding, noisy supervision, domain transfer, hallucinated annotations, privacy, computational cost, and limited explainability. Finally, we outline future directions for reliable semantic access to large-scale video archives, with emphasis on temporal adaptability, hybrid indexing, provenance tracking, explainability, and knowledge-aware representations.