Learning Selective Vision-Language Inference for Vehicle Make and Model Recognition

Vision-Language Models provide strong zero-shot capabilities for fine-grained recognition, but their high computational cost limits scalability in video-based applications. In this work, we address track-level vehicle make-and-model recognition under a constrained inference computational cost. Rather than exhaustively evaluating all frames within a vehicle track, we propose a learned selective inference framework that predicts the reliability of VLM predictions at the frame level and queries only the most informative instances. The method formulates track-level recognition as a budget-aware decision problem and introduces a lightweight correctness estimator that models the conditional probability of VLM accuracy using visual and geometric cues. To benchmark this setting, we introduce the Vehicle Brand Recognition Dataset (VBRD), designed for fine-grained vehicle recognition in realistic multi-frame tracking scenarios. Experimental results demonstrate that selective querying with as few as five VLM evaluations per track achieves competitive or superior accuracy compared to full-frame aggregation while reducing computational cost by more than 50x. The proposed framework enables scalable deployment of VLM-based recognition systems in long-duration video streams without modifying or fine-tuning the underlying foundation model.