TY - GEN
T1 - Training-Free Uncertainty Guidance for Complex Visual Tasks with MLLMs
AU - Kim, Sanghwan
AU - Xiao, Rui
AU - Alaniz, Stephan
AU - Xian, Yongqin
AU - Akata, Zeynep
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
PY - 2026/1/1
Y1 - 2026/1/1
N2 - Multimodal Large Language Models (MLLMs) often struggle with fine-grained perception, such as identifying small objects in high-resolution images or detecting key moments in long videos. Existing methods typically rely on complex, task-specific fine-tuning, which reduces generalizability and increases system complexity. In this work, we propose an effective, training-free framework that uses an MLLM’s intrinsic uncertainty as proactive guidance. Our core insight is that a model’s uncertainty decreases when provided with relevant visual information. We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most informative data. We apply this simple principle to three challenging visual tasks: Visual Search, Long Video Understanding, and Temporal Grounding, allowing off-the-shelf MLLMs to achieve performance competitive with specialized, fine-tuned systems. Our results demonstrate that leveraging intrinsic uncertainty is a powerful strategy for improving fine-grained multimodal performance. Code is available at https://github.com/ExplainableML/ug-framework.
AB - Multimodal Large Language Models (MLLMs) often struggle with fine-grained perception, such as identifying small objects in high-resolution images or detecting key moments in long videos. Existing methods typically rely on complex, task-specific fine-tuning, which reduces generalizability and increases system complexity. In this work, we propose an effective, training-free framework that uses an MLLM’s intrinsic uncertainty as proactive guidance. Our core insight is that a model’s uncertainty decreases when provided with relevant visual information. We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most informative data. We apply this simple principle to three challenging visual tasks: Visual Search, Long Video Understanding, and Temporal Grounding, allowing off-the-shelf MLLMs to achieve performance competitive with specialized, fine-tuned systems. Our results demonstrate that leveraging intrinsic uncertainty is a powerful strategy for improving fine-grained multimodal performance. Code is available at https://github.com/ExplainableML/ug-framework.
KW - Fine-Grained Perception
KW - Multimodal Large Language Models
KW - Uncertainty Estimation
UR - https://www.scopus.com/pages/publications/105054159640
U2 - 10.1007/978-3-032-37020-4_33
DO - 10.1007/978-3-032-37020-4_33
M3 - Conference contribution
AN - SCOPUS:105054159640
SN - 9783032370198
T3 - Lecture Notes in Computer Science
SP - 606
EP - 624
BT - Computer Vision – ECCV 2026 - 19th European Conference, 2026, Proceedings
A2 - Favaro, Paolo
A2 - Kukelova, Zuzana
A2 - Maki, Atsuto
A2 - Rohrbach, Anna
A2 - Schindler, Konrad
A2 - Tombari, Federico
PB - Springer Science and Business Media Deutschland GmbH
T2 - 19th European Conference on Computer Vision, ECCV 2026
Y2 - 8 September 2026 through 12 September 2026
ER -