Multimodal large language models (MLLMs) can generate fluent visual responses even when the underlying visual prediction is weak, ambiguous, or incorrect. This work presents a retrieval-augmented, reliability-aware selective inference method that evaluates the strength and consistency of visual evidence before a prediction is communicated through a downstream multimodal response. A pretrained ResNet-50 encoder extracts normalized visual embeddings, and FAISS retrieves the top𝑘 reference images from an ImageNet-100 evidence database. Prediction reliability is assessed using retrieval similarity, class-support agreement, evidence margin, entropy-based uncertainty, and an aggregate reliability score. A decision gate then determines whether the prediction should be accepted, presented cautiously, or rejected through abstention or fallback. The selected decision is used to control the final user-facing response generated by the model. Experiments on ImageNet-100 show that the proposed method improves the accuracy of returned predictions from 85.84% to 88.88% at 89.04% coverage. The accepted error rate decreases from 14.16% to 11.12%, corresponding to a 3.04-percentage-point absolute reduction and a 21.48% relative reduction. Expected Calibration Error decreases from 7.40% to 5.87%, while high-reliability wrong predictions decrease from 263 to 226. These results demonstrate the potential of retrieval-derived evidence and selective decision gating as a post-hoc reliability mechanism for controlling visual predictions before they are incorporated into multimodal responses. The current evaluation is conducted in a controlled visual-classification setting and does not constitute a complete assessment of free-form MLLM hallucination.

1. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, “Learning transferable visual models from natural language supervision”, “Proceedings of the 38th International Conference on Machine Learning”, vol. 139 of Proceedings of Machine Learning Research, pp. 8748–8763, PMLR, 2021.
2. H. Liu, C. Li, Q. Wu, Y. J. Lee, “Visual instruction tuning”, “Advances in Neural Information Processing Systems”, vol. 36, pp. 34892–34916, Curran Associates, Inc., 2023, doi: 10.52202/075280-1516.
3. J. Li, D. Li, S. Savarese, S. Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models”, “Proceedings of the 40th International Conference on Machine Learning”, vol. 202 of Proceedings of Machine Learning Research, pp. 19730–19742, PMLR, 2023.
4. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, S. C. H. Hoi, “InstructBLIP: Towards general-purpose vision-language models with instruction tuning”, “Advances in Neural Information Processing Systems”, vol. 36, pp. 49250–49267, Curran Associates, Inc., 2023, doi: 10.52202/075280-2142.
5. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Bińkowski, R. Barreira, O. Vinyals, A. Zisserman, K. Simonyan, “Flamingo: A visual language model for few-shot learning”, “Advances in Neural Information Processing Systems”, vol. 35, pp. 23716–23736, Curran Associates, Inc., 2022, doi: 10.52202/068431-1723.
6. D. Zhu, J. Chen, X. Shen, X. Li, M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models”, “International Conference on Learning Representations”, 2024.
7. A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, K. Saenko, “Object hallucination in image captioning”, “Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing”, pp. 4035–4045, Association for Computational Linguistics, Brussels, Belgium, 2018, doi: 10.18653/v1/D18-1437.
8. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks”, “Advances in Neural Information Processing Systems”, vol. 33, pp. 9459–9474, Curran Associates, Inc., 2020.
9. K. Guu, K. Lee, Z. Tung, P. Pasupat, M. Chang, “Retrieval augmented language model pre-training”, “Proceedings of the 37th International Conference on Machine Learning”, vol. 119 of Proceedings of Machine Learning Research, pp. 3929–3938, PMLR, 2020.
10. N. Papernot, P. McDaniel, “Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning”, arXiv preprint arXiv:1803.04765, 2018, doi: 10.48550/arXiv.1803.04765.
11. J. Johnson, M. Douze, H. Jégou, “Billion-scale similarity search with GPUs”, IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2021, doi: 10.1109/TBDATA.2019.2921572.
12. D. Yan, Y. Wang, J. Wang, H. Wang, Z. Li, “K-nearest neighbor search by random projection forests”, IEEE Transactions on Big Data, vol. 7, no. 1, pp. 147–157, 2021, doi: 10.1109/TBDATA.2019.2908178.
13. C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, “On calibration of modern neural networks”, “Proceedings of the 34th International Conference on Machine Learning”, vol. 70 of Proceedings of Machine Learning Research, pp. 1321–1330, PMLR, 2017.
14. Y. Gal, Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning”, “Proceedings of the 33rd International Conference on Machine Learning”, vol. 48 of Proceedings of Machine Learning Research, pp. 1050–1059, PMLR, 2016.
15. B. Lakshminarayanan, A. Pritzel, C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles”, “Advances in Neural Information Processing Systems”, vol. 30, pp. 6402–6413, Curran Associates, Inc., 2017.
16. Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, J. Snoek, “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift”, “Advances in Neural Information Processing Systems”, vol. 32, pp. 13991–14002, Curran Associates, Inc., 2019.
17. Y. Geifman, R. El-Yaniv, “Selective classification for deep neural networks”, “Advances in Neural Information Processing Systems”, vol. 30, pp. 4878–4887, Curran Associates, Inc., 2017.
18. D. Yan, P. Wang, M. Linden, B. S. Knudsen, T. W. Randolph, “Statistical methods for tissue array images—algorithmic scoring and co-training”, The Annals of Applied Statistics, vol. 6, no. 3, pp. 1280–1305, 2012, doi: 10.1214/12-AOAS543.

Citations by Dimensions

Citations by PlumX

Crossref Citations

No related articles were found.