Explainable Multimodal Hybrid Framework for Cervical Cancer Detection via Vision Transformers and LLM-Based Clinical Feature Fusion

Abstract

Cervical cancer is still among the major female causes of death globally, particularly in regions with limited access to healthcare. Although deep learning has been prospective for computer-aided diagnosis, its lack of interpretability and limited use of heterogeneous data types have limited clinical acceptance. In this paper, we present an interpretable multimodal hybrid deep learning architecture that combines Vision Transformers (ViT) with clinical metadata via Large Language Model (LLM)-assisted fusion. Visual features are learned from Pap smear images with ViT and enhanced predictive accuracy is achieved with the incorporation of structured clinical inputs such as patient history, HPV status, and cytology scores, regulated through an LLM-based attention mechanism.For feature imbalance handling and enhanced interpretability, we apply a dual-branch framework wherein image and text streams are conjoined via a semantic fusion layer for facilitating cross-modal alignment. Transparency is achieved by employing Lime, Saliency maps, and Grad-CAM visualizations that allow clinicians to trace-back predictions onto predictive image regions and metadata attributes. Comprehensive testing on benchmark datasets including SIPaKMeD and Herlev indicates that our approach achieves higher classification accuracy (99.93%) than single CNNs, ViTs, and classic machine learning models. Furthermore, hybrid models such as ViT-MobileNet and ViT-BERT are superior in generalization and outperform state-of-the-art methods for cell-level classification and patient-level classification tasks.Overall, the system enhances cervical cancer screening by providing a clear, stable, and clinically viable AI solution for early detection. Future directions include adding information on histopathology and real-time deployment through web-based diagnostic tools.