A Multi-Modal Deep Learning Framework for Breast Cancer Prediction and Stage Identification Using Mammography and Ultrasound Images

Abstract

Breast cancer remains one of the most prevalent causes of mortality among women worldwide, highlighting the urgent need for accurate, accessible, and automated diagnostic systems. This study investigates the effectiveness of deep learning models in classifying breast tumors as malignant or normal using two imaging modalities—mammography and ultrasound—sourced from publicly available Kaggle datasets. A combined dataset of 7,091 images was preprocessed through resizing, normalization, and augmentation before training four convolutional neural network architectures: a custom-designed CNN and three transfer learning models, DenseNet121, MobileNetV3, and ShuffleNetV2. The objective was to evaluate model performance, examine the benefits of multimodal imaging, and explore the feasibility of breast cancer stage prediction. Experimental results demonstrate that deep learning models can reliably classify breast tumors, with MobileNetV3 achieving the highest overall accuracy of 88.7%, followed closely by DenseNet121 at 88.2%. ShuffleNetV2 achieved the highest malignant recall (0.894), indicating stronger sensitivity in identifying cancer cases. These outcomes confirm that multimodal training enhances classification robustness by capturing complementary anatomical information from both imaging modalities. However, the study also reveals that stage prediction is not feasible using current public datasets, as they lack clinical annotations related to tumor staging. The findings underscore the potential of deep learning for breast cancer screening and highlight the need for stage-labeled datasets and richer clinical metadata. Future research should focus onmultimodal diagnostic frameworks that integrate imaging, clinical records, and advanced explainable AI techniques to support more comprehensive and clinically valuable breast cancer detection systems.