Transparent deep learning for medicinal plant recognition: A hybrid CNN–ViT approach with explainable AI on BDHerbalPlants

Abstract

Recognizing medicinal plants from images is crucial for biodiversity conservation and drug discovery, yet existing datasets are scarce and geographically limited. Ancient Bengal's medical system, Ayurveda, Unani, and Homeopathy, prominently rely on medicinal plants as natural product-based drugs. In this work, we present BDHerbalPlants, a curated dataset comprising 1792 high-quality images of 8 species of herbal plants native to Bangladesh. We benchmark five modern CNN-driven frameworks (EfficientNet-B3, EfficientNetV2-S, RegNetY-3.2GF, Xception, DenseNet201) and a transformer model (ViT-Base), along with a custom hybrid CNN to analyze performance variability. The models were optimized with Adam optimizer, cross-entropy loss, and training methodologies including early stopping and learning rate scheduling. We evaluate the impact of augmentation by comparing raw data training results (accuracy ranging from 96% to 99%) with augmented data training performance (accuracy ranging from 98% to 99.5%). ViT-Base gains the highest accuracy on raw data (99.3%, F1 score = 0.991), while with augmentation, Xception has the highest accuracy (99.5%, F1 score = 0.997). To interpret the models’ decisions, we apply LIME to the top-performing model, revealing the leaf regions with the most critical patterns and validating the models' emphasis on botanical features. Lastly, we demonstrate real-time inference via a Hugging Face Space, enabling user-friendly plant identification in field conditions. We demonstrate that in a consistent training protocol, model choice and augmentation have an influence on performance, and XAI and real-time deployment offer clarity and usefulness in the field of recognition