Real-Time Bengali Sign Language Recognition on Word and Read Out Loud The Speech Based on Computer Vision

Abstract

What if you are a foreigner among your own people due to not speaking in the same language as the community you’re born in, living in? And even it’s not your choice to make. It’s just a glimpse of the harshness of an average deaf person has to go through from the very moment he was born. He is a foreigner among his own people and it’s not by choice, but because of his disability. There are sign languages for such person to communicate with the world, but it’s also not helping him that much to avoid the communication isolation, due to the lack of interest in learning another language by his surrounding people. From this scenario the idea of this research arose, if we could mitigate the hassle of learning sign language for non-deaf person through an AI system that could interpret sign language for him, even if it means one way, at least the life of a deaf would be much easier than now. But the first problem we faced was lack of dataset to train state of art models like YOLO family, detectron2, 3d CNN etc. There is Ishara-Lipi but it’s only hand wrist images, to build a robust model we might need more variation and quantity. So, we managed to gather around 2000 dataset for Bengali alphabet and train and build basic model like VGG16, VGG19 with accuracy of 55% and 52% respectively, to more advance model like 3D CNN to detect signs in real-time and generate words to sentence and also read out loud for people who could not read.