Hand Gesture Recognition and Conversion to Speech for Speech Impaired

Introduction

Communication plays a crucial role in human interaction. However, for individuals with speech impairments, expressing themselves through traditional means can be challenging. This project focuses on developing a hand gesture recognition system that converts gestures into speech, facilitating easier communication between speech-impaired individuals and others. By leveraging Internet of Things (IoT) technology and machine learning, this system aims to bridge the communication gap through a wearable glove equipped with flex sensors, converting hand movements into text and speech outputs.

Literature Survey

Gesture recognition is a widely researched area with various approaches. Oudah et al. (2020) discuss the classification of hand gestures into static and dynamic, focusing on the position of the fingers and hand shape. Camera-based methods, as described by Ji et al. (2019), provide a contactless solution for hand gesture recognition, utilizing sensors like Kinect to improve interaction speed and accuracy. However, systems based on sensor gloves, like those mentioned by Sonwalker et al. (2019), offer precise control but often face challenges such as high costs and user discomfort. This project builds on the wearable sensor approach, using flex sensors integrated into a glove to capture gestures without the need for expensive equipment, thus offering a more affordable and practical solution.

Proposed Methods

The project employs an IoT-based framework to recognize and convert hand gestures into speech. It consists of several modules:

1. Hardware Setup: The system uses five flex sensors attached to the fingertips of a glove to capture hand gestures. These sensors are connected to an ESP32 microcontroller, which processes the sensor data. The recognized gestures are transmitted to a Firebase database and displayed on an OLED screen, and a mobile application (Kodular Companion) produces the corresponding speech.

2. Gesture Recognition: Flex sensors change their resistance based on the bending of the fingers. The system identifies specific gestures by mapping the sensor values to predefined patterns. Each gesture is assigned a unique value that triggers a corresponding text and audio output.

3. Speech Conversion: The recognized gestures are converted into text, which is updated in real-time on the Firebase database. The Kodular Companion mobile app is used to produce the speech, thus enabling communication between the speech-impaired individual and others.

Results and Discussion

The system successfully captured and recognized a variety of hand gestures, converting them into both text and speech outputs. The project demonstrated effective communication using a cost-efficient method. The real-time database updates, along with the OLED and Kodular Companion outputs, proved to be reliable for both text and speech translation.

However, the system has limitations in recognizing complex gestures and could benefit from integrating additional sensors or refining the algorithms for better accuracy. Processing speed was also noted as a challenge, especially when using more complex machine learning algorithms.

Conclusion and Future Enhancements

This project provides a practical solution for bridging the communication gap faced by speech-impaired individuals through gesture recognition. The system is user-friendly and cost-effective, using IoT to enable real-time conversion of hand movements into speech. Future research could focus on enhancing gesture recognition accuracy by incorporating more advanced machine learning techniques. Additionally, exploring alternative methods for reducing response time and integrating more gestures into the system would improve its usability in daily life.

→ Click here to view and download