Sign to Language AI Model

Cmotions is proud to present our innovative solution, the Sign to Language AI Model. This groundbreaking solution, developed in our Innovation Lab, leverages advanced AI technologies to bridge the communication gap for individuals who rely on sign language. Our model translates sign language into natural-sounding spoken words, significantly enhancing accessibility and inclusivity.

Solution Overview:

The Sign to Language AI Model solution by Cmotions is a groundbreaking technology that promotes inclusivity by breaking down communication barriers for individuals with hearing impairments. This solution leverages advanced AI technologies to translate sign language into spoken words in real-time, enabling seamless communication for those who rely on sign language. By addressing a critical need for communication accessibility, our solution fosters a more inclusive society where everyone has equal access to communication and information.

Technical Details: The solution was developed using a combination of pre-trained AI models, fine-tuning techniques, and large language models (LLMs) to ensure high accuracy and natural language output. The model was trained on the largest dataset for word-level American Sign Language (ASL) recognition, covering 2,000 common words in ASL. By utilizing Azure Databricks and Azure AI Foundry, the solution is scalable and can be easily integrated with other Microsoft technologies.

Data Collection and Preprocessing:

We utilized the Word-level Deep Sign Language Recognition from Video dataset, the largest video dataset for Word-Level American Sign Language (ASL) recognition, featuring 2,000 common words in ASL 

The dataset was used to train a pre-trained video model called 'VideoMAE' to classify individual signs (glosses).

Model Training and Fine-Tuning:

The model was fine-tuned to predict glosses from video clips of sign language. This process involved removing irrelevant elements and adding noise to improve accuracy, achieving an F1 score of 98.3% 

The glosses were then transformed into coherent sentences using a large language model (LLM) called qwen2.5 32b-instruct 

Speech Conversion:

The final step involved converting the written text into spoken words using a Text-to-Speech (gTTS) package .

Impact on the Community: 

The Sign to Language AI Model has the potential to significantly improve communication accessibility for individuals with hearing impairments. By enabling real-time translation of sign language into speech, our solution empowers individuals with hearing impairments to communicate more effectively with the world around them. This not only promotes inclusivity but also enhances the participation of this group in various aspects of daily life, such as education, work, and social interactions.

Impact:

This collaboration aims to integrate the Sign to Language AI Model into existing services, enhancing communication accessibility for their clients. The potential impact of this solution is immense, as it can significantly improve communication accessibility for individuals who rely on sign language. By enabling real-time translation of sign language into speech, our solution empowers individuals with hearing impairments to interact more effectively with the world around them.

Microsoft Technologies Used:

The solution utilizes the latest Microsoft technologies, such as Azure Databricks, Azure AI Foundry, and Azure Blob Storage, to provide a robust and scalable infrastructure. This ensures that the solution is not only technologically advanced but also reliable and secure for users.

Azure Databricks: The entire solution runs on Azure Databricks, ensuring scalability and integration with other Microsoft technologies 

Azure AI Foundry: Used for deploying the large language model (LLM) for transforming glosses into coherent sentences 

Blob Storage: Data is stored securely on Azure Blob Storage