IIIT-H students build browser plugin for Indian Sign Language

The tool uses a 3D avatar to convert selected online text into Indian Sign Language, without requiring a server-based AI model or GPU
IIIT-H students build browser plugin for Indian Sign Language
IIIT-H students build browser plugin for Indian Sign Language
Updated on: 

Two undergraduate students from the International Institute of Information Technology Hyderabad (IIIT-H) have developed a browser plugin that converts selected text on web pages into Indian Sign Language (ISL) using a 3D avatar.

The tool, developed by third-year BTech students Masthan Sharif Shaik and Mohammed Raza from the Centre for Visual Information Technology (CVIT), aims to make online content more accessible to people with hearing impairments, for whom sign language is often their first language.

Supervised by professors CV Jawahar and Vinay Namboodiri of the University of Bath, UK, the researchers designed the plugin to operate directly within a browser in real time, without requiring specialised hardware, a GPU or an AI model running on a server.

How does the plugin work?

The researchers used videos from the official Indian Sign Language Research and Training Centre (ISLRTC) dictionary to reconstruct the signers' three-dimensional body, hand and finger movements using computer vision.

However, automatic reconstruction could produce inaccuracies in finger movements, which can alter the meaning of a sign. To address this, the team developed an annotation tool called Gizmo to correct key poses rather than manually editing every frame.

The system also converts English sentences into Indian Sign Language word order before displaying them. For instance, the sentence "He will come home" is rearranged as "He home come" in ISL.

When a user highlights text and selects the 'Click To Sign' option, the plugin retrieves the stored movements for individual signs and blends them to produce a continuous signing sequence. Words without corresponding signs are represented through fingerspelling, with each letter signed individually.

Test results and future improvements

Unlike systems that generate movements from sign notation, the plugin replays motion captured from real signers, allowing for more natural movements. Since the computationally intensive processing takes place offline during the creation of the sign dictionary, displaying a sentence requires only the playback and blending of stored movements.

In tests involving deaf and hard-of-hearing signers, participants understood approximately 89 per cent of the sentences presented by the avatar. However, the absence of facial expressions was identified as a limitation, as facial movements also convey grammatical information in ISL.

The researchers plan to incorporate facial expressions in the next version of the tool.

The work has been documented in a paper titled SignPlug – ISL Plugin for the Web, which has been accepted for presentation at the 17th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2026), scheduled to be held in Kolkata in December.

Prof CV Jawahar said that the team was working in two areas: sign language synthesis and sign language understanding. The plugin focuses on synthesising sign language from text using affordable, readily available hardware.