Skip to content

Research

Published and academic work

Machine learning research from university, including a peer-reviewed publication on sequence modelling for gesture recognition.

Peer-reviewed publication

Indian Sign Language Recognition

Attention-based LSTM for recognising ISL gestures from video. Published at IEEE InC4 2025.

Role
Co-author — third of four authors
Domain
Machine Learning & Data Science
Stack
Python · TensorFlow · NumPy · pandas · OpenCV
Result
94% classification accuracy
Read the paper(opens in a new tab)
Still: video frames of a sign gesture weighted by an attention layer and classified into an ISL term.

Overview

An attention-based LSTM that recognises Indian Sign Language gestures from video, published at IEEE InC4 2025. The model learns which frames of a gesture matter, rather than treating every frame alike.

The question

Indian Sign Language is used by millions and supported by far less technology than American Sign Language. And a sign is a movement, not a pose — models that classify frames one at a time miss what makes a gesture mean what it means.

Not every frame of a sign matters equally. Attention lets the model weight the ones that do.

Method

  1. Record

    A new dataset: seven ISL terms, filmed at high resolution — no gloves or depth sensors.

  2. Extract

    Pose and hand keypoints taken from every frame, low-value points dropped to cut noise.

  3. Sequence

    Each clip becomes a fixed-length sequence of keypoint frames.

  4. Model

    Three stacked LSTM layers learn short- and long-range movement.

  5. Attend

    An attention layer weights frames by importance before dense layers and a softmax classify the sign.

Evaluation

Metrics
Accuracy, precision, recall and F1, averaged over k-fold cross-validation.
Baselines
Two stacked GRUs, a GRU-LSTM hybrid, and VGG19.
Outcome
The attention model led all three baselines on every metric. Full figures are in the paper.

My Role

Role

Co-author — third of four authors

Contribution

  • Contributed to the attention-based LSTM model for gesture classification
  • Worked on frame-weighting so the model focuses on the most informative frames of a sequence
  • Contributed to the empirical evaluation against a traditional LSTM baseline

Team Context

Third author on a four-author paper with faculty supervision at Bennett University.

Results

94%

classification accuracy

210

video samples, 7 ISL terms

Limits

Seven signs, a few hundred clips, and a small group of young signers recorded in controlled conditions. It recognises isolated signs, not continuous signing, and was not evaluated on signers held out from training — so the accuracy says how well it separates these seven gestures, not how it would do on a stranger. The paper itself names lighting, background and signer variation as future work.

What I'd Improve

  • Evaluate signer-independently — train on some people, test on others. First fix, and the one that would most change the headline number.
  • Add facial expression and body posture, which carry grammatical meaning in ISL and are ignored here.
  • Grow the vocabulary and move from isolated signs toward continuous sentences.