ASL Recognizer

Problem

Existing ASL recognition tools are either research-only or require expensive hardware. A lightweight, real-time system was needed that runs on consumer hardware with just a webcam, using open-source ML frameworks.

Constraints

• Must run in real-time on consumer hardware with just a webcam
• MediaPipe hand landmark extraction must be fast enough for live inference
• Model must handle varying lighting conditions and hand positions
• J and Z are motion-based letters, excluded from static classifier scope

Key Decisions

1. MediaPipe over custom CNN for hand detection

   Need reliable hand landmark extraction that works across different skin tones and lighting.

   Used Google's MediaPipe Hands for pre-built, production-quality hand landmark detection, then trained a lightweight classifier on extracted landmarks.

   Trade-off: Depends on MediaPipe accuracy, but dramatically reduces training data needs and computation compared to end-to-end CNN approaches.
2. Landmark features over raw pixel classification

   Raw image classification would require massive datasets and GPU training.

   Extract 21 hand landmarks (63 features) and train a simple dense network, making the model lightweight and fast for real-time inference.

   Trade-off: Loses some spatial information from raw images, but enables CPU-only real-time inference with high accuracy on static poses.