Sign Language Detection by Rasab

A production-grade, real-time American Sign Language (ASL) detection application powered by YOLOv5, FastAPI, and React.

---

🚀 Features

• Real-Time Detection: Low-latency video streaming and hand gesture inference utilizing WebSockets.
• High-Performance Backend: FastAPI-based server with YOLOv5 integration, automated GPU/CUDA acceleration, and singleton model loader.
• Sleek Front-End: Modern, glassmorphic UI built in React + TypeScript with Tailwind CSS and dynamic HTML5 Canvas overlay.
• Privacy-Centric: Video frames are processed entirely in-memory and are never stored or logged.
• Environment Driven: Hardened architecture fully configurable via environment variables.

---

🛠️ Tech Stack

• Frontend: React 18, Vite, TypeScript, Tailwind CSS, Lucide React, react-use-websocket.
• Backend: FastAPI, Python 3.11, PyTorch (torch), OpenCV (headless), NumPy, Uvicorn, Python-dotenv.
• Model: YOLOv5 Custom trained architecture (sign_language_best.pt).

---

📦 Installation

1. Clone the repository

git clone https://github.com/SyedRasab/Sign-Language-Detection.git
cd Sign-Language-Detection

2. Clone YOLOv5 Framework v7.0

To run the model inference, clone the official YOLOv5 v7.0 source code into the project root:

git clone https://github.com/ultralytics/yolov5.git --branch v7.0 --depth 1 yolov5

3. Backend Installation

Create a Python virtual environment, activate it, and install dependencies:

# Create virtual environment
python -m venv venv

# Activate it (Windows PowerShell)
.\venv\Scripts\Activate.ps1

# Activate it (Linux/macOS)
source venv/bin/activate

# Install requirements
pip install -r backend/requirements.txt

4. Frontend Installation

Install the Node.js packages:

cd frontend
npm install

---

⚙️ Environment Variables

Create a .env file in the root directory. Refer to .env.example for the required keys:

# Backend Server Configuration
HOST=0.0.0.0
PORT=8000
MODEL_PATH=sign_language_best.pt

# Frontend Configuration (used to locate the backend WebSocket server)
VITE_WS_URL=ws://localhost:8000/ws/detect

---

🚀 Usage

1. Run the Backend API Server

Activate your virtual environment, navigate to the backend directory, and start the FastAPI application:

cd backend
python main.py

The backend server will run on http://localhost:8000 (WebSocket endpoint available at /ws/detect).

2. Run the Frontend Development Server

From the frontend directory, start the Vite server:

cd frontend
npm run dev

The frontend development server will launch on http://localhost:5174 (or http://localhost:5173). Open the printed address in your web browser, click "Start Camera", and perform supported ASL gestures.

---

📁 Project Structure

Sign-Language-Detection/
├── backend/
│   ├── main.py              # FastAPI application server with WebSocket detection endpoint
│   └── requirements.txt     # Python backend dependencies
├── frontend/
│   ├── src/                 # React frontend application source
│   │   ├── CameraFeed.tsx   # Canvas overlays and real-time WebSocket video frame streaming
│   │   ├── GesturesGuide.tsx# UI Component detailing supported sign language gestures
│   │   └── App.tsx          # App structure and style layout
│   ├── index.html
│   ├── package.json
│   └── tailwind.config.js
├── sign_language_best.pt    # Custom trained YOLOv5 PyTorch model weights
├── Dockerfile               # Production Docker container setup
├── .env.example             # Template for environment configuration
├── .gitignore               # System-wide git tracking exclusions
└── README.md                # Professional project documentation

---

🔧 PyTorch 2.6+ Compatibility Fix

Important

PyTorch ≥ 2.6 changed the default loading behavior of torch.load() to weights_only=True, which blocks custom checkpoint formats from loading and raises a WeightsUnpickler error.

If you are running PyTorch 2.6 or later, please apply a quick patch to your cloned yolov5/ framework by running this terminal command:

python -c "
import pathlib
files = {
    'yolov5/models/experimental.py': [\"torch.load(attempt_download(w), map_location='cpu')\"],
    'yolov5/hubconf.py': [\"torch.load(attempt_download(path), map_location=device)\"],
    'yolov5/utils/general.py': [\"torch.load(f, map_location=torch.device('cpu'))\"],
    'yolov5/train.py': [
        \"torch.load(weights, map_location='cpu')\",
        \"torch.load(last, map_location='cpu')\"
    ],
}
for fpath, targets in files.items():
    p = pathlib.Path(fpath)
    if p.exists():
        text = p.read_text(encoding='utf-8')
        for t in targets:
            text = text.replace(t, t.replace(')', ', weights_only=False)'))
        p.write_text(text, encoding='utf-8')
        print(f'Patched: {fpath}')
print('Compatibility patch complete.')
"

---

🛠️ Troubleshooting

1. ModuleNotFoundError: No module named 'cv2'

Ensure you have installed the requirements inside your active virtual environment:

pip install opencv-python-headless

2. Slow Inference (2–5 FPS)

The system is running on CPU. Ensure a CUDA-capable NVIDIA graphics card is available and check your PyTorch installation:

python -c "import torch; print('CUDA Available:', torch.cuda.is_available())"

3. Black or Frozen Video Stream

• Verify that no other application (e.g. Zoom, Teams) is currently using your webcam.
• Grant camera access permissions in your browser.

---

⚠️ Dataset Exclusions

Please note that the custom raw training dataset and raw images folder (Data/) are not committed or included in this repository to keep the codebase lightweight and maintain production-ready focus.

---

🤝 Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

---

📄 License

This project is licensed under the terms of the MIT License. See LICENSE for details.