Eyemunicate

EyeMunicate is an open-source, zero-cost, privacy-first Augmentative and Alternative Communication (AAC) web application engineered for individuals with Motor Neuron Diseases (ALS), Locked-In Syndrome (LIS), cerebral palsy, and high-level spinal cord injuries.

It enables paralyzed and non-verbal patients to navigate virtual keyboards, form complete sentences, and synthesize local speech using only eye movements and deliberate eyelid blinks via standard commodity webcams (720p/1080p)—without requiring specialized infrared hardware, cloud servers, active internet connectivity, or user login.

🌟 Key Features

• 100% Client-Side & Zero-Cloud Privacy: All facial landmark processing (MediaPipe 478 3D landmarks), eye tracking, selection hysteresis, and speech synthesis run locally in volatile browser RAM. Zero video frames or personal data leave the device.
• Works Offline (Installable PWA): Operates completely offline post-installation via WebRTC, HTML5 Canvas, Web Speech API, and IndexedDB.
• Trilingual AAC Support: Instant runtime switching between:
  • English: Standard QWERTY grid and a 3x4 ETAOIN frequency-optimized grid (E T A O I N S H R D L U) placing high-frequency letters in central low-fatigue zones.
  • Urdu (اردو): Right-to-Left (RTL) Nasta'liq character matrix with pre-built native emergency phrases.
  • Chinese (中文): Pinyin initial/final matrix feeding into a candidate Hanzi selector.
• 9-Point Ridge-Regularized Calibration: Guided 3x3 calibration routine measuring Pupil Center Distance ($PCD_x, PCD_y$) with multi-sample vector averaging and 2nd-order 2D polynomial transform mapping.
• Midas-Touch & Jitter Prevention:
  • Speed-adaptive velocity filter (One-Euro style) for gaze jitter suppression.
  • Magnetic Target Snap Hysteresis (28px target snap margin around keys).
  • Dwell-time decay timer (600ms – 2000ms adjustable) with dynamic radial SVG countdown rings (DwellTimerRing).
  • Eye Aspect Ratio (EAR) deliberate selection blink override (400ms–1200ms).
• One-Glance Emergency Panel: One-dwell urgent phrase board (Pain, Water, Nurse, Move Me, Yes, No) with direct text-to-speech output.
• Caregiver Control Dashboard & Mouse Mode: Mouse/keyboard interface for adjusting sensitivity sliders, editing custom phrases, exporting CSV session logs, and testing in Mouse Simulation Mode (no camera required).

🛠️ Technology Stack

| Layer | Technology | Purpose | | :--- | :--- | :--- | | Framework | React 18 + TypeScript + Vite | High-performance, type-safe web application scaffold | | Vision Tracking | @mediapipe/tasks-vision | Client-side WASM 478 3D facial & iris landmark extraction | | Styling & UI | Tailwind CSS v4 + Lucide Icons | High-contrast dark theme (#121212) with enlarged 28px+ text | | Animations | Framer Motion + Canvas Confetti | Accessible, spring-based UI transitions & feedback | | Persistence | Dexie.js (IndexedDB) | Local client-side storage for settings, calibration, and session logs | | Speech Synthesis | Web Speech API | Offline local text-to-speech engine with visual captions | | PWA Caching | vite-plugin-pwa | Service worker offline caching for zero-network operation |

🚀 Quick Start & Installation

Prerequisites

• Node.js v18.0 or higher
• npm v9.0 or higher
• A standard laptop with built-in or USB RGB webcam (720p or 1080p)

Installation Steps

1. Clone the repository:

   git clone https://github.com/RasikhAli/EyeMunicate.git
   cd EyeMunicate
   

2. Install node dependencies:

   npm install
   

3. Start local development server:

   npm run dev
   

   Open http://localhost:3000 in Google Chrome or Microsoft Edge.

4. Build production PWA bundle:

   npm run build
   

📖 User & Caregiver Operational Guide

1. Patient Onboarding

1. Position the patient comfortably in bed or wheelchair (40 cm – 70 cm from the camera).
2. Ensure adequate ambient room lighting (avoid strong backlight behind the patient).
3. Click "Enable Camera & Begin Calibration" (or enable Mouse Simulation Mode for testing without a camera).

2. 9-Point Calibration Sequence

• Instruct the patient to fixate their gaze continuously on each animated target dot (1 to 9) as it appears on screen until the ring fills green.
• Upon completion, a Calibration Quality Score (0–100%) will be evaluated and stored in IndexedDB.

3. Patient Typing & Selection

• Gaze Dwell Selection: Looking at any virtual key fills the green radial countdown ring. Holding gaze for the set dwell time (default 1000ms) selects the character.
• Intentional Blink Selection: Executing a deliberate long blink (400ms – 1200ms) immediately selects the highlighted key.
• Emergency Panel: Looking at the persistent top panel (Pain, Water, Nurse) triggers immediate text-to-speech playback without requiring sentence construction.

4. Caregiver Dashboard

• Click the "Caregiver" button in the top right to access sensitivity sliders, language switching, phrase management, and session log exports (CSV).

📊 System Architecture & Gaze Mathematical Model

Eye Aspect Ratio (EAR) Blink Classification

$$\text{EAR} = \frac{|p_2 - p_6| + |p_3 - p_5|}{2 |p_1 - p_4|}$$

• $\text{EAR} \ge 0.19$: Eyes open
• $\text{Duration} < 150,\text{ms}$: Involuntary blink (ignored)
• $400,\text{ms} \le \text{Duration} \le 1200,\text{ms}$: Intentional selection blink
• $\text{Duration} > 1500,\text{ms}$: Eyes resting (dwell timers paused)

2nd-Order Polynomial Calibration Mapping

$$\begin{aligned} S_x &= a_0 + a_1 X + a_2 Y + a_3 X^2 + a_4 Y^2 + a_5 XY \ S_y &= b_0 + b_1 X + b_2 Y + b_3 X^2 + b_4 Y^2 + b_5 XY \end{aligned}$$ Coefficients $[a_0 \dots a_5]^T$ and $[b_0 \dots b_5]^T$ are solved using Ridge Regularized Least Squares $(M^T M + \lambda I)^{-1} M^T Y$.

📜 License & Grants

Developed as part of the Pakistan Science Foundation (PSF) – National Natural Science Foundation of China (NSFC) Joint Research Grant program for Assistive Biomedical Engineering.

Released under the open-source MIT License. Free for non-commercial global medical, clinical, and personal use.