Head Tracking & Blink Detection
A real-time computer vision system that enables hands-free computer control for motor-disabled users — mapping head orientation to cursor movement and detecting intentional blinks as mouse clicks.
Overview
What it does
Tracks 468 facial landmarks in real time from a standard webcam. Head pose moves the cursor; deliberate eye blinks trigger mouse clicks — full computer control, no hands.
Problem solved
Individuals with ALS, spinal cord injury, or other motor disabilities cannot use a mouse. Commercial solutions cost thousands. This runs on any laptop with a built-in webcam.
Built for
Motor-disabled users who need accessible computer input, assistive technology researchers, and accessibility tool developers looking for an open-source baseline.
MediaPipe's FaceMesh returns 468 3D landmark coordinates per frame. From these, two signals are extracted: head pose (rotation angles via solvePnP) drives the cursor via a scaled mapping, and the Eye Aspect Ratio (EAR) computed from eyelid landmarks triggers click events when it drops below a calibrated threshold for more than 3 consecutive frames.
The result is a system that runs at 60fps on a modern CPU, with cursor movement smooth enough for typical office tasks. A calibration mode lets each user set their comfortable blink threshold, accounting for individual differences in blinking behavior.
Technical Architecture
Real-time processing loop
Exponential Moving Average (EMA) smoothing with α=0.3 is applied to the cursor position before every PyAutoGUI move call — this eliminates the micro-jitter that appears when head position has small natural oscillations, making the cursor feel stable.
Key Features
Real-time 60fps tracking
MediaPipe FaceMesh runs at 60fps on CPU — no GPU needed. Latency under 50ms from movement to cursor update.
Per-user calibration
EAR blink threshold and cursor sensitivity are calibrated per session — accounts for different eye shapes and blink patterns.
Standard webcam only
Works with any 720p+ webcam. No depth sensor, IR camera, or specialized hardware required.
EMA-smoothed movement
Exponential moving average removes jitter from natural head tremors — cursor movement feels natural, not jerky.
Challenges & Solutions
False positive blinks from natural eye movement
Challenge: The average person blinks 15–20 times/minute involuntarily. Any blink-based click system must distinguish intentional clicks from natural reflexes.
Solution: Applied a dual filter: EAR must drop below threshold and stay below it for 3+ consecutive frames (~50ms). Natural reflexes are faster; intentional blinks are slower and meet both conditions.
Cursor jitter from head micro-tremors
Challenge: Raw head pose coordinates oscillate 2–4 pixels per frame even when the user is trying to hold still.
Solution: Exponential Moving Average with α=0.3 smooths the position stream: pos = α * raw + (1-α) * prev_pos. This dramatically reduces jitter while keeping response latency under 80ms.
Edge-of-screen precision
Challenge: The linear mapping of head rotation angles to screen coordinates made reaching screen edges require extreme head tilts.
Solution: Applied a nonlinear mapping function (sigmoid-like curve) that accelerates cursor speed near the screen edges while keeping fine control in the center — similar to how game joysticks handle dead zones.
Results & Impact
60fps
Real-time tracking
<50ms
End-to-end latency
468
Facial landmarks
$0
Hardware cost
The system enables full cursor navigation including text selection and scrolling on a standard laptop. Compared to commercial eye-tracking devices that cost $2,000–$5,000, this runs entirely on existing hardware with no additional cost — making accessible computing available to anyone with a webcam.
What I'd do differently
- →Add a dwell-click alternative to blink-click. Blink-to-click works but fatigues users in long sessions. Dwell-click — holding the cursor in one position for 800ms triggers a click — is often preferred by motor-disabled users and would be a better primary interaction mode.
- →Per-lighting-condition calibration. The EAR threshold varies with ambient lighting because pupil size and eyelid reflectance change. An adaptive threshold that recalibrates every 30 seconds based on recent EAR history would make it more robust across environments.
- →Package it properly for non-technical users. The current setup requires Python, MediaPipe, and a command-line launch. A desktop application with a system tray icon and one-click enable/disable would be the difference between a tool and a prototype.
Code Highlight
The Eye Aspect Ratio algorithm — the geometric formula that distinguishes intentional blinks from natural reflexes.
import numpy as np
def eye_aspect_ratio(landmarks, eye_indices):
"""
EAR = (||p2-p6|| + ||p3-p5||) / (2 * ||p1-p4||)
Drops sharply during a blink — threshold + frame count = click.
"""
pts = np.array([[landmarks[i].x, landmarks[i].y] for i in eye_indices])
A = np.linalg.norm(pts[1] - pts[5])
B = np.linalg.norm(pts[2] - pts[4])
C = np.linalg.norm(pts[0] - pts[3])
return (A + B) / (2.0 * C)
# EMA smoothing for stable cursor position
ALPHA = 0.3
prev_x, prev_y = screen_w // 2, screen_h // 2
def smooth_cursor(raw_x, raw_y):
global prev_x, prev_y
prev_x = ALPHA * raw_x + (1 - ALPHA) * prev_x
prev_y = ALPHA * raw_y + (1 - ALPHA) * prev_y
return int(prev_x), int(prev_y)