Build a Virtual Mouse with OpenCV and MediaPipe in Python

Want to build a virtual mouse with OpenCV and MediaPipe? This beginner Python tutorial shows you how to control your computer mouse with hand gestures — no special gloves, no expensive gear. Just a webcam, OpenCVA Python toolkit for working with cameras and images, and MediaPipeGoogle’s toolkit that can find hands, faces, and body points in video.

This guide is written for beginners. Technical Python / CV words are colored — hover them for a plain-English tip. Everyday words (webcam, finger, pinch) stay normal.

• Soft green = common library / beginner tech idea
• Warm orange = a bit more technical
• Cool blue = deeper concept

What you will build

A small Python app that:

1. Opens your webcam
2. Finds your hand in the video
3. Moves the mouse when you move your index finger
4. “Clicks” when you pinch (thumb + index fingertip close together)

This is computer visionTeaching a computer to understand images and video — useful on its own, and a friendly first step toward more advanced projects later.

We are not training a huge AI model here. We use ready-made tools (OpenCVA Python toolkit for cameras, images, and video + MediaPipeGoogle’s toolkit that can find hands, faces, and body points in video) that already know how to find hands.

What you need

Step 0 — Create a project folder

Open a terminal and run:

mkdir virtual-mouse
cd virtual-mouse
python -m venv .venv

Activate the virtual environment:

Windows (PowerShell):

.\.venv\Scripts\Activate.ps1

macOS / Linux:

source .venv/bin/activate

You should see (.venv) at the start of your prompt.

Step 1 — Install the packages

pip install opencv-python mediapipe pyautogui numpy

What each one does:

• opencv-python → OpenCVHelps Python open the webcam and draw on frames
• mediapipe → finds your hand and landmarksSpecial points on the hand (fingertips, knuckles, wrist)
• pyautogui → moves the real mouse cursor on your screen
• numpy → helps with numbers and math on points

On some Macs you may also need to allow Terminal (or your IDE) to control the computer in System Settings → Privacy & Security → Accessibility.

Step 2 — See yourself on camera (sanity check)

Create camera_test.py:

import cv2

# 0 usually means the default webcam
cap = cv2.VideoCapture(0)

if not cap.isOpened():
    raise SystemExit("Could not open webcam. Try another index: 1 or 2.")

print("Press Q to quit.")

while True:
    ok, frame = cap.read()
    if not ok:
        break

    # Mirror the image so it feels like a mirror
    frame = cv2.flip(frame, 1)

    cv2.imshow("Camera test", frame)

    # Wait 1ms for a key; quit on Q
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

Run it:

python camera_test.py

You should see a live video window. If the window is black, check that another app is not locking the camera, and try VideoCapture(1).

Photo to add: screenshot of the OpenCV window showing your face/desk from the webcam (mirror view).

Step 3 — Find your hand with MediaPipe

MediaPipe Hands returns up to 21 landmarksDots on the hand skeleton — fingertip, joints, wrist. Each landmark has x and y between 0 and 1 (relative to the image size).

Create hand_landmarks.py:

import cv2
import mediapipe as mp

mp_hands = mp.solutions.hands
mp_draw = mp.solutions.drawing_utils

cap = cv2.VideoCapture(0)

with mp_hands.Hands(
    static_image_mode=False,
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.6,
) as hands:
    while True:
        ok, frame = cap.read()
        if not ok:
            break

        frame = cv2.flip(frame, 1)
        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        result = hands.process(rgb)

        if result.multi_hand_landmarks:
            for hand in result.multi_hand_landmarks:
                # Draw the skeleton on the frame
                mp_draw.draw_landmarks(
                    frame,
                    hand,
                    mp_hands.HAND_CONNECTIONS,
                )

                # Landmark 8 = tip of the index finger
                h, w, _ = frame.shape
                tip = hand.landmark[8]
                cx, cy = int(tip.x * w), int(tip.y * h)
                cv2.circle(frame, (cx, cy), 10, (200, 213, 168), -1)

        cv2.imshow("Hand landmarks", frame)
        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

cap.release()
cv2.destroyAllWindows()

Tips if detection is flaky:

• Sit closer to the camera
• Keep your palm facing the camera
• Avoid busy backgrounds behind your hand

Photo to add: same webcam window with the MediaPipe hand skeleton drawn, and a green/leaf circle on the index fingertip.

Step 4 — Map finger position to the screen

Your webcam image is smaller than your monitor. We need coordinate mappingTurn a point from the camera image into a pixel on your monitor so the fingertip lines up with the real mouse.

import pyautogui

screen_w, screen_h = pyautogui.size()


def finger_to_screen(x_norm: float, y_norm: float, cam_w: int, cam_h: int):
    """x_norm / y_norm are MediaPipe values from 0 to 1."""
    # Optional: ignore the edges of the camera so the cursor can reach corners
    margin = 0.1
    x = (x_norm - margin) / (1 - 2 * margin)
    y = (y_norm - margin) / (1 - 2 * margin)

    # Clamp between 0 and 1
    x = max(0.0, min(1.0, x))
    y = max(0.0, min(1.0, y))

    return int(x * screen_w), int(y * screen_h)

Why the margin? Without it, you often cannot push the cursor into the very corners of the screen because your finger never reaches the extreme edges of the camera frame.

Step 5 — Move the mouse (gently)

Jumping the cursor every frame feels jittery. A simple exponential smoothingMix the old cursor position with the new one so movement looks calmer trick helps:

smooth_x, smooth_y = 0, 0
alpha = 0.35  # closer to 1 = snappier; closer to 0 = smoother


def smooth_move(target_x: int, target_y: int):
    global smooth_x, smooth_y
    smooth_x = int(smooth_x * (1 - alpha) + target_x * alpha)
    smooth_y = int(smooth_y * (1 - alpha) + target_y * alpha)
    pyautogui.moveTo(smooth_x, smooth_y)

Step 6 — Pinch to click

We treat a click as: distance between thumb tip (landmark 4) and index tip (landmark 8) gets small.

import math


def pinch_distance(hand, frame_w: int, frame_h: int) -> float:
    thumb = hand.landmark[4]
    index = hand.landmark[8]
    x1, y1 = thumb.x * frame_w, thumb.y * frame_h
    x2, y2 = index.x * frame_w, index.y * frame_h
    return math.hypot(x2 - x1, y2 - y1)

Then in the loop:

PINCH_THRESHOLD = 40  # pixels — tweak for your camera
clicked = False

dist = pinch_distance(hand, w, h)
if dist < PINCH_THRESHOLD and not clicked:
    pyautogui.click()
    clicked = True
elif dist >= PINCH_THRESHOLD:
    clicked = False

The clicked flag stops one long pinch from firing hundreds of clicks.

Photo to add: close-up of a hand pinching (thumb + index), optionally with the orange distance line between fingertips visible in the app.

Step 7 — Build the final app (piece by piece)

Do not try to memorize one giant script. We will stack the pieces you already built. Create a new file called virtual_mouse.py and add each block in order.

Part A — Imports (tools we will use)

import cv2          # camera + drawing on video
import mediapipe as mp
import pyautogui    # move / click the real mouse
import math         # distance between two points

Then grab the MediaPipe helpers:

mp_hands = mp.solutions.hands
mp_draw = mp.solutions.drawing_utils

Part B — Safety + starting values

# If the mouse runs away, slam it into a screen corner to stop the script
pyautogui.FAILSAFE = True
pyautogui.PAUSE = 0  # do not add extra delay after each mouse move

screen_w, screen_h = pyautogui.size()  # your monitor size in pixels
cap = cv2.VideoCapture(0)              # open webcam 0

# Cursor starts in the middle of the screen
smooth_x, smooth_y = screen_w // 2, screen_h // 2
alpha = 0.35           # smoothing: lower = calmer cursor
PINCH_THRESHOLD = 40   # how close fingertips must be (in pixels)
clicked = False        # stops one pinch from clicking forever

In plain English: we open the camera, learn how big your screen is, and set a few knobs you can tweak later (alpha, PINCH_THRESHOLD).

Part C — Finger → screen helper

Same idea as Step 4. Paste this under Part B:

def finger_to_screen(x_norm, y_norm):
    # MediaPipe gives x/y from 0 to 1. We stretch that to the monitor.
    margin = 0.1  # ignore camera edges so corners are reachable
    x = (x_norm - margin) / (1 - 2 * margin)
    y = (y_norm - margin) / (1 - 2 * margin)
    x = max(0.0, min(1.0, x))  # keep inside 0..1
    y = max(0.0, min(1.0, y))
    return int(x * screen_w), int(y * screen_h)

Part D — Start the hand tracker

with mp_hands.Hands(
    max_num_hands=1,              # one hand is enough
    min_detection_confidence=0.7, # how sure before we “see” a hand
    min_tracking_confidence=0.6,  # how sure while following it
) as hands:
    print("Virtual mouse running. Press Q to quit.")
    print("Move index finger to move cursor. Pinch to click.")

Everything that runs while the app is alive goes inside this with block (indented).

Part E — The main loop (the heart of the app)

Think of the loop as a tiny recipe that repeats many times per second:

1. Grab a camera picture
2. Find a hand
3. Move the mouse
4. Check for a pinch click
5. Show the video
6. Quit if you press Q

E1 — Read and prepare one frame

    while True:
        ok, frame = cap.read()
        if not ok:
            break

        frame = cv2.flip(frame, 1)                 # mirror view
        h, w, _ = frame.shape
        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)  # MediaPipe likes RGB
        result = hands.process(rgb)

E2 — If a hand is found: draw it, then move the cursor

        if result.multi_hand_landmarks:
            hand = result.multi_hand_landmarks[0]  # first (only) hand
            mp_draw.draw_landmarks(frame, hand, mp_hands.HAND_CONNECTIONS)

            tip = hand.landmark[8]  # index fingertip
            tx, ty = finger_to_screen(tip.x, tip.y)

            # Blend old position + new position (smoothing)
            smooth_x = int(smooth_x * (1 - alpha) + tx * alpha)
            smooth_y = int(smooth_y * (1 - alpha) + ty * alpha)
            pyautogui.moveTo(smooth_x, smooth_y)

Why smoothing? Raw fingertip tracking wiggles a little. Mixing old + new position makes the cursor feel calmer.

E3 — Measure pinch distance + draw a helper line

            thumb = hand.landmark[4]  # thumb tip
            x1, y1 = thumb.x * w, thumb.y * h
            x2, y2 = tip.x * w, tip.y * h
            dist = math.hypot(x2 - x1, y2 - y1)  # distance in pixels

            # Orange line between fingertips — helps you tune the threshold
            cv2.line(frame, (int(x1), int(y1)), (int(x2), int(y2)), (158, 93, 56), 2)
            cv2.putText(
                frame,
                f"pinch: {int(dist)}",
                (20, 40),
                cv2.FONT_HERSHEY_SIMPLEX,
                0.8,
                (32, 35, 31),
                2,
            )

Watch the pinch: NN number on screen. When your fingers are apart it is big; when you pinch it drops. That number tells you what PINCH_THRESHOLD should be.

E4 — Click once when the pinch starts

            if dist < PINCH_THRESHOLD and not clicked:
                pyautogui.click()
                clicked = True
                cv2.putText(
                    frame,
                    "CLICK!",
                    (20, 80),
                    cv2.FONT_HERSHEY_SIMPLEX,
                    0.9,
                    (49, 93, 71),
                    2,
                )
            elif dist >= PINCH_THRESHOLD:
                clicked = False  # ready for the next pinch

clicked means: “we already clicked for this pinch.” When fingers open again, we reset it so the next pinch can click.

E5 — Show the window + quit on Q

        cv2.imshow("Virtual mouse", frame)
        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

Part F — Clean up when you quit

These two lines sit after the with block (same indent as with):

cap.release()
cv2.destroyAllWindows()

They free the camera and close the OpenCV window.

Quick mental map

When Parts A–F are stacked in that order inside virtual_mouse.py, run it:

python virtual_mouse.py

Photo to add: the full app window with landmarks + “pinch: NN” text, next to a browser or desktop where the cursor is clearly following the hand (side-by-side is ideal).

How to use it

1. Hold your hand so the palm faces the camera
2. Move your index fingertip to move the cursor
3. Bring thumb + index close to click
4. Press Q in the OpenCV window to quit
5. If the mouse goes wild, shove the cursor into a screen corner (PyAutoGUI failsafeEmergency stop — moving the mouse to a corner raises an error and stops the script)

Common problems (and fixes)

Tiny upgrades (when you are ready)

• Right-click: pinch with middle finger + thumb instead
• Scroll: if two fingers move up/down, call pyautogui.scroll(...)
• On-screen HUD: draw a soft circle that turns green when a click happens
• Calibration mode: press C to set your own pinch threshold live

What you learned

• How to open a webcam with OpenCVPython toolkit for cameras, images, and video
• How MediaPipe finds hand landmarksThe 21 key points that describe a hand pose
• How coordinate mappingCamera point → monitor pixel lines the fingertip up with the cursor
• How a pinch gesture can trigger pyautogui.click()

You built a real HCIHuman–Computer Interaction — ways people control computers beyond keyboard/mouse demo — the same family of ideas behind touchscreens, VR controllers, and accessibility tools.

FAQ

How do I make a virtual mouse with OpenCV and MediaPipe?

Install opencv-python, mediapipe, and pyautogui, open the webcam with OpenCV, detect hand landmarks with MediaPipe Hands, map the index fingertip to screen coordinates, and use a pinch gesture (thumb + index) to trigger pyautogui.click().

What is MediaPipe Hands used for in a virtual mouse?

MediaPipeGoogle’s hand-tracking toolkit Hands finds 21 landmarks on your hand in each webcam frame. Landmark 8 (index fingertip) drives the cursor; landmark 4 (thumb tip) helps detect a pinch click.

Why is my OpenCV virtual mouse cursor jittery?

Raw fingertip positions wiggle a little. Use exponential smoothingMix old and new cursor positions so movement looks calmer (blend the previous cursor position with the new one) and keep lighting steady so MediaPipe tracking stays stable.

Can beginners build an OpenCV hand-tracking mouse?

Yes. If you can run a Python script and install packages with pip, you can follow this tutorial. You do not need deep machine learning knowledge — OpenCV and MediaPipe handle the hard vision parts.

OpenCV vs MediaPipe — which one moves the mouse?

OpenCV captures the webcam frames and draws helpers on screen. MediaPipe detects the hand. PyAutoGUIPython library that moves and clicks the real system mouse (via pyautogui) actually moves and clicks the system cursor.

Next reading on Sythra Articles

• Your first Python program
• Talk to an LLM in about 20 lines of Python
• Seven Python patterns for ML notebooks

Have fun — and keep your other hand near the keyboard the first time you run it.