Draw in the air with MediaPipe finger tracking

Point your index finger at the webcam and paint on thin air. This project is a good test of everything hand tracking teaches: reading landmarks, turning finger positions into gestures, and keeping a persistent drawing separate from the live video. By the end you will have a canvas with draw, pause and erase modes.

What we are building

• Index finger up alone → draw
• Index + middle fingers up → "hover" (move without drawing)
• Fist or all fingers up → clear the canvas
• Pressing c also clears it, q quits

Install

python -m pip install opencv-python mediapipe numpy

MediaPipe usually needs Python 3.8–3.12. If installation fails on a newer Python, create a virtual environment with a supported version.

The key idea: two layers

If you draw directly onto the video frame, the line is erased on the next frame because a new frame arrives. So keep a separate persistent canvas and combine it with each frame for display:

camera frame (new every frame)  ─┐
                                 ├──► merged image shown on screen
canvas (kept between frames)    ─┘

Hand landmarks you need

MediaPipe returns 21 landmarksNumbered key points on the hand — knuckles, joints and fingertips per hand with coordinates normalised to 0–1. The useful ones:

A finger is "up" when its tip is above its PIP joint — meaning a smaller y, because image y grows downward.

Step 1 — Setup and finger detection

import cv2
import mediapipe as mp
import numpy as np

mp_hands = mp.solutions.hands
mp_draw = mp.solutions.drawing_utils

hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.6,
)

TIPS = {"index": 8, "middle": 12, "ring": 16, "pinky": 20}
PIPS = {"index": 6, "middle": 10, "ring": 14, "pinky": 18}


def fingers_up(landmarks):
    """Return a dict like {'index': True, 'middle': False, ...}."""
    return {
        name: landmarks[TIPS[name]].y < landmarks[PIPS[name]].y
        for name in TIPS
    }

This ignores the thumb on purpose — thumb direction depends on which hand it is and is not needed here.

Step 2 — The main loop

cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)

canvas = None
prev_point = None
color = (0, 140, 255)        # BGR orange
thickness = 8

while True:
    ok, frame = cap.read()
    if not ok:
        break

    frame = cv2.flip(frame, 1)                     # mirror so movement feels natural
    h, w = frame.shape[:2]
    if canvas is None:
        canvas = np.zeros((h, w, 3), dtype=np.uint8)

    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)   # MediaPipe expects RGB
    result = hands.process(rgb)

    mode = "no hand"
    if result.multi_hand_landmarks:
        hand = result.multi_hand_landmarks[0]
        lm = hand.landmark
        up = fingers_up(lm)

        tip = (int(lm[8].x * w), int(lm[8].y * h))

        if all(up.values()) or not any(up.values()):
            mode = "clear"
            canvas[:] = 0
            prev_point = None
        elif up["index"] and not up["middle"]:
            mode = "draw"
            if prev_point is not None:
                cv2.line(canvas, prev_point, tip, color, thickness)
            prev_point = tip
        else:
            mode = "hover"
            prev_point = None

        cv2.circle(frame, tip, 10, (255, 255, 255), 2)
        mp_draw.draw_landmarks(frame, hand, mp_hands.HAND_CONNECTIONS)
    else:
        prev_point = None

    # merge canvas onto the frame
    gray = cv2.cvtColor(canvas, cv2.COLOR_BGR2GRAY)
    _, inv = cv2.threshold(gray, 1, 255, cv2.THRESH_BINARY_INV)
    inv = cv2.cvtColor(inv, cv2.COLOR_GRAY2BGR)
    frame = cv2.bitwise_and(frame, inv)       # cut a hole where the drawing is
    frame = cv2.bitwise_or(frame, canvas)     # paint the drawing in

    cv2.putText(frame, f"mode: {mode}", (20, 40),
                cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2)
    cv2.imshow("Air canvas", frame)

    key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break
    if key == ord("c"):
        canvas[:] = 0

cap.release()
cv2.destroyAllWindows()

Why prev_point matters

A line is drawn between the previous fingertip position and the current one. Setting prev_point = None whenever you are not drawing breaks the stroke, so lifting your finger does not leave a straight line jumping across the canvas.

Why the bitwise merge?

Adding the canvas to the frame would make the drawing translucent against bright backgrounds. The mask approach first blanks the frame wherever the canvas has ink, then paints the ink in, giving solid colour.

Step 3 — Make strokes smooth

Raw fingertip positions jitter by a few pixels. Blend each new point with the previous one:

SMOOTH = 0.6   # 0 = no smoothing, closer to 1 = heavy smoothing

def smooth(prev, new, amount=SMOOTH):
    if prev is None:
        return new
    return (
        int(prev[0] * amount + new[0] * (1 - amount)),
        int(prev[1] * amount + new[1] * (1 - amount)),
    )

# in draw mode:
tip = smooth(prev_point, tip)

This is exponential smoothingBlending the old value with the new one so movement looks calm, the same trick used in the virtual mouse guide. Too much smoothing makes the line lag behind your finger, so tune between 0.3 and 0.7.

Step 4 — Add a colour palette

Use hover mode as a "selection" gesture: when you hover over a swatch at the top of the screen, switch colour.

SWATCHES = [
    ((20, 70),  (0, 140, 255)),    # orange
    ((90, 140), (0, 200, 0)),      # green
    ((160, 210), (255, 100, 0)),   # blue
]

def draw_palette(frame):
    for (x1, x2), col in SWATCHES:
        cv2.rectangle(frame, (x1, 60), (x2, 110), col, -1)

def pick_color(tip, current):
    if 60 <= tip[1] <= 110:
        for (x1, x2), col in SWATCHES:
            if x1 <= tip[0] <= x2:
                return col
    return current

Call draw_palette(frame) before showing the frame, and in hover mode set color = pick_color(tip, color). Because drawing only happens in draw mode, you can reach the swatches without leaving a mark.

Troubleshooting

Ideas to extend it

• Add an eraser: a gesture where the fist draws in black with a thick brush
• Save the drawing with cv2.imwrite("drawing.png", canvas) on a key press
• Recognise what you drew by sending the canvas to a classifier or an LLM with vision

The same landmark-to-gesture approach drives the gesture volume controller and the finger counter.

FAQ

How does air drawing with MediaPipe work?

MediaPipe finds 21 hand landmarks in each frame. The code reads the index fingertip position, draws a line from its previous position onto a separate canvas, and overlays that canvas on the video.

How do I tell if a finger is up in MediaPipe?

Compare the fingertip landmark's y value with the joint below it. Because image y increases downward, the finger is up when the tip's y is smaller than the PIP joint's y.

Why does my drawing disappear on the next frame?

You are drawing on the video frame itself, which is replaced each loop. Draw on a persistent canvas array and merge it with each new frame.

Can I draw with both hands?

Yes. Set max_num_hands=2 and keep a separate previous-point variable for each hand.

Next reading on Sythra Articles

• Build a virtual mouse with OpenCV and MediaPipe
• Count fingers with MediaPipe
• Detect colors and track objects with OpenCV HSV