---
title: "Backpropagation worked out with real numbers"
description: "Follow one training step through a tiny neural network with real numbers: forward pass, loss, the chain rule backwards, and a weight update that cuts the loss."
url: https://articles.sythra.ai/articles/backpropagation-worked-example
slug: backpropagation-worked-example
author: "vaibhavkothari"
author_url: https://articles.sythra.ai/writers/vaibhavkothari
date_published: 2026-11-12T21:11:00.153Z
date_modified: 2026-11-12T21:11:00.153Z
topics: ["Machine Learning", "Deep Learning", "Numpy"]
reading_time_minutes: 7
publisher: "Sythra"
publisher_url: https://sythra.ai
access: free
language: en
---

# Backpropagation worked out with real numbers

> Follow one training step through a tiny neural network with real numbers: forward pass, loss, the chain rule backwards, and a weight update that cuts the loss.

Source: https://articles.sythra.ai/articles/backpropagation-worked-example · Author: vaibhavkothari · Published: 2026-11-12 · Reading time: 7 min · Topics: Machine Learning, Deep Learning, Numpy

Backpropagation is usually explained with symbols until your eyes glaze. Here we do the opposite: one tiny network, one training example, every number written out. By the end you will see that backprop is just the chain rule applied in a sensible order, and you will have code that proves the hand calculation right.

## The network

Two inputs, two hidden neurons, one output. All neurons use the sigmoid function; there are no biases, to keep the arithmetic short.

```
 input         hidden         output

  x1 ──┬──────► h1 ──(w5)──┐
       │   ╲  ╱            ├──► o
  x2 ──┴──────► h2 ──(w6)──┘

  every input connects to every hidden neuron (the four W1 weights)
```

The sigmoid and its derivative:

```
σ(z)  = 1 / (1 + e^-z)
σ'(z) = σ(z) · (1 − σ(z))
```

That derivative formula is the convenient fact that makes the algebra short: we can reuse the output we already computed.

Starting values:

```
x  = [0.5, 1.0]          target y = 1.0
W1 = [[0.1, 0.3],        # W1[i][j] connects input i to hidden neuron j
      [0.2, 0.4]]
W2 = [0.5, 0.6]          # hidden neuron j to the output
loss L = ½ (o − y)²
```

## Forward pass

Hidden pre-activations (`z = x · W1`):

```
z_h1 = 0.5·0.1 + 1.0·0.2 = 0.25
z_h2 = 0.5·0.3 + 1.0·0.4 = 0.55
```

Hidden activations:

```
h1 = σ(0.25) = 0.5622
h2 = σ(0.55) = 0.6341
```

Output:

```
z_o = 0.5622·0.5 + 0.6341·0.6 = 0.6616
o   = σ(0.6616) = 0.6596
```

Loss:

```
L = ½ (0.6596 − 1.0)² = 0.0579
```

The network predicts 0.66 where we wanted 1.0. Now: **how should each weight change to close that gap?**

## Backward pass

We want `∂L/∂w` for every weight. The chain rule says: multiply the local derivatives along the path from the loss back to that weight. Work from the output toward the inputs, reusing earlier results.

### Step 1 — Loss to output

```
∂L/∂o = o − y = 0.6596 − 1.0 = −0.3404
```

Negative: the output is too low, so increasing it reduces the loss.

### Step 2 — Through the output sigmoid

```
∂L/∂z_o = ∂L/∂o · σ'(z_o)
        = −0.3404 · 0.6596 · (1 − 0.6596)
        = −0.0764
```

Call this value `δ_o`. Everything upstream builds on it.

### Step 3 — Output-layer weights

`z_o = h1·w5 + h2·w6`, so the derivative of `z_o` with respect to each weight is simply the hidden activation feeding it:

```
∂L/∂w5 = δ_o · h1 = −0.0764 · 0.5622 = −0.0430
∂L/∂w6 = δ_o · h2 = −0.0764 · 0.6341 = −0.0485
```

### Step 4 — Send the error back to the hidden layer

How much did each hidden neuron contribute to the output error? Weight the error by the connection strength:

```
∂L/∂h1 = δ_o · w5 = −0.0764 · 0.5 = −0.0382
∂L/∂h2 = δ_o · w6 = −0.0764 · 0.6 = −0.0459
```

This is the step that gives backpropagation its name: the error signal travels backwards along the same weights the forward pass used.

### Step 5 — Through the hidden sigmoids

```
δ_h1 = ∂L/∂h1 · h1(1 − h1) = −0.0382 · 0.5622 · 0.4378 = −0.00941
δ_h2 = ∂L/∂h2 · h2(1 − h2) = −0.0459 · 0.6341 · 0.3659 = −0.01064
```

### Step 6 — First-layer weights

Each weight's gradient is the delta of the neuron it feeds, times the input it carries:

```
∂L/∂W1[0][0] = δ_h1 · x1 = −0.00941 · 0.5 = −0.00470
∂L/∂W1[0][1] = δ_h2 · x1 = −0.01064 · 0.5 = −0.00532
∂L/∂W1[1][0] = δ_h1 · x2 = −0.00941 · 1.0 = −0.00941
∂L/∂W1[1][1] = δ_h2 · x2 = −0.01064 · 1.0 = −0.01064
```

Notice how small these are compared with the output-layer gradients. Each sigmoid multiplied the signal by a factor of at most 0.25. Stack many layers and the gradient shrinks toward zero — the **vanishing gradient problem**, and a big reason modern networks prefer ReLU-style activations.

## The update

Gradient descent subtracts the gradient times a learning rate. With `lr = 0.5`:

```
w5 = 0.5 − 0.5·(−0.0430) = 0.5215
w6 = 0.6 − 0.5·(−0.0485) = 0.6242
W1[0][0] = 0.1 − 0.5·(−0.00470) = 0.10235
... and so on
```

Every gradient was negative, so every weight rose, pushing the output upward toward 1.0. After the update, re-running the forward pass gives `o ≈ 0.6661` and `L ≈ 0.0557` — down from `0.6596` and `0.0579`. One step, a measurable improvement.

## Prove it in NumPy

```python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

x = np.array([0.5, 1.0])
y = 1.0
W1 = np.array([[0.1, 0.3], [0.2, 0.4]])
W2 = np.array([0.5, 0.6])

# forward
z1 = x @ W1
h = sigmoid(z1)
z2 = h @ W2
o = sigmoid(z2)
loss = 0.5 * (o - y) ** 2
print("forward:", h, o, loss)

# backward
d_o = o - y
delta_o = d_o * o * (1 - o)
dW2 = delta_o * h

d_h = delta_o * W2
delta_h = d_h * h * (1 - h)
dW1 = np.outer(x, delta_h)

print("dW2:", dW2)
print("dW1:", dW1)

# update
lr = 0.5
W1 -= lr * dW1
W2 -= lr * dW2

o_new = sigmoid(sigmoid(x @ W1) @ W2)
print("new loss:", 0.5 * (o_new - y) ** 2)
```

The printed gradients match the hand calculation. Two lines of the backward pass carry the entire idea:

- `delta_o` — the error at a neuron after passing back through its activation
- `d_h = delta_o * W2` — the error pushed backwards through the weights

## Check your gradients numerically

Whenever you derive gradients yourself, verify them. A **gradient check** nudges one weight by a tiny amount and measures the real change in loss:

```python
def loss_fn(W1, W2):
    o = sigmoid(sigmoid(x @ W1) @ W2)
    return 0.5 * (o - y) ** 2

W1 = np.array([[0.1, 0.3], [0.2, 0.4]])
W2 = np.array([0.5, 0.6])
eps = 1e-6

W2_plus = W2.copy();  W2_plus[0] += eps
W2_minus = W2.copy(); W2_minus[0] -= eps
numeric = (loss_fn(W1, W2_plus) - loss_fn(W1, W2_minus)) / (2 * eps)
print(numeric)    # about -0.04296, matching dW2[0]
```

If the numeric and analytic values disagree beyond a few decimal places, there is a bug in your backward pass. This is the standard way people debug hand-written networks.

## What this scales to

Real frameworks do exactly this, automatically:

- PyTorch's `loss.backward()` records every operation in the forward pass, then walks the graph in reverse applying the chain rule.
- Matrices replace single numbers, so `delta * h` becomes a matrix product like the ones in our [matrix operations guide](https://articles.sythra.ai/articles/matrix-operations-for-machine-learning).
- The structure is identical in a [CNN](https://articles.sythra.ai/articles/cnn-from-scratch-numpy) or a [transformer](https://articles.sythra.ai/articles/transformer-from-scratch-pytorch).

Once you have seen one step with real numbers, `backward()` stops being magic.

## FAQ

### What is backpropagation in simple terms?

Backpropagation computes how much each weight in a neural network contributed to the final error. It applies the chain rule from the output layer backwards, reusing intermediate results, so all gradients come out in one backward sweep.

### Why is it called backpropagation?

Because the error signal flows backwards through the network, from the output toward the inputs, along the same connections the forward pass used.

### What is the vanishing gradient problem?

When gradients pass through many layers of saturating activations like sigmoid, each layer multiplies them by a small number, so early layers receive almost no learning signal. ReLU activations, residual connections and good initialisation reduce this.

### How do I know my gradients are correct?

Run a numerical gradient check: nudge a weight by a tiny epsilon, measure the change in loss, and compare with your analytic gradient. They should agree to several decimal places.

## Next reading on Sythra Articles

- [Linear regression from scratch with NumPy](https://articles.sythra.ai/articles/linear-regression-from-scratch-numpy)
- [A CNN from scratch in NumPy](https://articles.sythra.ai/articles/cnn-from-scratch-numpy)
- [Transformer from scratch in PyTorch](https://articles.sythra.ai/articles/transformer-from-scratch-pytorch)

## Glossary (terms defined in this article)
- **Backpropagation** (advanced) — The algorithm that computes how much each weight contributed to the error, using the chain rule from the output backwards
- **sigmoid** (basic) — A smooth function that squashes any number into the range 0 to 1