---
title: "Decision trees and random forests, explained simply"
description: "How a decision tree picks its questions using Gini impurity, why a single tree overfits, and how a random forest fixes it by averaging many different trees."
url: https://articles.sythra.ai/articles/decision-trees-random-forests-explained
slug: decision-trees-random-forests-explained
author: "vaibhavkothari"
author_url: https://articles.sythra.ai/writers/vaibhavkothari
date_published: 2026-11-12T21:11:00.153Z
date_modified: 2026-11-12T21:11:00.153Z
topics: ["Machine Learning", "Python", "Scikit Learn"]
reading_time_minutes: 6
publisher: "Sythra"
publisher_url: https://sythra.ai
access: free
language: en
---

# Decision trees and random forests, explained simply

> How a decision tree picks its questions using Gini impurity, why a single tree overfits, and how a random forest fixes it by averaging many different trees.

Source: https://articles.sythra.ai/articles/decision-trees-random-forests-explained · Author: vaibhavkothari · Published: 2026-11-12 · Reading time: 6 min · Topics: Machine Learning, Python, Scikit Learn

A decision tree is the one machine learning model you can read out loud. It is also the building block of random forests and gradient boosting, which win a huge share of real-world tabular-data problems. This guide explains how a tree chooses its questions, why it overfits, and how a forest repairs that.

## A tree is just a flowchart

Here is a tree trained on the classic iris flower data, printed by scikit-learn:

```
|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2
```

To classify a flower you start at the top and follow the answers: *is the petal narrower than 0.8 cm? If yes, species 0. Otherwise, is it narrower than 1.75 cm?* — and so on until you reach a leaf. No equations, no scaling, no black box.

You can produce this yourself:

```python
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=0)
tree.fit(iris.data, iris.target)

print(export_text(tree, feature_names=iris.feature_names))
```

## How does the tree choose its questions?

At every node the tree tries many possible splits ("is feature `f` ≤ threshold `t`?") and keeps the one that makes the resulting groups **purest**. A group is pure when it contains mostly one class.

The usual measure is Gini impurity:

```
gini = 1 - sum(p_class ** 2)
```

where `p_class` is the share of each class in the group.

### A worked example

A node holds 10 customers: 6 bought, 4 did not.

```
gini = 1 - (0.6² + 0.4²) = 1 - (0.36 + 0.16) = 0.48
```

Try splitting on "visited the pricing page":

- Left group: 6 customers, 5 bought, 1 did not → `1 - (5/6)² - (1/6)² = 0.278`
- Right group: 4 customers, 1 bought, 3 did not → `1 - (1/4)² - (3/4)² = 0.375`

The weighted average impurity after the split is:

```
(6/10) × 0.278 + (4/10) × 0.375 = 0.317
```

The split reduced impurity from 0.48 to 0.317, a gain of about 0.163. The tree computes this for every feature and threshold and picks the biggest gain. Then it repeats inside each child. That is the entire algorithm: **greedy, recursive purity-chasing**.

## Why one tree overfits

Left alone, a tree keeps splitting until every leaf is pure — which can mean one leaf per training example. It memorises. On the breast-cancer dataset:

```python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

for depth in [None, 3]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_train, y_train)
    print(depth, tree.score(X_train, y_train), tree.score(X_test, y_test))
```

Typical results:

| `max_depth` | Train accuracy | Test accuracy |
|---|---|---|
| `None` (unlimited) | 1.000 | about 0.92 |
| `3` | about 0.98 | about 0.94 |

The unlimited tree scores a perfect 100% on data it has seen and *worse* on new data. That gap is [overfitting](https://articles.sythra.ai/articles/how-to-detect-overfitting). Limiting depth (or `min_samples_leaf`) is called **pruning** and trades a little training accuracy for better generalisation.

Trees are also unstable: change a few training rows and the top question can change, producing a completely different tree. That instability is the key to the next idea.

## Random forests: many trees, one vote

A random forest trains hundreds of trees and lets them vote. To make the trees *different* (voting on identical trees is pointless) it adds two kinds of randomness:

1. **Bootstrap sampling.** Each tree trains on a random sample of the rows, drawn with replacement, so it sees a slightly different dataset.
2. **Random feature subsets.** At each split, only a random handful of features are considered, so trees cannot all copy the same strongest question.

Each tree overfits in its *own* way. Averaging 200 of them cancels most of the individual noise while keeping the signal. This idea of reducing variance by averaging is called **bagging**.

```python
from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)

print(forest.score(X_train, y_train))   # 1.0
print(forest.score(X_test, y_test))     # about 0.96
```

On this split the forest reaches about 96% test accuracy against 92% for the unlimited single tree — better, without any tuning. (Exact numbers shift with the split; run it on several seeds before drawing conclusions. [Cross-validation](https://articles.sythra.ai/articles/cross-validation-vs-single-split) gives you a steadier estimate.)

## Which features mattered?

Forests report how much each feature reduced impurity across all trees:

```python
from sklearn.datasets import load_breast_cancer

names = load_breast_cancer().feature_names
ranked = sorted(zip(forest.feature_importances_, names), reverse=True)[:5]
for score, name in ranked:
    print(f"{score:.3f}  {name}")
```

For this data the top features are measurements like `worst perimeter`, `worst area` and `worst concave points`. Treat importances as a hint, not proof: they favour features with many distinct values and can split credit between correlated features.

## The settings that matter

| Parameter | Effect |
|---|---|
| `n_estimators` | More trees = steadier results, slower. 100–500 is typical |
| `max_depth` | Smaller = simpler, less overfitting |
| `min_samples_leaf` | Larger = smoother predictions |
| `max_features` | Fewer = more diverse trees |
| `class_weight="balanced"` | Helps with imbalanced classes |

## Strengths and weaknesses

**Good at:** mixed numeric and categorical-style data, no scaling needed, captures non-linear patterns and feature interactions, hard to badly misconfigure.

**Weak at:** extrapolating beyond the range of training data (a tree can only output values it has seen), very high-dimensional sparse data like raw text, and giving a model small enough to read — a 200-tree forest is not interpretable the way one tree is.

## FAQ

### How does a decision tree decide where to split?

It tests many candidate questions and picks the one that makes the child groups purest, usually measured by the drop in Gini impurity or entropy. It then repeats on each child.

### Why is a random forest better than a single decision tree?

A single tree overfits and changes a lot with small data changes. A forest averages many different trees, which cancels their individual errors and generalises better.

### Do decision trees need feature scaling?

No. Splits compare a feature to a threshold, so standardising or normalising the data changes nothing.

### How many trees should a random forest have?

Start with 100–200. Accuracy usually plateaus, so more trees mainly cost time. Check that the score stops improving rather than guessing.

## Next reading on Sythra Articles

- [Cross-validation: why one train/test split lies](https://articles.sythra.ai/articles/cross-validation-vs-single-split)
- [How to detect overfitting](https://articles.sythra.ai/articles/how-to-detect-overfitting)
- [Your first machine learning model with scikit-learn](https://articles.sythra.ai/articles/first-machine-learning-model-scikit-learn)

## Glossary (terms defined in this article)
- **decision tree** (basic) — A model that predicts by asking a chain of yes/no questions about the input, like a flowchart
- **Gini impurity** (medium) — A score from 0 (all one class) upward measuring how mixed a group is
- **random forest** (medium) — An ensemble of many decision trees, each trained on a different random sample, whose predictions are averaged