--- title: "What matrices actually do in machine learning" description: "Your dataset is a matrix, a model is a matrix, and training is matrix multiplication repeated. Six operations, drawn out, with the ML job each one does." url: https://articles.sythra.ai/articles/matrix-operations-for-machine-learning slug: matrix-operations-for-machine-learning author: "vaibhavkothari" author_url: https://articles.sythra.ai/writers/vaibhavkothari date_published: 2026-08-26T13:46:51.908Z date_modified: 2026-08-29T10:22:15.966Z topics: ["Machine Learning", "Python", "Numpy", "Learning"] reading_time_minutes: 7 publisher: "Sythra" publisher_url: https://sythra.ai access: free language: en --- # What matrices actually do in machine learning > Your dataset is a matrix, a model is a matrix, and training is matrix multiplication repeated. Six operations, drawn out, with the ML job each one does. Source: https://articles.sythra.ai/articles/matrix-operations-for-machine-learning · Author: vaibhavkothari · Published: 2026-08-26 · Reading time: 7 min · Topics: Machine Learning, Python, Numpy, Learning Open any machine learning paper and you drown in symbols. Open the code and it is four lines of NumPy. The gap is smaller than it looks. Almost everything in machine learning is one of six matrix operations, and each one does a job you can describe in a sentence. ## Your data is already a matrix A spreadsheet is a matrix. Rows are examples, columns are features: ```matrix # X — 4 houses, 3 features (4×3) 1200 3 15 1800 4 8 950 2 22 2400 5 3 ``` Size in square feet, bedrooms, age in years. Every dataset you will ever load has this shape: ```python import numpy as np X = np.array([ [1200, 3, 15], [1800, 4, 8], [950, 2, 22], [2400, 5, 3], ]) print(X.shape) # (4, 3) -> 4 samples, 3 features ``` Read `(4, 3)` as *"four rows of data, three measurements each."* Getting into the habit of reading shapes out loud removes most of the mystery from ML code. ## Operation 1 — Multiplication: apply a model A linear model has one weight per feature. Multiplying gives a prediction for every row at once: ```matrix # X (4×3) [1200] [3] [15] 1800 4 8 950 2 22 2400 5 3 × # w (3×1) [100] [5000] [-800] = # predictions (4×1) [123000] 201600 95000 264400 ``` The first prediction, written out: ```text 1200×100 + 3×5000 + 15×(-800) = 120000 + 15000 - 12000 = 123000 ``` ```python w = np.array([[100], [5000], [-800]]) predictions = X @ w print(predictions.ravel()) # [123000 201600 95000 264400] ``` That is the entire idea of a linear model: **one weight per feature, multiply, add up.** A neural network layer is exactly this, followed by a nonlinear squash — and repeated a few times. Notice what you did *not* write: a loop over houses. Matrix multiplication handles all four rows in one operation, which is why GPUs make deep learning fast. They are multiplication machines. ## Operation 2 — Transpose: swap what you are comparing `.T` flips rows and columns: ```matrix # X (4×3) 1200 3 15 1800 4 8 950 2 22 2400 5 3 → # X.T (3×4) 1200 1800 950 2400 3 4 2 5 15 8 22 3 ``` Each row of `X.T` is now one *feature* across all houses, instead of one house across all features. That is the setup for asking feature-level questions: ```python covariance = X.T @ X print(covariance.shape) # (3, 3) -> one cell per feature pair ``` `X.T @ X` gives a 3×3 table of how each feature relates to each other feature. This single expression is at the heart of linear regression, PCA, and correlation analysis. When you see `X.T @ X` in code, it means *"compare features to features."* ## Operation 3 — Addition: shift everything Adding a row vector applies one number per column, across every row: ```matrix # X (2×3) 1200 3 15 1800 4 8 + # bias (1×3) 100 1 -5 = # result (2×3) 1300 4 10 1900 5 3 ``` ```python bias = np.array([100, 1, -5]) print(X[:2] + bias) ``` That is broadcasting. It is the `+ b` in every neural network layer: ```python output = X @ W + b ``` One line, two operations, an entire layer. ## Operation 4 — Elementwise multiply: mask and weight `*` multiplies matching positions. It looks similar to `@` and does something completely different: ```matrix # values 1 2 3 4 5 6 · # mask 1 0 1 0 1 1 = # result 1 0 3 0 5 6 ``` ```python values = np.array([[1, 2, 3], [4, 5, 6]]) mask = np.array([[1, 0, 1], [0, 1, 1]]) print(values * mask) ``` This is how dropout works — multiply by a random 0/1 mask. It is also how you apply sample weights, ignore padded positions in a sequence, and zero out invalid entries. Whenever you want to keep some numbers and silence others, this is the operation. ## Operation 5 — The identity: multiply by nothing The identity matrix has 1s on the diagonal and 0s elsewhere: ```matrix # I (3×3) 1 0 0 0 1 0 0 0 1 ``` `A @ I = A`. It is the matrix equivalent of multiplying by 1, and it earns its place in one very practical spot — ridge regression: ```python n_features = X.shape[1] alpha = 1.0 w = np.linalg.solve(X.T @ X + alpha * np.eye(n_features), X.T @ y) ``` Adding `alpha * np.eye(n)` nudges the diagonal upwards. That small nudge is what stops the solution exploding when two features are nearly identical — regularisation, in one term. The connection to [overfitting](https://articles.sythra.ai/articles/how-to-detect-overfitting) is direct: bigger `alpha` means smaller weights means less memorising. ## Operation 6 — Solve: fit the model in one line Linear regression has a closed-form answer, and it is pure matrix algebra: ```python import numpy as np X = np.array([[1, 1200], [1, 1800], [1, 950], [1, 2400]], dtype=float) y = np.array([123000, 201600, 95000, 264400], dtype=float) w = np.linalg.solve(X.T @ X, X.T @ y) print(w) # [intercept, price per square foot] ``` That column of 1s at the front is the intercept trick: it lets the constant term ride along as just another feature. Use `np.linalg.solve`, not `np.linalg.inv(...) @ ...`. Solving is faster and numerically stabler than inverting. For anything remotely real, prefer the least-squares routine, which copes when the matrix is singular: ```python w, residuals, rank, singular = np.linalg.lstsq(X, y, rcond=None) ``` ## Reading shapes like sentences Once shapes make sense, unfamiliar ML code becomes readable: | Code | Shapes | Read as | |---|---|---| | `X @ W` | (32, 10) @ (10, 4) → (32, 4) | 32 samples, 10 features in, 4 out | | `X.T @ X` | (100, 5) → (5, 5) | Every feature against every feature | | `X @ X.T` | (100, 5) → (100, 100) | Every sample against every sample | | `X - X.mean(0)` | (100, 5) − (5,) | Centre each feature | | `W1 @ W2` | (10, 8) @ (8, 4) | Stack two layers | The middle number always cancels. That cancellation *is* the computation: 10 features meet 10 weights and collapse into 4 outputs. ## A two-layer network, entirely in NumPy ```python import numpy as np rng = np.random.default_rng(42) X = rng.normal(size=(32, 10)) # 32 samples, 10 features W1 = rng.normal(size=(10, 16)) * 0.1 b1 = np.zeros(16) W2 = rng.normal(size=(16, 1)) * 0.1 b2 = np.zeros(1) hidden = np.maximum(0, X @ W1 + b1) # ReLU output = hidden @ W2 + b2 print(hidden.shape, output.shape) # (32, 16) (32, 1) ``` Six lines. Two matrix multiplies, two broadcast additions, one `maximum` for the nonlinearity. Every deep learning framework is an optimised, differentiable version of exactly this, and `np.maximum(0, z)` is the only part that is not linear algebra — without it, stacked layers would collapse back into a single matrix. ```remember # Remember this `X @ W` -> apply a model to every row at once `X.T` -> swap what you are comparing `X * mask` -> keep some values, zero the rest --- Your data is already a matrix: rows are examples, columns are features. ``` ## FAQ ### Why does machine learning use matrices? Because a dataset is naturally a table of rows and columns, and matrix multiplication applies a model to every row at once. That removes loops and maps directly onto hardware built for parallel multiplication. ### What does X.T @ X mean in machine learning? It produces a square matrix with one cell per pair of features, describing how features relate to each other. It appears in linear regression, PCA and correlation analysis. ### What is the difference between matrix multiplication and elementwise multiplication? Matrix multiplication combines rows with columns and changes the shape. Elementwise multiplication multiplies values in matching positions and keeps the shape, and is used for masks, dropout and sample weights. ### Why add the identity matrix in ridge regression? Adding `alpha * np.eye(n)` raises the diagonal, which keeps the solution stable when features are nearly duplicated and shrinks the weights. That shrinkage is the regularisation. ### Do I need to learn linear algebra before machine learning? You need shapes, matrix multiplication, transpose and broadcasting. That is enough to read most model code. Eigenvalues and decompositions can wait until you need PCA. ## Next reading on Sythra Articles - [Matrix multiplication in NumPy: @ vs *](https://articles.sythra.ai/articles/numpy-matrix-multiplication-shapes) - [Train your first ML model in 20 lines](https://articles.sythra.ai/articles/first-machine-learning-model-scikit-learn) - [How to detect overfitting in your own model](https://articles.sythra.ai/articles/how-to-detect-overfitting) ## Glossary (terms defined in this article) - **features** (basic) — The input measurements the model learns from - **broadcasting** (medium) — NumPy stretching a smaller array across a bigger one automatically - **dropout** (advanced) — Randomly switching off some neurons during training so the model cannot over-rely on any one - **ridge regression** (advanced) — Linear regression with a penalty that keeps the weights small