A neural network sounds exotic, but you already know every part of it: matrix-vector products, ramps and composition. Networks built this way, with many more units and with weights found by training, are the math inside modern speech and image recognition and language models. Here is the whole idea at a size you can trace by hand.
Idea clave
A layer takes a vector $\mathbf{x}$, computes $\mathbf{z} = W\mathbf{x} + \mathbf{b}$, then applies ReLU to each entry: $\mathbf{h} = \operatorname{ReLU}(W\mathbf{x} + \mathbf{b})$. Each hidden unit is a switch: on when its $z$ is positive (it passes $z$ through), off when $z \le 0$ (it outputs $0$). A two-layer network with one output computes $$y = \mathbf{v} \cdot \operatorname{ReLU}(W\mathbf{x} + \mathbf{b}) + c.$$
Por qué funciona
Why the switches matter: without them, $$\mathbf{v} \cdot (W\mathbf{x} + \mathbf{b}) + c = (\mathbf{v}^{T}W)\,\mathbf{x} + (\mathbf{v} \cdot \mathbf{b} + c),$$ a single affine map, which is just a line or a plane. A composition of affine maps is affine, so stacking any number of switch-free layers never makes anything new. The ReLU between the layers breaks this. With one input, hidden unit $j$ adds the ramp $v_{j}\operatorname{ReLU}(w_{j}x + b_{j})$, which bends at $x = -b_{j}/w_{j}$, so the output is a sum of ramps: a bent-line graph with at most one bend per unit.
Worked examples
Ejemplo resuelto: A forward pass by hand
Let $W = \begin{bmatrix} 1 & 2 \\ -1 & 1 \end{bmatrix}$, $\mathbf{b} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$, $\mathbf{v} = (2, 5)$ and $c = -1$. Find $y$ for $\mathbf{x} = \begin{bmatrix} 3 \\ -1 \end{bmatrix}$.
- $$W\mathbf{x} = \begin{bmatrix} 1 \cdot 3 + 2 \cdot (-1) \\ (-1) \cdot 3 + 1 \cdot (-1) \end{bmatrix} = \begin{bmatrix} 1 \\ -4 \end{bmatrix}.$$
- Add the bias: $\mathbf{z} = \begin{bmatrix} 1 \\ -3 \end{bmatrix}$.
- Switches: $\mathbf{h} = \operatorname{ReLU}(\mathbf{z}) = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$. Unit 2 is off.
- Output: $y = 2 \cdot 1 + 5 \cdot 0 - 1 = 1$. For this input the weight $5$ never mattered.
Respuesta: $y = 1$
Ejemplo resuelto: Turning a rule into weights
Build a network, with no training, that gives $10$ when $x \ge 5$ and $12$ otherwise, for every whole number $x$.
- A sum of ramps is continuous, so it cannot jump. But it can slide down by $2$ between $x = 4$ and $x = 5$, where there are no whole numbers to get wrong.
- $\operatorname{ReLU}(x - 4) - \operatorname{ReLU}(x - 5)$ is $0$ for $x \le 4$, climbs to $1$ at $x = 5$, and then stays at $1$, because after $x = 5$ both ramps rise together.
- So $y = 12 - 2\operatorname{ReLU}(x - 4) + 2\operatorname{ReLU}(x - 5)$. As a network: $W = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$, $\mathbf{b} = \begin{bmatrix} -4 \\ -5 \end{bmatrix}$, $\mathbf{v} = (-2, 2)$ and $c = 12$.
- Check: $x = 3$ gives $12$; $x = 5$ gives $12 - 2 \cdot 1 + 2 \cdot 0 = 10$; $x = 9$ gives $12 - 2 \cdot 5 + 2 \cdot 4 = 10$.
- Between $4$ and $5$ the output is a ramp: $x = 4.5$ gives $11$, a value the rule never gives. The network is exact on the inputs it was built for, the whole numbers, and only there.
Respuesta: $y = 12 - 2\operatorname{ReLU}(x - 4) + 2\operatorname{ReLU}(x - 5)$
This idea scales up. Handwired, a project by Cheng Lou, compiles ordinary if-else and loop code into fixed network weights built from affine maps and ReLU switches like these, with no training at all. The weights are exact for the inputs the program supports, and because they are ordinary weights, training could still adjust them later.
Ejemplo resuelto: Logic with one switch
The inputs $x_{1}$ and $x_{2}$ are bits, $0$ or $1$. What does $y = x_{1} + x_{2} - 2\operatorname{ReLU}(x_{1} + x_{2} - 1)$ compute?
- The ramp $\operatorname{ReLU}(x_{1} + x_{2} - 1)$ is $1$ when both bits are $1$, and $0$ otherwise.
- Table: $(0, 0) \to 0$, $(0, 1) \to 1$, $(1, 0) \to 1$, $(1, 1) \to 2 - 2 = 0$.
- That is XOR: $1$ when exactly one bit is $1$. No affine map $px_{1} + qx_{2} + r$ can do this, because its values always satisfy $y(0, 0) + y(1, 1) = y(0, 1) + y(1, 0)$, while XOR has $0 + 0 \ne 1 + 1$. The switch is what makes it possible.
Respuesta: XOR (exclusive or)
Error común
Switching before adding the bias. $\operatorname{ReLU}(W\mathbf{x}) + \mathbf{b}$ is not a layer. The bias goes in first, because it decides where each switch flips. In the first example, $\operatorname{ReLU}(W\mathbf{x}) + \mathbf{b} = (1, 1)$, but the layer gives $(1, 0)$. Check: every hidden output should be $0$ or positive.
Error común
Expecting more layers to help without switches. Two layers with no ReLU collapse into one affine map, as the box above shows. Check: if your network has no switches, multiply it out; you will get one matrix and one bias.
The output layer usually has no switch, so $y$ can be negative. Training means choosing $W$, $\mathbf{b}$, $\mathbf{v}$ and $c$ to fit examples. It uses gradient descent, with the derivatives found by backpropagation. The picture below does it on a small data set.