Classification
The Kernel Trick
Data that isn't linearly separable in 2D becomes so in a lifted feature space
φ(x₁, x₂) = (x₁, x₂, x₁² + x₂²)
+1 (outer ring)
−1 (inner ring)
separating plane / circle
φ paraboloid
Drag the lift slider or tap Lift to raise the points into feature space.
Key ideas
- Non-linearly separable in 2D: two concentric rings have no straight line between them. A perceptron would thrash forever
- The feature map: $\phi(x_1, x_2) = (x_1, x_2, x_1^2 + x_2^2)$ lifts each point onto a paraboloid. Points farther from the origin sit higher
- Linear in the lifted space: a flat plane at some height $z = c$ cleanly separates the outer ring (which lives high) from the inner ring (which lives low)
- Project back to 2D: that plane's intersection with the paraboloid is a horizontal circle — the boundary is nonlinear in the original space even though it's linear in feature space
- The kernel trick: we never have to explicitly compute $\phi(x)$ — an inner product $K(x, x') = \phi(x)\cdot \phi(x')$ is enough for many learners to work in the lifted space implicitly