4509 - Bridging Mathematics · intuition session
We already did linear algebra the way you will need it: subspaces, rank, projection, the spectral theorem, and the Hessian.
That lecture stays. It is the one you study.
But several of you asked the same kind of question afterwards, and they were all really one question…
What do these things look like?
We already wrote:
\operatorname{Ker}(A) is the directions A flattens to zero.
That sentence is doing a lot of work. To flatten something, A has to do something to the plane.
And we never said what a matrix does. We only said what it is: a box of numbers, and a rule to multiply it by a vector.
Today we fix that, and only that. No new theorems, no new definitions to memorize. Same material, with pictures.
Every one of these is a line you have already seen. We are just going to look at them.
We will use the same matrix from beginning to end:
A=\begin{bmatrix}4&3\\1&2\end{bmatrix}
It is small enough to compute in your head, and it happens to have very clean answers to every question we will ask of it.
Every widget you will see today opens with exactly this A already loaded.
Take the vector
v=\begin{bmatrix}-2\\1\end{bmatrix}
What does that notation actually say? It says: go -2 along the first basis vector, and 1 along the second.
v=-2\times\hat{\imath}+1\times\hat{\jmath}=-2\times\begin{bmatrix}1\\0\end{bmatrix}+1\times\begin{bmatrix}0\\1\end{bmatrix}
So far this is just bookkeeping. The numbers -2 and 1 are instructions relative to \hat{\imath} and \hat{\jmath}.
Those instructions assume \hat{\imath} and \hat{\jmath} are where we left them.
So: what if we moved \hat{\imath} and \hat{\jmath}, and kept the instructions?
That is the entire idea of a linear transformation. Keep the recipe, move the ingredients.
We can define a linear transformation as a transformation of every vector on the plane that must satisfy two things:
We will deal with the formal definition and the rigor later. For now this is enough to see what is going on.
Say \hat{\imath} lands on (4,1), and \hat{\jmath} lands on (3,2):
\hat{\imath}\,'=\begin{bmatrix}4\\1\end{bmatrix}\qquad \hat{\jmath}\,'=\begin{bmatrix}3\\2\end{bmatrix}
Where does our v=(-2,1) go? Use the same recipe, with the new ingredients:
v'=-2\times\begin{bmatrix}4\\1\end{bmatrix}+1\times\begin{bmatrix}3\\2\end{bmatrix}=\begin{bmatrix}-8+3\\-2+2\end{bmatrix}=\begin{bmatrix}-5\\0\end{bmatrix}
Any w=(x,y) lands on x times the new \hat{\imath}, plus y times the new \hat{\jmath}:
w=\begin{bmatrix}x\\y\end{bmatrix}\ \text{ lands on }\ x\begin{bmatrix}4\\1\end{bmatrix}+y\begin{bmatrix}3\\2\end{bmatrix}=\begin{bmatrix}4x+3y\\x+2y\end{bmatrix}
Look at that right-hand side, and now look at ordinary matrix multiplication:
\begin{bmatrix}4&3\\1&2\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}4x+3y\\x+2y\end{bmatrix}
The columns of a matrix are simply where the basis vectors land.
A=\begin{bmatrix}\color{#0e7c6b}{4}&\color{#eb6834}{3}\\\color{#0e7c6b}{1}&\color{#eb6834}{2}\end{bmatrix}\qquad \color{#0e7c6b}{A\hat{\imath}=(4,1)}\qquad \color{#eb6834}{A\hat{\jmath}=(3,2)}
Matrix times vector is not a rule to memorize. It is “read the recipe, use the new ingredients”.
Everything else today follows from this one sentence.
Drag the sliders. The two coloured arrows are the columns of A, and the whole plane follows them.
The shaded parallelogram is the image of the unit square. Its area is \det A.
For our A: \ \det A = 4\times 2 - 3\times 1 = 5. The transformation multiplies every area by 5.
Now try the singular preset. The parallelogram has no area left, the plane has been squashed onto a line, and \det A = 0. That is “flattening”.
Take v=(-2,1) again, and take z=(1,-0.5), which sits on the same line through the origin (it is -\tfrac12 v).
Av=\begin{bmatrix}-5\\0\end{bmatrix}\qquad Az=\begin{bmatrix}2.5\\0\end{bmatrix}
Both land on the horizontal axis, and 2.5=-\tfrac12\times(-5). They started on one line, they ended on one line.
Keep this in mind. It is exactly why, when the columns are linearly dependent, a whole line of vectors gets sent to a single point.
(1) The columns of M are (0,1) and (-1,0). What does M do to the plane?
b). \hat{\imath} goes up, \hat{\jmath} goes left, so the whole plane turns a quarter turn.
(2) If \det M = 0, then…
a). Zero area left means the columns are linearly dependent.
Let M=\begin{bmatrix}2&1\\0&3\end{bmatrix}. Where does w=(1,2) land, and by how much are areas scaled?
Use the recipe: 1 times the first column, plus 2 times the second.
Mw=1\begin{bmatrix}2\\0\end{bmatrix}+2\begin{bmatrix}1\\3\end{bmatrix}=\begin{bmatrix}4\\6\end{bmatrix}
And \det M = 2\times3-1\times0=6, so areas are multiplied by 6. ✅
Suppose we first rotate the plane, and then apply our A.
Every vector gets moved twice. Is there a single matrix that does both at once?
Yes, and you already know its name. It is the product of the two matrices.
AB means: apply B first, then apply A to whatever came out.
Students always find this annoying, and they are right to. The one you apply first is written on the right.
The reason is just how we write “apply B to v”:
A(Bv)=(AB)v
B is the one sitting next to v, so B touches v first. Reading right to left is reading in time order.
The green arrows are where \hat{\imath} and \hat{\jmath} go after the first step. The orange ones are where they end up after the second.
And the single matrix AB sends them straight to the orange arrows, in one go. Two steps and one matrix agree, on the basis vectors and on any other point you pick.
Now tick the swap box. AB and BA land in different places, so AB \neq BA.
“Rotate then stretch” is not the same as “stretch then rotate”. Order matters, because it is a sequence of actions, not a product of numbers.
Determinants multiply.
\det(AB)=\det A\cdot\det B
If the first step multiplies areas by 2 and the second by 5, the pair multiplies them by 10. Obvious, once areas are what determinants are.
Inverses reverse.
(AB)^{-1}=B^{-1}A^{-1}
To undo “socks, then shoes”, you take off the shoes first. Undoing a sequence means undoing it backwards.
For A\in\mathbb{R}^{m\times n} we defined:
\operatorname{Im}(A)=\{Ax\}\qquad \operatorname{Ker}(A)=\{x : Ax=0\}
With the picture in hand these stop being symbols:
\operatorname{Im}(A) is everywhere the plane can reach after A acts. It is the span of the columns, which is to say the span of where the basis vectors landed.
\operatorname{Ker}(A) is what got crushed to the origin. If nothing is crushed, the kernel is just \{0\}.
At full rank the image is the whole plane and only the origin maps to the origin. Nothing is lost.
Make the columns line up. Now the image is a line (dimension 1), and a whole line of vectors is crushed to zero (dimension 1).
\underbrace{1}_{\operatorname{rank}}+\underbrace{1}_{\operatorname{nullity}}=\underbrace{2}_{n}
Rank-nullity is conservation of dimension. What you lose down the kernel, you keep in the image.
We already saw that:
\text{no perfect multicollinearity}\iff \operatorname{Ker}(X)=\{0\}\iff X^\top X \text{ invertible}
Read it with the picture. Perfect multicollinearity means one regressor is a combination of the others, so the columns of X are linearly dependent.
Dependent columns means X crushes some direction to zero, so there is a \beta \neq 0 with X\beta = 0.
Then \hat\beta and \hat\beta + \beta fit the data equally well, and there is no unique answer. The invertibility condition is just “nothing gets crushed”. 👎
(1) M maps \mathbb{R}^2\to\mathbb{R}^2 and squashes the plane onto a line. Then \operatorname{nullity}(M) is
b). Rank is 1, and 1 + \text{nullity} = 2.
(2) A design matrix X has a column equal to the sum of two others. Then
b). The columns are dependent, so \operatorname{Ker}(X)\neq\{0\}.
Let M=\begin{bmatrix}1&2\\2&4\end{bmatrix}. Find \det M, the rank, and a vector in the kernel.
\det M = 1\times4-2\times2 = 0, so M is singular and the rank is 1 (the second column is twice the first).
For the kernel solve x + 2y = 0, so any multiple of (2,-1) works:
M\begin{bmatrix}2\\-1\end{bmatrix}=\begin{bmatrix}2-2\\4-4\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}\ ✅
Rank 1 plus nullity 1 equals 2. ✅
For X,Y\in\mathbb{R}^n:
\langle X,Y\rangle=X^\top Y=\sum_i x_iy_i,\qquad \lVert X\rVert=\sqrt{\langle X,X\rangle}
The norm is length. Fine. But what is the inner product itself measuring?
\cos\theta=\frac{\langle X,Y\rangle}{\lVert X\rVert\,\lVert Y\rVert}
It measures how much the two vectors agree in direction, scaled by their lengths. Zero means they agree not at all, and that is orthogonality:
X\perp Y\iff \langle X,Y\rangle = 0
Given X and Y, split Y into the part that lies along X, and the part that does not:
\operatorname{proj}_X Y=\frac{\langle X,Y\rangle}{\lVert X\rVert^2}X,\qquad \operatorname{rej}_X Y = Y-\operatorname{proj}_X Y
The rejection is orthogonal to X, by construction. That is the whole content of the split.
In econometrics the endogenous variable is Y. We explain it with X, so we project Y onto X. What is not explained, the error, is exactly \operatorname{rej}_X Y.
We already saw it: for a subspace S and a point y, there is a unique closest \hat y \in S, characterised by
(y-\hat y)\perp S
In the widget, drag the vector around. The shortest connection to the line is always the perpendicular one.
Any other point on the line gives a longer residual. That is not a coincidence, it is Pythagoras: going along the line adds length in a direction that was already accounted for.
“Closest” and “perpendicular residual” are the same condition. Minimising is geometry.
Let S=\operatorname{col}(X) and \hat y = X\beta. Perpendicular residual means the residual is orthogonal to every column of X:
X^\top(y-X\beta)=0
\Longrightarrow\ \underbrace{X^\top X\beta = X^\top y}_{\text{normal equations}}\ \Longrightarrow\ \hat\beta = (X^\top X)^{-1}X^\top y
The formula is not an algebraic accident. It is the perpendicular dropped from y onto the span of the regressors, written out.
There is a whole separate widget for this one, The geometry of OLS, with \hat y, the residual, R^2 as \cos^2\theta, and Frisch-Waugh-Lovell. Work through it on your own after the session.
(1) If \langle X,Y\rangle=0 then \operatorname{proj}_X Y is
b). Nothing of Y lies along X, so there is nothing to keep.
(2) R^2 = \lVert\hat y\rVert^2/\lVert y\rVert^2 is the squared cosine of the angle between y and the regressor space. So R^2=1 means
a). Angle zero, nothing to project, residual zero.
Let X=(3,4) and Y=(5,0). Find \operatorname{proj}_X Y and check the rejection is orthogonal to X.
\langle X,Y\rangle = 15 and \lVert X\rVert^2 = 25, so
\operatorname{proj}_X Y=\tfrac{15}{25}(3,4)=(1.8,\,2.4)
\operatorname{rej}_X Y=(5,0)-(1.8,2.4)=(3.2,\,-2.4)
Check: \langle X,\operatorname{rej}_X Y\rangle = 3(3.2)+4(-2.4)=9.6-9.6=0 ✅
Apply A to a vector and two things generally change: its length, and its direction.
But some vectors only change length. They stay on their own line.
Take e=(-1,1) with our running A:
\begin{bmatrix}4&3\\1&2\end{bmatrix}\begin{bmatrix}-1\\1\end{bmatrix}=\begin{bmatrix}-4+3\\-1+2\end{bmatrix}=\begin{bmatrix}-1\\1\end{bmatrix}
It did not move at all. So e is an eigenvector, with \lambda_1 = 1.
Take f=(0.9487,\,0.3162):
\begin{bmatrix}4&3\\1&2\end{bmatrix}\begin{bmatrix}0.9487\\0.3162\end{bmatrix}=\begin{bmatrix}4.7434\\1.5811\end{bmatrix}=5\begin{bmatrix}0.9487\\0.3162\end{bmatrix}
Same line, stretched by 5. So \lambda_2 = 5.
Two quick checks against what we already saw:
\operatorname{tr}A = 4+2 = 6 = 1+5 = \textstyle\sum\lambda_i
\det A = 5 = 1\times 5 = \textstyle\prod\lambda_i
The determinant being the product of the eigenvalues is now readable: stretch by 1 one way and by 5 the other, and areas grow by 5. ✅
We already wrote, in one line, “in the eigenbasis x=Qz”, and moved on.
That line is a change of basis, and it is worth a picture, because the whole definiteness argument rests on it.
The vector does not move. We are only changing the ruler we describe it with, from \hat{\imath},\hat{\jmath} to the eigenvectors.
If A=A^\top, then the eigenvalues are real, eigenvectors of distinct eigenvalues are orthogonal, and
A=Q\Lambda Q^\top,\qquad Q^\top Q = I
Read the three factors right to left, as a sequence of actions, exactly like section 2:
Q^\top rewrites the vector in the eigenbasis. \Lambda stretches each axis by its own \lambda_i. Q puts it back.
A symmetric matrix is just a stretch along perpendicular axes. Q names the axes, \Lambda gives the stretch factors.
With A symmetric, Q(x)=x^\top A x. In the eigenbasis this becomes
Q(x)=\sum_i \lambda_i z_i^2
Every z_i^2 is non-negative, so the signs of the \lambda_i decide the sign of the whole form:
Turn on the quadratic form in the widget and change the eigenvalue signs. The surface changes shape exactly when a sign flips.
At a stationary point \nabla f = 0, the second-order Taylor term is all that is left:
f(x_0+h)\approx f(x_0)+\tfrac12 h^\top H h
So the Hessian’s eigenvalues decide max, min, or saddle. Curvature is eigenvalues.
H\prec 0 gives a local max, H\succ 0 a local min, and H indefinite a saddle.
This is the bridge into next lecture’s concavity, and later into classifying steady states from a Jacobian. Same reading every time.
(1) A is symmetric with eigenvalues 3 and -2. Then x^\top A x is
c). Mixed signs, so a saddle.
(2) v is an eigenvector of A with \lambda = 1. Applying A to v
a). Same line, stretch factor 1, so nothing moves. Our e=(-1,1) was this case.
Let H=\begin{bmatrix}2&0\\0&-3\end{bmatrix} be a Hessian at a stationary point. Classify the point.
H is diagonal, so the eigenvalues are on the diagonal: \lambda_1 = 2, \lambda_2 = -3.
x^\top H x = 2x_1^2 - 3x_2^2
Positive along the first axis, negative along the second.
Mixed signs, so H is indefinite and the point is a saddle. ✅
Everything here was intuition for the linear algebra lecture, not a replacement for it. The definitions, theorems, and proofs you are examined on are in 03_linear_algebra_projection.
There is one more thing to work through on your own:
It puts the projection picture together with \hat\beta, R^2 as \cos^2\theta, multicollinearity and Frisch-Waugh-Lovell, in one place. Click the title above to open it.