Linear algebra, translated

Vectors

An arrow, or a list of numbers. It's the same thing.

The idea

A vector such as $\bm{u} = \begin{bmatrix} 3 \\ 1 \end{bmatrix}$ is a set of directions: "go 3 right and 1 up". It doesn't care where you start, only how far and which way. You can also read it as a list of data: 3 apples, 1 banana. Both readings are correct, and switching between them is half of this course.

• Adding vectors means doing one move, then the other. Put them tip to tail.
• Scaling (multiplying by a number) stretches or shrinks the arrow. A negative number flips it around.
• Length is Pythagoras: $\left\| \begin{bmatrix} 3 \\ 4 \end{bmatrix} \right\| = \sqrt{3^2 + 4^2} = 5$. More in Norm.
• $\R^n$ means "lists of $n$ numbers". $(2, 0, -1, 5)$ lives in $\R^4$. You can't draw it, but every rule works exactly the same: entry by entry.

Linear combinations

A recipe: some of this vector, plus some of that one.

The idea

A linear combination of $\bm{v}_1$ and $\bm{v}_2$ is any vector of the form

where the weights $c_1, c_2$ are ordinary numbers. You scale each ingredient, then add.

Practical hook: a smoothie. A banana gives 100 calories and 1 g of protein; a scoop of yogurt gives 60 calories and 5 g. Two bananas and three scoops give

The interesting question runs backwards: "I want exactly this nutrition. How much of each ingredient do I need?" That's asking for the weights, and it's where systems of equations come from.

The norm (length)

Pythagoras, extended to as many entries as you like.

The idea

• In 2D it's the hypotenuse. In 3D it's the diagonal of a box. In $\R^n$ it's the same formula with more terms.
• Distance between two points $P$ and $Q$ is $\|\bm{p} - \bm{q}\|$. Subtract, then take the length.
• Unit vector in the direction of $\bm{v}$: $\dfrac{\bm{v}}{\|\bm{v}\|}$. It keeps the direction and sets the length to 1. Think "just the compass heading, no distance".
• Practical hook: comparing two customers' shopping lists, or two songs' features, by the distance between their vectors. Small distance means similar. Recommendation apps do exactly this in $\R^{1000}$.

Rules worth knowing

The dot product

One number that says how much two arrows agree.

The idea

Multiply matching entries and add them up:

The answer is a plain number, not a vector.

Practical hook: you already use dot products at the till. Buy 2 coffees and 3 muffins at \$4 and \$3 each, and your bill is

Geometric meaning:

where $\theta$ is the angle between them. So the sign tells you the direction relationship at a glance:

Physics example you may know: work $=$ force $\cdot$ displacement. Only the part of your push that goes along the direction of motion counts. That "part along a direction" idea becomes projections.

It works the same in $\R^n$: $(1, 0, 2, 3)\cdot(2, 5, 1, -1) = 2 + 0 + 2 - 3 = 1$.

Vector projections

The shadow one arrow casts on another.

The idea

Shine a light straight down onto the line of $\bm{v}$. The shadow $\bm{u}$ casts on it is the projection of $\bm{u}$ onto $\bm{v}$. It answers: "how much of $\bm{u}$ points in $\bm{v}$'s direction?"

Every vector splits into two pieces: the part along $\bm{v}$ (the projection) and the part perpendicular to $\bm{v}$ (what's left over).

Practical hook: a box on a ramp. Gravity pulls straight down, but you split it into the part along the ramp (makes the box slide) and the part into the ramp (presses it against the surface). That split is exactly a projection.

Formulas

The perpendicular part is $\bm{u} - \operatorname{proj}_{\bm{v}}\bm{u}$. Reading the vector formula: $\dfrac{\bm{u}\cdot\bm{v}}{\bm{v}\cdot\bm{v}}$ is just a number, "how many $\bm{v}$'s long the shadow is". Then you multiply $\bm{v}$ by it.

The cross product

Give it two arrows in 3D, get back an arrow perpendicular to both.

The idea

• $\bm{u}\times\bm{v}$ is a vector that sticks straight out of the flat surface containing $\bm{u}$ and $\bm{v}$.
• Its length equals the area of the parallelogram $\bm{u}$ and $\bm{v}$ make: $\|\bm{u}\times\bm{v}\| = \|\bm{u}\|\,\|\bm{v}\|\sin\theta$.
• Its direction follows the right-hand rule: fingers along $\bm{u}$, curl toward $\bm{v}$, thumb points to $\bm{u}\times\bm{v}$.
• Real-world: torque. Push a wrench handle, and the bolt turns along the axis perpendicular to both the handle and your push.

The formula is easiest to remember as a determinant (Unit 7), expanded along the top row:

The minus sign on the middle entry is the part people get wrong.

Lines and planes in ℝⁿ

A line is "start here, walk one way". A plane is "start here, walk two ways".

Lines

Think GPS: a starting point $\bm{p}$, a direction $\bm{d}$, and a time $t$. Where are you at time $t$?

At $t = 0$ you're at $(1, 0, 2)$; at $t = 1$, at $(3, 1, 5)$; at $t = -1$, at $(-1, -1, -1)$. This formula doesn't care about dimension: a line in $\R^4$ is just $(1, 0, 2, -1) + t(1, 1, 0, 3)$.

Planes, version 1: parametric form (any $\R^n$)

A plane needs a starting point and two directions that don't point the same way. Two parameters, because a plane has two degrees of freedom:

Every point on the plane is "start at $\bm{p}$, go $s$ steps along $\bm{u}$ and $t$ steps along $\bm{v}$": $\bm{p}$ plus a linear combination of $\bm{u}$ and $\bm{v}$.

Planes, version 2: one equation ($\R^3$ only)

In 3D a plane can also be pinned down by one point on it and one normal vector $\bm{n} = (a, b, c)$ that sticks straight out of it, like a pencil standing upright on a table. Every arrow lying in the table is perpendicular to the pencil, so

The coefficients $(a, b, c)$ are the normal vector. Read it straight off.

Distance from a point to a plane

This is a projection onto the normal in disguise:

For example, from the origin to $2x - y + z = 3$: $\dfrac{|0 - 3|}{\sqrt6} = \dfrac{3}{\sqrt6} \approx 1.22$.

Systems of linear equations

Several conditions that all have to be true at once.

The idea

Each equation in two unknowns is a line. Solving the system means finding where all the lines meet. In three unknowns each equation is a plane. There are only three possibilities, and it's worth being able to draw all three:

"Linear" means each variable appears only multiplied by a number and added: no $x^2$, no $xy$, no $\sin x$, no $1/x$. That's what keeps the pictures flat (lines and planes, never curves).

Augmented matrices, REF and RREF

High-school elimination, with tidy bookkeeping.

The augmented matrix

The variable names never change during elimination, so stop writing them. Keep only the numbers, one row per equation and one column per variable, with a bar where the $=$ signs were:

A missing variable gets a 0 in its column. Don't skip it.

The three elementary row operations

Only ever do these, because none of them change the solution:

1. Interchange two rows, $R_i \leftrightarrow R_j$. (Listing the equations in a different order.)
2. Scale a row by a non-zero number, $R_i \to kR_i$. (Multiplying both sides of an equation.)
3. Replace a row by itself plus a multiple of another, $R_i \to R_i + kR_j$. (Classic elimination.)

REF and RREF

The same system at both stages:

Strategy: work left to right, creating zeros below each pivot (this gives REF). Then work right to left, scaling pivots to 1 and creating zeros above them (this gives RREF).

Reading the final matrix

Matrix–vector and matrix–matrix multiplication

A matrix is a machine: vector goes in, vector comes out.

The most useful idea: the column view

Multiplying a matrix by a vector means "take this much of column 1, plus this much of column 2". A bakery example makes it concrete. The columns are recipes (one cake, one batch of cookies); the rows are ingredients (eggs, cups of flour):

The vector says how many of each recipe you're making (2 cakes, 5 batches). The answer is your shopping list. That's a linear combination of the columns.

Two ways to compute $A\bm{x}$, same answer

Take $A = \begin{bmatrix} 1 & 0 & 2 \\ -1 & 3 & 1 \end{bmatrix}$ and $\bm{x} = \begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}$.

Row view (dot each row with $\bm{x}$), the fastest by hand:

Column view (weights times columns), which gives the meaning:

Size check: $A$ is $2\times3$ and $\bm{x}$ has 3 entries, so the answer has 2 entries. The number of columns of $A$ must equal the number of entries of $\bm{x}$.

Matrix times matrix

• Size rule: $(m\times n)(n\times p) = (m\times p)$. The inner numbers must match; the outer numbers give the answer's size.
• The $(i, j)$ entry of $AB$ is (row $i$ of $A$) $\cdot$ (column $j$ of $B$).
• Order matters. $AB$ is usually not $BA$. Socks-then-shoes is not shoes-then-socks.

Matrix algebra basics

• Adding, subtracting: the matrices must be the same size; work entry by entry.
• Scalar times matrix: multiply every entry.
• Transpose $A^T$: rows become columns, so a $2\times3$ becomes a $3\times2$: $\begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}^T = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}$.
• Identity $I$: 1s on the diagonal, 0s elsewhere. $AI = IA = A$, like multiplying by 1.
• Powers: $A^2 = AA$. Only square matrices have powers.

A surprising example:

even though the matrix itself isn't zero.

Matrix inverse

The undo button.

The idea

If $A$ is a machine, $A^{-1}$ is the machine that reverses it: $A^{-1}A = AA^{-1} = I$. You can't divide by a matrix, so to solve $A\bm{x} = \bm{b}$ you multiply both sides by the undo button: $\bm{x} = A^{-1}\bm{b}$.

Some matrices have no inverse. If $A$ squashes the whole plane onto a line, lots of different inputs land on the same output, and there's no way to know which one you started from. You can't unscramble an egg.

The $2\times2$ formula

In words: swap $a$ and $d$, change the signs of $b$ and $c$, and divide by $ad - bc$. If $ad - bc = 0$, there is no inverse.

The $[\,A \mid I\,]$ method (any size)

Write $A$ and $I$ side by side and row reduce until the left half is $I$. The right half is then $A^{-1}$:

Rules worth knowing

• $(A^{-1})^{-1} = A$. Undoing the undo gets you back where you started.
• $(AB)^{-1} = B^{-1}A^{-1}$. The order reverses: to undo "socks then shoes", you take off the shoes first.
• $(A^T)^{-1} = (A^{-1})^T$.
• Only square matrices can have inverses.

Span, independence and basis

Where can these vectors take you, and is any of them wasted?

Span: where can you get to?

Imagine you can only move along certain arrows, as far as you like, forwards or backwards. The span is every place you can reach, i.e. all the linear combinations:

• One non-zero arrow: you can reach a line through the origin.
• Two arrows pointing different ways: you can reach a plane through the origin (in 2D, that's everything).
• Two arrows pointing the same way (one a multiple of the other): still only a line. The second one added nothing.

Is $\bm{b}$ in the span? That asks "can I find weights that make $\bm{b}$?", which is a system. Put the vectors in as columns with $\bm{b}$ after the bar, and row reduce. Consistent means yes.

Independence: is anything redundant?

A set of vectors is linearly independent if none of them can be built from the others. Paint analogy: red, blue and yellow are independent. Add purple and the set becomes dependent, because you could already mix purple from red and blue.

The textbook definition: $\{\bm{v}_1, \dots, \bm{v}_k\}$ is independent if

In words: the only way to walk along these arrows and end up back at the start is not to move at all.

How to test: put the vectors in as the columns of a matrix and row reduce. A pivot in every column means independent; any column without a pivot means dependent. For $n$ vectors in $\R^n$ you can instead check that the determinant is non-zero.

Basis and dimension

A basis of a space is a set of vectors that is independent (no waste) and spans the space (reaches everywhere). It's a coordinate system. The standard basis of $\R^2$ is $\left\{\begin{bmatrix} 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 1 \end{bmatrix}\right\}$, but $\left\{\begin{bmatrix} 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \end{bmatrix}\right\}$ works just as well.

Every basis of a space has the same number of vectors. That number is the dimension: $\dim\R^n = n$, a plane has dimension 2, a line has dimension 1.

The weights needed to build a vector from a basis are its coordinates in that basis. In section 1.2 we found $(7, 4) = 1(1, 1) + 3(2, 1)$, so in the basis $\{(1, 1), (2, 1)\}$ its coordinates are $(1, 3)$.

Subspaces of ℝⁿ

Flat things through the origin, and the two that every matrix comes with.

What counts as a subspace

A subspace $U$ of $\R^n$ is a flat thing through the origin: just $\{\bm{0}\}$, a line through $\bm{0}$, a plane through $\bm{0}$, and so on up to all of $\R^n$. The subspace test has three parts:

1. $\bm{0} \in U$. (The fastest way to rule things out.)
2. If $\bm{x}, \bm{y} \in U$, then $\bm{x} + \bm{y} \in U$. (Closed under addition.)
3. If $\bm{x} \in U$ and $c \in \R$, then $c\bm{x} \in U$. (Closed under scalar multiplication.)

A useful shortcut: the span of any set of vectors is always a subspace. So if you can write the set as "all combinations of these vectors", you're done.

The two subspaces every matrix has

The row space (the span of the rows) has the same dimension as the column space. A basis for it is the non-zero rows of the RREF.

Rank and the rank–nullity theorem

For an $m\times n$ matrix $A$,

In words: every column is either a pivot column (a "decided" variable) or a free column (a free variable). Count both and you've counted every column.

Concept map on the board

• Everything in Units 1–5 connects to $A\bm{x} = \bm{b}$: linear combinations of columns (U1), row reduction (U2), inverse (U3), independence of columns (U4), null and column space (U5).
• Partial Invertible Matrix Theorem: invertible, a pivot in every column, independent columns, columns span, $\operatorname{null}A = \{\bm{0}\}$.

Exam technique

• Answer the question asked (number, vector, matrix, yes/no). Show the row operations. Check every answer: substitute back, dot with the originals, multiply out.
• Exams usually allow only a basic calculator, so practise the arithmetic by hand.

Linear transformations

Moves that keep grid lines straight, parallel and evenly spaced, and keep the origin still.

The single most useful fact

The columns of the matrix are where the standard basis vectors land. For $T : \R^2 \to \R^2$,

To build any transformation matrix, just ask "where do $\bm{e}_1$ and $\bm{e}_2$ end up?" Real-world: every rotation, zoom and flip in a video game or photo editor is a matrix doing exactly this.

Is it linear?

Officially, $T$ is linear if for all vectors $\bm{u}, \bm{v}$ and scalars $c$:

Quick screening tests:

• Does $\bm{0}$ go to $\bm{0}$? If not, it's not linear. So "shift everything right by 1" is out.
• Any squares, products of variables, $\sin x$, or added constants? Not linear.
• If each output is just "numbers times variables, added up", it's linear, and you can read off the matrix.

Kernel and image

• The kernel of $T$ is every input sent to $\bm{0}$. It's the null space of $T$'s matrix (Unit 5).
• The image of $T$ is every output $T$ can produce. It's the column space of $T$'s matrix.
• $T$ is one-to-one (no two inputs share an output) exactly when $\ker T = \{\bm{0}\}$.
• $T$ is onto (every output can be reached) exactly when the image is the whole target space.
• Example: projecting 3D onto the floor, $T(x, y, z) = (x, y, 0)$. The kernel is the $z$-axis (everything squashed to $\bm{0}$), and the image is the floor. $T$ is neither one-to-one nor onto $\R^3$.

Determinants

How much the matrix stretches area (or volume).

The idea

Take the unit square and push it through the matrix. It becomes a parallelogram. The determinant is that parallelogram's (signed) area.

• $\det A = 6$: areas get 6 times bigger.
• $\det A \lt 0$: space got flipped over, like a mirror image.
• $\det A = 0$: everything got squashed flat. That's exactly why there's no inverse.

How to compute

$2\times2$:

$3\times3$, cofactor expansion along the first row (signs alternate $+\,-\,+$):

Tip: you may expand along any row or column. Choose the one with the most zeros.

Or row reduce to a triangular matrix, whose determinant is the product of the diagonal entries, keeping track of what the row operations did:

Properties

• $\det(AB) = \det A \cdot \det B$ and $\det(A^T) = \det A$.
• $\det(A^{-1}) = \dfrac{1}{\det A}$.
• $A$ is invertible exactly when $\det A \ne 0$.

Cramer's rule

Solve for one variable at a time using determinants.

The rule

If $A$ is $n\times n$ with $\det A \ne 0$, the solution of $A\bm{x} = \bm{b}$ is

where $A_i(\bm{b})$ is $A$ with column $i$ replaced by $\bm{b}$. In words: to get a variable, swap the right-hand side into that variable's column, take the determinant, and divide by the original determinant.

When to use it

Eigenvalues and eigenvectors

Find the directions a matrix only stretches, then use them as your axes.

Eigenvectors: the directions that don't turn

Most vectors get knocked off their line when a matrix acts on them. A few special ones stay on their own line and just get longer, shorter or flipped. A non-zero vector $\bm{v}$ is an eigenvector of $A$ with eigenvalue $\lambda$ if

In words: "$A$ does to $\bm{v}$ what a plain number would." Picture spinning a globe: every point moves except those on the axis. The axis is an eigenvector with eigenvalue 1. Real uses include vibration modes of bridges and buildings, Google's original PageRank, and population models.

Recipe

1. Solve the characteristic equation $\det(A - \lambda I) = 0$ for $\lambda$. Why: we need $(A - \lambda I)\bm{v} = \bm{0}$ to have a non-zero solution, so $A - \lambda I$ must squash something, so its determinant is 0.
2. For each $\lambda$, solve $(A - \lambda I)\bm{v} = \bm{0}$ by row reduction. The solution set is the eigenspace; its basis vectors are the eigenvectors.

Diagonalization

Diagonalization: switching to the matrix's favourite axes

Use the eigenvectors as your coordinate axes, and the matrix does nothing but stretch along each axis. In those coordinates it becomes a diagonal matrix, the simplest kind there is:

with the eigenvalues in $D$ in the same order as their eigenvectors in $P$.

Why bother? Powers. Computing $A^{100}$ directly takes 99 matrix multiplications. With diagonalization, $A^{100} = PD^{100}P^{-1}$, and powering a diagonal matrix just powers each diagonal entry. Engineers use this to predict where a system ends up after many steps.

When can you diagonalize?

• An $n\times n$ matrix is diagonalizable exactly when it has $n$ independent eigenvectors (enough to fill $P$).
• $n$ distinct eigenvalues: guaranteed yes.
• A repeated eigenvalue: check. If $\lambda$ is a root of multiplicity $k$, you need $k$ independent eigenvectors for it.
• The classic "no": the shear $\begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$. $\lambda = 1$ twice, but only one eigenvector direction, $(1, 0)$.

Orthogonal and orthonormal bases

Bases where every direction is at right angles to the others, like the lines on graph paper.

The idea

• A set $\{\bm{v}_1, \dots, \bm{v}_k\}$ is orthogonal if $\bm{v}_i\cdot\bm{v}_j = 0$ whenever $i \ne j$.
• It is orthonormal if, in addition, every $\|\bm{v}_i\| = 1$.
• An orthogonal set of non-zero vectors is automatically independent. Perpendicular directions can't be built from each other.

Why they're so convenient: in an ordinary basis, finding coordinates means solving a system. In an orthogonal basis, each coordinate is a single projection:

If the basis is orthonormal, this is simply $c_i = \bm{x}\cdot\bm{v}_i$.

The Gram–Schmidt process

Given any basis $\{\bm{x}_1, \bm{x}_2, \dots\}$, Gram–Schmidt turns it into an orthogonal basis of the same space. The idea: keep the first vector; from each next vector, subtract its shadows on the ones you've already fixed. What's left is perpendicular to all of them.

and so on. Divide each $\bm{v}_i$ by its length at the end if you need an orthonormal basis.

Orthogonal matrices

A square matrix $Q$ whose columns are orthonormal is called orthogonal. Its big perk:

The inverse comes for free. Rotations and reflections are orthogonal matrices: they keep every length and angle the same, and $\det Q = \pm1$.

Projecting onto a subspace

If $\{\bm{v}_1, \dots, \bm{v}_k\}$ is an orthogonal basis of a subspace $U$, then

The leftover $\bm{x} - \operatorname{proj}_U\bm{x}$ is perpendicular to all of $U$. The set of all vectors perpendicular to $U$ is the orthogonal complement $U^\perp$.

Symmetric matrices

Matrices that are pure stretching along perpendicular axes.

The idea

A matrix is symmetric if $A^T = A$: it's a mirror image of itself across the main diagonal. $\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$ is symmetric; $\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$ isn't.

They come up whenever "the effect of $i$ on $j$ equals the effect of $j$ on $i$": stiffness matrices in structures, stress in materials, covariance in data, distance tables.

The Principal Axes (Spectral) Theorem

For a real symmetric matrix $A$:

1. All its eigenvalues are real.
2. Eigenvectors for different eigenvalues are automatically orthogonal.
3. $A$ is orthogonally diagonalizable: $A = QDQ^T$ with $Q$ orthogonal. Since $Q^{-1} = Q^T$, there's no inverse to compute.

In plain English: every symmetric matrix just stretches space along some set of perpendicular axes. Those axes are its eigenvectors.

Recipe for $A = QDQ^T$

1. Find the eigenvalues and eigenvectors as usual.
2. Normalize the eigenvectors to length 1. (If an eigenvalue repeats, run Gram–Schmidt on its eigenvectors first.)
3. Put them in $Q$ as columns, and the eigenvalues in $D$ in the same order.

Quadratic forms

An expression like $2x^2 + 2xy + 2y^2$ can be written as $\bm{x}^TA\bm{x}$ with a symmetric $A$. The squared terms go on the diagonal, and the $xy$ coefficient is split in half between the two off-diagonal entries:

Switching to the eigenvector axes removes the cross term: the expression becomes $3u^2 + v^2$. If every eigenvalue is positive, the form is positive definite: it's positive for every $\bm{x} \ne \bm{0}$, and its level curves are ellipses like the one above.

Least squares

When there's no exact answer, find the closest one.

The idea

Real measurements are noisy. Fit a straight line through 4 lab data points and you get 4 equations in 2 unknowns, with no exact solution. A least-squares solution $\hat{\bm{x}}$ makes $A\hat{\bm{x}}$ as close as possible to $\bm{b}$, that is, it makes $\|A\bm{x} - \bm{b}\|$ as small as it can be.

Geometric picture: $A\bm{x}$ can only land in the column space of $A$. The closest point there to $\bm{b}$ is the projection of $\bm{b}$ onto the column space. The leftover error is perpendicular to every column, i.e. $A^T(\bm{b} - A\hat{\bm{x}}) = \bm{0}$, which gives the normal equations:

Recipe for a best-fit line $y = a + bx$

1. Write one equation per data point: $a + bx_i = y_i$.
2. $A$ has a column of 1s (for $a$) and a column of the $x$-values (for $b$). The vector $\bm{y}$ holds the $y$-values.
3. Compute $A^TA$ and $A^T\bm{y}$, then solve the $2\times2$ system $A^TA\begin{bmatrix} a \\ b \end{bmatrix} = A^T\bm{y}$.

One matrix, three lenses

• Take $A = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix}$: as a transformation (where do $\bm{e}_1, \bm{e}_2$ go?), as an area scale ($\det = 10$), and by its eigenvectors (stretching by 5 and 2).
• Add to the Invertible Matrix Theorem: $\det A \ne 0$, 0 is not an eigenvalue, $T$ is one-to-one and onto.

Typical exam traps

• Composition order; $\det(kA) = k^n\det A$; mismatched $P$ and $D$; repeated eigenvalues.

The course in one picture

• Unit by unit around $A\bm{x} = \bm{b}$: columns (U1), solving (U2–3), structure (U4–5), geometry of $A$ (U6–7), best axes (U8, U10), perpendicularity (U9), best approximation (U11).
• The full Invertible Matrix Theorem (Appendix B).

Revision plan

• Rework one problem per lecture; do the self-check table; MATLAB command sheet.