Change of basis with Python

The lesson on linear maps stated that the matrix of a transformation depends on the basis, and that the relation between two representations is a similarity. This lesson answers the question that one left open: among all possible bases, which produces the simplest matrix, and what is preserved in moving from one to another.

Eigenvectors

The course has used eigenvectors on several occasions — in describing principal component analysis, in justifying the symmetry of $A^\top A$ — without defining them. The definition is as follows.

Let $A \in \mathbb{R}^{n\times n}$ . A vector $\vec{v} \neq \vec{0}$ is an eigenvector of $A$ if there is a scalar $\lambda$ such that

A\vec{v} = \lambda\vec{v}

and $\lambda$ is the associated eigenvalue. The condition $\vec{v} \neq \vec{0}$ is part of the definition: the zero vector satisfies the equality for every $\lambda$ and admitting it would empty the notion of content.

Geometrically, $\vec{v}$ spans a line the transformation leaves invariant. On it, the map does nothing but multiply by $\lambda$ .

Rewriting the definition as $(A - \lambda I)\vec{v} = \vec{0}$ with $\vec{v} \neq \vec{0}$ , the null space of $A - \lambda I$ must be non-trivial, which by the determinant criterion of the lesson on the inverse and the transpose is equivalent to

\det(A - \lambda I) = 0

This is the characteristic equation. For $n = 2$ it is a polynomial of degree two, so there are at most two real eigenvalues, and there may be none.

The lines are the directions the map leaves invariant. The grey arrow is a unit vector; the red one is its image.

a2b1c0d3

two independent eigen-directions: diagonalisable

λ = 3.00 , 2.00

tr = 5.00 · det = 6.00

In its eigenbasis the matrix is diag(3.00, 2.00)

Diagonalisation

If $A$ has $n$ linearly independent eigenvectors, placing them as the columns of a matrix $S$ gives

S^{-1} A S = \Lambda = \operatorname{diag}(\lambda_1,\dots,\lambda_n)

This is the similarity of the previous lesson with a particular choice of basis: the one formed by the eigenvectors themselves. In that basis the map does not mix directions, it merely scales each axis by its eigenvalue. It is the simplest form the matrix can take.

import numpy as np

A = np.array([[2., 1.],
              [0., 3.]])

vals, S = np.linalg.eig(A)
np.allclose(np.linalg.inv(S) @ A @ S, np.diag(vals))     # True

Not every matrix is diagonalisable, and the figure reaches all three cases that prevent or permit it.

With distinct real eigenvalues the eigenvectors are automatically independent and the diagonalisation exists. A multiple of the identity is already diagonal in any basis. But a shear such as $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ has a repeated eigenvalue and a single eigen-direction: there are not two independent eigenvectors with which to form $S$ , and the matrix is called defective. A rotation has no real eigenvalues, because no line survives the turn.

The practical consequence is that np.linalg.eig may return complex values, and that the matrix $S$ it produces can be badly conditioned when the matrix approaches the defective case. Diagonalisation is neither always available nor always stable.

What similarity preserves

Two matrices related by $\tilde{A} = S^{-1}AS$ represent the same map in different bases. The quantities that do not depend on the basis are therefore properties of the map, and come out identical in both.

2.001.0003.00

→

S⁻¹AS

2.571.710.142.43

tr	5.00	5.00
det	6.00	6.00
λ	3.00 , 2.00	3.00 , 2.00

S is the basis; A stays fixed.

s₁₁2s₁₂-1s₂₁1s₂₂3

The entries change; the trace, the determinant and the eigenvalues do not.

One particular choice of $S$ returns the diagonal form: the one made of the eigenvectors. For $A = \begin{bmatrix}2&1\\0&3\end{bmatrix}$ that matrix is $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ , and the result is $\operatorname{diag}(2, 3)$ . Any other $S$ produces different entries with the same invariants.

The proof for the determinant is immediate from the multiplicativity of its own lesson:

\det(S^{-1}AS) = \det(S^{-1})\det(A)\det(S) = \frac{1}{\det S}\det(A)\det(S) = \det(A)

For the trace the cyclic property $\operatorname{tr}(XY) = \operatorname{tr}(YX)$ gives $\operatorname{tr}(S^{-1}AS) = \operatorname{tr}(ASS^{-1}) = \operatorname{tr}(A)$ . The eigenvalues agree because the characteristic polynomial is the same, and with them the rank.

Two identities connecting these quantities follow:

\operatorname{tr}(A) = \sum_i \lambda_i, \qquad \det(A) = \prod_i \lambda_i

In the eigenbasis both are evident, since the matrix is diagonal; and as neither depends on the basis, they hold in any.

The spectral theorem

For symmetric matrices the situation improves substantially. The spectral theorem states that every real symmetric matrix is diagonalisable, that its eigenvalues are real, and that its eigenvectors can be chosen orthonormal:

A = A^\top \quad\Longrightarrow\quad A = Q\Lambda Q^\top, \qquad Q^\top Q = I

The difference from the general case is that $S^{-1}$ is replaced by $Q^\top$ . There is no inverse to compute, the condition number of $Q$ is exactly $1$ by the lesson on basis and dimension, and the defective and complex cases are excluded.

C = np.array([[2., 1.],
              [1., 3.]])                    # symmetric

vals, Q = np.linalg.eigh(C)                 # eigh, not eig
np.allclose(Q @ np.diag(vals) @ Q.T, C)     # True
np.allclose(Q.T @ Q, np.eye(2))             # True

np.linalg.eigh is not a convenience but the correct routine: it exploits the symmetry, guarantees real eigenvalues by construction, and returns an ordered orthonormal basis. Using eig on a symmetric matrix gives the same result with more work and with rounding error in the imaginary part.

Application: principal component analysis

The lessons on subspaces and on basis and dimension presented principal component analysis two ways: as the search for the subspace minimising the residual, and as the change to the basis where the data need fewest coordinates. The spectral theorem explains why both descriptions coincide and why the problem has a solution at all.

The covariance matrix $C = \tfrac{1}{n}X^\top X$ of centred data is symmetric and positive semidefinite, as the lesson on the inverse and the transpose established. The spectral theorem then guarantees an orthonormal basis of eigenvectors with real, non-negative eigenvalues.

X = X - X.mean(axis=0)
C = X.T @ X / len(X)

vals, Q = np.linalg.eigh(C)         # ascending
Z = X @ Q[:, ::-1]                  # coordinates in the eigenbasis

In that basis the covariance is diagonal: the new coordinates are uncorrelated, and the $i$ -th eigenvalue is the variance along direction $i$ . Retaining the $k$ largest is exactly the low-rank approximation of the lesson on rank, and the truncation error is the sum of the discarded eigenvalues.

Principal component analysis is therefore not a separate algorithm. It is the diagonalisation of a symmetric matrix, with a statistical reading of its eigenvalues.

Exercise. Build the shear $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ and check that np.linalg.eig returns a repeated eigenvalue and two practically identical columns in $S$ . Compute np.linalg.cond(S) and explain why the diagonalisation is unusable in that case. Repeat with $\begin{bmatrix}1&1\\0&1.01\end{bmatrix}$ and observe how the conditioning changes.