Notes

The Geometry of Normality

13 min read · Linear algebra · Polar decomposition

Stretching, turning, and polar decomposition: a geometric view of when a matrix is normal.

I’ve always found that when a linear algebra concept is either geometrically motivated or has geometrical implications, I’m able to understand it and see its significance more than if I’d only approached it algebraically. I don’t have much natural skill with geometric reasoning, but nonetheless, when the geometry of the problem is revealed to me, a much deeper understanding comes along with it.

One of the most satisfying examples was in graduate school reading Trefethen & Bau’s presentation of the Singular Value Decomposition in Numerical Linear Algebra [1]. I’d studied SVD in a couple of undergraduate courses and didn’t think there was much more juice to be squeezed, but Trefethen & Bau introduced SVD by pointing out that the image of the unit sphere SS under a real m×nm{\times}n matrix AA with m≥nm\geq n and full column rank, is a hyperellipse and the nn singular values of AA are the lengths of the principal semiaxes of ASAS.1 This fact seemed like it should’ve been obvious once it was pointed out to me, but it wasn’t. I hadn’t encountered this perspective in my previous linear algebra courses. Figure 1 illustrates this in two dimensions.

Two perpendicular singular directions on a unit circle map to the principal semiaxes of a tilted ellipse. Each image vector has length equal to its singular value.
Figure 1. Geometry of the SVD — The geometry of the SVD for a real 2×22\times2 matrix. The orthonormal right singular vectors v1,v2v_1,v_2 on the unit circle SS map to the principal semiaxes Avi=σiuiAv_i=\sigma_i u_i of the ellipse ASAS. Their lengths are the singular values σ1\sigma_1 and σ2\sigma_2. Inspired by Figure 4.1 of [1].

That revelation completely changed the way I viewed SVD, and in fact, all matrix decompositions. It is in that spirit of geometric understanding I want to discuss another important matrix decomposition: the polar decomposition. Polar decomposition gives a geometric way to understand normality by asking when stretching and turning can be performed in either order.

The Polar Decomposition

The polar decomposition is generalizable to rectangular matrices with m≥nm \geq n, but for our purposes we will stick to square matrices. Any A∈Cn×nA \in \mathbb{C}^{n\times n} has a factorization A=UPA = UP, where U∈Cn×nU \in \mathbb{C}^{n\times n} is unitary and P∈Cn×nP \in \mathbb{C}^{n\times n} is Hermitian positive semidefinite. [2]

This decomposition is a natural extension of the polar representation z=reiθz = re^{i\theta} of a complex number. In fact, as we shall see, PP functions like rr and UU like eiθe^{i\theta}.

From here on, let AA be a real invertible square matrix. Its polar decomposition A=UPA=UP, with UU orthogonal and PP symmetric positive definite (SPD), is unique. Recall a matrix PP is SPD if P=PTP=P^{\mathsf T} and xTPx>0x^{\mathsf T}Px>0 for every nonzero xx.

And since UU is orthogonal, this demonstrates an interesting fact. Any invertible linear transformation in space can be decomposed into two distinct steps: stretching of space along principal axes (application of PP) followed by a rigid rotation (or reflection) with no distortion (application of UU).

Stretching and turning

Geometrically, PP acts on a vector by stretching components along perpendicular axes by positive factors. If viv_i is a unit vector along one of those axes, then

Pvi=σivi,σi>0. Pv_i=\sigma_i v_i, \qquad \sigma_i>0.

The σi\sigma_i are also the singular values of AA. σi>1\sigma_i>1 expands; 0<σi<10<\sigma_i<1 contracts. So, when I say "stretch", I really mean "scale". In the plane, PP’s image of the unit circle becomes an ellipse whose semiaxes lie along the viv_i and have lengths σi\sigma_i.

A symmetric stretch can still change a vector’s direction. For example, P=diag⁡(2,1)P=\operatorname{diag}(2,1) maps (1,1)T(1,1)^{\mathsf T} to (2,1)T(2,1)^{\mathsf T}. Its principal axes stay fixed, but the components aren’t scaled the same. So vectors not on the principal axes can get turned. Contrast this with a rigid rotation, which turns all vectors through the same angle in the plane.

But that seems to imply the amount of stretching depends on direction. If that is the case, wouldn’t rotating first change the result? So order matters here. Or, rather, order CAN matter here. But when does it not? That is, when do we have UP=PUUP = PU?

When can we reverse the order?

First, let’s consider when UP≠PUUP \neq PU. It’s easy to construct such cases. Consider a stretch that doubles horizontal components and leaves vertical components alone, followed by a counterclockwise quarter-turn:

P=[2001],U=[0−110]. P=\begin{bmatrix}2&0\\0&1\end{bmatrix}, \qquad U=\begin{bmatrix}0&-1\\1&0\end{bmatrix}.

Applied to the horizontal unit vector e1=(1,0)Te_1=(1,0)^{\mathsf T}, the two orders give

[10]→P[20]→U[02],[10]→U[01]→P[01]. \begin{aligned} \begin{bmatrix}1\\0\end{bmatrix} &\xrightarrow{P}\begin{bmatrix}2\\0\end{bmatrix} \xrightarrow{U}\begin{bmatrix}0\\2\end{bmatrix},\\[0.5em] \begin{bmatrix}1\\0\end{bmatrix} &\xrightarrow{U}\begin{bmatrix}0\\1\end{bmatrix} \xrightarrow{P}\begin{bmatrix}0\\1\end{bmatrix}. \end{aligned}

The stretch treats horizontal and vertical directions differently. Rotating first changes which treatment this vector receives. The final vectors point in the same direction, but they aren’t the same length.

Now replace PP with the uniform stretch 2I2I. Every direction gets the same factor, so a rotation can’t move a vector into a differently stretched direction. The two orders agree for every vector (Figure 2).

Six panels compare input, stretch then quarter-turn, and quarter-turn then stretch. With horizontal scaling by two, the output ellipses have different orientations and the marked vectors have different lengths. With uniform scaling by two, both outputs coincide point for point.
Figure 2. Reversing the order — Reversing the same stretch and a counterclockwise quarter-turn. Unequal stretches give different results in the upper group. Uniform stretching makes the two orders agree point for point in the lower group.

But does this mean we need uniform stretching for the operations to commute? Not necessarily. Consider our original P=diag⁡(2,1)P=\operatorname{diag}(2,1), but this time replace the quarter-turn with a reflection across the horizontal axis, U=diag⁡(1,−1)U=\operatorname{diag}(1,-1). Here the order doesn’t matter. The reflection reverses one of the principal directions, but it doesn’t swap the axes. The horizontal component is doubled, and the vertical component changes sign without changing magnitude, regardless of which operation acts first. So maybe the important thing isn’t whether the stretching is uniform, but whether the orthogonal operator preserves the stretching pattern.

A shear hides both operations

So far we’ve started with a stretch and a turn and asked what happens if we reverse their order. But what if we start with a transformation whose stretching and turning aren’t obvious in the first place? Consider a shear. A horizontal shear slides each horizontal line by an amount proportional to its height.

A=[1201],A[xy]=[x+2yy]. A=\begin{bmatrix}1&2\\0&1\end{bmatrix}, \qquad A\begin{bmatrix}x\\y\end{bmatrix} =\begin{bmatrix}x+2y\\y\end{bmatrix}.

It leaves the horizontal axis fixed. There doesn’t appear to be a rotation involved. Nevertheless, its polar factors are

U=12[11−11],P=12[1113]. U=\frac{1}{\sqrt{2}}\begin{bmatrix}1&1\\-1&1\end{bmatrix}, \qquad P=\frac{1}{\sqrt{2}}\begin{bmatrix}1&1\\1&3\end{bmatrix}.

The matrix UU is a clockwise rotation through 45∘45^\circ. The shear is not itself a rotation, but its rotation factor is exactly what undoes the stretch’s action on the horizontal axis.

The fixed horizontal vector makes the interaction clearer:

Pe1=12[11],UPe1=e1. Pe_1=\frac{1}{\sqrt{2}}\begin{bmatrix}1\\1\end{bmatrix}, \qquad UPe_1=e_1.

You can see the stretch tilts e1e_1 upward through 45∘45^\circ without changing this vector’s length. But then the clockwise turn brings it back. So we can have a vector that is fixed in place by the complete transformation even though both stages move it (Figure 3).

Three panels follow a unit circle through a symmetric stretch and a clockwise 45-degree turn to a shear. The blue horizontal unit vector first tilts upward and then returns to its original position; the orange vertical vector ends at coordinates (2, 1).
Figure 3. The polar factors of a shear — The shear as a symmetric stretch followed by a clockwise 45∘45^\circ turn. Follow the blue circle from e1e_1 to (1,1)T/2(1,1)^{\mathsf T}/\sqrt{2} and back to e1e_1. The orange square follows e2e_2 to (1,3)T/2(1,3)^{\mathsf T}/\sqrt{2} and then to (2,1)T(2,1)^{\mathsf T}.

What changes if we turn first? The same two factors now give

UP=[1201],PU=[01−12]. UP=\begin{bmatrix}1&2\\0&1\end{bmatrix}, \qquad PU=\begin{bmatrix}0&1\\-1&2\end{bmatrix}.

In particular,

UPe1=[10],PUe1=[0−1]. UPe_1=\begin{bmatrix}1\\0\end{bmatrix}, \qquad PUe_1=\begin{bmatrix}0\\-1\end{bmatrix}.

The coordinated cancellation that left e1e_1 fixed previously disappears when we swap the factors. Figure 4 shows the change. The clockwise 45∘45^\circ rotation doesn’t preserve the stretching pattern of PP, so these factors don’t commute.

Two panels compare stretch then turn with turn then stretch for the shear. The blue input point ends at (1, 0) in the first panel and (0, -1) in the second. The ellipses have equal semiaxis lengths and different orientations.
Figure 4. Reversing the shear’s factors — Reversing the shear’s polar factors changes the transformation. The same blue input point ends at (1,0)T(1,0)^{\mathsf T} under UPUP and at (0,−1)T(0,-1)^{\mathsf T} under PUPU. The two ellipses have the same semiaxis lengths but different orientations in the fixed coordinate system.

Why this is normality

Turning first changes the stretch

So, we can’t generally switch the same\emph{same} two factors. But can we always turn first then stretch? Yes, but with a different stretch factor, QQ.

A=UP=QU,Q=UPUT. A=UP=QU, \qquad Q=UPU^{\mathsf T}.

This is the left polar decomposition. Its stretch QQ generally differs from the right stretch PP. The identity UP=QUUP=QU always holds with this adjusted stretch. It scales by the same amounts as PP, but along axes moved by UU.

But under what conditions can we turn first and still use the same stretch factor, PP?

Where normality comes in

Recall a real, square matrix is called normal if ATA=AATA^{\mathsf T}A = AA^{\mathsf T}. The two sides have a direct geometric interpretation. From A=UP=QUA=UP=QU, we get

ATA=P2,AAT=Q2. A^{\mathsf T}A=P^2, \qquad AA^{\mathsf T}=Q^2.

ATA=P2A^{\mathsf T}A=P^2 describes stretching along the original principal axes. AAT=Q2AA^{\mathsf T}=Q^2 describes the same stretch magnitudes but along axes moved by UU.

We have AA is normal if and only if P2=Q2P^2=Q^2. And because of the uniqueness of positive-definite square roots, we have P=QP=Q. Since Q=UPUTQ=UPU^{\mathsf T}, P=QP=Q is the same statement as PU=UPPU=UP.

So we have the non-obvious (to me at least!) fact that normality is precisely the condition that the polar factors of AA commute [3, Chapter 8]:

ATA=AAT  ⟺  P=Q  ⟺  UP=PU \boxed{A^{\mathsf T}A=AA^{\mathsf T} \iff P=Q \iff UP=PU}

This reminds me of what I found so satisfying about the sphere-to-ellipse picture of SVD I mentioned in the intro. I’d known the algebraic definition of normality for years, but viewing normality through this geometric construal lets me understand it from a different perspective.

Where the scalar analogy survives

I began by stating the polar decomposition is analogous to the polar form z=reiθz=re^{i\theta} of a complex number, whose magnitude and phase factors, of course, always commute. For a normal matrix, we can recover that scalar picture one coordinate at a time.

We’ve just shown that normality makes UU and PP commute. Both factors are normal themselves, so the simultaneous diagonalization theorem gives us a shared orthonormal eigenbasis over C\mathbb C [4, Theorem 2].2 If VV is the unitary matrix whose columns are those eigenvectors, then

P=Vdiag⁡(r1,…,rn)V∗,U=Vdiag⁡(eiθ1,…,eiθn)V∗. \begin{aligned} P&=V\operatorname{diag}(r_1,\ldots,r_n)V^*,\\ U&=V\operatorname{diag}(e^{i\theta_1},\ldots,e^{i\theta_n})V^*. \end{aligned}

Here V∗V^* is the conjugate transpose of VV. The rj>0r_j>0 are the eigenvalues of PP, hence the singular values of AA, and the eigenvalues of UU lie on the unit circle because UU is orthogonal.

Multiplying the two diagonal representations gives

A=Vdiag⁡(r1eiθ1,…,rneiθn)V∗. A=V\operatorname{diag} (r_1e^{i\theta_1},\ldots,r_ne^{i\theta_n})V^*.

So in this basis, applying AA just multiplies the jjth coordinate by rjeiθjr_je^{i\theta_j}, independently of all the others. We started with an analogy to z=reiθz=re^{i\theta}; now each coordinate has exactly that form. The requirement that stretching and turning commute has brought us back to ordinary complex multiplication. In particular,

σj(A)=∣λj(A)∣,j=1,…,n, \sigma_j(A)=|\lambda_j(A)|,\qquad j=1,\ldots,n,

where the singular values coincide with the magnitudes of the eigenvalues, ordered largest to smallest.

Our shear doesn’t fit this picture. Its two eigenvalues are both 11, but its singular values are

1+2≈2.414,2−1≈0.414. 1+\sqrt{2}\approx2.414,\qquad \sqrt{2}-1\approx0.414.

Some directions are amplified while others are contracted. And the eigenvalues alone conceal this stretching. For a nonnormal matrix, they don’t necessarily determine the singular values. This becomes noticeably important when we apply the transformation repeatedly.

What happens on the next application?

Applying the transformation twice gives

A2=UPUP. A^2=UPUP.

After the first application, a vector has been turned. The next stretch sees it in that new direction. When the factors commute, their effects can be collected:

Ak=UkPk,k=0,1,2,…. A^k=U^kP^k, \qquad k=0,1,2,\ldots.

Otherwise, this rearrangement isn’t generally valid. Once stretching depends on direction, the direction presented to the next stretch matters.

Polar decomposition makes that apparent. Normality identifies the case in which the orthogonal motion preserves the stretching pattern, so their order can be ignored. In another article I want to explore what happens under repeated application when that compatibility fails.

Appendix: Recovering the factors

For the figures, I computed UU and PP using scipy.linalg.polar. But it’s useful to know why the factors exist and are unique when AA is invertible. For nonsingular AA, let B=ATAB=A^{\mathsf T}A. It is positive definite, since for nonzero xx,

xTBx=∥Ax∥22>0. x^{\mathsf T}Bx=\lVert Ax\rVert_2^2>0.

Write its orthogonal spectral decomposition as

B=Vdiag⁡(λ1,…,λn)VT,λi>0. B=V\operatorname{diag}(\lambda_1,\ldots,\lambda_n)V^{\mathsf T}, \qquad \lambda_i>0.

Then

P=B1/2=Vdiag⁡(λ1,…,λn)VT,U=AP−1. P=B^{1/2} =V\operatorname{diag}(\sqrt{\lambda_1},\ldots,\sqrt{\lambda_n})V^{\mathsf T}, \qquad U=AP^{-1}.

These satisfy UP=AUP=A and

UTU=P−1BP−1=I. U^{\mathsf T}U=P^{-1}BP^{-1}=I.

Now we can show they are unique. Any SPD square root RR of BB commutes with B=R2B=R^2, and therefore preserves each eigenspace of BB. On the eigenspace for λ\lambda, λ\sqrt{\lambda} is the only positive eigenvalue of the symmetric matrix RR, so it acts as λI\sqrt{\lambda}I on that space. Thus R=PR=P. This fixes PP, and then U=AP−1U=AP^{-1} is fixed as well.

For numerical computation, an SVD avoids explicitly forming ATAA^{\mathsf T}A or a matrix inverse. If A=WΣVTA=W\Sigma V^{\mathsf T}, then

U=WVT,P=VΣVT. U=WV^{\mathsf T}, \qquad P=V\Sigma V^{\mathsf T}.

The columns of VV are input singular directions; UU carries them to the output singular directions in WW. In the code for generating the figures in this article, I used scipy.linalg.polar to compute the right polar factors.

For a singular square matrix, the stretch is positive semidefinite and remains unique, but the orthogonal factor doesn’t have to be: a collapsed direction can’t determine its subsequent motion. For example, both II and diag⁡(1,−1)\operatorname{diag}(1,-1) can serve as the orthogonal factor of diag⁡(1,0)\operatorname{diag}(1,0).

Notes

  1. Trefethen & Bau continue providing excellent geometric insight throughout the rest of the text as well. I highly recommend it for building intuition with linear algebra even if numerical computing isn’t your goal.

    ↩
  2. Even though AA is real, basis might need complex vectors. A quarter-turn in the plane doesn’t have real eigendirections. Over C\mathbb C, its eigenvalues are ii and −i-i.

    ↩

References

  1. Lloyd N. Trefethen, David Bau III. Numerical Linear Algebra. Society for Industrial and Applied Mathematics (SIAM), 1997.↩1 ↩2
  2. Nick Higham. What Is the Polar Decomposition?. 2020. https://nhigham.com/2020/07/28/what-is-the-polar-decomposition/.↩
  3. Nicholas J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics (SIAM), 2008. doi:10.1137/1.9780898717778.↩
  4. Morris Newman. Two Classical Theorems on Commuting Matrices. Journal of Research of the National Bureau of Standards, Section B: Mathematics and Mathematical Physics, 71B(2–3), pp. 69-71, 1967. https://nvlpubs.nist.gov/nistpubs/jres/71b/jresv71bn2-3p69_a1b.pdf.↩