Lecture 30

[[lecture-data]]

2024-11-11

Nearing the end of:

5. Chapter 5

Last time, we saw how condition number determines the amount of noise in a linear system. We also saw a similar thing when we are inverting a matrix with some noise thrown in!

Absolute Vector Norm

The vector norm on $C^{n}$ is called absolute precisely when for all $x \in C^{n}$ it holds that

| | | x | | | = | | x | |

where $| x |$ is taken component wise

Example

All $ℓ_{p}$ norms are absolute since $ℓ_{p} (x) = {(\sum | x |^{p})}^{1 / p}$

(see absolute vector norm)

Monotone Vector Norm

We call a vector norm monotone precisely when for all $x, y \in C^{n}$ ,

| x | \leq | y | ⟹ | | x | | \leq | | y | |

again, where $| x |$ is taken component wise.

Example

All $ℓ_{p}$ norms are also monotone!

(see monotone vector norm)

Theorem

Suppose $| | \cdot | |$ is a norm on $C^{n}$ . The following are equivalent:

$| | \cdot | |$ is monotone
$| | \cdot | |$ is absolute
The matrix norm $| | \cdot | |^{'}$ induced by $| | \cdot | |$ satisfies the following: for all diagonal $D \in M_{n}$

| | D | |^{'} = max {| d_{i i} |}

this is called the "funky diagonal property" 😊

Example

$| | \cdot | |_{1, 1}$ since the matrix 1 norm is max column sum
$| | \cdot | |_{2, 2}$
$| | \cdot | |_{\infty, \infty}$ since matrix infinity norm is max row sum

Proof

(1) ⟹ (2)

Suppose $| | \cdot | |$ is monotone. For all $x \in C^{n}$ , we have that $| x | \leq | | x | | ⟹ | | x | | \leq | x | ⟹ | | | x | | | = | | x | |$

(2) ⟹ (1)

Read the book :)

(1) ⟹ (3)

Suppose $| | \cdot | |$ is monotone. Let $D \in M_{n}$ be diagonal. For any $x \in C^{n} \neq 0$ , $D x = [d_{11} x_{1} d_{22} x_{2} \dots d_{n n} x_{n}]$ . So $| D x | \leq | max_{i} | d_{i i} | x |$ .
So by monotonicity, we have

\begin{aligned} = max_{i} | d_{i i} | \cdot | | x | | \\ ⟹ \frac{| | D x | |}{| | x | |} \leq max_{i} | d_{ᵢ} | \end{aligned}

Equality is attained with $x = e_{k}$ the standard basis vector with $1$ in the $k$ th position, and where $k$ is the index of the largest $| d_{i, i} |$ . Thus we get

$| | D | |^{'} := max \frac{| | D x | |}{| | x | |} = max_{i} | d_{i i} |$

(3) ⟹ (1)

Suppose $| | \cdot | |$ induces $| | \cdot | |^{'}$ with the funky diagonal property. Suppose $x, y \in C^{n}$ such that $| x | \leq y$ . We define the following diagonal matrix $D$ :

for $i = 1, 2, \dots, n$ , let $d_{i i} = {\begin{cases} \frac{x_{i}}{y_{i}} if y_{i} \neq 0 \\ 0 if y_{i} = 0 \end{cases}$
So, $D y = x$ and $| | D | |^{'} = max_{i} | d_{i i} | \leq 1$ . So

\begin{array}{r} | | x | | = | | D y | | \leq (*) | | D | |^{'} | | y | | \leq (* *) | | y | | \end{array}

Where $(*)$ is by definition of the induced matrix norm and $(* *)$ is because $| | D | |^{'} \leq 1$ .

And the result follows!

Thus the result follows :)

(see funky diagonal property)

Berer-Fike Theorem

Let $| | \cdot | |$ be a matrix norm on $M_{n}$ induced by a monotone vector norm. Let $A, Δ A \in M_{n}$ be such that $A$ is diagonalizable as $A = S D S^{- 1}$ .

Then for all $λ \in σ (A + Δ A)$ , there exists a $τ \in σ (A)$ such that $| λ - τ | \leq | | Δ A | | κ_{| | \cdot | |} (S)$

where $κ$ is the condition number

If $A$ is also normal, then $| | λ - τ | | \leq | | Δ A | |_{2, 2}$

Proof

Assume $λ \notin σ (A)$ . (otherwise, the result is trivial).
$λ I - [A + Δ A]$ is singular (since $λ$ is an eigenvalue of $A + Δ A$ ). So
$S^{- 1} [λ I - [A + Δ A]] S$ is also singular. But note that $λ I - D$ is invertible (since $λ \notin σ (A)$ ) and this inverse has diagonal entries $\frac{1}{λ - d_{i i}}$ .

\begin{aligned} S^{- 1} [λ I - [A + Δ A]] S & = λ I - D - S^{- 1} Δ A S \\ ⟹ (λ I - D)^{- 1} [λ I - D - S^{- 1} Δ A S] & = I - [λ I - D]^{- 1} S^{- 1} Δ A S is singular \\ ⟹ 1 & \leq | | (λ I - D)^{- 1} S^{- 1} Δ A S | | (*) \\ ⟹ 1 & \leq | | (λ I - D)^{- 1} | | \cdot | | S^{- 1} Δ A S | | \\ = \frac{1}{| λ - τ |} | | S^{- 1} A S | | (* *) \end{aligned}

$(*)$ follows since matrices with norm less than 1 define an infinite series inverse, but we know that the matrix $I - [λ I - D]^{- 1} S^{- 1} Δ A S$ is singular.
$(* *)$ follows by the funky diagonal property! We take the largest element in our definition of $[λ I - D]^{- 1}$

If $A$ is normal, then $S$ can be chosen to be unitary! So

| | S | |_{2, 2}^{2} = ρ (S^{*} S) = ρ (I) = 1

So ${κ_{| | \cdot | |}}_{2, 2} (S) = 1 \cdot 1 = 1$

(see Berer-Fike Theorem)

Note

If $A, Δ A$ are hermitian, Weyl's Theorem says

\begin{aligned} λ_{1} (Δ A) + λ_{k} (A) & \leq λ_{k} (A + Δ A) \leq λ_{n} (Δ A) + λ_{k} (A) \\ ⟹ λ_{1} (Δ A) & \leq λ_{k} (A + Δ A) - λ_{k} (A) \leq λ_{n} (A) \\ ⟹ | λ_{k} (A + Δ A) - λ_{k} (A) | & \leq ρ (Δ A) \leq | | Δ A | |_{2, 2} \end{aligned}

ie, we can say WHICH eigenvalue each eigenvalue of the perturbed matrix is "close to"