Singular Value Decomposition

序言：對於任意矩陣 A，可以運用 SVD 將其分解成 $USV^T$
其中U是 $AA^T$ 之eigenvectors；V是 $A^TA$ 之eigenvectors

4. For any matrix A, $AA^T$ and $A^TA$ are both symmetric.

sol. $(AA^T)^T=AA^T$; $(A^TA)^T = A^T A$

5. For arbitrary matrices A, B, AB and BA have same nonzero eigenvalues. (if AB, BA both exist)

sol. $AB\vec{u} = \lambda \vec{u} \rightarrow since \lambda, \vec{u} \neq 0, B\vec{u}\neq 0$ and

$BA(B\vec{u})_{(\rightarrow set=\vec{y})} = B(AB\vec{u}) = \lambda B \vec{u} = \lambda \vec{y}$
$$\begin{cases} AB\vec{u} = \lambda\vec{u} \\ BA\vec{y} = \lambda\vec{y} \end{cases}$$

From 4. and 5., we can write $A^TA$ and $AA^T$ as:

 $AA^T = UDU^T$; $A^TA = V \tilde{D}V$ and D, $\tilde{D}$ have the same nonzero eigenvalues.

6. Eigenvalues of $AA^T$ are all $\geq$ 0

sol. Let $\vec{v}$ be an arbitrary vector with shape (A.shape[0], 1).

$\vec{v}^T_{1\times m}A_{m\times n}A^T_{n\times m} \vec{v}_{m\times 1} = \vec{y}^T\vec{y}\geq 0$ (set $y=\vec{v}^TA$)

Now we set $\vec{v} =$ eigenvector of  $AA^T$ $(\vec{u})$

$\vec{u}^TAA^T\vec{u} = \vec{u}^T\lambda\vec{u} = \lambda\vec{u}^T\vec{u}$

Since $\vec{u}^T\vec{u}\geq 0$ and $\vec{u}^TAA^T\vec{u}\geq 0$, $\lambda$ always $\geq 0$ 

And from 6. we can rewrite A as $USV^T$ such that 

$AA^T = USV^TVS^TU^T = UDU^T$ and $A^TA = VS^TU^TUSV^T = V\tilde{D}V^T$

$S_{i,i}$的值 = 

$$\begin{cases} \sqrt{\lambda_i}必存在  \text{ (for nonzero eigenvalue }\lambda_i\text{)}\\ 0 \end{cases}$$