tags math linear algebra

linear algebra

matrix

diagonal 對角 commutative law associative law distributive law

sample

$$A=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]$$

$$B=\left[\begin{array}{cc}1& -2\\ 2& -4\end{array}\right]$$

乘積

$$\begin{gathered} A=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right] \\ B=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right] \\ A\ast B=\left[\begin{array}{cc}a\ast 1+b\ast 3& a\ast 2+b\ast 4\\ c\ast 1+d\ast 3& c\ast 2+d\ast 4\end{array}\right] \\ A=\left[\begin{array}{cc}2& 3\\ 1& -5\end{array}\right]B=\left[\begin{array}{ccc}4& 3& 6\\ 1& -2& 3\end{array}\right] \\ A\ast B=\left[\begin{array}{ccc}2\ast 4+3\ast 1& 2\ast 3+3\ast -2& 2\ast 6+3\ast 3\\ 1\ast 4+-5\ast 1& 1\ast 3+-5\ast -2& 1\ast 6+-5\ast 3\end{array}\right]=\left[\begin{array}{ccc}11& 0& 21\\ -1& 13& -9\end{array}\right] \\ \underset{[n\times m]}{A}\ast \underset{[m\times s]}{B}=\underset{[n\times s]}{C} \end{gathered}$$

反矩陣

$$\begin{gathered} A=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right] \\ a\ast d-b\ast c=0thennotinverable \\ A\ast A^{-1}=I=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right] \\ {A}^{-1}=\frac{1}{\left|\begin{array}{cc}a& b\\ c& d\end{array}\right|}\ast \left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right] \\ \text{high level matrix} \\ \left[\begin{array}{cc}A& I\end{array}\right]=\left[\begin{array}{cc}A\ast A^{-1}& I\ast A^{-1}\end{array}\right]=\left[\begin{array}{cc}I& A^{-1}\end{array}\right] \\ \left[\begin{array}{ccccccc}1& 0& 0& |& 1& 0& 0\\ 1& 1& 0& |& 0& 1& 0\\ 1& 1& 1& |& 0& 0& 1\end{array}\right]=\left[\begin{array}{ccccccc}1& 0& 0& |& 1& 0& 0\\ 0& 1& 0& |& -1& 1& 0\\ 0& 0& 1& |& 0& -1& 1\end{array}\right] \\ \left[\begin{array}{ccccccc}1& 0& 0& |& 1& 0& 0\\ 1& 1& 0& |& 0& 1& 0\\ 1& 1& 1& |& 0& 0& 1\end{array}\right]=\left[\begin{array}{ccccccc}1& 0& 0& |& 1& 0& 0\\ 0& 1& 0& |& -1& 1& 0\\ 0& 0& 1& |& 0& -1& 1\end{array}\right] \end{gathered}$$

identity matrix

$$I=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$$

transpose matrix

$$\begin{gathered} A=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right] \\ {A}^T=\left[\begin{array}{cc}1& 3\\ 2& 4\end{array}\right](A^{-1}{)}^T=(A^T{)}^{-1} \end{gathered}$$

inner change (change row)

$$\left[\begin{array}{cc}1& 2\\ 3& 4\\ 5& 6\end{array}\right]==\left[\begin{array}{cc}5& 6\\ 3& 4\\ 1& 2\end{array}\right]$$

math to matrix

$$\{\begin{array}{llccc}1\ast x_1& +& 2\ast x_2& +& 3\ast x_3=5\\ 2\ast x_1& +& 3\ast x_2& +& 4\ast x_3=7\end{array}=\left[\begin{array}{cccc}1& 2& 3& 5\\ 2& 3& 4& 7\end{array}\right]$$

reduced echelon form(gaussian elimination)

$$\begin{gathered} \left[\begin{array}{cccccc}0& 3& -6& 6& 4& -5\\ 3& -7& 8& -5& 8& 9\\ 3& -9& 12& -9& 6& 15\end{array}\right]==\left[\begin{array}{cccccc}3& -9& 12& -9& 6& 15\\ 3& -7& 8& -5& 8& 9\\ 0& 3& -6& 6& 4& -5\end{array}\right]=\left[\begin{array}{cccccc}3& -9& 12& -9& 6& 15\\ 0& 2& -4& 4& 2& -6\\ 0& 3& -6& 6& 4& -5\end{array}\right] \\ =\left[\begin{array}{cccccc}3& -9& 12& -9& 6& 15\\ 0& 2& -4& 4& 2& -6\\ 0& 3& -6& 6& 4& -5\end{array}\right]=\left[\begin{array}{cccccc}3& -9& 12& -9& 6& 15\\ 0& 2& -4& 4& 2& -6\\ 0& 0& 0& 0& 1& 4\end{array}\right] \\ =\left[\begin{array}{cccccc}3& -9& 12& -9& 6& 15\\ 0& 1& -2& 2& 1& -3\\ 0& 0& 0& 0& 1& 4\end{array}\right]=\left[\begin{array}{cccccc}1& -3& 4& -3& 2& 5\\ 0& 1& -2& 2& 1& -3\\ 0& 0& 0& 0& 1& 4\end{array}\right] \\ =\left[\begin{array}{cccccc}1& -3& 4& -3& 0& -3\\ 0& 1& -2& 2& 0& -7\\ 0& 0& 0& 0& 1& 4\end{array}\right]=\left[\begin{array}{cccccc}1& 0& -2& -3& 0& -24\\ 0& 1& -2& 2& 0& -7\\ 0& 0& 0& 0& 1& 4\end{array}\right] \\ =\{\begin{array}{l}1\ast x_1-2\ast x_3+-3\ast x_4=24\\ 1\ast x_2-2\ast x_3+2\ast x_4=-7\\ x_5=4\end{array} \end{gathered}$$

vector equation and matrix equation

$$\begin{gathered} \{\begin{array}{llcccccc}& x_1& +& 2x_2& -& x_3& =& 3\\ -& 2x_1& -& 2x_2& +& 4x_3& =& 0\\ & & & 2x_2& +& 2x_3& =& 6\end{array}== \\ {x}_1\left[\begin{array}{c}1\\ -2\\ 0\end{array}\right]+x_2\left[\begin{array}{c}2\\ -2\\ 2\end{array}\right]+x_3\left[\begin{array}{c}-1\\ 4\\ 2\end{array}\right]=\left[\begin{array}{c}3\\ 0\\ 6\end{array}\right] \\ A\left[\begin{array}{ccc}1& 2& -1\\ -2& -2& 4\\ 0& 2& 2\end{array}\right]\ast x\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=\left[\begin{array}{c}3\\ 0\\ 6\end{array}\right] \\ Ax=b \\ {A}_{matrix}\ast x_{vector}=b_{vector} \end{gathered}$$

span

$$\begin{gathered} v_1=\left[\begin{array}{c}1\\ -2\\ 0\end{array}\right]v_2=\left[\begin{array}{c}2\\ -2\\ 2\end{array}\right]v_3=\left[\begin{array}{c}-1\\ 4\\ 2\end{array}\right] \\ set\{\begin{array}{cccc}& v_1,& v_2,& v_3\end{array}\}span\text{: mean all the combination vector of can combine by }{v}_1、v_2、v_3. \\ set\{\begin{array}{cccc}& v_1,& v_2,& v_3\end{array}\}span{R}^3\text{: mean all the vector can be combine by }{v}_1、v_2、v_3. \\ \text{b are the all the vector} \\ \text{x are the all the possible vector} \\ example: \\ if\{\begin{array}{cccc}& v_1,& v_2,& v_3\end{array}\}span{R}^{3}then \\ A\left[\begin{array}{ccc}1& 2& -1\\ -2& -2& 4\\ 0& 2& 2\end{array}\right]\ast x\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=b\left[\begin{array}{c}{b}_1\\ b_2\\ b_3\end{array}\right] \\ and\left[\begin{array}{cccc}1& 2& -1& 0\\ -2& -2& 4& 0\\ 0& 2& 2& 0\end{array}\right]\text{have pivot position in every row} \\ but\left[\begin{array}{cccc}1& 2& -1& b_1\\ -2& -2& 4& b_2\\ 0& 2& 2& b_3\end{array}\right]=\left[\begin{array}{cccc}1& 0& -3& b_1\\ 0& 1& 1& b_2\\ 0& 0& 0& b_3\end{array}\right]\text{did not have pivot point in row3} \\ So\{\begin{array}{cccc}& v_1,& v_2,& v_3\end{array}\}notspan{R}^3 \end{gathered}$$

linear dependence

$$if\{\begin{array}{ccc}{v}_1,& v_2,& v_3\end{array}\}\text{ linear dependence then at least one of vector is linear combination of another.}$$

linear depandence vs. span:

$$\begin{gathered} A_{matrix}\ast x_{vector}=b_{vector} \\ A\left[\begin{array}{ccc}0& 2& -2\\ 1& 1& 1\\ -2& 0& -4\\ 1& 2& 0\end{array}\right]x\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=b\left[\begin{array}{c}{b}_1\\ b_2\\ b_3\\ b_4\end{array}\right] \end{gathered}$$

echelon form

$$\begin{gathered} \left[\begin{array}{cccc}0& 2& -2& b_1\\ 1& 1& 1& b_2\\ -2& 0& -4& b_3\\ 1& 2& 0& b_4\end{array}\right]=\left[\begin{array}{cccc}1& 1& 1& b_2\\ 0& 2& -2& b_1\\ -2& 0& -4& b_3\\ 1& 2& 0& b_4\end{array}\right]=\left[\begin{array}{cccc}1& 1& 1& b_2\\ 0& 2& -2& b_1\\ 0& 2& -2& 2b_2+b_3\\ 0& 1& -1& -b_2+b_4\end{array}\right]= \\ \left[\begin{array}{cccc}1& 0& 2& b_2+\frac{-b_1}{2}\\ 0& 2& -2& b_1\\ 0& 0& 0& 2b_2+b_3-b_1\\ 0& 0& 0& -b_2+b_4+\frac{-b_1}{2}\end{array}\right] \end{gathered}$$

vector equation

$$x_1\left[\begin{array}{c}1\\ 0\\ 0\\ 0\end{array}\right]+x_2\left[\begin{array}{c}0\\ 2\\ 0\\ 0\end{array}\right]+x_3\left[\begin{array}{c}2\\ -2\\ 0\\ 0\end{array}\right]=b\left[\begin{array}{c}{b}_2+\frac{-b_1}{2}\\ b_1\\ 2b_2+b_3-b_1\\ -b_2+b_4+\frac{-b_1}{2}\end{array}\right]$$

span R^4

NO,did not have pivot point in row3 and row4

linear independence?

NO,did not have pivot point in column 3

transform

one-to-one

$$\begin{gathered} T(x)=b \\ \text{for all b only have one x that T(x)=b} \end{gathered}$$

onto

$$\begin{gathered} T(x)=b \\ \text{for all b have at least one x that T(x)=b} \end{gathered}$$

linear transformation

$$\begin{gathered} T(x)=b \\ \text{for all b have at least one x that T(x)=b} \\ T(a+b)=T(a)+T(b) \end{gathered}$$

vectorspace

$$\begin{gathered} \text{If A is vector space then} \\ \{\begin{array}{l}\text{the zero vector or zero matrix must be in side of A}\\ \text{if u and v is inside A then (u+v) should be inside A}\\ \text{if v is inside A then n}\in \text{R and n*v should be also inside A}\end{array} \end{gathered}$$

child space

$$x=\left[\begin{array}{ccc}2& 2& 4\\ -2& 2& 4\\ 0& 4& 8\end{array}\right]\stackrel{\text{rref}}{\to }\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 2\\ 0& 0& 0\end{array}\right]$$

$$\begin{gathered} Col(A)=Row(A^T) \\ Col(A^T)=Row(A) \end{gathered}$$

null space

$$\begin{gathered} \text{null space : all the solution make }Av=0. \\ \text{orth to columns space} \\ \{\begin{array}{l}\left[\begin{array}{c}0\\ -2\\ 1\end{array}\right]\end{array} \end{gathered}$$

columns space

$$\begin{gathered} \text{columns space :all the combination of column vector } \\ Ax=b \\ \text{The pivot columns of the origin columns.} \\ \{\begin{array}{l}\left[\begin{array}{c}2\\ -2\\ 0\end{array}\right],\left[\begin{array}{c}2\\ 2\\ 4\end{array}\right]\end{array} \end{gathered}$$

rows space

$$\begin{gathered} \text{rows space :the space that is orth to columns space} \\ \text{the pivot row} \\ \{\begin{array}{l}\left[\begin{array}{c}2\\ 2\\ 4\end{array}\right],\left[\begin{array}{c}-2\\ 2\\ 4\end{array}\right]\end{array}or\{\begin{array}{l}\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\\ 2\end{array}\right]\end{array} \end{gathered}$$

rank nullity

$$\begin{gathered} A\in R^{m\times n} \\ rank(A)+nullity(A)=n \\ rank(A^T)+nullity(A^T)=m \end{gathered}$$

rank

the number of non-zero pivot column

$$\begin{gathered} rank(A)=dim(Col(A)) \\ rank(A)=rank(A^T) \end{gathered}$$

nullity/kernel

the number of zero pivot column

$$nullity(A)=dim(Nul(A))$$

Eigenvector and Eigenvalue

$$\begin{gathered} \lambda =\text{eigenvalue (nature number could be zero)} \\ v=\text{eigenvector(not zero vector)} \\ Av=\lambda v \\ Av=\lambda Iv \\ Av-\lambda Iv=0 \\ |A-\lambda I|=0 \end{gathered}$$

eigen value

$$\begin{gathered} A=\left[\begin{array}{cc}6& 9\\ 2& 3\end{array}\right] \\ det\left[\begin{array}{cc}6& 9\\ 2& 3\end{array}\right]=6\ast 3-9\ast 2=0 \\ |A-\lambda I|=0 \\ det\left[\begin{array}{cc}6-\lambda & 9\\ 2& 3-\lambda \end{array}\right]=(6-\lambda )(3-\lambda )-18=0 \\ \lambda =0or9 \end{gathered}$$

eigen vector

$$\begin{gathered} \lambda =0or9 \\ Av=\lambda v \\ \left[\begin{array}{cc}6& 9\\ 2& 3\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=9\left[\begin{array}{c}x\\ y\end{array}\right] \\ \{\begin{array}{l}6x+9y=9x\\ 2x+3y=9y\end{array} \\ 6x=18y\to x=-3y \\ Av=\lambda v \\ \left[\begin{array}{cc}6& 9\\ 2& 3\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=0\left[\begin{array}{c}x\\ y\end{array}\right] \\ x\in R \\ y\in R \\ v=\left[\begin{array}{c}3\\ 1\end{array}\right]or\left[\begin{array}{c}a\\ b\end{array}\right] \end{gathered}$$

diagonalization

$$\begin{gathered} \text{Find P and D that} \\ A=PD{P}^{-1} \\ \text{So }{A}^2=(PD{P}^{-1})(PD{P}^{-1}) \\ {A}^2=P{D}^2{P}^{-1} \\ {A}^n=P{D}^n{P}^{-1} \\ \underset{n\times m}{P}=\left[\begin{array}{ccccc}V{1}_{EigenVector}& V{2}_{EigenVector}& & V{3}_{EigenVector}& \dots \end{array}\right] \\ \underset{n\times m}{D}=\left[\begin{array}{cccc}\lambda 1_{EigenValue}& 0& 0& \dots \\ 0& \lambda 2_{EigenValue}& 0& \dots \\ 0& 0& \lambda 3_{EigenValue}& \dots \\ ⋮& ⋮& ⋮& \ddots \end{array}\right] \end{gathered}$$

orthogonal

$$\begin{gathered} \text{If }a\cdot b=0\text{ then }a\perp b\text{ (orthogonal)} \\ \left[\begin{array}{c}1\\ 2\end{array}\right]\cdot \left[\begin{array}{c}-8\\ 4\end{array}\right]=1\times -8+2\times 4=0 \\ \left[\begin{array}{c}1\\ 2\end{array}\right]\perp \left[\begin{array}{c}-8\\ 4\end{array}\right] \end{gathered}$$

dot

$$\begin{gathered} \underset{n\ast m}{A}\cdot \underset{n\ast m}{B} \\ \text{columns space vector: }A\cdot B=A^T\times B \\ \text{rows space vector: }A\cdot B=A\times B^T \end{gathered}$$

|A|

vector

$$\begin{gathered} |A|=\sqrt{\sum _{i=1}^n\sum _{j=1}^m{(A_{ij})}^2} \\ |A-B|=|A|-2A\cdot B+|B| \end{gathered}$$

matrix

link

$$\begin{gathered} |A-B|=\text{B to A distance or Similarity between two matrices} \\ distance=\sum _{i=1}^n\sum _{j=1}^m|A_{ij}-B_{ij}| \\ distance=\sqrt{\sum _{i=1}^n\sum _{j=1}^m{(A_{ij}-B_{ij})}^2} \end{gathered}$$

orthogonal vs orthonormal

orthogonal matrix

$$\text{If column vector are orth to each other(dot product is 0)}$$

orthonormal

$$\begin{gathered} \text{If column vector are orth to each other and the length of column vector is 1 then is orthonormal} \\ \text{then is orthogonal matrix} \\ \left[\begin{array}{c}\frac{1}{\sqrt{2}}\\ 0\\ \frac{1}{\sqrt{2}}\end{array}\right]\perp \left[\begin{array}{c}\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}\\ 0\end{array}\right]\perp \left[\begin{array}{c}0\\ \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}\end{array}\right] \\ \left[\begin{array}{ccc}\frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}& 0\\ 0& \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& 0& \frac{1}{\sqrt{2}}\end{array}\right]\text{is orthogonal matrix} \\ \text{if A is orthogonal matrix then } \\ {A}^T\times A=I \\ {A}^T=A^{-1} \\ det(A{)}^2=1 \end{gathered}$$

find orth basis(gram-schmidt process)

derivative of a Matrix

$$\begin{gathered} A=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right] \\ Ax=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]=\left[\begin{array}{c}1x_1+2x_2\\ 3x_1+4x_2\end{array}\right] \\ {f}_1(x_1,x_2)=1x_1+2x_2 \\ {f}_2(x_1,x_2)=3x_1+4x_2 \\ \frac{d}{dx}Ax=\left[\begin{array}{cc}\frac{d}{d{x}_1}{f}_1& \frac{d}{d{x}_2}{f}_1\\ \frac{d}{d{x}_1}{f}_2& \frac{d}{d{x}_2}{f}_2\end{array}\right]=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]=A \\ \therefore \frac{d}{dx}Ax=A \\ {X}^{T}AX=\left[\begin{array}{cc}{x}_1& x_2\end{array}\right]\left[\begin{array}{c}{a}_{11}{x}_1+a_{12}{x}_2\\ a_{21}{x}_1+a_{22}{x}_2\end{array}\right]=\left[\begin{array}{c}{a}_{11}{x_1}^2+a_{12}{x}_1{x}_2+a_{21}{x}_1{x}_2+a_{22}{x_2}^2\end{array}\right] \\ {f}_3(x_1,x_2)=a_{11}{x_1}^2+a_{12}{x}_1{x}_2+a_{21}{x}_1{x}_2+a_{22}{x_2}^2 \\ \frac{d}{dx}{X}^{T}AX=\left[\begin{array}{cc}\frac{d}{d{x}_1}{f}_3& \frac{d}{d{x}_2}{f}_3\end{array}\right]=\left[\begin{array}{cc}2x_1+a_{12}{x}_2& 2x_2+a_{21}{x}_1\end{array}\right] \\ \because z\ne x \end{gathered}$$

projection

$$\begin{gathered} \hat{y}=\frac{A}{A^{T}A}\frac{A^{T}y}{A^{T}A} \\ \text{define }z=y-\hat{y} \\ so\{\begin{array}{l}y=\hat{y}+z\\ \hat{y}\perp z\\ \hat{y}\in \text{A columns space}\end{array} \end{gathered}$$

curve fitting | regression | normal equation

$$\begin{gathered} (\dim \beta )n=2,\text{(test number)}m=4 \\ {\beta }_0+x{\beta }_1=y,\text{(x,y) } \\ {\beta }_0+2{\beta }_1=1,\text{(2,1) } \\ {\beta }_0+5{\beta }_1=2,\text{(5,2) } \\ {\beta }_0+7{\beta }_1=3,\text{(7,3) } \\ {\beta }_0+8{\beta }_1=3,\text{(8,3) } \\ A\times \beta =y \\ \underset{A[m\times n]}{\left[\begin{array}{cc}1& 2\\ 1& 5\\ 1& 7\\ 1& 8\end{array}\right]}\times \underset{\beta [n\times 1]}{\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\end{array}\right]}=\underset{y[m\times 1]}{\left[\begin{array}{c}1\\ 2\\ 3\\ 3\end{array}\right]} \\ \hat{y}=y\text{ projection on }A \\ z=y-\hat{y} \\ z\in \text{nul }{A}^T \\ \text{distance of y to A columns space is }z(\text{because }z\perp A) \end{gathered}$$

way 1

$$\begin{gathered} \text{Find }\beta let{|A\beta -y|}^2\text{have}\min \\ {|A\beta -y|}^2=(|A\beta |-2A\beta \cdot y+|y|{)}^2 \\ {|A\beta -y|}^2=(|A\beta |-2A\beta \cdot (\hat{y}+z)+|\hat{y}+z|{)}^2 \\ {|A\beta -y|}^2=(|A\beta |-2A\beta \cdot (\hat{y}+z)+|\hat{y}+z|{)}^2 \end{gathered}$$

way 2

$$\begin{gathered} \text{Find }\beta let{|A\beta -y|}^2\text{have}\min \\ \text{if derivative }{f}^{{}^{\prime }}(a)=0thenf(a)\text{ is max or min value } \\ \frac{d}{d\beta }{|y-A\beta |}^2=0 \\ 2|y-A\beta |\times \frac{d}{d\beta }|y-A\beta |=0 \\ 2|y-A\beta |\times \frac{d}{d\beta }(|y|-2y\cdot A\beta +|A\beta |)=0 \\ 2|y-A\beta |\times (A+|A\beta |)=0 \end{gathered}$$

way 3

$$\begin{gathered} \text{Find }\beta let{|A\beta -y|}^2\text{have}\min \\ \begin{array}{rl}|A^{T}y-A^{T}A\beta |& =A^T\times distance\\ |A^T(\hat{y}+z)-A^{T}A\beta |& =A^T\times distance\\ |A^T\hat{y}+A^{T}z-A^{T}A\beta |& =A^T\times distance\\ |A^T\hat{y}-A^{T}A\beta |& =A^T\times distance\end{array} \\ \hat{y}\in \text{A columns space so can find }A\beta =\hat{y} \\ |A^T\hat{y}-A^{T}A\beta |=0 \\ {A}^{T}y=A^{T}A\beta \\ \beta =(A^{T}A{)}^{-1}{A}^{T}y \end{gathered}$$