分类与回归梯度下降公式推导

1.相关函数公式及求导

• 链式推导法则
  ∂ J ( θ ) ∂ θ = ∂ J ( h ) ∂ h ∗ ∂ h ( z ) ∂ z ∗ ∂ z ( θ ) ∂ θ
  ∂J(θ)∂θ=∂J(h)∂h∗∂h(z)∂z∗∂z(θ)∂θ" role="presentation" style="position: relative;">∂J(θ)∂θ=∂J(h)∂h∗∂h(z)∂z∗∂z(θ)∂θ$$\begin{array}{r}\frac{\mathrm{\partial }{J}_{(\theta )}}{\mathrm{\partial }\theta }=\frac{\mathrm{\partial }{J}_{(h)}}{\mathrm{\partial }h}\ast \frac{\mathrm{\partial }{h}_{(z)}}{\mathrm{\partial }z}\ast \frac{\mathrm{\partial }{z}_{(\theta )}}{\mathrm{\partial }\theta }\end{array}$$
  ∂θ∂J(θ)​​=∂h∂J(h)​​∗∂z∂h(z)​​∗∂θ∂z(θ)​​​

1.1.线性回归公式

• 原函数
  $$z_{(\theta )}={\theta }^{T}x$$
• 求导过程
  ∂ z ( θ ) ∂ θ = ∂ θ T x ∂ θ = x
  ∂z(θ)∂θ=∂θTx∂θ=x" role="presentation" style="position: relative;">∂z(θ)∂θ=∂θTx∂θ=x$$\begin{array}{rl}\frac{\mathrm{\partial }{z}_{(\theta )}}{\mathrm{\partial }{\theta }_{}}& =\frac{\mathrm{\partial }{\theta }^{T}x}{\mathrm{\partial }\theta }\\ & =x\end{array}$$
  ∂θ​∂z(θ)​​​=∂θ∂θTx​=x​

1.2.sigmoid函数

• 原函数
  h ( z ) = 1 1 + e − z
  h(z)=11+e−z" role="presentation" style="position: relative;">h(z)=11+e−z$$\begin{array}{rl}{h}_{(z)}& =\frac{1}{1+e^{-z}}\end{array}$$
  h(z)​​=1+e−z1​​
• 求导过程

∂ h ( z ) ∂ z = ∂ ( 1 1 + e − z ) ∂ z = 0 − ∂ ( 1 + e − z ) ( 1 + e − z ) 2 = e − z ( 1 + e − z ) 2 = 1 + e − z − 1 ( 1 + e − z ) ( 1 + e − z ) = 1 + e − z − 1 ( 1 + e − z ) . 1 ( 1 + e − z ) = [ 1 − 1 ( 1 + e − z ) ] . 1 ( 1 + e − z ) = ( 1 − h ( z ) ) ∗ h ( z )

∂z∂h(z)​​​=∂z∂(1+e−z1​)​=(1+e−z)20−∂(1+e−z)​=(1+e−z)2e−z​=(1+e−z)(1+e−z)1+e−z−1​=(1+e−z)1+e−z−1​.(1+e−z)1​=[1−(1+e−z)1​].(1+e−z)1​=(1−h(z)​)∗h(z)​​

2. 逻辑回归

2.1损失函数公式【交叉熵公式】

• 原函数
  L o s s ( w ) = − 1 m [ ∑ i = 1 m ( y i ∗ l o g h ( z ) + ( 1 − y i ) ∗ l o g ( 1 − h ( z ) ) ]
  Loss(w)=−1m[∑i=1m(yi∗logh(z)+(1−yi)∗log(1−h(z))]" role="presentation" style="position: relative;">Loss(w)=−1m[∑i=1m(yi∗logh(z)+(1−yi)∗log(1−h(z))]$$\begin{array}{rl}Los{s}_{(w)}& =-\frac{1}{m}[\sum _{i=1}^m(y^i\ast lo{g}^{h(z)}+(1-y^i)\ast lo{g}^{(1-h(z)})]\end{array}$$
  Loss(w)​​=−m1​[i=1∑m​(yi∗logh(z)+(1−yi)∗log(1−h(z))]​
• 求偏导过程
  ∂ L o s s ( h ) ∂ h = − ∂ 1 m [ ∑ i = 1 m ( y i ∗ l o g h ( z ) + ( 1 − y i ) ∗ l o g ( 1 − h ( z ) ) ] ∂ h = − 1 m [ ∑ i = 1 m ( y i ∗ 1 h ( z ) + ( 1 − y i ) ∗ 1 1 − h ( z ) ∗ ( − 1 ) ) ] = − 1 m [ ∑ i = 1 m ( y i h ( z ) + ( 1 − y i ) 1 − h ( z ) ∗ ( − 1 ) ) ] = − 1 m [ ∑ i = 1 m ( y i ∗ ( 1 − h ( z ) ) + ( y i − 1 ) ∗ h ( z ) h ( z ) ∗ ( 1 − h ( z ) ) ) ] = − 1 m [ ∑ i = 1 m ( y i − h ( z ) h ( z ) ∗ ( 1 − h ( z ) ) ) ] = 1 m [ ∑ i = 1 m ( h ( z ) − y i h ( z ) ∗ ( 1 − h ( z ) ) ) ]
  ∂Loss(h)∂h=−∂1m[∑i=1m(yi∗logh(z)+(1−yi)∗log(1−h(z))]∂h=−1m[∑i=1m(yi∗1h(z)+(1−yi)∗11−h(z)∗(−1))]=−1m[∑i=1m(yih(z)+(1−yi)1−h(z)∗(−1))]=−1m[∑i=1m(yi∗(1−h(z))+(yi−1)∗h(z)h(z)∗(1−h(z)))]=−1m[∑i=1m(yi−h(z)h(z)∗(1−h(z)))]=1m[∑i=1m(h(z)−yih(z)∗(1−h(z)))]" role="presentation" style="position: relative;">∂Loss(h)∂h=−∂1m[∑mi=1(yi∗logh(z)+(1−yi)∗log(1−h(z))]∂h=−1m[∑i=1m(yi∗1h(z)+(1−yi)∗11−h(z)∗(−1))]=−1m[∑i=1m(yih(z)+(1−yi)1−h(z)∗(−1))]=−1m[∑i=1m(yi∗(1−h(z))+(yi−1)∗h(z)h(z)∗(1−h(z)))]=−1m[∑i=1m(yi−h(z)h(z)∗(1−h(z)))]=1m[∑i=1m(h(z)−yih(z)∗(1−h(z)))]$$\begin{array}{rl}\frac{\mathrm{\partial }Los{s}_{(h)}}{\mathrm{\partial }h}& =-\frac{\mathrm{\partial }\frac{1}{m}[\sum _{i=1}^m(y^i\ast lo{g}^{h(z)}+(1-y^i)\ast lo{g}^{(1-h(z)})]}{\mathrm{\partial }h}\\ & =-\frac{1}{m}[\sum _{i=1}^m(y^i\ast \frac{1}{h(z)}+(1-y^i)\ast \frac{1}{1-h(z)}\ast (-1))]\\ & =-\frac{1}{m}[\sum _{i=1}^m(\frac{y^i}{h(z)}+\frac{(1-y^i)}{1-h(z)}\ast (-1))]\\ & =-\frac{1}{m}[\sum _{i=1}^m(\frac{y^i\ast (1-h_{(z)})+(y^i-1)\ast h_{(z)}}{h_{(z)}\ast (1-h_{(z)})})]\\ & =-\frac{1}{m}[\sum _{i=1}^m(\frac{y^i-h_{(z)}}{h_{(z)}\ast (1-h_{(z)})})]\\ & =\frac{1}{m}[\sum _{i=1}^m(\frac{h_{(z)}-y^i}{h_{(z)}\ast (1-h_{(z)})})]\end{array}$$
  ∂h∂Loss(h)​​​=−∂h∂m1​[∑i=1m​(yi∗logh(z)+(1−yi)∗log(1−h(z))]​=−m1​[i=1∑m​(yi∗h(z)1​+(1−yi)∗1−h(z)1​∗(−1))]=−m1​[i=1∑m​(h(z)yi​+1−h(z)(1−yi)​∗(−1))]=−m1​[i=1∑m​(h(z)​∗(1−h(z)​)yi∗(1−h(z)​)+(yi−1)∗h(z)​​)]=−m1​[i=1∑m​(h(z)​∗(1−h(z)​)yi−h(z)​​)]=m1​[i=1∑m​(h(z)​∗(1−h(z)​)h(z)​−yi​)]​

2.2 求导

∂ L o s s ( θ ) ∂ θ j = ∂ L o s s ( h ) ∂ h ∗ ∂ h ( z ) ∂ z ∗ ∂ z ( θ ) ∂ θ j = 1 m [ ∑ i = 1 m ( h ( z ) − y i h ( z ) ∗ ( 1 − h ( z ) ) ) ] ∗ ( 1 − h ( z ) ) ∗ h ( z ) ∗ x j i = 1 m ∑ i = 1 m ( h ( z ) − y i ) ∗ x j i

∂θj​∂Loss(θ)​​​=∂h∂Loss(h)​​∗∂z∂h(z)​​∗∂θj​∂z(θ)​​=m1​[i=1∑m​(h(z)​∗(1−h(z)​)h(z)​−yi​)]∗(1−h(z)​)∗h(z)​∗xji​=m1​i=1∑m​(h(z)​−yi)∗xji​​

2.2 逻辑回归梯度下降公式

• 对$\theta$求偏导
  θ j : = θ j − α ∗ ∂ L o s s ( θ ) ∂ θ j : = θ j − α ∗ 1 m ∑ i = 1 m ( h ( z ) − y i ) ∗ x j i
  θj:=θj−α∗∂Loss(θ)∂θj:=θj−α∗1m∑i=1m(h(z)−yi)∗xji" role="presentation" style="position: relative;">θj:=θj−α∗∂Loss(θ)∂θj:=θj−α∗1m∑i=1m(h(z)−yi)∗xij$$\begin{array}{rl}{\theta }_j& :={\theta }_j-\alpha \ast \frac{\mathrm{\partial }Los{s}_{(\theta )}}{\mathrm{\partial }{\theta }_j}\\ & :={\theta }_j-\alpha \ast \frac{1}{m}\sum _{i=1}^m(h_{(z)}-y^i)\ast x_j^i\end{array}$$
  θj​​:=θj​−α∗∂θj​∂Loss(θ)​​:=θj​−α∗m1​i=1∑m​(h(z)​−yi)∗xji​​

3. 线性回归

3.1损失函数公式【MSE公式】

• 原函数
  L o s s ( θ ) = 1 2 ( z ( θ ) − y ) 2
  Loss(θ)=12(z(θ)−y)2" role="presentation" style="position: relative;">Loss(θ)=12(z(θ)−y)2$$\begin{array}{rl}Los{s}_{(\theta )}& =\frac{1}{2}(z_{(\theta )}-y{)}^2\end{array}$$
  Loss(θ)​​=21​(z(θ)​−y)2​
• 求导过程
  ∂ L o s s ( θ ) ∂ θ j = ∂ 1 2 ( z ( θ ) − y i ) 2 ∂ θ j = 1 2 ∗ 2 ∗ ( z ( θ j ) − y i ) ∗ ∂ z ( θ j ) = ( z ( θ j ) − y i ) ∗ x j i
  ∂Loss(θ)∂θj=∂12(z(θ)−yi)2∂θj=12∗2∗(z(θj)−yi)∗∂z(θj)=(z(θj)−yi)∗xji" role="presentation" style="position: relative;">∂Loss(θ)∂θj=∂12(z(θ)−yi)2∂θj=12∗2∗(z(θj)−yi)∗∂z(θj)=(z(θj)−yi)∗xij$$\begin{array}{rl}\frac{\mathrm{\partial }Los{s}_{(\theta )}}{\mathrm{\partial }{\theta }_j}& =\frac{\mathrm{\partial }\frac{1}{2}(z_{(\theta )}-y^i{)}^2}{\mathrm{\partial }{\theta }_j}\\ & =\frac{1}{2}\ast 2\ast (z_{({\theta }_j)}-y^i)\ast \mathrm{\partial }{z}_{({\theta }_j)}\\ & =(z_{({\theta }_j)}-y^i)\ast x_j^i\end{array}$$
  ∂θj​∂Loss(θ)​​​=∂θj​∂21​(z(θ)​−yi)2​=21​∗2∗(z(θj​)​−yi)∗∂z(θj​)​=(z(θj​)​−yi)∗xji​​

3.2 线性回归梯度下降公式

θ j : = θ j − α ∗ ∂ L o s s ( θ ) ∂ θ j : = θ j − α ∗ ( z ( θ j ) − y i ) ∗ x j i

θj​​:=θj​−α∗∂θj​∂Loss(θ)​​:=θj​−α∗(z(θj​)​−yi)∗xji​​