深度学习-一个简单的深度学习推导

前言

本章主要推导一个简单的两层神经网络。
其中公式入口【入口】

---

1.sigmod函数

激活函数我们选择sigmod，其如下：
$$f(x)=\frac{1}{1+e^{-x}}$$
其图形为：

可以用python表示：

def sigmoid(x):
	return 1.0/(1.0+np.exp(-x))
1
2

2.sigmoid求导

先看一个复合函数求导：
$$如果y(u)=f(u),u(x)=g(x),那么\frac{dy}{dx}=\frac{dy}{du}\ast \frac{du}{dx}$$
那么对于sigmoid函数求导：
$$\begin{gathered} f(x)=\frac{1}{1+e^{-x}}, \\ 那么假设g(x)=1+e^{-x}, \\ f(x)=\frac{1}{g(x)} \\ f(x{)}^{‘}=\frac{-1}{g(x{)}^2}\ast (-e^{-x})=\frac{e^{-x}}{(1+e^{-x}{)}^2}=f(x)\ast (1-f(x)) \end{gathered}$$
如果用python表达：

def sigmoid_prime(x):
	"""sigmoid 函数的导数"""
	return sigmoid(x)*(1-sigmoid(x))
1
2
3

3.损失函数loss

$$Loss=\frac{1}{2}\ast {(\breve{y}-y)}^2$$
它的导数，
$$Los{s}^{‘}=\breve{y}-y$$

4.神经网络

1.神经网络结构

本次我们采用如下神经网络：

2.公式表示-正向传播

$$\begin{gathered} w_{13}\ast x_1+w_{23}\ast x_2+b_1={\sigma }_3,那么\breve{y_3}=sigmoid({\sigma }_3) \\ {w}_{14}\ast x_1+w_{24}\ast x_2+b_2={\sigma }_4,那么\breve{y_4}=sigmoid({\sigma }_4) \\ {w}_{15}\ast x_1+w_{25}\ast x_2+b_3={\sigma }_5,那么\breve{y_5}=sigmoid({\sigma }_5) \\ 同理可得， \\ {w}_{36}\ast \breve{y_3}+w_{46}\ast \breve{y_4}+w_{56}\ast \breve{y_5}+b_4={\sigma }_6,那么\breve{y_6}=sigmoid({\sigma }_6) \end{gathered}$$
上面的公式我们用矩阵表示：
$$\begin{gathered} \left[\begin{array}{cc}{x}_1& x_2\end{array}\right]\cdot \left[\begin{array}{ccc}{w}_{13}& w_{14}& w_{15}\\ w_{23}& w_{24}& w_{25}\end{array}\right]+\left[\begin{array}{c}{b}_1\\ b_2\\ b_3\end{array}\right]=\left[\begin{array}{c}{w}_{13}\ast x_1+w_{23}\ast x_2+b_1\\ w_{14}\ast x_1+w_{24}\ast x_2+b_2\\ w_{15}\ast x_1+w_{25}\ast x_2+b_3\end{array}\right]=\left[\begin{array}{c}{\sigma }_3\\ {\sigma }_4\\ {\sigma }_5\end{array}\right] \\ 代入激活函数， \\ \left[\begin{array}{c}sigmoid({\sigma }_3)\\ sigmoid({\sigma }_4)\\ sigmoid({\sigma }_5)\end{array}\right]=\left[\begin{array}{c}\breve{y_3}\\ \breve{y_4}\\ \breve{y_5}\end{array}\right] \\ \left[\begin{array}{ccc}& & \\ \breve{y_3}& \breve{y_4}& \breve{y_5}\end{array}\right]\cdot \left[\begin{array}{c}{w}_{36}\\ w_{46}\\ w_{56}\end{array}\right]+\left[\begin{array}{c}{b}_4\end{array}\right]=\left[\begin{array}{c}{w}_{36}\ast \breve{y_3}+w_{46}\ast \breve{y_4}+w_{56}\ast \breve{y_5}+b_4\end{array}\right]={\sigma }_6 \\ , \\ sigmoid({\sigma }_6)={\breve{y}}_6 \end{gathered}$$

3.梯度计算

1.Loss 函数

$$Loss=\frac{1}{2}\ast {({\breve{y}}_6-y_6)}^2$$

2.梯度

1.反向传播第2-3层

$$\left[\begin{array}{c}\frac{\partial l}{\partial w_{36}}\\ \\ \frac{\partial l}{\partial w_{46}}\\ \\ \frac{\partial l}{\partial w_{56}}\end{array}\right]=\left[\begin{array}{c}\frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial w_{36}}\\ \\ \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial w_{46}}\\ \\ \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial w_{56}}\end{array}\right]=\left[\begin{array}{c}({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast {\breve{y}}_3\\ \\ ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast {\breve{y}}_4\\ \\ ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast {\breve{y}}_5\end{array}\right]$$

上面的式子中$S(x)=\frac{1}{1+e^{-x}}$，其中${\sigma }_6$通过正向传播可以计算出来，具体细节看2式。

根据公式2，我们已经知道${\breve{y}}_6$和${\breve{y}}_3$的值，所以上面的权重偏导数就能计算出来了。
下面求bias的偏导数，$\frac{\partial l}{\partial b_4}$.
$$\frac{\partial l}{\partial b_4}=\frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial b_4}=({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))$$

2.反向传播第1-2层

权重

$$\begin{gathered} \left[\begin{array}{cc}\frac{\partial l}{\partial w_{13}}& \frac{\partial l}{\partial w_{23}}\\ & \\ \frac{\partial l}{\partial w_{14}}& \frac{\partial l}{\partial w_{24}}\\ & \\ \frac{\partial l}{\partial w_{15}}& \frac{\partial l}{\partial w_{25}}\end{array}\right]=\left[\begin{array}{cc}\frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_3}\ast \frac{\partial {\breve{y}}_3}{\partial {\sigma }_3}\ast \frac{\partial {\sigma }_3}{\partial w_{13}}& \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_3}\ast \frac{\partial {\breve{y}}_3}{\partial {\sigma }_3}\ast \frac{\partial {\sigma }_3}{\partial w_{23}}\\ & \\ \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_4}\ast \frac{\partial {\breve{y}}_4}{\partial {\sigma }_4}\ast \frac{\partial {\sigma }_4}{\partial w_{14}}& \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_4}\ast \frac{\partial {\breve{y}}_4}{\partial {\sigma }_4}\ast \frac{\partial {\sigma }_4}{\partial w_{24}}\\ & \\ \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_5}\ast \frac{\partial {\breve{y}}_5}{\partial {\sigma }_5}\ast \frac{\partial {\sigma }_5}{\partial w_{15}}& \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_5}\ast \frac{\partial {\breve{y}}_5}{\partial {\sigma }_5}\ast \frac{\partial {\sigma }_5}{\partial w_{25}}\end{array}\right]= \\ . \\ . \\ \left[\begin{array}{cc}({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{36}\ast S({\sigma }_3)\ast (1-S({\sigma }_3))\ast x_1& ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{36}\ast S({\sigma }_3)\ast (1-S({\sigma }_3))\ast x_2\\ & \\ ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{46}\ast S({\sigma }_4)\ast (1-S({\sigma }_4))\ast x_1& ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{46}\ast S({\sigma }_4)\ast (1-S({\sigma }_4))\ast x_2\\ & \\ ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{56}\ast S({\sigma }_5)\ast (1-S({\sigma }_5))\ast x_1& ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{56}\ast S({\sigma }_5)\ast (1-S({\sigma }_5))\ast x_2\end{array}\right] \end{gathered}$$
偏置
$$\begin{gathered} \left[\begin{array}{c}\frac{\partial l}{\partial b_1}\\ \\ \frac{\partial l}{\partial b_2}\\ \\ \frac{\partial l}{\partial b_3}\end{array}\right]=\left[\begin{array}{c}\frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_3}\ast \frac{\partial {\breve{y}}_3}{\partial {\sigma }_3}\ast \frac{\partial {\sigma }_3}{\partial b_1}\\ \\ \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_4}\ast \frac{\partial {\breve{y}}_4}{\partial {\sigma }_4}\ast \frac{\partial {\sigma }_4}{\partial b_2}\\ \\ \frac{\partial l}{\partial {\breve{y}}_6}\ast \frac{\partial {\breve{y}}_6}{\partial {\sigma }_6}\ast \frac{\partial {\sigma }_6}{\partial {\breve{y}}_5}\ast \frac{\partial {\breve{y}}_5}{\partial {\sigma }_5}\ast \frac{\partial {\sigma }_5}{\partial b_3}\end{array}\right]= \\ . \\ \left[\begin{array}{c}({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{36}\ast S({\sigma }_3)\ast (1-S({\sigma }_3))\\ \\ ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{46}\ast S({\sigma }_4)\ast (1-S({\sigma }_4))\\ \\ ({\breve{y}}_6-y_6)\ast S({\sigma }_6)\ast (1-S({\sigma }_6))\ast w_{56}\ast S({\sigma }_5)\ast (1-S({\sigma }_5))\end{array}\right] \end{gathered}$$

综上所述，通过反向传播，就可以计算出偏导数了。

3.python代码

根据上面的分析，下面我们写一下python代码，代码就很简单了

import numpy as np
import random
import os


"""核心就是如何布局biases和weights这两个矩阵"""

class Network(object):
    """
    列表sizes包含对应层的神经元数目,如果列表是[2,3,1],那么就是指一个三层神经网络,第一层有
    2个神经元,第二层有3个神经元,第三次有1个神经元.
    """
    def __init__(self, sizes):
        """这里num_layers是3"""
        self.num_layers=len(sizes)
        self.sizes=sizes

        """
        随机初始化偏差,初始化后如下
       [array(
       [[-1.17963885],
       [ 0.41953645],
       [-0.88551629]]), 
       array([[0.20600121]])]
       特别注意这里是3x1的一个矩阵
       """
        self.biases=[np.random.randn(y,1) for y in sizes[1:]]

        """
        随机初始化权重
        [array(
       [[-0.25009885, -0.33699188],
       [-0.53513364, -1.57623694],
       [ 1.89456316,  0.66985265]]), 
       array([[-0.18411963, -0.08143799,  0.53533203]])]
       上面两个矩阵是3x2,1x3
        """
        self.weights=[np.random.randn(y,x) for x,y in zip(sizes[:-1],sizes[1:])]

    def feedforward(self,x):
        """
        输入可以认为是一个2x1的向量,因为列才是向量
        比如下面的点积,[3x2]*[2*1] + [3*1] = [3*1]
        """
        a=np.array(x).reshape(len(x),1)
        for b, w in zip(self.biases,self.weights):
            a=sigmoid(np.dot(w,a)+b)
        return a


    def SGD(self,training_data,epochs,mini_batch_size,eta,test_data=None):
        """
        使用小批量随机梯度下降算法训练神经网络,使用training_data是由训练输入和目标输出的元组(x,y)
        组成。
        """
        if(test_data):
            n_test=len(test_data)
        n=len(training_data)
        for j in range(epochs):
            random.shuffle(training_data)
            mini_batchs=[
                training_data[k:k+mini_batch_size]
                for k in range(0,n,mini_batch_size)
            ]

            for mini_batch in mini_batchs:
                self.update_mini_batch(mini_batch,eta)
            
            if test_data:
                print("Epoch {0}:{1}/{2}".format(j,self.evaluate(test_data),n_test))
            else:
                print("Epoch {0} complete.".format(j))

    
    def update_mini_batch(self,mini_batch,eta):
        """
        使用小批量应用梯度下降算法和反向传播算法来更新神经网络的权重和偏置。
        mini_batch是又若干元组组成的(x,y)组成的列表,eta为学习率。
        其中x为batch * 2 * 1
        """
        nabla_b=[np.zeros(b.shape) for b in self.biases]
        nablea_w=[np.zeros(w.shape) for w in self.weights]

        for x,y in mini_batch:
            """计算梯度"""
            delta_nabla_b,delta_nable_w=self.backprob(x,y)
            nabla_b=[nb+dnb for nb,dnb in zip(nabla_b,delta_nabla_b)]
            nablea_w=[nw+dnw for nw,dnw in zip(nablea_w,nablea_w)]
        
        self.weights=[w-(eta/len(mini_batch)) * nw for w,nw in zip(self.weights,nablea_w)]
        self.biases=[b-(eta/len(mini_batch)) * nb for b,nb in zip(self.biases,nabla_b)]
      
    def backprob(self,a,b):
        nabla_b=[np.zeros(b.shape) for b in self.biases]
        nabla_w=[np.zeros(w.shape) for w in self.weights]
        x=np.array(a).reshape(len(a),1)
        y=np.array(b).reshape(len(b),1)
        activation=x
        activations=[x]
        zs=[]

        """
        正向传播
        biases 是[3x1,1x1]
        weights是[3x2,1x3]
        第1-2层的计算
        [3x2] * [2*1] + [3x1] = [3x1]
        第2-3层的计算
        [1x3] * [3x1] + [1x1] = [1x1] 
        """
        for b,w in zip(self.biases,self.weights):
            z=np.dot(w,activation) + b
            """未激活"""
            zs.append(z)
            """激活函数"""
            activation=sigmoid(z)
            activations.append(activation)

        """反向传播,计算最后2层的梯度"""
        delta=self.cost_derivative(activations[-1],y) * sigmoid_prime(zs[-1])
        nabla_b[-1]=delta
        nabla_w[-1]=np.dot(delta,activations[-2].transpose())

        """反向传播,计算其余层梯度"""
        for l in range(2,self.num_layers):
            z=zs[-l]
            sp=sigmoid_prime(z)
            delta=np.dot(self.weights[-l+1].transpose(),delta) * sp
            nabla_b[-l] =delta
            nabla_w[-l] = np.dot(delta,activations[-l-1].transpose())
        return (nabla_b,nabla_w)


    def evaluate(self,test_data):
        """argmax返回的是a中元素最大值所对应的索引值"""
        # test_results=[(np.argmax(self.feedforward(x),y)) for x,y in test_data] 
        test_results=[(self.feedforward(x),y) for x,y in test_data] 
        return sum(int(compare_float(x,y,0.001)) for x,y in test_results)

    def cost_derivative(self,output_activations,y):
        """loss函数的导数 loss=1/2 * (y^ - y)^2"""
        return (output_activations)

def compare_float(a, b, precision):
    if abs(a - b) <= precision:
        return 1
    return 0
        
def sigmoid(x):
    return 1.0/(1.0+np.exp(-x))

"""sigmoid的导数"""
def sigmoid_prime(x):
    return sigmoid(x)*(1-sigmoid(x))

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155

4.MNIST 数据集

写好代码后我们用测试集测试一下
链接: https://pan.baidu.com/s/1gSeRPwDODK4IeZLVsmPBfQ?pwd=6zcp
提取码: 6zcp

import MNIST.mnist as mnist

if __name__=="__main__":
    dataset=mnist.load_mnist()
    training_data=dataset[0][0]
    training_label=dataset[0][1]
    test_data=dataset[1][0]
    test_lable=dataset[1][1]

    net = Network([784,30,1])
    td=[(np.array(x.copy()),[np.array(y.copy())]) for (x,y) in zip(training_data,training_label)]
    tt_d=[(np.array(x.copy()),[np.array(y.copy())]) for (x,y) in zip(test_data,test_lable)]
    net.SGD(td,30,10,3.0,tt_d)
1
2
3
4
5
6
7
8
9
10
11
12
13

结果如下，可以看到最后精度稳定在98%，还可以：