神经网络和深度学习-梯度下降Gradient Descent（下）

梯度下降gradient descent

我们接着用数学公式来看一下梯度下降

首先是梯度

$\frac{\partial \cos t}{\partial \omega}$

然后我们对权重进行更新

$\omega=\omega-\alpha \frac{\partial \cos t}{\partial \omega}$

解析一下梯度求导公式

\begin{aligned} \frac{\partial cost (ω)}{\partial ω} & = \frac{\partial}{\partial ω} \frac{1}{N} \sum_{n = 1}^{N} {(x_{n} \cdot ω - y_{n})}^{2} \\ = \frac{1}{N} \sum_{n = 1}^{N} \frac{\partial}{\partial ω} {(x_{n} \cdot ω - y_{n})}^{2} \\ = \frac{1}{N} \sum_{n = 1}^{N} 2 \cdot (x_{n} \cdot ω - y_{n}) \frac{\partial (x_{n} \cdot ω - y_{n})}{\partial ω} \\ = \frac{1}{N} \sum_{n = 1}^{N} 2 \cdot x_{n} \cdot (x_{n} \cdot ω - y_{n}) \end{aligned}

\frac{\partial c o s t ( ω )}{\partial ω} = \frac{\partial}{\partial ω} \frac{1}{N} n = 1 \sum N (x_{n} \cdot ω - y_{n})^{2} = \frac{1}{N} n = 1 \sum N \frac{\partial}{\partial ω} (x_{n} \cdot ω - y_{n})^{2} = \frac{1}{N} n = 1 \sum N 2 \cdot (x_{n} \cdot ω - y_{n}) \frac{\partial ( x _{n} \cdot ω - y _{n} )}{\partial ω} = \frac{1}{N} n = 1 \sum N 2 \cdot x_{n} \cdot (x_{n} \cdot ω - y_{n})

我们继续进行更新

$\omega=\omega-\alpha \frac{1}{N} \sum_{n=1}^{N} 2 \cdot x_{n} \cdot\left(x_{n} \cdot \omega-y_{n}\right)$

接下来我们来看一下梯度下降的详细代码实现

定义训练集

在这里插入图片描述

定义训练模型

在这里插入图片描述

定义损失函数

在这里插入图片描述

定义梯度函数

在这里插入图片描述

训练过程，每次都要拿权重-学习率*梯度，学习率这里是超参数，设置为0.01，进行100个epoch

在这里插入图片描述

最后输出训练日志

import numpy as np
import matplotlib.pyplot as plt

x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = 1.0


def forward(x):
    return x * w


def cost(xs, ys):
    cost = 0
    for x, y in zip(xs, ys):
        y_pred = forward(x)
        cost += (y_pred - y) ** 2
    return cost / len(xs)


def gradient(xs, ys):
    grad = 0
    for x, y in zip(xs, ys):
        grad += 2 * x * (x * w - y)
    return grad / len(xs)


epoch_list = []
cost_list = []

print('Predict(before training)', 4, forward(4))
for epoch in range(100):
    epoch_list.append(epoch)
    cost_val = cost(x_data, y_data)
    cost_list.append(cost_val)
    grad_val = gradient(x_data, y_data)
    w -= 0.01 * grad_val
    print('Epoch:', epoch, 'w=', w, 'loss=', cost_val)
print('Predict(after training)', 4, forward(4))

plt.plot(epoch_list, cost_list)
plt.ylabel('Cost')
plt.xlabel('Epoch')
plt.show()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

下面展示运行结果

Predict(before training) 4 4.0
Epoch: 0 w= 1.0933333333333333 loss= 4.666666666666667
Epoch: 1 w= 1.1779555555555554 loss= 3.8362074074074086
Epoch: 2 w= 1.2546797037037036 loss= 3.1535329869958857
Epoch: 3 w= 1.3242429313580246 loss= 2.592344272332262
Epoch: 4 w= 1.3873135910979424 loss= 2.1310222071581117
Epoch: 5 w= 1.4444976559288012 loss= 1.7517949663820642
Epoch: 6 w= 1.4963445413754464 loss= 1.440053319920117
Epoch: 7 w= 1.5433523841804047 loss= 1.1837878313441108
Epoch: 8 w= 1.5859728283235668 loss= 0.9731262101573632
Epoch: 9 w= 1.6246153643467005 loss= 0.7999529948031382
Epoch: 10 w= 1.659651263674342 loss= 0.6575969151946154
Epoch: 11 w= 1.6914171457314033 loss= 0.5405738908195378
Epoch: 12 w= 1.7202182121298057 loss= 0.44437576375991855
Epoch: 13 w= 1.7463311789976905 loss= 0.365296627844598
Epoch: 14 w= 1.7700069356245727 loss= 0.3002900634939416
Epoch: 15 w= 1.7914729549662791 loss= 0.2468517784170642
Epoch: 16 w= 1.8109354791694263 loss= 0.2029231330489788
Epoch: 17 w= 1.8285815011136133 loss= 0.16681183417217407
Epoch: 18 w= 1.8445805610096762 loss= 0.1371267415488235
Epoch: 19 w= 1.8590863753154396 loss= 0.11272427607497944
Epoch: 20 w= 1.872238313619332 loss= 0.09266436490145864
Epoch: 21 w= 1.8841627376815275 loss= 0.07617422636521683
Epoch: 22 w= 1.8949742154979183 loss= 0.06261859959338009
Epoch: 23 w= 1.904776622051446 loss= 0.051475271914629306
Epoch: 24 w= 1.9136641373266443 loss= 0.04231496130368814
Epoch: 25 w= 1.9217221511761575 loss= 0.03478477885657844
Epoch: 26 w= 1.9290280837330496 loss= 0.02859463421027894
Epoch: 27 w= 1.9356521292512983 loss= 0.023506060193480772
Epoch: 28 w= 1.9416579305211772 loss= 0.01932302619282764
Epoch: 29 w= 1.9471031903392007 loss= 0.015884386331668398
Epoch: 30 w= 1.952040225907542 loss= 0.01305767153735723
Epoch: 31 w= 1.9565164714895047 loss= 0.010733986344664803
Epoch: 32 w= 1.9605749341504843 loss= 0.008823813841374291
Epoch: 33 w= 1.9642546069631057 loss= 0.007253567147113681
Epoch: 34 w= 1.9675908436465492 loss= 0.005962754575689583
Epoch: 35 w= 1.970615698239538 loss= 0.004901649272531298
Epoch: 36 w= 1.9733582330705144 loss= 0.004029373553099482
Epoch: 37 w= 1.975844797983933 loss= 0.0033123241439168096
Epoch: 38 w= 1.9780992835054327 loss= 0.0027228776607060357
Epoch: 39 w= 1.980143350378259 loss= 0.002238326453885249
Epoch: 40 w= 1.9819966376762883 loss= 0.001840003826269386
Epoch: 41 w= 1.983676951493168 loss= 0.0015125649231412608
Epoch: 42 w= 1.9852004360204722 loss= 0.0012433955919298103
Epoch: 43 w= 1.9865817286585614 loss= 0.0010221264385926248
Epoch: 44 w= 1.987834100650429 loss= 0.0008402333603648631
Epoch: 45 w= 1.9889695845897222 loss= 0.0006907091659248264
Epoch: 46 w= 1.9899990900280147 loss= 0.0005677936325753796
Epoch: 47 w= 1.9909325082920666 loss= 0.0004667516012495216
Epoch: 48 w= 1.9917788075181404 loss= 0.000383690560742734
Epoch: 49 w= 1.9925461188164473 loss= 0.00031541069384432885
Epoch: 50 w= 1.9932418143935788 loss= 0.0002592816085930997
Epoch: 51 w= 1.9938725783835114 loss= 0.0002131410058905752
Epoch: 52 w= 1.994444471067717 loss= 0.00017521137977565514
Epoch: 53 w= 1.9949629871013967 loss= 0.0001440315413480261
Epoch: 54 w= 1.9954331083052663 loss= 0.0001184003283899171
Epoch: 55 w= 1.9958593515301082 loss= 9.733033217332803e-05
Epoch: 56 w= 1.9962458120539648 loss= 8.000985883901657e-05
Epoch: 57 w= 1.9965962029289281 loss= 6.57716599593935e-05
Epoch: 58 w= 1.9969138906555615 loss= 5.406722767150764e-05
Epoch: 59 w= 1.997201927527709 loss= 4.444566413387458e-05
Epoch: 60 w= 1.9974630809584561 loss= 3.65363112808981e-05
Epoch: 61 w= 1.9976998600690001 loss= 3.0034471708953996e-05
Epoch: 62 w= 1.9979145397958935 loss= 2.4689670610172655e-05
Epoch: 63 w= 1.9981091827482769 loss= 2.0296006560253656e-05
Epoch: 64 w= 1.9982856590251044 loss= 1.6684219437262796e-05
Epoch: 65 w= 1.9984456641827613 loss= 1.3715169898293847e-05
Epoch: 66 w= 1.9985907355257035 loss= 1.1274479219506377e-05
Epoch: 67 w= 1.9987222668766378 loss= 9.268123006398985e-06
Epoch: 68 w= 1.9988415219681517 loss= 7.61880902783969e-06
Epoch: 69 w= 1.9989496465844576 loss= 6.262999634617916e-06
Epoch: 70 w= 1.9990476795699081 loss= 5.1484640551938914e-06
Epoch: 71 w= 1.9991365628100501 loss= 4.232266273994499e-06
Epoch: 72 w= 1.999217150281112 loss= 3.479110977946351e-06
Epoch: 73 w= 1.999290216254875 loss= 2.859983851026929e-06
Epoch: 74 w= 1.9993564627377531 loss= 2.3510338359374262e-06
Epoch: 75 w= 1.9994165262155628 loss= 1.932654303533636e-06
Epoch: 76 w= 1.999470983768777 loss= 1.5887277332523938e-06
Epoch: 77 w= 1.9995203586170245 loss= 1.3060048068548734e-06
Epoch: 78 w= 1.9995651251461022 loss= 1.0735939958924364e-06
Epoch: 79 w= 1.9996057134657994 loss= 8.825419799121559e-07
Epoch: 80 w= 1.9996425135423248 loss= 7.254887315754342e-07
Epoch: 81 w= 1.999675878945041 loss= 5.963839812987369e-07
Epoch: 82 w= 1.999706130243504 loss= 4.902541385825727e-07
Epoch: 83 w= 1.9997335580874436 loss= 4.0301069098738336e-07
Epoch: 84 w= 1.9997584259992822 loss= 3.312926995781724e-07
Epoch: 85 w= 1.9997809729060159 loss= 2.723373231729343e-07
Epoch: 86 w= 1.9998014154347876 loss= 2.2387338352920307e-07
Epoch: 87 w= 1.9998199499942075 loss= 1.8403387118941732e-07
Epoch: 88 w= 1.9998367546614149 loss= 1.5128402140063082e-07
Epoch: 89 w= 1.9998519908930161 loss= 1.2436218932547864e-07
Epoch: 90 w= 1.9998658050763347 loss= 1.0223124683409346e-07
Epoch: 91 w= 1.9998783299358769 loss= 8.403862850836479e-08
Epoch: 92 w= 1.9998896858085284 loss= 6.908348768398496e-08
Epoch: 93 w= 1.9998999817997325 loss= 5.678969725349543e-08
Epoch: 94 w= 1.9999093168317574 loss= 4.66836551287917e-08
Epoch: 95 w= 1.9999177805941268 loss= 3.8376039345125727e-08
Epoch: 96 w= 1.9999254544053418 loss= 3.154680994333735e-08
Epoch: 97 w= 1.9999324119941766 loss= 2.593287985380858e-08
Epoch: 98 w= 1.9999387202080534 loss= 2.131797981222471e-08
Epoch: 99 w= 1.9999444396553017 loss= 1.752432687141379e-08
Predict(after training) 4 7.999777758621207
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

输出图像：

在这里插入图片描述

我们通过上面可以发现，随着训练次数增加，cost的值越来越小，接近于0，权重w在训练到一定的次数之后无限接近2

tip:数据中存在噪声，无法真正到0

下面我们来看一下梯度下降的衍生，随机梯度下降Stochastic Gradient Descent

在梯度下降中损失函数取得是所有损失的均值

而随机梯度下降的意思就是在N个样本中随机选一个，拿单个样本的损失对梯度求导，然后更新

在这里插入图片描述

引用随机梯度下降的原因：当陷入梯度消失时，用单个样本的梯度下降，可能会产生噪声，使得模型向前推动，引用这种随机性，就有可能跨越过鞍点，向最优值前进

在这里插入图片描述

接下来我们看一下代码根据梯度下降进行了哪些修改

定义单个损失loss

在这里插入图片描述

梯度上进行了修改

在这里插入图片描述

训练过程中对每一个样本来求梯度，然后进行更新

在这里插入图片描述

下面我们看一下完整的代码

import numpy as np
import matplotlib.pyplot as plt

x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = 1.0


def forward(x):
    return x * w


def loss(x, y):
    y_pred = forward(x)
    return (y_pred - y) ** 2


def gradient(x, y):
    return 2 * x * (x * w - y)


epoch_list = []
loss_list = []

print('Predict(before training)', 4, forward(4))
for epoch in range(100):
    epoch_list.append(epoch)
    for x, y in zip(x_data, y_data):
        grad = gradient(x, y)
        w -= 0.01 * grad
        print('\tgrad:', x, y, grad)
        l = loss(x, y)
    loss_list.append(l)
    print('Epoch:', epoch, 'w=', w, 'loss=', l)
print('Predict(after training)', 4, forward(4))

plt.plot(epoch_list, loss_list)
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.show()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

运行结果：

Predict(before training) 4 4.0
  grad: 1.0 2.0 -2.0
  grad: 2.0 4.0 -7.84
  grad: 3.0 6.0 -16.2288
Epoch: 0 w= 1.260688 loss= 4.919240100095999
  grad: 1.0 2.0 -1.478624
  grad: 2.0 4.0 -5.796206079999999
  grad: 3.0 6.0 -11.998146585599997
Epoch: 1 w= 1.453417766656 loss= 2.688769240265834
  grad: 1.0 2.0 -1.093164466688
  grad: 2.0 4.0 -4.285204709416961
  grad: 3.0 6.0 -8.87037374849311
Epoch: 2 w= 1.5959051959019805 loss= 1.4696334962911515
  grad: 1.0 2.0 -0.8081896081960389
  grad: 2.0 4.0 -3.1681032641284723
  grad: 3.0 6.0 -6.557973756745939
Epoch: 3 w= 1.701247862192685 loss= 0.8032755585999681
  grad: 1.0 2.0 -0.59750427561463
  grad: 2.0 4.0 -2.3422167604093502
  grad: 3.0 6.0 -4.848388694047353
Epoch: 4 w= 1.7791289594933983 loss= 0.43905614881022015
  grad: 1.0 2.0 -0.44174208101320334
  grad: 2.0 4.0 -1.7316289575717576
  grad: 3.0 6.0 -3.584471942173538
Epoch: 5 w= 1.836707389300983 loss= 0.2399802903801062
  grad: 1.0 2.0 -0.3265852213980338
  grad: 2.0 4.0 -1.2802140678802925
  grad: 3.0 6.0 -2.650043120512205
Epoch: 6 w= 1.8792758133988885 loss= 0.1311689630744999
  grad: 1.0 2.0 -0.241448373202223
  grad: 2.0 4.0 -0.946477622952715
  grad: 3.0 6.0 -1.9592086795121197
Epoch: 7 w= 1.910747160155559 loss= 0.07169462478267678
  grad: 1.0 2.0 -0.17850567968888198
  grad: 2.0 4.0 -0.6997422643804168
  grad: 3.0 6.0 -1.4484664872674653
Epoch: 8 w= 1.9340143044689266 loss= 0.03918700813247573
  grad: 1.0 2.0 -0.13197139106214673
  grad: 2.0 4.0 -0.5173278529636143
  grad: 3.0 6.0 -1.0708686556346834
Epoch: 9 w= 1.9512159834655312 loss= 0.021418922423117836
  grad: 1.0 2.0 -0.09756803306893769
  grad: 2.0 4.0 -0.38246668963023644
  grad: 3.0 6.0 -0.7917060475345892
Epoch: 10 w= 1.9639333911678687 loss= 0.01170720245384975
  grad: 1.0 2.0 -0.07213321766426262
  grad: 2.0 4.0 -0.2827622132439096
  grad: 3.0 6.0 -0.5853177814148953
Epoch: 11 w= 1.9733355232910992 loss= 0.006398948863435593
  grad: 1.0 2.0 -0.05332895341780164
  grad: 2.0 4.0 -0.2090494973977819
  grad: 3.0 6.0 -0.4327324596134101
Epoch: 12 w= 1.9802866323953892 loss= 0.003497551760830656
  grad: 1.0 2.0 -0.039426735209221686
  grad: 2.0 4.0 -0.15455280202014876
  grad: 3.0 6.0 -0.3199243001817109
Epoch: 13 w= 1.9854256707695 loss= 0.001911699652671057
  grad: 1.0 2.0 -0.02914865846100012
  grad: 2.0 4.0 -0.11426274116712065
  grad: 3.0 6.0 -0.2365238742159388
Epoch: 14 w= 1.9892250235079405 loss= 0.0010449010656399273
  grad: 1.0 2.0 -0.021549952984118992
  grad: 2.0 4.0 -0.08447581569774698
  grad: 3.0 6.0 -0.17486493849433593
Epoch: 15 w= 1.9920339305797026 loss= 0.0005711243580809696
  grad: 1.0 2.0 -0.015932138840594856
  grad: 2.0 4.0 -0.062453984255132156
  grad: 3.0 6.0 -0.12927974740812687
Epoch: 16 w= 1.994110589284741 loss= 0.0003121664271570621
  grad: 1.0 2.0 -0.011778821430517894
  grad: 2.0 4.0 -0.046172980007630926
  grad: 3.0 6.0 -0.09557806861579543
Epoch: 17 w= 1.9956458879852805 loss= 0.0001706246229305199
  grad: 1.0 2.0 -0.008708224029438938
  grad: 2.0 4.0 -0.03413623819540135
  grad: 3.0 6.0 -0.07066201306448505
Epoch: 18 w= 1.9967809527381737 loss= 9.326038746484765e-05
  grad: 1.0 2.0 -0.006438094523652627
  grad: 2.0 4.0 -0.02523733053271826
  grad: 3.0 6.0 -0.052241274202728505
Epoch: 19 w= 1.9976201197307648 loss= 5.097447086306101e-05
  grad: 1.0 2.0 -0.004759760538470381
  grad: 2.0 4.0 -0.01865826131080439
  grad: 3.0 6.0 -0.03862260091336722
Epoch: 20 w= 1.998240525958391 loss= 2.7861740127856012e-05
  grad: 1.0 2.0 -0.0035189480832178432
  grad: 2.0 4.0 -0.01379427648621423
  grad: 3.0 6.0 -0.028554152326460525
Epoch: 21 w= 1.99869919972735 loss= 1.5228732143933469e-05
  grad: 1.0 2.0 -0.002601600545300009
  grad: 2.0 4.0 -0.01019827413757568
  grad: 3.0 6.0 -0.021110427464781978
Epoch: 22 w= 1.9990383027488265 loss= 8.323754426231206e-06
  grad: 1.0 2.0 -0.001923394502346909
  grad: 2.0 4.0 -0.007539706449199102
  grad: 3.0 6.0 -0.01560719234984198
Epoch: 23 w= 1.9992890056818404 loss= 4.549616284094891e-06
  grad: 1.0 2.0 -0.0014219886363191492
  grad: 2.0 4.0 -0.005574195454370212
  grad: 3.0 6.0 -0.011538584590544687
Epoch: 24 w= 1.999474353368653 loss= 2.486739429417538e-06
  grad: 1.0 2.0 -0.0010512932626940419
  grad: 2.0 4.0 -0.004121069589761106
  grad: 3.0 6.0 -0.008530614050808794
Epoch: 25 w= 1.9996113831376856 loss= 1.3592075910762856e-06
  grad: 1.0 2.0 -0.0007772337246287897
  grad: 2.0 4.0 -0.0030467562005451754
  grad: 3.0 6.0 -0.006306785335127074
Epoch: 26 w= 1.9997126908902887 loss= 7.429187207079447e-07
  grad: 1.0 2.0 -0.0005746182194226179
  grad: 2.0 4.0 -0.002252503420136165
  grad: 3.0 6.0 -0.00466268207967957
Epoch: 27 w= 1.9997875889274812 loss= 4.060661735575354e-07
  grad: 1.0 2.0 -0.0004248221450375844
  grad: 2.0 4.0 -0.0016653028085471533
  grad: 3.0 6.0 -0.0034471768136938863
Epoch: 28 w= 1.9998429619451539 loss= 2.2194855602869353e-07
  grad: 1.0 2.0 -0.00031407610969225175
  grad: 2.0 4.0 -0.0012311783499932005
  grad: 3.0 6.0 -0.0025485391844828342
Epoch: 29 w= 1.9998838998815958 loss= 1.213131374411496e-07
  grad: 1.0 2.0 -0.00023220023680847746
  grad: 2.0 4.0 -0.0009102249282886277
  grad: 3.0 6.0 -0.0018841656015560204
Epoch: 30 w= 1.9999141657892625 loss= 6.630760559646474e-08
  grad: 1.0 2.0 -0.00017166842147497974
  grad: 2.0 4.0 -0.0006729402121816719
  grad: 3.0 6.0 -0.0013929862392156878
Epoch: 31 w= 1.9999365417379913 loss= 3.624255915449335e-08
  grad: 1.0 2.0 -0.0001269165240174175
  grad: 2.0 4.0 -0.0004975127741477792
  grad: 3.0 6.0 -0.0010298514424817995
Epoch: 32 w= 1.9999530845453979 loss= 1.9809538924707548e-08
  grad: 1.0 2.0 -9.383090920422887e-05
  grad: 2.0 4.0 -0.00036781716408107457
  grad: 3.0 6.0 -0.0007613815296476645
Epoch: 33 w= 1.9999653148414271 loss= 1.0827542027017377e-08
  grad: 1.0 2.0 -6.937031714571162e-05
  grad: 2.0 4.0 -0.0002719316432120422
  grad: 3.0 6.0 -0.0005628985014531906
Epoch: 34 w= 1.999974356846045 loss= 5.9181421028034105e-09
  grad: 1.0 2.0 -5.1286307909848006e-05
  grad: 2.0 4.0 -0.00020104232700646207
  grad: 3.0 6.0 -0.0004161576169003922
Epoch: 35 w= 1.9999810417085633 loss= 3.2347513278475087e-09
  grad: 1.0 2.0 -3.7916582873442906e-05
  grad: 2.0 4.0 -0.0001486330048638962
  grad: 3.0 6.0 -0.0003076703200690645
Epoch: 36 w= 1.9999859839076413 loss= 1.7680576050779005e-09
  grad: 1.0 2.0 -2.8032184717474706e-05
  grad: 2.0 4.0 -0.0001098861640933535
  grad: 3.0 6.0 -0.00022746435967313516
Epoch: 37 w= 1.9999896377347262 loss= 9.6638887447731e-10
  grad: 1.0 2.0 -2.0724530547688857e-05
  grad: 2.0 4.0 -8.124015974608767e-05
  grad: 3.0 6.0 -0.00016816713067413502
Epoch: 38 w= 1.999992339052936 loss= 5.282109892545845e-10
  grad: 1.0 2.0 -1.5321894128117464e-05
  grad: 2.0 4.0 -6.006182498197177e-05
  grad: 3.0 6.0 -0.00012432797771566584
Epoch: 39 w= 1.9999943361699042 loss= 2.887107421958329e-10
  grad: 1.0 2.0 -1.1327660191629008e-05
  grad: 2.0 4.0 -4.4404427951505454e-05
  grad: 3.0 6.0 -9.191716585732479e-05
Epoch: 40 w= 1.9999958126624442 loss= 1.5780416225633037e-10
  grad: 1.0 2.0 -8.37467511161094e-06
  grad: 2.0 4.0 -3.282872643772805e-05
  grad: 3.0 6.0 -6.795546372551087e-05
Epoch: 41 w= 1.999996904251097 loss= 8.625295142578772e-11
  grad: 1.0 2.0 -6.191497806007362e-06
  grad: 2.0 4.0 -2.4270671399762023e-05
  grad: 3.0 6.0 -5.0240289795056015e-05
Epoch: 42 w= 1.999997711275687 loss= 4.71443308235547e-11
  grad: 1.0 2.0 -4.5774486259198e-06
  grad: 2.0 4.0 -1.794359861406747e-05
  grad: 3.0 6.0 -3.714324913239864e-05
Epoch: 43 w= 1.9999983079186507 loss= 2.5768253628059826e-11
  grad: 1.0 2.0 -3.3841626985164908e-06
  grad: 2.0 4.0 -1.326591777761621e-05
  grad: 3.0 6.0 -2.7460449796734565e-05
Epoch: 44 w= 1.9999987490239537 loss= 1.4084469615916932e-11
  grad: 1.0 2.0 -2.5019520926150562e-06
  grad: 2.0 4.0 -9.807652203264183e-06
  grad: 3.0 6.0 -2.0301840059744336e-05
Epoch: 45 w= 1.9999990751383971 loss= 7.698320862431846e-12
  grad: 1.0 2.0 -1.8497232057157476e-06
  grad: 2.0 4.0 -7.250914967116273e-06
  grad: 3.0 6.0 -1.5009393983689279e-05
Epoch: 46 w= 1.9999993162387186 loss= 4.20776540913866e-12
  grad: 1.0 2.0 -1.3675225627451937e-06
  grad: 2.0 4.0 -5.3606884460322135e-06
  grad: 3.0 6.0 -1.109662508014253e-05
Epoch: 47 w= 1.9999994944870796 loss= 2.299889814334344e-12
  grad: 1.0 2.0 -1.0110258408246864e-06
  grad: 2.0 4.0 -3.963221296032771e-06
  grad: 3.0 6.0 -8.20386808086937e-06
Epoch: 48 w= 1.9999996262682318 loss= 1.2570789110540446e-12
  grad: 1.0 2.0 -7.474635363990956e-07
  grad: 2.0 4.0 -2.930057062755509e-06
  grad: 3.0 6.0 -6.065218119744031e-06
Epoch: 49 w= 1.999999723695619 loss= 6.870969979249939e-13
  grad: 1.0 2.0 -5.526087618612507e-07
  grad: 2.0 4.0 -2.166226346744793e-06
  grad: 3.0 6.0 -4.484088535150477e-06
Epoch: 50 w= 1.9999997957248556 loss= 3.7555501141274804e-13
  grad: 1.0 2.0 -4.08550288710785e-07
  grad: 2.0 4.0 -1.6015171322436572e-06
  grad: 3.0 6.0 -3.3151404608133817e-06
Epoch: 51 w= 1.9999998489769344 loss= 2.052716967104274e-13
  grad: 1.0 2.0 -3.020461312175371e-07
  grad: 2.0 4.0 -1.1840208351543424e-06
  grad: 3.0 6.0 -2.4509231284497446e-06
Epoch: 52 w= 1.9999998883468353 loss= 1.1219786256679713e-13
  grad: 1.0 2.0 -2.2330632942768602e-07
  grad: 2.0 4.0 -8.753608113920563e-07
  grad: 3.0 6.0 -1.811996877876254e-06
Epoch: 53 w= 1.9999999174534755 loss= 6.132535848018759e-14
  grad: 1.0 2.0 -1.6509304900935717e-07
  grad: 2.0 4.0 -6.471647520100987e-07
  grad: 3.0 6.0 -1.3396310407642886e-06
Epoch: 54 w= 1.999999938972364 loss= 3.351935118167793e-14
  grad: 1.0 2.0 -1.220552721115098e-07
  grad: 2.0 4.0 -4.784566662863199e-07
  grad: 3.0 6.0 -9.904052991061008e-07
Epoch: 55 w= 1.9999999548815364 loss= 1.8321081844499955e-14
  grad: 1.0 2.0 -9.023692726373156e-08
  grad: 2.0 4.0 -3.5372875473171916e-07
  grad: 3.0 6.0 -7.322185204827747e-07
Epoch: 56 w= 1.9999999666433785 loss= 1.0013977760018664e-14
  grad: 1.0 2.0 -6.671324292994996e-08
  grad: 2.0 4.0 -2.615159129248923e-07
  grad: 3.0 6.0 -5.413379398078177e-07
Epoch: 57 w= 1.9999999753390494 loss= 5.473462367088053e-15
  grad: 1.0 2.0 -4.932190122985958e-08
  grad: 2.0 4.0 -1.9334185274999527e-07
  grad: 3.0 6.0 -4.002176350326181e-07
Epoch: 58 w= 1.9999999817678633 loss= 2.991697274308627e-15
  grad: 1.0 2.0 -3.6464273378555845e-08
  grad: 2.0 4.0 -1.429399514307761e-07
  grad: 3.0 6.0 -2.9588569994132286e-07
Epoch: 59 w= 1.9999999865207625 loss= 1.6352086111474931e-15
  grad: 1.0 2.0 -2.6958475007887728e-08
  grad: 2.0 4.0 -1.0567722164012139e-07
  grad: 3.0 6.0 -2.1875184863517916e-07
Epoch: 60 w= 1.999999990034638 loss= 8.937759877335403e-16
  grad: 1.0 2.0 -1.993072418216002e-08
  grad: 2.0 4.0 -7.812843882959442e-08
  grad: 3.0 6.0 -1.617258700292723e-07
Epoch: 61 w= 1.9999999926324883 loss= 4.885220495987371e-16
  grad: 1.0 2.0 -1.473502342363986e-08
  grad: 2.0 4.0 -5.7761292637792394e-08
  grad: 3.0 6.0 -1.195658771990793e-07
Epoch: 62 w= 1.99999999455311 loss= 2.670175009618106e-16
  grad: 1.0 2.0 -1.0893780100218464e-08
  grad: 2.0 4.0 -4.270361841918202e-08
  grad: 3.0 6.0 -8.839649012770678e-08
Epoch: 63 w= 1.9999999959730488 loss= 1.4594702493172377e-16
  grad: 1.0 2.0 -8.05390243385773e-09
  grad: 2.0 4.0 -3.1571296688071016e-08
  grad: 3.0 6.0 -6.53525820126788e-08
Epoch: 64 w= 1.9999999970228268 loss= 7.977204100704301e-17
  grad: 1.0 2.0 -5.9543463493128e-09
  grad: 2.0 4.0 -2.334103754719763e-08
  grad: 3.0 6.0 -4.8315948575350376e-08
Epoch: 65 w= 1.9999999977989402 loss= 4.360197735196887e-17
  grad: 1.0 2.0 -4.402119557767037e-09
  grad: 2.0 4.0 -1.725630838222969e-08
  grad: 3.0 6.0 -3.5720557178819945e-08
Epoch: 66 w= 1.9999999983727301 loss= 2.3832065197304227e-17
  grad: 1.0 2.0 -3.254539748809293e-09
  grad: 2.0 4.0 -1.2757796596929438e-08
  grad: 3.0 6.0 -2.6408640607655798e-08
Epoch: 67 w= 1.9999999987969397 loss= 1.3026183953845832e-17
  grad: 1.0 2.0 -2.406120636067044e-09
  grad: 2.0 4.0 -9.431992964437086e-09
  grad: 3.0 6.0 -1.9524227568012975e-08
Epoch: 68 w= 1.999999999110563 loss= 7.11988308874388e-18
  grad: 1.0 2.0 -1.7788739370416806e-09
  grad: 2.0 4.0 -6.97318647269185e-09
  grad: 3.0 6.0 -1.4434496264925656e-08
Epoch: 69 w= 1.9999999993424284 loss= 3.89160224698574e-18
  grad: 1.0 2.0 -1.3151431055291596e-09
  grad: 2.0 4.0 -5.155360582875801e-09
  grad: 3.0 6.0 -1.067159693945996e-08
Epoch: 70 w= 1.9999999995138495 loss= 2.1270797208746147e-18
  grad: 1.0 2.0 -9.72300906454393e-10
  grad: 2.0 4.0 -3.811418736177075e-09
  grad: 3.0 6.0 -7.88963561149103e-09
Epoch: 71 w= 1.9999999996405833 loss= 1.1626238773828175e-18
  grad: 1.0 2.0 -7.18833437218791e-10
  grad: 2.0 4.0 -2.8178277489132597e-09
  grad: 3.0 6.0 -5.832902161273523e-09
Epoch: 72 w= 1.999999999734279 loss= 6.354692062078993e-19
  grad: 1.0 2.0 -5.314420015167798e-10
  grad: 2.0 4.0 -2.0832526814729135e-09
  grad: 3.0 6.0 -4.31233715403323e-09
Epoch: 73 w= 1.9999999998035491 loss= 3.4733644793346653e-19
  grad: 1.0 2.0 -3.92901711165905e-10
  grad: 2.0 4.0 -1.5401742103904326e-09
  grad: 3.0 6.0 -3.188159070077745e-09
Epoch: 74 w= 1.9999999998547615 loss= 1.8984796531526204e-19
  grad: 1.0 2.0 -2.9047697580608656e-10
  grad: 2.0 4.0 -1.1386696030513122e-09
  grad: 3.0 6.0 -2.3570478902001923e-09
Epoch: 75 w= 1.9999999998926234 loss= 1.0376765851119951e-19
  grad: 1.0 2.0 -2.1475310418850313e-10
  grad: 2.0 4.0 -8.418314934033333e-10
  grad: 3.0 6.0 -1.7425900722400911e-09
Epoch: 76 w= 1.9999999999206153 loss= 5.671751114309842e-20
  grad: 1.0 2.0 -1.5876944203796484e-10
  grad: 2.0 4.0 -6.223768167501476e-10
  grad: 3.0 6.0 -1.2883241140571045e-09
Epoch: 77 w= 1.9999999999413098 loss= 3.100089617511693e-20
  grad: 1.0 2.0 -1.17380327679939e-10
  grad: 2.0 4.0 -4.601314884666863e-10
  grad: 3.0 6.0 -9.524754318590567e-10
Epoch: 78 w= 1.9999999999566096 loss= 1.6944600977692705e-20
  grad: 1.0 2.0 -8.678080476443029e-11
  grad: 2.0 4.0 -3.4018121652934497e-10
  grad: 3.0 6.0 -7.041780492045291e-10
Epoch: 79 w= 1.9999999999679208 loss= 9.2616919156479e-21
  grad: 1.0 2.0 -6.415845632545825e-11
  grad: 2.0 4.0 -2.5150193039280566e-10
  grad: 3.0 6.0 -5.206075570640678e-10
Epoch: 80 w= 1.9999999999762834 loss= 5.062350511130293e-21
  grad: 1.0 2.0 -4.743316850408519e-11
  grad: 2.0 4.0 -1.8593837580738182e-10
  grad: 3.0 6.0 -3.8489211817704927e-10
Epoch: 81 w= 1.999999999982466 loss= 2.7669155644059242e-21
  grad: 1.0 2.0 -3.5067948545020045e-11
  grad: 2.0 4.0 -1.3746692673066718e-10
  grad: 3.0 6.0 -2.845563784603655e-10
Epoch: 82 w= 1.9999999999870368 loss= 1.5124150106147723e-21
  grad: 1.0 2.0 -2.5926372160256506e-11
  grad: 2.0 4.0 -1.0163070385260653e-10
  grad: 3.0 6.0 -2.1037571684701106e-10
Epoch: 83 w= 1.999999999990416 loss= 8.26683933105326e-22
  grad: 1.0 2.0 -1.9167778475548403e-11
  grad: 2.0 4.0 -7.51381179497912e-11
  grad: 3.0 6.0 -1.5553425214420713e-10
Epoch: 84 w= 1.9999999999929146 loss= 4.518126871054872e-22
  grad: 1.0 2.0 -1.4170886686315498e-11
  grad: 2.0 4.0 -5.555023108172463e-11
  grad: 3.0 6.0 -1.1499068364173581e-10
Epoch: 85 w= 1.9999999999947617 loss= 2.469467919185614e-22
  grad: 1.0 2.0 -1.0476508549572827e-11
  grad: 2.0 4.0 -4.106759377009439e-11
  grad: 3.0 6.0 -8.500933290633839e-11
Epoch: 86 w= 1.9999999999961273 loss= 1.349840097651456e-22
  grad: 1.0 2.0 -7.745359908994942e-12
  grad: 2.0 4.0 -3.036149109902908e-11
  grad: 3.0 6.0 -6.285105769165966e-11
Epoch: 87 w= 1.999999999997137 loss= 7.376551550022107e-23
  grad: 1.0 2.0 -5.726086271806707e-12
  grad: 2.0 4.0 -2.2446045022661565e-11
  grad: 3.0 6.0 -4.646416584819235e-11
Epoch: 88 w= 1.9999999999978835 loss= 4.031726170507742e-23
  grad: 1.0 2.0 -4.233058348290797e-12
  grad: 2.0 4.0 -1.659294923683774e-11
  grad: 3.0 6.0 -3.4351188560322043e-11
Epoch: 89 w= 1.9999999999984353 loss= 2.2033851437431755e-23
  grad: 1.0 2.0 -3.1294966618133913e-12
  grad: 2.0 4.0 -1.226752033289813e-11
  grad: 3.0 6.0 -2.539835008974478e-11
Epoch: 90 w= 1.9999999999988431 loss= 1.2047849775995315e-23
  grad: 1.0 2.0 -2.3137047833188262e-12
  grad: 2.0 4.0 -9.070078021977679e-12
  grad: 3.0 6.0 -1.8779644506139448e-11
Epoch: 91 w= 1.9999999999991447 loss= 6.5840863393251405e-24
  grad: 1.0 2.0 -1.7106316363424412e-12
  grad: 2.0 4.0 -6.7057470687359455e-12
  grad: 3.0 6.0 -1.3882228699912957e-11
Epoch: 92 w= 1.9999999999993676 loss= 3.5991747246272455e-24
  grad: 1.0 2.0 -1.2647660696529783e-12
  grad: 2.0 4.0 -4.957811938766099e-12
  grad: 3.0 6.0 -1.0263789818054647e-11
Epoch: 93 w= 1.9999999999995324 loss= 1.969312363793734e-24
  grad: 1.0 2.0 -9.352518759442319e-13
  grad: 2.0 4.0 -3.666400516522117e-12
  grad: 3.0 6.0 -7.58859641791787e-12
Epoch: 94 w= 1.9999999999996543 loss= 1.0761829795642296e-24
  grad: 1.0 2.0 -6.914468997365475e-13
  grad: 2.0 4.0 -2.7107205369247822e-12
  grad: 3.0 6.0 -5.611511255665391e-12
Epoch: 95 w= 1.9999999999997444 loss= 5.875191475205477e-25
  grad: 1.0 2.0 -5.111466805374221e-13
  grad: 2.0 4.0 -2.0037305148434825e-12
  grad: 3.0 6.0 -4.1460168631601846e-12
Epoch: 96 w= 1.999999999999811 loss= 3.2110109830478153e-25
  grad: 1.0 2.0 -3.779199175824033e-13
  grad: 2.0 4.0 -1.4814816040598089e-12
  grad: 3.0 6.0 -3.064215547965432e-12
Epoch: 97 w= 1.9999999999998603 loss= 1.757455879087579e-25
  grad: 1.0 2.0 -2.793321129956894e-13
  grad: 2.0 4.0 -1.0942358130705543e-12
  grad: 3.0 6.0 -2.2648549702353193e-12
Epoch: 98 w= 1.9999999999998967 loss= 9.608404711682446e-26
  grad: 1.0 2.0 -2.0650148258027912e-13
  grad: 2.0 4.0 -8.100187187665142e-13
  grad: 3.0 6.0 -1.6786572132332367e-12
Epoch: 99 w= 1.9999999999999236 loss= 5.250973729513143e-26
Predict(after training) 4 7.9999999999996945
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402

显示图像为：

在这里插入图片描述

tips：在目前的深度学习中，我们使用的更多的时小批量梯度下降（MBGD），采用的就是mini-batch

相关阅读:
cookie、session、tooken
flutter实现上拉到底部加载更多数据
如何在Spring 中将文件作为字符串加载？
pycharm安装（windows）
mac安装redis及springboot整合redis，简单快速
vite.config.js文件配置代理设置VITE_APP_BASE_API
VMWare安装Ubuntu
【操作系统】1.3.1 操作系统的运行机制
C++11（列表初始化，声明，范围for）
C# Winform编程（9）网络编程

原文地址：https://blog.csdn.net/weixin_55500281/article/details/128038288