【深度学习】实验13 使用Dropout抑制过拟合

使用Dropout抑制过拟合

Dropout是一种常用的神经网络正则化方法，主要用于防止过拟合。在深度学习中，由于网络层数过多，参数数量庞大，模型容易过拟合，并且在测试时产生较大的泛化误差。Dropout方法借鉴了集成学习中的Bagging思想，通过随机的方式，将一部分神经元的输出设置为0，从而减少过拟合的可能。

Dropout方法最早由Hinton等人提出，其基本思想是在训练时，以一定的概率随机地将网络中某些神经元的输出置为0。这种随机的行为可以被看作是一种对网络进行了部分剪枝，从而增加了网络的容忍性，使网络更加健壮，同时也避免了网络中某些特定的神经元对整个网络的过度依赖。

Dropout方法的具体实现如下：在每次训练过程中，以一定的概率p随机选择一部分神经元并将其置为0，被选择的神经元不参与后续的训练和反向传播。在测试时，为了保持模型的稳定性和一致性，一般不会采取随机化的方式，而是将每个神经元的权重乘以概率p，这里的p是在训练时选择的那个概率。

Dropout方法不仅可用于全连接网络，也可用于卷积神经网络和循环神经网络中，以减少过拟合现象。并且，它的实现简单，仅需要在模型训练时对每个神经元以概率p随机地进行挑选和保留，所以Dropout方法得到了广泛的应用和推广。

总之，Dropout方法可以在一定程度上提高模型的准确性和泛化能力，对于防止过拟合有着较好的效果。但是需要注意的是，Dropout方法会导致训练过程中每个mini-batch的梯度都不同，所以在使用Dropout方法时需要调整学习率，以保证模型的收敛速度和效果。

1. 环境准备

# 导入库
import keras
from keras import layers
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
1
2
3
4
5
6
7

Using TensorFlow backend.
1

2. 导入数据集

# 导入数据集
data = pd.read_csv('./dataset/credit-a.csv', header=None)
data
1
2
3

653 rows × 16 columns

data.iloc[:, -1].unique()
1

array([-1,  1])
1

3. 对所有数据的预测

3.1 数据集

x = data.iloc[:, :-1].values
y = data.iloc[:, -1].replace(-1, 0).values.reshape(-1, 1)
1
2

x.shape, y.shape
1

((653, 15), (653, 1))
1

3.2 构建神经网络

model = keras.Sequential()
model.add(layers.Dense(128, input_dim=15, activation='relu'))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
1
2
3
4
5

model.summary()
1

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 128)               2048      
_________________________________________________________________
dense_2 (Dense)              (None, 128)               16512     
_________________________________________________________________
dense_3 (Dense)              (None, 128)               16512     
_________________________________________________________________
dense_4 (Dense)              (None, 1)                 129       
=================================================================
Total params: 35,201
Trainable params: 35,201
Non-trainable params: 0
_________________________________________________________________
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['acc']
)
1
2
3
4

WARNING:tensorflow:From /home/nlp/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
1
2
3

3.3 训练模型

history = model.fit(x, y, epochs=1000)
1

WARNING:tensorflow:From /home/nlp/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

Epoch 1/1000
653/653 [==============================] - 0s 434us/step - loss: 7.5273 - acc: 0.5988
Epoch 2/1000
653/653 [==============================] - 0s 92us/step - loss: 3.7401 - acc: 0.6187
Epoch 3/1000
653/653 [==============================] - 0s 75us/step - loss: 3.6464 - acc: 0.5712
Epoch 4/1000
653/653 [==============================] - 0s 56us/step - loss: 10.2291 - acc: 0.6631
Epoch 5/1000
653/653 [==============================] - 0s 63us/step - loss: 2.0400 - acc: 0.6233
Epoch 6/1000
653/653 [==============================] - 0s 120us/step - loss: 2.4279 - acc: 0.6217
Epoch 7/1000
653/653 [==============================] - 0s 105us/step - loss: 2.3289 - acc: 0.6325
Epoch 8/1000
653/653 [==============================] - 0s 159us/step - loss: 3.2521 - acc: 0.6294
Epoch 9/1000
653/653 [==============================] - 0s 89us/step - loss: 2.6005 - acc: 0.6294
Epoch 10/1000
653/653 [==============================] - 0s 118us/step - loss: 1.3997 - acc: 0.6738
……
Epoch 1000/1000
653/653 [==============================] - 0s 106us/step - loss: 0.2630 - acc: 0.9326
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

3.4 分析模型

history.history.keys()
1

dict_keys(['loss', 'acc'])
1

plt.plot(history.epoch, history.history.get('loss'), c='r')
plt.plot(history.epoch, history.history.get('acc'), c='b')
1
2

[]
1

4. 对未见过数据的预测

4.1 划分数据集

x_train = x[:int(len(x)*0.75)]
x_test = x[int(len(x)*0.75):]
y_train = y[:int(len(x)*0.75)]
y_test = y[int(len(x)*0.75):]
1
2
3
4

x_train.shape, x_test.shape, y_train.shape, y_test.shape
1

((489, 15), (164, 15), (489, 1), (164, 1))
1

4.2 构建神经网络

model = keras.Sequential()
1

model.add(layers.Dense(128, input_dim=15, activation='relu'))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
1
2
3
4

#admam:利用梯度的一阶矩估计和二阶矩估计动态调整每个参数的学习率.
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['acc']
)
1
2
3
4
5

4.3 训练模型

history = model.fit(x_train, y_train, epochs=1000, validation_data=(x_test, y_test))
1

Train on 489 samples, validate on 164 samples
Epoch 1/1000
489/489 [==============================] - 0s 487us/step - loss: 14.4564 - acc: 0.5951 - val_loss: 3.3778 - val_acc: 0.7256
Epoch 2/1000
489/489 [==============================] - 0s 110us/step - loss: 6.0909 - acc: 0.6012 - val_loss: 2.0924 - val_acc: 0.7195
Epoch 3/1000
489/489 [==============================] - 0s 195us/step - loss: 2.4527 - acc: 0.6074 - val_loss: 1.0763 - val_acc: 0.7378
Epoch 4/1000
489/489 [==============================] - 0s 183us/step - loss: 1.0751 - acc: 0.6585 - val_loss: 0.8990 - val_acc: 0.7134
Epoch 5/1000
489/489 [==============================] - 0s 155us/step - loss: 1.2669 - acc: 0.6503 - val_loss: 1.8094 - val_acc: 0.6585
Epoch 6/1000
489/489 [==============================] - 0s 202us/step - loss: 3.6742 - acc: 0.6892 - val_loss: 1.1836 - val_acc: 0.4573
Epoch 7/1000
489/489 [==============================] - 0s 166us/step - loss: 1.7544 - acc: 0.7301 - val_loss: 2.0060 - val_acc: 0.4573
Epoch 8/1000
489/489 [==============================] - 0s 185us/step - loss: 1.4768 - acc: 0.6605 - val_loss: 0.8917 - val_acc: 0.5427
Epoch 9/1000
489/489 [==============================] - 0s 163us/step - loss: 1.6829 - acc: 0.6667 - val_loss: 4.7695 - val_acc: 0.4573
Epoch 10/1000
489/489 [==============================] - 0s 157us/step - loss: 8.4323 - acc: 0.7239 - val_loss: 2.0879 - val_acc: 0.7439
……
Epoch 1000/1000
489/489 [==============================] - 0s 97us/step - loss: 0.0272 - acc: 0.9877 - val_loss: 2.2746 - val_acc: 0.8049
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

4.4 分析模型

history.history.keys()
1

dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])
1

plt.plot(history.epoch, history.history.get('val_acc'), c='r', label='val_acc')
plt.plot(history.epoch, history.history.get('acc'), c='b', label='acc')
plt.legend()
1
2
3

1

model.evaluate(x_train, y_train)
1

489/489 [==============================] - 0s 37us/step
[0.021263938083038197, 0.9897750616073608]
1
2

model.evaluate(x_test, y_test)
1

164/164 [==============================] - 0s 46us/step
[2.274633582976715, 0.8048780560493469]
1
2

过拟合：在训练数据正确率非常高， 在测试数据上比较低

5. 使用Dropout抑制过拟合

5.1 构建神经网络

model = keras.Sequential()
model.add(layers.Dense(128, input_dim=15, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(1, activation='sigmoid'))
1
2
3
4
5
6
7
8

model.summary()
1

Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_9 (Dense)              (None, 128)               2048      
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_10 (Dense)             (None, 128)               16512     
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_11 (Dense)             (None, 128)               16512     
_________________________________________________________________
dropout_3 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_12 (Dense)             (None, 1)                 129       
=================================================================
Total params: 35,201
Trainable params: 35,201
Non-trainable params: 0
________________________________________________________________
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['acc']
)
1
2
3
4

5.2 训练模型

history = model.fit(x_train, y_train, epochs=1000, validation_data=(x_test, y_test))
1

Train on 489 samples, validate on 164 samples
Epoch 1/1000
489/489 [==============================] - 1s 1ms/step - loss: 41.6885 - acc: 0.5378 - val_loss: 9.9666 - val_acc: 0.6768
Epoch 2/1000
489/489 [==============================] - 0s 298us/step - loss: 53.1358 - acc: 0.5358 - val_loss: 11.0265 - val_acc: 0.6951
Epoch 3/1000
489/489 [==============================] - 0s 173us/step - loss: 36.9899 - acc: 0.5828 - val_loss: 11.6578 - val_acc: 0.6890
Epoch 4/1000
489/489 [==============================] - 0s 177us/step - loss: 43.3404 - acc: 0.5808 - val_loss: 7.5652 - val_acc: 0.6890
Epoch 5/1000
489/489 [==============================] - 0s 197us/step - loss: 23.3085 - acc: 0.6196 - val_loss: 7.9913 - val_acc: 0.6890
Epoch 6/1000
489/489 [==============================] - 0s 254us/step - loss: 24.1833 - acc: 0.6155 - val_loss: 5.5747 - val_acc: 0.7073
Epoch 7/1000
489/489 [==============================] - 0s 229us/step - loss: 19.7051 - acc: 0.5890 - val_loss: 5.5711 - val_acc: 0.7012
Epoch 8/1000
489/489 [==============================] - 0s 180us/step - loss: 22.1131 - acc: 0.5849 - val_loss: 7.0290 - val_acc: 0.6890
Epoch 9/1000
489/489 [==============================] - 0s 172us/step - loss: 23.2305 - acc: 0.6115 - val_loss: 4.2624 - val_acc: 0.6951
Epoch 10/1000
……
Epoch 1000/1000
489/489 [==============================] - 0s 137us/step - loss: 0.3524 - acc: 0.8200 - val_loss: 0.7290 - val_acc: 0.7012
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

5.3 分析模型

model.evaluate(x_train, y_train)
1

489/489 [==============================] - 0s 41us/step
[0.3090217998422728, 0.8548057079315186]
1
2

model.evaluate(x_test, y_test)
1

164/164 [==============================] - 0s 64us/step
[0.7289713301309725, 0.7012194991111755]
1
2

plt.plot(history.epoch, history.history.get('val_acc'), c='r', label='val_acc')
plt.plot(history.epoch, history.history.get('acc'), c='b', label='acc')
plt.legend()
1
2
3

1

6. 正则化

l1：loss = s*abs(w1 + w2 + …) + mse

l2：loss = s*(w12 + w22 + …) + mse

from keras import regularizers
1

6.1 神经网络太过复杂容易过拟合

#神经网络太过复杂容易过拟合
model = keras.Sequential()
model.add(layers.Dense(128, kernel_regularizer=regularizers.l2(0.001), input_dim=15, activation='relu'))
model.add(layers.Dense(128, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
model.add(layers.Dense(128, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
1
2
3
4
5
6

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['acc']
)
1
2
3
4

history = model.fit(x_train, y_train, epochs=1000, validation_data=(x_test, y_test))
1

Train on 489 samples, validate on 164 samples
Epoch 1/1000
489/489 [==============================] - 0s 752us/step - loss: 22.2560 - acc: 0.5910 - val_loss: 9.1111 - val_acc: 0.6524
Epoch 2/1000
489/489 [==============================] - 0s 137us/step - loss: 6.8963 - acc: 0.6217 - val_loss: 3.2886 - val_acc: 0.4573
Epoch 3/1000
489/489 [==============================] - 0s 161us/step - loss: 5.0407 - acc: 0.6830 - val_loss: 1.1973 - val_acc: 0.7256
Epoch 4/1000
489/489 [==============================] - 0s 218us/step - loss: 6.6088 - acc: 0.6421 - val_loss: 7.4651 - val_acc: 0.7012
Epoch 5/1000
489/489 [==============================] - 0s 233us/step - loss: 8.3945 - acc: 0.6973 - val_loss: 2.5579 - val_acc: 0.7317
Epoch 6/1000
489/489 [==============================] - 0s 192us/step - loss: 7.0204 - acc: 0.6196 - val_loss: 3.6758 - val_acc: 0.6829
Epoch 7/1000
489/489 [==============================] - 0s 152us/step - loss: 3.9961 - acc: 0.7382 - val_loss: 1.6183 - val_acc: 0.7317
Epoch 8/1000
489/489 [==============================] - 0s 94us/step - loss: 2.3441 - acc: 0.6237 - val_loss: 1.1523 - val_acc: 0.7256
Epoch 9/1000
489/489 [==============================] - 0s 114us/step - loss: 1.8178 - acc: 0.6442 - val_loss: 1.3449 - val_acc: 0.7073
Epoch 10/1000
489/489 [==============================] - 0s 157us/step - loss: 1.6122 - acc: 0.7117 - val_loss: 1.2869 - val_acc: 0.6646
……
Epoch 1000/1000
489/489 [==============================] - 0s 130us/step - loss: 0.1452 - acc: 0.9775 - val_loss: 1.0515 - val_acc: 0.7866
model.evaluate(x_train, y_train)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

model.evaluate(x_train, y_train)
1

489/489 [==============================] - 0s 34us/step
[0.17742264538942426, 0.9611452221870422]
1
2

model.evaluate(x_test, y_test)
1

164/164 [==============================] - 0s 77us/step
[1.0514701096023, 0.7865853905677795]
1
2

6.2 太简单容易欠拟合

#太简单容易欠拟合
model = keras.Sequential()
model.add(layers.Dense(4, input_dim=15, activation='relu'))
model.add(layers.Dense(1,  activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
1
2
3
4
5

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['acc']
)
1
2
3
4

history = model.fit(x_train, y_train, epochs=1000, validation_data=(x_test, y_test))
1

Train on 489 samples, validate on 164 samples
Epoch 1/1000
489/489 [==============================] - 0s 502us/step - loss: 0.6932 - acc: 0.4765 - val_loss: 0.6931 - val_acc: 0.6341
Epoch 2/1000
489/489 [==============================] - 0s 91us/step - loss: 0.6931 - acc: 0.5174 - val_loss: 0.6930 - val_acc: 0.6341
Epoch 3/1000
489/489 [==============================] - 0s 107us/step - loss: 0.6931 - acc: 0.5174 - val_loss: 0.6924 - val_acc: 0.6341
Epoch 4/1000
489/489 [==============================] - 0s 91us/step - loss: 0.6930 - acc: 0.5174 - val_loss: 0.6916 - val_acc: 0.6341
Epoch 5/1000
489/489 [==============================] - 0s 101us/step - loss: 0.6930 - acc: 0.5174 - val_loss: 0.6914 - val_acc: 0.6341
Epoch 6/1000
489/489 [==============================] - 0s 113us/step - loss: 0.6930 - acc: 0.5174 - val_loss: 0.6914 - val_acc: 0.6341
Epoch 7/1000
489/489 [==============================] - 0s 147us/step - loss: 0.6929 - acc: 0.5174 - val_loss: 0.6908 - val_acc: 0.6341
Epoch 8/1000
489/489 [==============================] - 0s 166us/step - loss: 0.6929 - acc: 0.5174 - val_loss: 0.6905 - val_acc: 0.6341
Epoch 9/1000
489/489 [==============================] - 0s 162us/step - loss: 0.6929 - acc: 0.5174 - val_loss: 0.6904 - val_acc: 0.6341
Epoch 10/1000
489/489 [==============================] - 0s 129us/step - loss: 0.6928 - acc: 0.5174 - val_loss: 0.6901 - val_acc: 0.6341
……
Epoch 1000/1000
489/489 [==============================] - 0s 86us/step - loss: 0.6926 - acc: 0.5174 - val_loss: 0.6849 - val_acc: 0.6341
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

model.evaluate(x_train, y_train)
1

489/489 [==============================] - 0s 43us/step
[0.6925447341854587, 0.5173823833465576]
1
2

model.evaluate(x_test, y_test)
1

164/164 [==============================] - 0s 39us/step
[0.684889389247429, 0.6341463327407837]
1
2

6.3 选取适当的神经网络

# 选取适当的神经网络
model = keras.Sequential()
model.add(layers.Dense(4, input_dim=15, activation='relu'))
model.add(layers.Dense(4,  activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
1
2
3
4
5

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['acc']
)
1
2
3
4

history = model.fit(x_train, y_train, epochs=1000, validation_data=(x_test, y_test))
1

Train on 489 samples, validate on 164 samples
Epoch 1/1000
489/489 [==============================] - 0s 575us/step - loss: 40.1825 - acc: 0.5317 - val_loss: 17.6376 - val_acc: 0.6098
Epoch 2/1000
489/489 [==============================] - 0s 104us/step - loss: 30.0785 - acc: 0.5337 - val_loss: 12.6986 - val_acc: 0.6159
Epoch 3/1000
489/489 [==============================] - 0s 148us/step - loss: 20.0469 - acc: 0.5112 - val_loss: 8.3732 - val_acc: 0.5671
Epoch 4/1000
489/489 [==============================] - 0s 151us/step - loss: 12.5171 - acc: 0.4908 - val_loss: 3.8925 - val_acc: 0.5061
Epoch 5/1000
489/489 [==============================] - 0s 113us/step - loss: 4.4324 - acc: 0.4294 - val_loss: 0.9156 - val_acc: 0.4573
Epoch 6/1000
489/489 [==============================] - 0s 79us/step - loss: 1.0313 - acc: 0.5419 - val_loss: 0.9974 - val_acc: 0.4695
Epoch 7/1000
489/489 [==============================] - 0s 88us/step - loss: 1.0071 - acc: 0.5562 - val_loss: 0.8852 - val_acc: 0.5183
Epoch 8/1000
489/489 [==============================] - 0s 88us/step - loss: 0.9085 - acc: 0.5808 - val_loss: 0.7934 - val_acc: 0.5366
Epoch 9/1000
489/489 [==============================] - 0s 107us/step - loss: 0.8235 - acc: 0.5992 - val_loss: 0.7390 - val_acc: 0.5366
Epoch 10/1000
489/489 [==============================] - 0s 114us/step - loss: 0.7711 - acc: 0.5971 - val_loss: 0.7174 - val_acc: 0.5366
……
Epoch 1000/1000
489/489 [==============================] - 0s 141us/step - loss: 0.3095 - acc: 0.8732 - val_loss: 0.3971 - val_acc: 0.8537
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

model.evaluate(x_train, y_train)
1

489/489 [==============================] - 0s 68us/step
[0.30120014958464536, 0.8813905715942383]
1
2

model.evaluate(x_test, y_test)
1

164/164 [==============================] - 0s 45us/step
[0.39714593858253666, 0.8536585569381714]
1
2

附：系列文章