deployment滚动更新时副本减少的分析

1 概述

1.1 环境

版本信息如下：
a、操作系统：centos 7.6
b、kubernetes版本：v1.15.0

1.2 滚动原理

deployment的.spec.strategy字段来设置滚动更新策略。rollingUpdate字段下有两个字段可以设置：maxSurge和maxUnavailable，分别表示滚动更新过程中超越.spec.replicas的最大数量和最小可用副本数量。针对一个deployment对象，在滚动更新时，版本最新的rs对象的.spec.replicas的值会逐渐增大，所有旧版本的rs对象的.spec.replicas的值会逐渐缩小，因此滚动更新的本质只是调节新旧rs对象的.spec.replicas的值，maxSurge和maxUnavailable是为了给副本范围做一个“框定”。

Check if we can scale down. We can scale down in the following 2 cases:
1) Some old replica sets have unhealthy replicas, we could safely scale down those unhealthy replicas since that won't further increase unavailability.
2) New replica set has scaled up and it's replicas becomes ready, then we can scale down old replica sets in a further step.

maxScaledDown := allPodsCount - minAvailable - newReplicaSetPodsUnavailable
1
2
3
4
5

滚动更新时，增加副本不会减少微服务的可用性，但减少副本会导致微服务的可用性变弱（并且新增的副本是冷的，不能等效于运行很久的pod，微服务可用性进一步弱化），因此减少副本的过程往往更被关心。

2 滚动更新的主流程

先尝试扩容，后尝试缩容，一个周期只扩容或者缩容。

func (dc *DeploymentController) rolloutRolling(d *apps.Deployment, rsList []*apps.ReplicaSet) error {
	// 获取最新版本rs对象和所有旧版本对象
	newRS, oldRSs, err := dc.getAllReplicaSetsAndSyncRevision(d, rsList, true)
	if err != nil {
		return err
	}
	allRSs := append(oldRSs, newRS)

	// 尝试扩容，如果可以的话
	scaledUp, err := dc.reconcileNewReplicaSet(allRSs, newRS, d)
	if err != nil {
		return err
	}
	// 如果确实发生了扩容（新版本rs对象的.spec.replicas增大），则更新deployment对象的.status字段并退出方法
	if scaledUp {
		return dc.syncRolloutStatus(allRSs, newRS, d)
	}

	// 尝试缩容，如果可以的话
	scaledDown, err := dc.reconcileOldReplicaSets(allRSs, controller.FilterActiveReplicaSets(oldRSs), newRS, d)
	if err != nil {
		return err
	}
	// 如果确实发生了缩容（新版本rs对象的.spec.replicas减小），则更新deployment对象的.status字段并退出方法
	if scaledDown {
		// Update DeploymentStatus
		return dc.syncRolloutStatus(allRSs, newRS, d)
	}

	// deployment下的副本都完成更新，则清理deployment底下的旧版本rs对象
	if deploymentutil.DeploymentComplete(d, &d.Status) {
		if err := dc.cleanupDeployment(oldRSs, d); err != nil {
			return err
		}
	}

	// 更新deployment对象的.status字段
	return dc.syncRolloutStatus(allRSs, newRS, d)
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

3 什么条件下会发生实际的缩容操作

在reconcileOldReplicaSets( )中，
maxScaledDown := allPodsCount - minAvailable - newRSUnavailablePodCount
其中allPodsCount是所有rs对象的.spec.replicas之和，minAvailable是最小可用副本数，newRSUnavailablePodCount是最新版本rs对象的非可用副本的数量。

如果maxScaledDown小于等于0，则不会发生实际的缩容操作，因为没机会执行到方法dc.scaleDownOldReplicaSetsForRollingUpdate(allRSs, oldRSs, deployment)。一个显著的情景是新版本rs对象的pod处于unavailable状态，有机会导致maxScaledDown小于等于0，如此一来就不会发生实际的缩容操作。

func (dc *DeploymentController) reconcileOldReplicaSets(allRSs []*apps.ReplicaSet, oldRSs []*apps.ReplicaSet, newRS *apps.ReplicaSet, deployment *apps.Deployment) (bool, error) {
	/*
		其他代码
	*/
	
	// maxScaledDown小于等于0，则直接退出方法，不会发生缩容
	maxScaledDown := allPodsCount - minAvailable - newRSUnavailablePodCount
	if maxScaledDown <= 0 {
		return false, nil
	}


	// 下面开始缩容
	
	/*
		步骤1：对包含不健康pod的rs对象进行缩容
		oldRSs, cleanupCount, err := dc.cleanupUnhealthyReplicas(oldRSs, deployment, maxScaledDown)
	*/

	/*
		步骤2：对旧版本rs对象进行缩容，入参oldRSs在步骤1中已被更新
	*/
	allRSs = append(oldRSs, newRS)
	scaledDownCount, err := dc.scaleDownOldReplicaSetsForRollingUpdate(allRSs, oldRSs, deployment)
	

	// 缩容总数
	totalScaledDown := cleanupCount + scaledDownCount
	return totalScaledDown > 0, nil
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

4 一次缩容操作时的缩容上限

在scaleDownOldReplicaSetsForRollingUpdate( )中，
totalScaleDownCount := availablePodCount - minAvailable
其中availablePodCount是所有版本的rs对象的所有可用pod数量之和，minAvailable是最小可用的pod数量，可见availablePodCount越大，能缩容的上限就越大。
算出totalScaleDownCount 后，遍历所有旧版本rs对象进行缩容，被缩的副本数量之和不超过totalScaleDownCount。

5 日志例子分析

期望副本数量是10，maxUnavailable是8，maxSurge是3。

rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 0 available pods.

# 进入了reconcileOldReplicaSets(  )方法，
# 由于maxScaledDown大于0（13 - 8 - 3），则发生缩容，其实此时连一个新版本的pod都没有available
rolling.go:169] Found 10 available pods in old RS default/echo-gz-57597fbff9
rolling.go:140] Cleaned up unhealthy replicas from old RSes by 0
rolling.go:203] Found 10 available pods in deployment echo-gz, scaling down old RSes
# 对旧版本rs缩容2个，因为 totalScaleDownCount := 10 - 8
rolling.go:148] Scaled down old RSes of deployment echo-gz by 2

# 虽然再次进入了reconcileOldReplicaSets(  )方法，
# 由于maxScaledDown小于等于0（11 - 8 - 3），不会发生实际的缩容
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 0 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 0 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 0 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 0 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 0 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 0 available pods.

# 再次进入了reconcileOldReplicaSets(  )方法，
# 由于maxScaledDown大于0（11 - 8 - 2），则发生缩容
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 1 available pods.
rolling.go:169] Found 8 available pods in old RS default/echo-gz-57597fbff9
rolling.go:140] Cleaned up unhealthy replicas from old RSes by 0
rolling.go:203] Found 9 available pods in deployment echo-gz, scaling down old RSes
# 对旧版本rs缩容1个，因为 totalScaleDownCount := 9 - 8
rolling.go:148] Scaled down old RSes of deployment echo-gz by 1

# 虽然再次进入了reconcileOldReplicaSets(  )方法，
# 由于maxScaledDown小于等于0（10 - 8 - 2），不会发生实际的缩容
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 1 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 1 available pods.

# 再次进入了reconcileOldReplicaSets(  )方法，
# 由于maxScaledDown大于0（10 - 8 - 1），则发生缩容
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 2 available pods.
rolling.go:169] Found 7 available pods in old RS default/echo-gz-57597fbff9
rolling.go:140] Cleaned up unhealthy replicas from old RSes by 0
rolling.go:203] Found 9 available pods in deployment echo-gz, scaling down old RSes
# 对旧版本rs缩容1个，因为 totalScaleDownCount := 9 - 8
rolling.go:148] Scaled down old RSes of deployment echo-gz by 1

# 虽然再次进入了reconcileOldReplicaSets(  )方法，
# 由于maxScaledDown小于等于0（9 - 8 - 1），不会发生实际的缩容
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 2 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 2 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 2 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 2 available pods.
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 2 available pods.

# 再次进入了reconcileOldReplicaSets(  )方法，
# 由于maxScaledDown大于0（9 - 8 - 0），则发生缩容
rolling.go:94] New replica set default/echo-gz-5f85ccdcc9 has 3 available pods.
rolling.go:169] Found 6 available pods in old RS default/echo-gz-57597fbff9
rolling.go:140] Cleaned up unhealthy replicas from old RSes by 0
rolling.go:203] Found 9 available pods in deployment echo-gz, scaling down old RSes
# 对旧版本rs缩容1个，因为 totalScaleDownCount := 9 - 8
rolling.go:148] Scaled down old RSes of deployment echo-gz by 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

6 小结

totalScaleDownCount := availablePodCount - minAvailable，availablePodCount是所有版本的rs对象的所有可用pod数量之和，availablePodCount越大则一次缩容数量的上限就越大。而maxSurge越大，则大概率让availablePodCount越大，因此一次缩容数量越大，微服务的可用性在短时间内变弱。