-
K8S- Deployment 的滚动更新 Rolling Update
滚动更新
这里的更新指的不是更新deployment 本身的属性(label/ replicas)等， 而是更新POD 的container 的版本
更新方法通常有两种
- 是直接update deployment配置， 注意只有update了template中的内容(与container相关) 才会触发更新
- 用kubectl set image 命令
构造新版本的service
为了更好地测试，
我们构造1个新的版本的spring boot的service更新很简单
直接更新pom里面的version就好， 由1.1.1 to 1.1.2<groupId>com.homegroupId> <artifactId>bq_apiartifactId> <version>1.1.2version> <name>bq-api-servicename>- 1
- 2
- 3
- 4
里面已经利用springboot actuator 构造了1个接口/actuator/info 可以获得当前版本
[gateman@manjaro-x13 ~]$ curl 127.0.0.1:8080/actuator/info {"app":"Sales API","version":"1.1.2","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}- 1
- 2
当把docker image 推送到GAR后
可以用下面命令确认[gateman@manjaro-x13 ~]$ gcloud artifacts docker images list europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo --include-tags Listing items under project jason-hsbc, location europe-west2, repository my-docker-repo. IMAGE DIGEST TAGS CREATE_TIME UPDATE_TIME SIZE europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:30fb2cebd2bf82863608037ce41048114c061acbf1182261a748dadefff2372f 2024-03-18T02:45:02 2024-03-18T02:45:02 366980350 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:55737b9c2efc7115d878c8b8b036dd0fe8ae5e66b2154fe05ba015b28a31c7aa 1.0.0 2024-03-18T01:40:34 2024-03-18T01:40:34 366980348 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:569397b5d13264988800ad5af15359b4ab085eaca209484e6d859fc2a9e6b6ab 1.1.1 2024-03-31T12:03:57 2024-03-31T12:03:57 366981699 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:648920b7aba0b5cdeeca9e1237d9eeb62ee29188801a95adf7619bcee94e9eb1 2024-03-15T02:29:55 2024-03-15T02:29:55 367470655 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:6c4129c5a6938004b14cc555518b52a18e8177ce6940b52db7c514783ef9325b 1.1.2 2024-04-04T21:08:09 2024-04-04T21:08:09 366981681 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:7db14476085376db27a2814f8348fdbe002516c576d396c02030094c859f279c 2024-03-15T20:51:58 2024-03-15T20:51:58 367470658 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:88c7f68df0dd0508b551d53ad22cf4395fea233b5e77be016f13ce9c8c4401fa 2024-03-30T00:12:15 2024-03-30T00:12:15 366981231 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:a42cd0cd3a3483599b17d94022d1ccd234003491db217b4055f64997478149e1 2024-03-15T00:03:21 2024-03-15T00:03:21 367470619 europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service sha256:f1e5a8064b134aedd1a5840c16af85633e98a9649c830b6a3f6788a8b00bf16f 2024-03-25T00:35:48 2024-03-25T00:35:48 366980337- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
可以看见到1.1.2 的image 已经在GAR了
update deployment配置 测试
先看当前的状态
root@k8s-master:~# kubectl get deploy -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 10/10 10 10 3h27m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service app=bq-api-service root@k8s-master:~# kubectl get rs -o wide --show-labels NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample-9f8d9c988 10 10 10 3h27m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service,pod-template-hash=9f8d9c988 app=bq-api-service,pod-template-hash=9f8d9c988 root@k8s-master:~# kubectl get pods -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS bq-api-service-deploy-sample-9f8d9c988-25tqr 1/1 Running 0 3h27m 10.244.1.20 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-67psf 1/1 Running 0 3h27m 10.244.3.30 k8s-node3 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-cvm4z 1/1 Running 0 3h27m 10.244.3.32 k8s-node3 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-f77tx 1/1 Running 0 3h27m 10.244.1.23 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-f98s6 1/1 Running 0 3h27m 10.244.2.82 k8s-node0 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-g6627 1/1 Running 0 3h27m 10.244.1.22 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-jlc9w 1/1 Running 0 3h27m 10.244.3.31 k8s-node3 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-mnqmf 1/1 Running 0 3h27m 10.244.2.80 k8s-node0 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-pw468 1/1 Running 0 3h27m 10.244.2.81 k8s-node0 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-v565f 1/1 Running 0 3h27m 10.244.1.21 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 root@k8s-master:~#- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
可以见到只有1个deploy 对象， 1个rs 对象， 和 10 个pods, 版本是1.1.1
测试1, 只更新 deployment 的label
命令
kubectl edit deployment bq-api-service-deploy-sample
结果， 单纯地添加了1个label在deployment 对象， rs 和 pod 都没有触发更新
root@k8s-master:~# kubectl edit deployment bq-api-service-deploy-sample deployment.apps/bq-api-service-deploy-sample edited root@k8s-master:~# kubectl get deploy -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 10/10 10 10 3h38m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service app=bq-api-service,author=Gateman root@k8s-master:~# kubectl get rs -o wide --show-labels NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample-9f8d9c988 10 10 10 3h38m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service,pod-template-hash=9f8d9c988 app=bq-api-service,pod-template-hash=9f8d9c988 root@k8s-master:~# kubectl get pods -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS bq-api-service-deploy-sample-9f8d9c988-25tqr 1/1 Running 0 3h39m 10.244.1.20 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-67psf 1/1 Running 0 3h39m 10.244.3.30 k8s-node3 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-cvm4z 1/1 Running 0 3h39m 10.244.3.32 k8s-node3 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-f77tx 1/1 Running 0 3h39m 10.244.1.23 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-f98s6 1/1 Running 0 3h39m 10.244.2.82 k8s-node0 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-g6627 1/1 Running 0 3h39m 10.244.1.22 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-jlc9w 1/1 Running 0 3h39m 10.244.3.31 k8s-node3 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-mnqmf 1/1 Running 0 3h39m 10.244.2.80 k8s-node0 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-pw468 1/1 Running 0 3h39m 10.244.2.81 k8s-node0 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-v565f 1/1 Running 0 3h39m 10.244.1.21 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=9f8d9c988 root@k8s-master:~#- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
注意， 也可以更新本地的yaml ， 然后用kubectl apply -f xxx.yaml 来更新
root@k8s-master:~/k8s-s/deployments# git pull hint: Pulling without specifying how to reconcile divergent branches is hint: discouraged. You can squelch this message by running one of the following hint: commands sometime before your next pull: hint: hint: git config pull.rebase false # merge (the default strategy) hint: git config pull.rebase true # rebase hint: git config pull.ff only # fast-forward only hint: hint: You can replace "git config" with "git config --global" to set a default hint: preference for all repositories. You can also pass --rebase, --no-rebase, hint: or --ff-only on the command line to override the configured default per hint: invocation. remote: Enumerating objects: 7, done. remote: Counting objects: 100% (7/7), done. remote: Compressing objects: 100% (4/4), done. remote: Total 4 (delta 3), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (4/4), 322 bytes | 6.00 KiB/s, done. From e.coding.net:nvd11/k8s/k8s-s 12b5990..f24fd58 master -> origin/master Updating 12b5990..f24fd58 Fast-forward deployments/bq-api-service-sample.yaml | 1 + 1 file changed, 1 insertion(+) root@k8s-master:~/k8s-s/deployments# kubectl apply -f bq-api-service-sample.yaml deployment.apps/bq-api-service-deploy-sample configured- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
测试2, 只更新 deployment 的replicas 期望副本数量
replicas: 10 -> 12
结果是 deployment 和 rs 的属性都更新了， 而且pod 也增加了两个， 但是实际上rs 的hash 没变， 还是那个RS, 而且pod 的版本仍然是1.1.1 并没有触发滚动更新
oot@k8s-master:~/k8s-s/deployments# kubectl edit deployment bq-api-service-deploy-sample deployment.apps/bq-api-service-deploy-sample edited root@k8s-master:~/k8s-s/deployments# kubectl get deploy -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 12/12 12 12 3h47m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~/k8s-s/deployments# kubectl get rs -o wide --show-labels NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample-9f8d9c988 12 12 12 3h47m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service,pod-template-hash=9f8d9c988 app=bq-api-service,pod-template-hash=9f8d9c988 root@k8s-master:~/k8s-s/deployments# kubectl get po -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS bq-api-service-deploy-sample-9f8d9c988-25tqr 1/1 Running 0 3h48m 10.244.1.20 k8s-node1app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-2zlnw 1/1 Running 0 81s 10.244.2.83 k8s-node0 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-67psf 1/1 Running 0 3h48m 10.244.3.30 k8s-node3 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-cvm4z 1/1 Running 0 3h48m 10.244.3.32 k8s-node3 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-f77tx 1/1 Running 0 3h48m 10.244.1.23 k8s-node1 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-f98s6 1/1 Running 0 3h48m 10.244.2.82 k8s-node0 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-g6627 1/1 Running 0 3h48m 10.244.1.22 k8s-node1 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-jlc9w 1/1 Running 0 3h48m 10.244.3.31 k8s-node3 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-mnqmf 1/1 Running 0 3h48m 10.244.2.80 k8s-node0 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-pw468 1/1 Running 0 3h48m 10.244.2.81 k8s-node0 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-qntj8 1/1 Running 0 81s 10.244.3.33 k8s-node3 app=bq-api-service,pod-template-hash=9f8d9c988 bq-api-service-deploy-sample-9f8d9c988-v565f 1/1 Running 0 3h48m 10.244.1.21 k8s-node1 app=bq-api-service,pod-template-hash=9f8d9c988 - 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
测试3, 只更新 deployment 的 template.container.image的version
结果， 触发了滚动更新
这里我多次执行了 get deploy 命令， 可以见到version 由1.1.1 逐渐变成了1.1.2至于ReplicaSet 则是直接多了1个， 足以看出k8s 是新建1个 RS 作为临时的POD 容器进行滚动更新， 当新的RS 完成所有pod 更新时， 旧的RS 就被丢弃放在一边了
root@k8s-master:~# kubectl get deployments -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 12/12 12 12 3h52m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~# kubectl get deployments -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 12/12 12 12 3h52m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~# kubectl get deployments -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 9/12 12 9 3h54m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~# kubectl get deployments -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 9/12 12 9 3h54m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~# kubectl get deployments -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 11/12 12 11 3h54m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~# kubectl get deployments -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 12/12 12 12 3h54m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~# kubectl get deployments -o wide --show-labels NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample 12/12 12 12 3h54m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 app=bq-api-service app=bq-api-service,author=Jason root@k8s-master:~# kubectl get rs -o wide --show-labels NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR LABELS bq-api-service-deploy-sample-8d49d9845 12 12 12 40s bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 app=bq-api-service,pod-template-hash=8d49d9845 app=bq-api-service,pod-template-hash=8d49d9845 bq-api-service-deploy-sample-9f8d9c988 0 0 0 3h55m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service,pod-template-hash=9f8d9c988 app=bq-api-service,pod-template-hash=9f8d9c988 root@k8s-master:~#- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
kubectl set image 命令更新测试
kubectl set image deployment/
=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:<新版本号> root@k8s-master:~/k8s-s/deployments# kubectl set image deployment/bq-api-service-deploy-sample bq-api-service-container=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.3 deployment.apps/bq-api-service-deploy-sample image updated root@k8s-master:~# curl 10.244.2.91:8080/actuator/info {"app":"Sales API","version":"1.1.3","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}root@k8s-master:~#- 1
- 2
- 3
- 4
- 5
查看deployment 更新历史
- 利用 Kubectl rollout 命令
root@k8s-master:~/k8s-s/deployments# kubectl rollout history deployment/bq-api-service-deploy-sample deployment.apps/bq-api-service-deploy-sample REVISION CHANGE-CAUSE 1 <none> 2 <none> 3 <none> root@k8s-master:~/k8s-s/deployments# kubectl rollout history deployment/bq-api-service-deploy-sample --revision=2 deployment.apps/bq-api-service-deploy-sample with revision #2 Pod Template: Labels: app=bq-api-service pod-template-hash=8d49d9845 Containers: bq-api-service-container: Image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 Port: <none> Host Port: <none> Environment: <none> Mounts: <none> Volumes: <none>- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 查看rs 的数量和版本
root@k8s-master:~/k8s-s/deployments# kubectl get rs -o wide -l app=bq-api-service NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR bq-api-service-deploy-sample-7bff8fbc4b 12 12 12 6m11s bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.3 app=bq-api-service,pod-template-hash=7bff8fbc4b bq-api-service-deploy-sample-8d49d9845 0 0 0 13m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.2 app=bq-api-service,pod-template-hash=8d49d9845 bq-api-service-deploy-sample-9f8d9c988 0 0 0 4h8m bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.1 app=bq-api-service,pod-template-hash=9f8d9c988- 1
- 2
- 3
- 4
- 5
这里的-l 就是label selector 的意思啦！
-
相关阅读:
rocketmq是如何消费
Vue + element-ui 【前端项目一】初始化创建项目 1
mixup--学习笔记
Vue 中什么阶段（生命周期）才能访问操作dom？为什么？
一周时间深扒事务 总结代码演示篇 拿捏事务
如何压缩视频？视频压缩变小方法汇总
c++ 指针
spring webflux的简单使用
Variational graph auto-encoders (VGAE)
相机投影矩阵计算
- 原文地址：https://blog.csdn.net/nvd11/article/details/137382092
