k8s异常：The node was low on resource: [DiskPressure].

1、现象

node2节点上所有Pod都无法正常工作，kube-proxy都有异常，登陆dashboard页面在容器组里面下是有The node was low on resource: [DiskPressure]. 报错信息说是节点资源不足，disk磁盘的意思

登陆在node2服务器节点上,根节点达到85% 使用率

/dev/mapper/centos-root  6.2G  5.3G  994M   85% /
1

在通过node2上的msg日志,磁盘达到85%，需要释放磁盘空间低于80%

Oct  9 13:37:23 k8s-node2 kubelet: I1009 13:37:23.861570     658 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 85% which is over the high threshold (85%). Trying to free 287558041 bytes down to the low threshold (80%).
1

通过过滤关键字

[root@k8s-node2 log]# grep Percentage /var/log/messages



Oct 12 09:37:36 uat-181 kubelet: I1012 09:37:36.113821   15318 container_manager_linux.go:253] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:5 Percentage:0} GracePeriod:0s MinReclaim:}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms}

1
2
3
4
5
6

2、解决方法

node的kubelet负责顶起采集资源占用数据，并和预先设置的threshold值进行比较，如果超过threshold值，kubelet会杀掉一些Pod来回收相关资源，K8sg官网解读kubernetes配置资源不足处理

考虑到磁盘空余85% k8s就运行异常，从上面nodefs.available percentage: 0.15也就是说nodees的可用空间要低于15%的时候，该节点上讲会执行eviction操作，由于磁盘已经达到了85%,在怎么驱逐也无法正常启动就会一直重启，Pod状态也是pending中

修改配置文件增加传参数,添加此配置项–eviction-hard=nodefs.available<5%

[root@k8s-node01 ~]#  systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In:/usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf

[root@k8s-node01 ~]# vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 


# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --eviction-hard=nodefs.available<5%"
1
2
3
4
5
6
7
8
9
10
11
12

重启kubelet,验证修改值是否生效。

root@k8s-node02 ~]#systemctl daemon-reload
[root@k8s-node02 ~]#systemctl  restart kubelet


[root@k8s-node01 ~]# grep Percentage  /var/log/messages #最新日志，已经是0.05 也就是剩余5%,才会出现驱逐的情况。
{Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.05}
1
2
3
4
5
6

最终还是得配合清除业务程序日志，减少磁盘得使用率。