• hdfs - balancer学习

    balance是啥，顾名思义 是个平衡器

    主要是平衡各个datanode之间的使用

     网上的文档一个比一个写的6结果，有的命令都拼错了。。。而且你知道究竟平衡的是啥么

    直接上官网

    Apache Hadoop 3.2.2 – HDFS Commands Guideicon-default.png?t=M5H6https://hadoop.apache.org/docs/r3.2.2/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer

    --查看balance 也就是集群之间转移数据的速度

    hdfs dfsadmin -getBalancerBandwidth node17:9867 

    Balancer bandwidth is 10485760 bytes per second.  --10M嫌慢 设置20M

    这里权限有点问题。。。认证hdfs

    hdfs dfsadmin -setBalancerBandwidth 20971520

    1. [root@worker01 /home/devuser]# hdfs dfsadmin -setBalancerBandwidth  20971520
    2. NumberFormatException: For input string: " 20971520"
    3. Usage: hdfs dfsadmin [-setBalancerBandwidth <bandwidth in bytes per second>]
    4. [root@worker01 /home/devuser]# hdfs dfsadmin -setBalancerBandwidth 20971520
    5. setBalancerBandwidth: Access denied for user hive. Superuser privilege is required
    6. [root@worker01 /home/devuser]# kinit hdfs 
    7. Password for hdfs@CDH.COM: 
    8. [root@worker01 /home/devuser]# hdfs dfsadmin -setBalancerBandwidth 20971520
    9. [root@worker01 /home/devuser]# hdfs dfsadmin -setBalancerBandwidth 20971520
    10. Balancer bandwidth is set to 20971520 for master.data.com/9.134.64.234:8020
    11. Balancer bandwidth is set to 20971520 for node01.data.com/9.134.66.48:8020
    12.

    --这里的时候我遇到一个问题 ip我知道，这个端口是啥。。但是我注意这个是ipc_port 

    开始准备balancer

    1. [root@worker01 /home/devuser]# hdfs balancer --help
    2. Usage: hdfs balancer
    3. 	[-policy <policy>]	the balancing policy: datanode or blockpool
    4. 	[-threshold <threshold>]	Percentage of disk capacity
    5. 	[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]	Excludes the specified datanodes.
    6. 	[-include [-f <hosts-file> | <comma-separated list of hosts>]]	Includes only the specified datanodes.
    7. 	[-source [-f <hosts-file> | <comma-separated list of hosts>]]	Pick only the specified datanodes as source nodes.
    8. 	[-blockpools <comma-separated list of blockpool ids>]	The balancer will only run on blockpools included in this list.
    9. 	[-idleiterations <idleiterations>]	Number of consecutive idle iterations (-1 for Infinite) before exit.
    10. 	[-runDuringUpgrade]	Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.
    12. Generic options supported are:
    13. -conf <configuration file>        specify an application configuration file
    14. -D <property=value>               define a value for a given property
    15. -fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
    16. -jt <local|resourcemanager:port>  specify a ResourceManager
    17. -files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
    18. -libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
    19. -archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machines
    20.  
    21. The general command line syntax is:
    22. command [genericOptions] [commandOptions]

     这时候要考虑一个问题？怎么样才算平衡？

    比如10个dn，每个100G容量，共计1T， 总共使用了200G 其中 dn1使用了1k dn2使用了99G

    那么我要怎么平衡？ dn1和dn2平衡到20G还是50G。我也不知道。

    开始实验

    threshold 

    hdfs balancer -threshold 5  --阈值=5 也就是容忍datanode数据的差距是5%

    1. [root@worker01 /home/devuser]# hdfs balancer -threshold 5
    2. 22/06/27 11:29:18 INFO balancer.Balancer: Using a threshold of 5.0
    3. 22/06/27 11:29:18 INFO balancer.Balancer: namenodes  = [hdfs://s2cluster]
    4. 22/06/27 11:29:18 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
    5. 22/06/27 11:29:18 INFO balancer.Balancer: included nodes = []
    6. 22/06/27 11:29:18 INFO balancer.Balancer: excluded nodes = []
    7. 22/06/27 11:29:18 INFO balancer.Balancer: source nodes = []
    8. Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
    9. 22/06/27 11:29:19 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
    10. 22/06/27 11:29:19 INFO block.BlockTokenSecretManager: Setting block keys
    11. 22/06/27 11:29:19 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
    12. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
    13. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
    14. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
    15. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 50 (default=50)
    16. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
    17. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
    18. 22/06/27 11:29:19 INFO block.BlockTokenSecretManager: Setting block keys
    19. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
    20. 22/06/27 11:29:19 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)
    21. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.117.90:1004
    22. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.68.200:1004
    23. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.80.60:1004
    24. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.81.221:1004
    25. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.124.14:1004
    26. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.122.87:1004
    27. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.83.33:1004
    28. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.163.60:1004
    29. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.115.141:1004
    30. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.124.36:1004
    31. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.71.192:1004
    32. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.123.37:1004
    33. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.117.73:1004
    34. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.163.238:1004
    35. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.116.73:1004
    36. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.77.69:1004
    37. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.123.190:1004
    38. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.81.127:1004
    39. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.163.174:1004
    40. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.80.64:1004
    41. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.122.224:1004
    42. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.123.215:1004
    43. 22/06/27 11:29:19 INFO net.NetworkTopology: Adding a new node: /default/9.134.122.6:1004
    44. 22/06/27 11:29:19 INFO balancer.Balancer: 3 over-utilized: [9.134.117.90:1004:DISK, 9.134.124.14:1004:DISK, 9.134.116.73:1004:DISK]
    45. 22/06/27 11:29:19 INFO balancer.Balancer: 5 underutilized: [9.134.80.60:1004:DISK, 9.134.81.221:1004:DISK, 9.134.124.36:1004:DISK, 9.134.81.127:1004:DISK, 9.134.80.64:1004:DISK]
    46. 22/06/27 11:29:19 INFO balancer.Balancer: Need to move 51.86 GB to make the cluster balanced.
    47. 22/06/27 11:29:19 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => underUtilized
    48. 22/06/27 11:29:19 INFO balancer.Balancer: Decided to move 10 GB bytes from 9.134.117.90:1004:DISK to 9.134.80.60:1004:DISK
    49. 22/06/27 11:29:19 INFO balancer.Balancer: Decided to move 10 GB bytes from 9.134.124.14:1004:DISK to 9.134.81.221:1004:DISK
    50. 22/06/27 11:29:19 INFO balancer.Balancer: Decided to move 10 GB bytes from 9.134.116.73:1004:DISK to 9.134.124.36:1004:DISK
    51. 22/06/27 11:29:19 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => belowAvgUtilized
    52. 22/06/27 11:29:19 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: underUtilized => aboveAvgUtilized
    53. 22/06/27 11:29:19 INFO balancer.Balancer: Decided to move 10 GB bytes from 9.134.68.200:1004:DISK to 9.134.81.127:1004:DISK
    54. 22/06/27 11:29:19 INFO balancer.Balancer: Decided to move 10 GB bytes from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK
    55. 22/06/27 11:29:19 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => underUtilized
    56. 22/06/27 11:29:19 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => belowAvgUtilized
    57. 22/06/27 11:29:19 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: underUtilized => aboveAvgUtilized
    58. 22/06/27 11:29:19 INFO balancer.Balancer: Will move 50 GB in this iteration
    59. 22/06/27 11:29:19 INFO balancer.Dispatcher: Limiting threads per target to the specified max.
    60. 22/06/27 11:29:19 INFO balancer.Dispatcher: Allocating 50 threads per target.
    61. 22/06/27 11:29:19 INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
    62. 	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88)
    63. 	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1962)
    64. 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1421)
    65. 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocks(FSNamesystem.java:1716)
    66. 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlocks(NameNodeRpcServer.java:631)
    67. 	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.getBlocks(NamenodeProtocolServerSideTranslatorPB.java:89)
    68. 	at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12966)
    69. 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
    70. 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
    71. 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
    72. 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
    73. 	at java.security.AccessController.doPrivileged(Native Method)
    74. 	at javax.security.auth.Subject.doAs(Subject.java:422)
    75. 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
    76. 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
    77. , while invoking NamenodeProtocolTranslatorPB.getBlocks over master.data.com/9.134.64.234:8020. Trying to failover immediately.
    78. 22/06/27 11:29:19 WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #13 NamenodeProtocolTranslatorPB.getBlocks over node01.data.com/9.134.66.48:8020
    79. 22/06/27 11:29:19 WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #16 NamenodeProtocolTranslatorPB.getBlocks over node01.data.com/9.134.66.48:8020
    80. 22/06/27 11:29:19 WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #14 NamenodeProtocolTranslatorPB.getBlocks over node01.data.com/9.134.66.48:8020
    81. 22/06/27 11:29:19 WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #12 NamenodeProtocolTranslatorPB.getBlocks over node01.data.com/9.134.66.48:8020
    82. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090172187_16434653 with size=112350808 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    83. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090197454_16459920 with size=134217728 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    84. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090172224_16434690 with size=92790751 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    85. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090200946_16463412 with size=134217728 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    86. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090243729_16506195 with size=10589105 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    87. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090383941_16646407 with size=73175462 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    88. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090243736_16506202 with size=10589105 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.68.200:1004
    89. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090384199_16646665 with size=265221529 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    90. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1090374653_16637119 with size=15559033 from 9.134.122.87:1004:DISK to 9.134.80.64:1004:DISK through 9.134.122.87:1004
    91. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1087149463_13410403 with size=70067422 from 9.134.117.90:1004:DISK to 9.134.80.60:1004:DISK through 9.134.163.174:1004
    92. 22/06/27 11:29:19 INFO balancer.Dispatcher: Start moving blk_1085874680_12135195 with size=18098983 from 9.134.68.200:1004:DISK to 9.134.81.127:1004:DISK through 9.134.83.33:1004
    93.

    结果很有意思的是打印日志的时候报错了。来分析下日志

    1.Using a threshold of 5.0 说明获取到阈值

    2.namenodes  = [hdfs://s2cluster] 获取到集群信息 并且是HA

    3.included nodes = [] ，excluded nodes = [] 说明我们没有设置指定哪些节点或者排除哪些节点

    4.dfs.balancer.xx 打印了很多默认参数

    5. 3 over-utilized: [9.134.117.90:1004:DISK, 9.134.124.14:1004:DISK, 9.134.116.73:1004:DISK]

    5 underutilized: [9.134.80.60:1004:DISK, 9.134.81.221:1004:DISK, 9.134.124.36:1004:DISK, 9.134.81.127:1004:DISK, 9.134.80.64:1004:DISK]
    这里说的是3个超过了限额的。5个低于限额的。我们来先看下这3个over发现其实有问题的。

     

    发现其实这三个没有相似处。暂且不管。

    6.开始分配over->under

    7开始move结果报错了，Operation category READ is not supported in state standby 这个不用说肯定是用到了standby的节点 

    8.关键日志, while invoking NamenodeProtocolTranslatorPB.getBlocks over master.data.com/9.134.64.234:8020. Trying to failover immediately. 这个b怎么就用的mater，下图可以看到master处于备用阶段，按照ha应该是可以用node01的。淦！

     9、但是后面的日志惊奇的发现他又开始用node01的active nn了。并且有开始move的日志了

    10、Failed to move blk_1074919453_1178890 with size=134217728 from 9.134.68.200:1004:DISK to 9.134.81.127:1004:DISK through 9.134.80.60:1004
    java.io.IOException: Got error, status=ERROR, status message Not able to receive block 1074919453 from /9.134.70.1:44227 because threads quota is exceeded., reportedBlock move is failed 好像是thread超过限额了？

    -----------------------

    上面的命令说明了啥？

    只要你设置了阈值那么就会立马起作用！！！ 

    于是改变策略我只移动两个nn的。

     发现根本就没有低于阈值或者高于阈值的。。感觉这里有问题。

    hdfs balancer -threshold 5 再次使用

    之前取销是因为一直报错。。。后面发现这些报错不用管。等十几秒就会正常。前面都是连接错误啥的。

    后面正常了就会显示 正在从a 移到b 

     这里显示完成了。但是我在cdh的控制页面发现还是一样的。

     其实我这里放了一个错误！！

    磁盘的使用情况不是hdfs的容量的使用情况！！

    磁盘的使用情况df -h以node17为例！cdh显示 376G/491G

     

     node17所有容量相加 差不多是500多G，但是/data目录是394G，使用的差不多是302+51=353G

     

    hdfs dfs -report 搜索node17

     

     这里显示configured Capacity 是383G， dfs实际使用116G 非dfs使用175G 

    一般来说hdfs目录是挂载在/data目录的。

    可以看到dfs占用了117G 其余的非hdfs差不多是5.7+7.1+8.7+137+11+其余的=175G

    此时我们可以得到如下结论

    /data目录本身有394G内存 与这个report的383G差不多。（这个误差我不知道为啥。）

    /data目录有比如kudu的tablet log啥的这些属于非hdfs的文件，占用了175G和report一样！！

    /data/dfs =117G和我们report的116.08G一样！

    这时才明白 我为什么我上面看到node17 node18差别那么大，怎么不能balance了把，

    截图node18 /data目录容量。因为这上面他本身dfs占用内存也不少

    而node17的tablet目录也有100多G 让我误以为两者相差很多

    当前这个都是我balance后的截图。。balance前两者是否平衡，我也不能回到过去了。。

    总结

    我们平衡hdfs，是为了平衡什么呢？

    /data目录？/data/dfs下的容量？/data/dfs的容量占/data的百分比？

    个人认为是/data/dfs的容量占/data的百分比。

    由于我之前已经balance成功了。

    可以看到所有的dn节点DFS Used% 差不多都是25-35之间，刚好符合我们的比例+_5%

    Configured Capacity: 10424513773568 (9.48 TB)
    Present Capacity: 7435669957834 (6.76 TB)
    DFS Remaining: 4419545119946 (4.02 TB)
    DFS Used: 3016124837888 (2.74 TB)
    DFS Used%: 40.56%

    9.48T是所有机器的/data目录容量

    Present Capacity 是什么？先留下疑问

    2.74 T是所有机器的/data/dfs目录容量

    40.56%是 2.74/6.72 那么平均的百分比应该是40呀我们的threhold=5那么其余的dn应该是35-45%呀

    注意下面的dn 以node19为例

    Name: 9.134.115.141:1004 (node19.data.com)
    Configured Capacity: 411883184128 (383.60 GB)
    DFS Used: 119300038656 (111.11 GB)
    Non DFS Used: 51429728256 (47.90 GB)
    DFS Remaining: 219661856768 (204.58 GB)
    DFS Used%: 28.96%
    DFS Remaining%: 53.33%
    383.60 GB 是/data目录的

    111.11 GB 是/data/dfs目录的

    28.96% =111.11G/383G

    这个怎么百分比会这么少呢？按道理来说，每天机器差不多都是25-35%，那么total也应该是30左右呀怎么会是40%

    有意思的事情出来了。可以看到我们

    每个dn的dfs used差不多是110-120G那么23个节点就是2.xT 等于我们上面的2.74T。

    dn中容量为383G的有14台 482G的有9台，总容量=383*14+482*9=9.7T

    所以这里存在一个问题！！！！

    所有dn的 DFS Used%= DFS Used/Present Capacity

    每个dn的 DFS Used%=DFS Used/Configured Capacity

    所以这里是有出入的。

    所以关键在present Capacity 个人猜测是 Configured Capacity -所有的Non DFS Used

    所以Present Capacity=9.48T-2.25T= 7.23T貌似不对...我也没办法了。

    虽然这个问题没解决。但是有了如下收获！！

    1. 设置转移速度， 设置balance阈值

    2.记住balance之前 hdfs dfsadmin -report 查看各个dn的dfs used%

    3.这里balance的是 /data/dfs目录 占/data目录的百分比。

    4.一般来说要么设置dfs.datanode.data.dir = /data 那么就不要把其余工作目录设置到data。

    因为比如每个dn容量为500G，但是某个dn 的/data目录里有个400G的文件，那么这个dn的dfs used在用到20%的时候就会出现问题。这个时候可以通过df -h 来排查。当然你排查了也没用。因为这个400G的文件就是在这！！！

    1. [root@worker01 /home/devuser]# hdfs dfsadmin -report
    2. [root@worker01 /home/devuser]# hdfs dfsadmin -report
    3. Configured Capacity: 10424513773568 (9.48 TB)
    4. Present Capacity: 7435669957834 (6.76 TB)
    5. DFS Remaining: 4419545119946 (4.02 TB)
    6. DFS Used: 3016124837888 (2.74 TB)
    7. DFS Used%: 40.56%
    8. Replicated Blocks:
    9. 	Under replicated blocks: 0
    10. 	Blocks with corrupt replicas: 0
    11. 	Missing blocks: 0
    12. 	Missing blocks (with replication factor 1): 0
    13. 	Low redundancy blocks with highest priority to recover: 0
    14. 	Pending deletion blocks: 0
    15. Erasure Coded Block Groups: 
    16. 	Low redundancy block groups: 0
    17. 	Block groups with corrupt internal blocks: 0
    18. 	Missing block groups: 0
    19. 	Low redundancy blocks with highest priority to recover: 0
    20. 	Pending deletion blocks: 0
    21.  
    22. -------------------------------------------------
    23. Live datanodes (23):
    24.  
    25. Name: 9.134.115.141:1004 (node19.data.com)
    26. Hostname: node19.data.com
    27. Rack: /default
    28. Decommission Status : Normal
    29. Configured Capacity: 411883184128 (383.60 GB)
    30. DFS Used: 119300038656 (111.11 GB)
    31. Non DFS Used: 51429728256 (47.90 GB)
    32. DFS Remaining: 219661856768 (204.58 GB)
    33. DFS Used%: 28.96%
    34. DFS Remaining%: 53.33%
    35. Configured Cache Capacity: 4294967296 (4 GB)
    36. Cache Used: 0 (0 B)
    37. Cache Remaining: 4294967296 (4 GB)
    38. Cache Used%: 0.00%
    39. Cache Remaining%: 100.00%
    40. Xceivers: 2
    41. Last contact: Mon Jun 27 13:50:43 CST 2022
    42. Last Block Report: Mon Jun 27 09:02:06 CST 2022
    43.  
    44.  
    45. Name: 9.134.116.73:1004 (node22.data.com)
    46. Hostname: node22.data.com
    47. Rack: /default
    48. Decommission Status : Normal
    49. Configured Capacity: 411883184128 (383.60 GB)
    50. DFS Used: 139341885440 (129.77 GB)
    51. Non DFS Used: 4911681536 (4.57 GB)
    52. DFS Remaining: 246138056704 (229.23 GB)
    53. DFS Used%: 33.83%
    54. DFS Remaining%: 59.76%
    55. Configured Cache Capacity: 4294967296 (4 GB)
    56. Cache Used: 0 (0 B)
    57. Cache Remaining: 4294967296 (4 GB)
    58. Cache Used%: 0.00%
    59. Cache Remaining%: 100.00%
    60. Xceivers: 2
    61. Last contact: Mon Jun 27 13:50:42 CST 2022
    62. Last Block Report: Mon Jun 27 11:49:39 CST 2022
    63.  
    64.  
    65. Name: 9.134.117.73:1004 (node17.data.com)
    66. Hostname: node17.data.com
    67. Rack: /default
    68. Decommission Status : Normal
    69. Configured Capacity: 411883184128 (383.60 GB)
    70. DFS Used: 124636291072 (116.08 GB)
    71. Non DFS Used: 188638429184 (175.68 GB)
    72. DFS Remaining: 77116903424 (71.82 GB)
    73. DFS Used%: 30.26%
    74. DFS Remaining%: 18.72%
    75. Configured Cache Capacity: 4294967296 (4 GB)
    76. Cache Used: 0 (0 B)
    77. Cache Remaining: 4294967296 (4 GB)
    78. Cache Used%: 0.00%
    79. Cache Remaining%: 100.00%
    80. Xceivers: 2
    81. Last contact: Mon Jun 27 13:50:41 CST 2022
    82. Last Block Report: Mon Jun 27 10:26:56 CST 2022
    83.  
    84.  
    85. Name: 9.134.117.90:1004 (node16.data.com)
    86. Hostname: node16.data.com
    87. Rack: /default
    88. Decommission Status : Normal
    89. Configured Capacity: 411883184128 (383.60 GB)
    90. DFS Used: 127383461888 (118.64 GB)
    91. Non DFS Used: 107218415616 (99.85 GB)
    92. DFS Remaining: 155668209253 (144.98 GB)
    93. DFS Used%: 30.93%
    94. DFS Remaining%: 37.79%
    95. Configured Cache Capacity: 4294967296 (4 GB)
    96. Cache Used: 0 (0 B)
    97. Cache Remaining: 4294967296 (4 GB)
    98. Cache Used%: 0.00%
    99. Cache Remaining%: 100.00%
    100. Xceivers: 4
    101. Last contact: Mon Jun 27 13:50:42 CST 2022
    102. Last Block Report: Mon Jun 27 12:35:15 CST 2022
    103.  
    104.  
    105. Name: 9.134.122.224:1004 (node14.data.com)
    106. Hostname: node14.data.com
    107. Rack: /default
    108. Decommission Status : Normal
    109. Configured Capacity: 411883184128 (383.60 GB)
    110. DFS Used: 139456868352 (129.88 GB)
    111. Non DFS Used: 139715756032 (130.12 GB)
    112. DFS Remaining: 111218999296 (103.58 GB)
    113. DFS Used%: 33.86%
    114. DFS Remaining%: 27.00%
    115. Configured Cache Capacity: 4294967296 (4 GB)
    116. Cache Used: 0 (0 B)
    117. Cache Remaining: 4294967296 (4 GB)
    118. Cache Used%: 0.00%
    119. Cache Remaining%: 100.00%
    120. Xceivers: 2
    121. Last contact: Mon Jun 27 13:50:43 CST 2022
    122. Last Block Report: Mon Jun 27 12:37:22 CST 2022
    123.  
    124.  
    125. Name: 9.134.122.6:1004 (node12.data.com)
    126. Hostname: node12.data.com
    127. Rack: /default
    128. Decommission Status : Normal
    129. Configured Capacity: 517572132864 (482.03 GB)
    130. DFS Used: 142010712064 (132.26 GB)
    131. Non DFS Used: 138266742784 (128.77 GB)
    132. DFS Remaining: 210434408448 (195.98 GB)
    133. DFS Used%: 27.44%
    134. DFS Remaining%: 40.66%
    135. Configured Cache Capacity: 4294967296 (4 GB)
    136. Cache Used: 0 (0 B)
    137. Cache Remaining: 4294967296 (4 GB)
    138. Cache Used%: 0.00%
    139. Cache Remaining%: 100.00%
    140. Xceivers: 2
    141. Last contact: Mon Jun 27 13:50:42 CST 2022
    142. Last Block Report: Mon Jun 27 12:51:06 CST 2022
    143.  
    144.  
    145. Name: 9.134.122.87:1004 (node23.data.com)
    146. Hostname: node23.data.com
    147. Rack: /default
    148. Decommission Status : Normal
    149. Configured Capacity: 411883184128 (383.60 GB)
    150. DFS Used: 126853881856 (118.14 GB)
    151. Non DFS Used: 5369483264 (5.00 GB)
    152. DFS Remaining: 258168258560 (240.44 GB)
    153. DFS Used%: 30.80%
    154. DFS Remaining%: 62.68%
    155. Configured Cache Capacity: 4294967296 (4 GB)
    156. Cache Used: 0 (0 B)
    157. Cache Remaining: 4294967296 (4 GB)
    158. Cache Used%: 0.00%
    159. Cache Remaining%: 100.00%
    160. Xceivers: 2
    161. Last contact: Mon Jun 27 13:50:43 CST 2022
    162. Last Block Report: Mon Jun 27 13:01:49 CST 2022
    163.  
    164.  
    165. Name: 9.134.123.190:1004 (node13.data.com)
    166. Hostname: node13.data.com
    167. Rack: /default
    168. Decommission Status : Normal
    169. Configured Capacity: 411883184128 (383.60 GB)
    170. DFS Used: 120316370944 (112.05 GB)
    171. Non DFS Used: 135664951296 (126.35 GB)
    172. DFS Remaining: 134410301440 (125.18 GB)
    173. DFS Used%: 29.21%
    174. DFS Remaining%: 32.63%
    175. Configured Cache Capacity: 4294967296 (4 GB)
    176. Cache Used: 0 (0 B)
    177. Cache Remaining: 4294967296 (4 GB)
    178. Cache Used%: 0.00%
    179. Cache Remaining%: 100.00%
    180. Xceivers: 2
    181. Last contact: Mon Jun 27 13:50:41 CST 2022
    182. Last Block Report: Mon Jun 27 10:26:38 CST 2022
    183.  
    184.  
    185. Name: 9.134.123.215:1004 (node21.data.com)
    186. Hostname: node21.data.com
    187. Rack: /default
    188. Decommission Status : Normal
    189. Configured Capacity: 411883184128 (383.60 GB)
    190. DFS Used: 123294625792 (114.83 GB)
    191. Non DFS Used: 49384546304 (45.99 GB)
    192. DFS Remaining: 217712451584 (202.76 GB)
    193. DFS Used%: 29.93%
    194. DFS Remaining%: 52.86%
    195. Configured Cache Capacity: 4294967296 (4 GB)
    196. Cache Used: 0 (0 B)
    197. Cache Remaining: 4294967296 (4 GB)
    198. Cache Used%: 0.00%
    199. Cache Remaining%: 100.00%
    200. Xceivers: 2
    201. Last contact: Mon Jun 27 13:50:43 CST 2022
    202. Last Block Report: Mon Jun 27 11:11:12 CST 2022
    203.  
    204.  
    205. Name: 9.134.123.37:1004 (node18.data.com)
    206. Hostname: node18.data.com
    207. Rack: /default
    208. Decommission Status : Normal
    209. Configured Capacity: 411883184128 (383.60 GB)
    210. DFS Used: 132322639872 (123.24 GB)
    211. Non DFS Used: 19139104768 (17.82 GB)
    212. DFS Remaining: 238929879040 (222.52 GB)
    213. DFS Used%: 32.13%
    214. DFS Remaining%: 58.01%
    215. Configured Cache Capacity: 4294967296 (4 GB)
    216. Cache Used: 0 (0 B)
    217. Cache Remaining: 4294967296 (4 GB)
    218. Cache Used%: 0.00%
    219. Cache Remaining%: 100.00%
    220. Xceivers: 2
    221. Last contact: Mon Jun 27 13:50:41 CST 2022
    222. Last Block Report: Mon Jun 27 10:27:47 CST 2022
    223.  
    224.  
    225. Name: 9.134.124.14:1004 (node15.data.com)
    226. Hostname: node15.data.com
    227. Rack: /default
    228. Decommission Status : Normal
    229. Configured Capacity: 411883184128 (383.60 GB)
    230. DFS Used: 134688702464 (125.44 GB)
    231. Non DFS Used: 146982957056 (136.89 GB)
    232. DFS Remaining: 108719964160 (101.25 GB)
    233. DFS Used%: 32.70%
    234. DFS Remaining%: 26.40%
    235. Configured Cache Capacity: 4294967296 (4 GB)
    236. Cache Used: 0 (0 B)
    237. Cache Remaining: 4294967296 (4 GB)
    238. Cache Used%: 0.00%
    239. Cache Remaining%: 100.00%
    240. Xceivers: 2
    241. Last contact: Mon Jun 27 13:50:42 CST 2022
    242. Last Block Report: Mon Jun 27 09:44:21 CST 2022
    243.  
    244.  
    245. Name: 9.134.124.36:1004 (node20.data.com)
    246. Hostname: node20.data.com
    247. Rack: /default
    248. Decommission Status : Normal
    249. Configured Capacity: 411883184128 (383.60 GB)
    250. DFS Used: 100490969088 (93.59 GB)
    251. Non DFS Used: 74869800960 (69.73 GB)
    252. DFS Remaining: 215030853632 (200.26 GB)
    253. DFS Used%: 24.40%
    254. DFS Remaining%: 52.21%
    255. Configured Cache Capacity: 4294967296 (4 GB)
    256. Cache Used: 0 (0 B)
    257. Cache Remaining: 4294967296 (4 GB)
    258. Cache Used%: 0.00%
    259. Cache Remaining%: 100.00%
    260. Xceivers: 2
    261. Last contact: Mon Jun 27 13:50:41 CST 2022
    262. Last Block Report: Mon Jun 27 08:49:47 CST 2022
    263.  
    264.  
    265. Name: 9.134.163.174:1004 (node26.data.com)
    266. Hostname: node26.data.com
    267. Rack: /default
    268. Decommission Status : Normal
    269. Configured Capacity: 517572132864 (482.03 GB)
    270. DFS Used: 149608824832 (139.33 GB)
    271. Non DFS Used: 113226809344 (105.45 GB)
    272. DFS Remaining: 227876229120 (212.23 GB)
    273. DFS Used%: 28.91%
    274. DFS Remaining%: 44.03%
    275. Configured Cache Capacity: 4294967296 (4 GB)
    276. Cache Used: 0 (0 B)
    277. Cache Remaining: 4294967296 (4 GB)
    278. Cache Used%: 0.00%
    279. Cache Remaining%: 100.00%
    280. Xceivers: 2
    281. Last contact: Mon Jun 27 13:50:40 CST 2022
    282. Last Block Report: Mon Jun 27 13:34:49 CST 2022
    283.  
    284.  
    285. Name: 9.134.163.238:1004 (node24.data.com)
    286. Hostname: node24.data.com
    287. Rack: /default
    288. Decommission Status : Normal
    289. Configured Capacity: 517572132864 (482.03 GB)
    290. DFS Used: 149926281216 (139.63 GB)
    291. Non DFS Used: 177427165184 (165.24 GB)
    292. DFS Remaining: 163358416896 (152.14 GB)
    293. DFS Used%: 28.97%
    294. DFS Remaining%: 31.56%
    295. Configured Cache Capacity: 4294967296 (4 GB)
    296. Cache Used: 0 (0 B)
    297. Cache Remaining: 4294967296 (4 GB)
    298. Cache Used%: 0.00%
    299. Cache Remaining%: 100.00%
    300. Xceivers: 2
    301. Last contact: Mon Jun 27 13:50:41 CST 2022
    302. Last Block Report: Mon Jun 27 11:00:49 CST 2022
    303.  
    304.  
    305. Name: 9.134.163.60:1004 (node25.data.com)
    306. Hostname: node25.data.com
    307. Rack: /default
    308. Decommission Status : Normal
    309. Configured Capacity: 517572132864 (482.03 GB)
    310. DFS Used: 148323336192 (138.14 GB)
    311. Non DFS Used: 139817668608 (130.22 GB)
    312. DFS Remaining: 202570858496 (188.66 GB)
    313. DFS Used%: 28.66%
    314. DFS Remaining%: 39.14%
    315. Configured Cache Capacity: 4294967296 (4 GB)
    316. Cache Used: 0 (0 B)
    317. Cache Remaining: 4294967296 (4 GB)
    318. Cache Used%: 0.00%
    319. Cache Remaining%: 100.00%
    320. Xceivers: 2
    321. Last contact: Mon Jun 27 13:50:42 CST 2022
    322. Last Block Report: Mon Jun 27 12:50:30 CST 2022
    323.  
    324.  
    325. Name: 9.134.68.200:1004 (node11.data.com)
    326. Hostname: node11.data.com
    327. Rack: /default
    328. Decommission Status : Normal
    329. Configured Capacity: 411883184128 (383.60 GB)
    330. DFS Used: 132859736064 (123.74 GB)
    331. Non DFS Used: 53714735104 (50.03 GB)
    332. DFS Remaining: 203817152512 (189.82 GB)
    333. DFS Used%: 32.26%
    334. DFS Remaining%: 49.48%
    335. Configured Cache Capacity: 4294967296 (4 GB)
    336. Cache Used: 0 (0 B)
    337. Cache Remaining: 4294967296 (4 GB)
    338. Cache Used%: 0.00%
    339. Cache Remaining%: 100.00%
    340. Xceivers: 2
    341. Last contact: Mon Jun 27 13:50:41 CST 2022
    342. Last Block Report: Mon Jun 27 09:35:01 CST 2022
    343.  
    344.  
    345. Name: 9.134.71.192:1004 (node05.data.com)
    346. Hostname: node05.data.com
    347. Rack: /default
    348. Decommission Status : Normal
    349. Configured Capacity: 411883184128 (383.60 GB)
    350. DFS Used: 123511422976 (115.03 GB)
    351. Non DFS Used: 50063003648 (46.62 GB)
    352. DFS Remaining: 216817197056 (201.93 GB)
    353. DFS Used%: 29.99%
    354. DFS Remaining%: 52.64%
    355. Configured Cache Capacity: 4294967296 (4 GB)
    356. Cache Used: 0 (0 B)
    357. Cache Remaining: 4294967296 (4 GB)
    358. Cache Used%: 0.00%
    359. Cache Remaining%: 100.00%
    360. Xceivers: 2
    361. Last contact: Mon Jun 27 13:50:42 CST 2022
    362. Last Block Report: Mon Jun 27 09:38:05 CST 2022
    363.  
    364.  
    365. Name: 9.134.77.69:1004 (node07.data.com)
    366. Hostname: node07.data.com
    367. Rack: /default
    368. Decommission Status : Normal
    369. Configured Capacity: 411883184128 (383.60 GB)
    370. DFS Used: 131336716288 (122.32 GB)
    371. Non DFS Used: 94864969728 (88.35 GB)
    372. DFS Remaining: 164189937664 (152.91 GB)
    373. DFS Used%: 31.89%
    374. DFS Remaining%: 39.86%
    375. Configured Cache Capacity: 4294967296 (4 GB)
    376. Cache Used: 0 (0 B)
    377. Cache Remaining: 4294967296 (4 GB)
    378. Cache Used%: 0.00%
    379. Cache Remaining%: 100.00%
    380. Xceivers: 2
    381. Last contact: Mon Jun 27 13:50:40 CST 2022
    382. Last Block Report: Mon Jun 27 11:48:10 CST 2022
    383.  
    384.  
    385. Name: 9.134.80.60:1004 (node27.data.com)
    386. Hostname: node27.data.com
    387. Rack: /default
    388. Decommission Status : Normal
    389. Configured Capacity: 517572132864 (482.03 GB)
    390. DFS Used: 125220708352 (116.62 GB)
    391. Non DFS Used: 154762784768 (144.13 GB)
    392. DFS Remaining: 210728370176 (196.26 GB)
    393. DFS Used%: 24.19%
    394. DFS Remaining%: 40.71%
    395. Configured Cache Capacity: 4294967296 (4 GB)
    396. Cache Used: 0 (0 B)
    397. Cache Remaining: 4294967296 (4 GB)
    398. Cache Used%: 0.00%
    399. Cache Remaining%: 100.00%
    400. Xceivers: 2
    401. Last contact: Mon Jun 27 13:50:41 CST 2022
    402. Last Block Report: Mon Jun 27 08:58:05 CST 2022
    403.  
    404.  
    405. Name: 9.134.80.64:1004 (node29.data.com)
    406. Hostname: node29.data.com
    407. Rack: /default
    408. Decommission Status : Normal
    409. Configured Capacity: 517572132864 (482.03 GB)
    410. DFS Used: 134744162304 (125.49 GB)
    411. Non DFS Used: 154976854016 (144.33 GB)
    412. DFS Remaining: 200990846976 (187.19 GB)
    413. DFS Used%: 26.03%
    414. DFS Remaining%: 38.83%
    415. Configured Cache Capacity: 4294967296 (4 GB)
    416. Cache Used: 0 (0 B)
    417. Cache Remaining: 4294967296 (4 GB)
    418. Cache Used%: 0.00%
    419. Cache Remaining%: 100.00%
    420. Xceivers: 2
    421. Last contact: Mon Jun 27 13:50:41 CST 2022
    422. Last Block Report: Mon Jun 27 11:06:53 CST 2022
    423.  
    424.  
    425. Name: 9.134.81.127:1004 (node31.data.com)
    426. Hostname: node31.data.com
    427. Rack: /default
    428. Decommission Status : Normal
    429. Configured Capacity: 517572132864 (482.03 GB)
    430. DFS Used: 131076886528 (122.07 GB)
    431. Non DFS Used: 151701401600 (141.28 GB)
    432. DFS Remaining: 207812038245 (193.54 GB)
    433. DFS Used%: 25.33%
    434. DFS Remaining%: 40.15%
    435. Configured Cache Capacity: 4294967296 (4 GB)
    436. Cache Used: 0 (0 B)
    437. Cache Remaining: 4294967296 (4 GB)
    438. Cache Used%: 0.00%
    439. Cache Remaining%: 100.00%
    440. Xceivers: 4
    441. Last contact: Mon Jun 27 13:50:41 CST 2022
    442. Last Block Report: Mon Jun 27 08:35:59 CST 2022
    443.  
    444.  
    445. Name: 9.134.81.221:1004 (node30.data.com)
    446. Hostname: node30.data.com
    447. Rack: /default
    448. Decommission Status : Normal
    449. Configured Capacity: 517572132864 (482.03 GB)
    450. DFS Used: 126044688384 (117.39 GB)
    451. Non DFS Used: 141402562560 (131.69 GB)
    452. DFS Remaining: 223264612352 (207.93 GB)
    453. DFS Used%: 24.35%
    454. DFS Remaining%: 43.14%
    455. Configured Cache Capacity: 4294967296 (4 GB)
    456. Cache Used: 0 (0 B)
    457. Cache Remaining: 4294967296 (4 GB)
    458. Cache Used%: 0.00%
    459. Cache Remaining%: 100.00%
    460. Xceivers: 2
    461. Last contact: Mon Jun 27 13:50:41 CST 2022
    462. Last Block Report: Mon Jun 27 10:09:56 CST 2022
    463.  
    464.  
    465. Name: 9.134.83.33:1004 (node28.data.com)
    466. Hostname: node28.data.com
    467. Rack: /default
    468. Decommission Status : Normal
    469. Configured Capacity: 517572132864 (482.03 GB)
    470. DFS Used: 133375627264 (124.22 GB)
    471. Non DFS Used: 152426917888 (141.96 GB)
    472. DFS Remaining: 204909318144 (190.84 GB)
    473. DFS Used%: 25.77%
    474. DFS Remaining%: 39.59%
    475. Configured Cache Capacity: 4294967296 (4 GB)
    476. Cache Used: 0 (0 B)
    477. Cache Remaining: 4294967296 (4 GB)
    478. Cache Used%: 0.00%
    479. Cache Remaining%: 100.00%
    480. Xceivers: 2
    481. Last contact: Mon Jun 27 13:50:43 CST 2022
    482. Last Block Report: Mon Jun 27 08:24:57 CST 2022
    483.  
    484.  
    485. Dead datanodes (2):
    486.  
    487. Name: 9.134.74.52:1004 (node06.data.com)
    488. Hostname: node06.data.com
    489. Decommission Status : Decommissioned
    490. Configured Capacity: 0 (0 B)
    491. DFS Used: 0 (0 B)
    492. Non DFS Used: 0 (0 B)
    493. DFS Remaining: 0 (0 B)
    494. DFS Used%: 100.00%
    495. DFS Remaining%: 0.00%
    496. Configured Cache Capacity: 0 (0 B)
    497. Cache Used: 0 (0 B)
    498. Cache Remaining: 0 (0 B)
    499. Cache Used%: 100.00%
    500. Cache Remaining%: 0.00%
    501. Xceivers: 0
    502. Last contact: Thu Jan 01 08:00:00 CST 1970
    503. Last Block Report: Never
    504.  
    505.  
    506. Name: 9.134.70.49:1004 (node04.data.com)
    507. Hostname: node04.data.com
    508. Decommission Status : Decommissioned
    509. Configured Capacity: 0 (0 B)
    510. DFS Used: 0 (0 B)
    511. Non DFS Used: 0 (0 B)
    512. DFS Remaining: 0 (0 B)
    513. DFS Used%: 100.00%
    514. DFS Remaining%: 0.00%
    515. Configured Cache Capacity: 0 (0 B)
    516. Cache Used: 0 (0 B)
    517. Cache Remaining: 0 (0 B)
    518. Cache Used%: 100.00%
    519. Cache Remaining%: 0.00%
    520. Xceivers: 0
    521. Last contact: Thu Jan 01 08:00:00 CST 1970
    522. Last Block Report: Never
    523.  
    524.  
    525. [root@worker01 /home/devuser]#  df -h
    526. 文件系统         容量  已用  可用 已用% 挂载点
    527. /dev/vda1         99G   56G   39G   59% /
    528. devtmpfs         7.7G     0  7.7G    0% /dev
    529. tmpfs            7.8G     0  7.8G    0% /dev/shm
    530. tmpfs            7.8G  266M  7.5G    4% /run
    531. tmpfs            7.8G     0  7.8G    0% /sys/fs/cgroup
    532. /dev/vdb1        394G  161G  214G   43% /data
    533. cm_processes     7.8G   66M  7.7G    1% /run/cloudera-scm-agent/process
    534. tmpfs            1.6G     0  1.6G    0% /run/user/1000
    535. 9.135.223.197:/   15T   12T  2.5T   83% /data/share/files/cfs
    536. tmpfs            1.6G     0  1.6G    0% /run/user/0
    537. [root@worker01 /home/devuser]#  df -h /data
    538. 文件系统        容量  已用  可用 已用% 挂载点
    539. /dev/vdb1       394G  161G  214G   43% /data
    540. [root@worker01 /home/devuser]#

  • 相关阅读:
    uniapp中用canvas实现小球碰撞的小动画
    传统社区如何进行数字化转型？快鲸智慧社区解决方案为你支招
    《安富莱嵌入式周报》第290期：开源静电便携测试仪，开源音频功放，CAN高波特率设计，超级铁电产品，小米Vela系统，65W USB PD充电器参考设计
    厨神之蛋糕制作
    算法训练营day17
    【蓝桥杯2025备赛】素数判断:从O(n^2)到O(n)学习之路
    【Iceberg＋Alluxio】助力加速数据通道（下篇）
    WinForm，可能是Windows上手最快的图形框架了
    交换机/防火墙-基础配置-23.10.11
    (计算机组成原理)第五章中央处理器-第七节1：多处理器的基本概念
  • 原文地址：https://blog.csdn.net/cclovezbf/article/details/125479880