HDFS机架感知配置（block放置策略/NodeGroup分配策略配置）

HDFS机架感知配置（block放置策略/NodeGroup分配策略配置）

1、配置hdfs-site和core-site文件

• core-site.xml

<property>
    <name>net.topology.implname>
    <value>org.apache.hadoop.net.NetworkTopologyWithNodeGroupvalue>
property>
<property>
    <name>net.topology.script.file.namename>
    <value>/tmp/rack.pyvalue>	
property>
<property>
    <name>net.topology.nodegroup.awarename>
    <value>truevalue>
property>
1
2
3
4
5
6
7
8
9
10
11
12

• hdfs-site.xml

<property>
    <name>dfs.block.replicator.classnamename>
    <value>
        org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithNodeGroup
    value>
property>
<property>
    <name>dfs.use.dfs.network.topologyname>
    <value>falsevalue>
property>
1
2
3
4
5
6
7
8
9
10

2、nodegroup示例

192.168.0.1 /rack1/nodegroup1
192.168.0.2 /rack1/nodegroup1
192.168.0.3 /rack1/nodegroup2
192.168.0.4 /rack1/nodegroup2
192.168.0.5 /rack2/nodegroup3
192.168.0.6 /rack2/nodegroup3
1
2
3
4
5
6

添加脚本

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import sys
rack = {
    "192.168.0.1": "/rack1/nodegroup1",
    "192.168.0.2": "/rack1/nodegroup1",
    "192.168.0.3": "/rack1/nodegroup2",
    "192.168.0.4": "/rack1/nodegroup2",
    "192.168.0.5": "/rack2/nodegroup3",
    "192.168.0.6": "/rack2/nodegroup3",
}
if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit(1)
        ip_address = sys.argv[1]
        result = rack.get(ip_address, "/default-rack")
        print(result)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

给脚本授权

chmod +777 /tmp/rack.py
1

3、通过命令查看是否配置成功

hdfs dfsadmin -printTopology
1