Hadoop集群配置

准备三台主机
192.168.1.244、192.168.1.245、192.168.1.246
安装hadoop-3.1.3
修改host文件

vi /etc/hosts
1

输入内容

192.168.1.244	hadoop244
192.168.1.245	hadoop245
192.168.1.246	hadoop246
1
2
3

修改主机名
分别修改主机名为hadoop244、hadoop245、hadoop246
直接在终端输入nmtui，有可视化界面修改主机名

nmtui
1

编辑hdfs-site.xml文件

/opt/hadoop-3.1.3/etc/hadoop
vi hdfs-site.xml
1
2

输入内容（三台主机一律一致）

<configuration>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>hadoop244:9870</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop246:9868</value>
    </property>
</configuration>

1
2
3
4
5
6
7
8
9
10
11

编辑core-site.xml文件

vi core-site.xml
1

输入内容（三台主机都一样）

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop244:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop-3.1.3/data</value>
    </property>
</configuration>

1
2
3
4
5
6
7
8
9
10
11

编辑workers文件

vi workers
1

输入内容（三台主机均一致）

hadoop244
hadoop245
hadoop246
1
2
3

编辑yarn-site.xml（三台主机均一致）

<configuration>
    <property>
        <name>yarn.nodemanagere.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop245</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log.server.url</name>
        <value>http://192.168.1.245:19888/jobhistory/logs</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>
</configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

配置环境变量（三台主机同步）

# hadoop
export HADOOP_HMOE=/opt/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HMOE/bin
export PATH=$PATH:$HADOOP_HMOE/sbin
export HADOOP_COMMON_HOME=/opt/hadoop-3.1.3
export HADOOP_HDFS_HOME=/opt/hadoop-3.1.3
export HADOOP_CONF_DIR=/opt/hadoop-3.1.3/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop-3.1.3
export HADOOP_MAPRED_HOME=/opt/hadoop-3.1.3

1
2
3
4
5
6
7
8
9
10