Hadoop - 安装

---

关于 Hadoop

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

• 官网: https://hadoop.apache.org
• 官方教程: https://hadoop.apache.org/docs/r1.0.4/cn/
• W3C教程：https://www.w3cschool.cn/hadoop/
• 菜鸟教程：https://www.runoob.com/w3cnote/hadoop-tutorial.html

---

发行版本

• Apache 开源社区版: https://hadoop.apache.org
• 商业发行版本
  • Cloudera: https://www.cloudera.com
    Hortonworks 和 Cloudera 合并了

---

架构变迁 1.0 --> 2.0 --> 3.0

---

---

安装配置

官方安装配置：
https://hadoop.apache.org/docs/r1.0.4/cn/quickstart.html

---

安装

macOS 安装 Hadoop：使用 brew
brew 安装、使用方法可见：https://blog.csdn.net/lovechris00/article/details/121613647

brew install hadoop
1

---

如果出现报错：

Error: Cannot install hadoop because conflicting formulae are installed.
yarn: because both install yarn binaries
Please brew unlink yarn before continuing.

根据提示执行命令即可：

brew unlink yarn
1

---

安装成功，查看版本

# 查看 hadoop 安装目录
brew info hadoop 
1
2

---

配置环境变量

根据你的环境变量文件，编辑

vim ~/.zshrc
# vim ~/.bash_profile
1
2

# Hadoop
export HADOOP_HOME=/usr/local/Cellar/hadoop/3.3.4/libexec
export PATH=$PATH:HADOOP_HOME
1
2
3

---

使环境变量在当前窗口生效。
你也可以使用 ctrl + tab 新开一个终端窗口。

source ~/.zshrc
# source ~/.bash_profile
1
2

---

配置

cd /usr/local/Cellar/hadoop/3.3.3/libexec/etc/hadoop
ls
1
2

修改配置文件

---

core-site.xml

vim  core-site.xml
1

进入文件后加入：

<configuration>
    <property>
    <name>fs.defaultFSname>
    <value>hdfs://localhost:8020value>
  property>
 
  
  <property>
    <name>hadoop.tmp.dirname>
    <value>file:/usr/local/Cellar/hadoop/tmpvalue>
  property>
configuration>
1
2
3
4
5
6
7
8
9
10
11
12

---

hdfs-site.xml

配置副本数

vim  hdfs-site.xml
1

<configuration>
        <property>
                <name>dfs.replicationname>
                <value>1value>
        property>
        
        <property>
                <name>dfs.permissionsname>
                <value>falsevalue>    
        property>
        
        <property>
                <name>dfs.namenode.name.dirname>
                <value>file:/usr/local/Cellar/hadoop/tmp/dfs/namevalue>
        property>
        
        <property>
                <name>dfs.datanode.data.dirname>
                <value>file:/usr/local/Cellar/hadoop/tmp/dfs/datavalue>
        property>
configuration> 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

---

mapped-site.xml

vim  mapped-site.xml
1

 <configuration>
  <property>
    
    <name>mapreduce.framework.namename>
    <value>yarnvalue>
  property>
  <property>
     <name>mapred.job.trackername>
     <value>localhost:9010value>
  property>
  
  
  <property>
     <name>yarn.app.mapreduce.am.envname>
     <value>HADOOP_MAPRED_HOME=/usr/local/Cellar/hadoop/3.3.3/libexecvalue>
  property>
  <property>
     <name>mapreduce.map.envname>
     <value>HADOOP_MAPRED_HOME=/usr/local/Cellar/hadoop/3.3.3/libexecvalue>
  property>
  <property>
     <name>mapreduce.reduce.envname>
     <value>HADOOP_MAPRED_HOME=/usr/local/Cellar/hadoop/3.3.3/libexecvalue>
property>
 
configuration> 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

---

yarn-site.xml

vim   yarn-site.xml
1

<configuration>
    <property>
        <name>yarn.nodemanager.aux-servicesname>
        <value>mapreduce_shufflevalue>
      property>
    <property>
        <name>yarn.resourcemanager.addressname>
        <value>localhost:9000value>
    property> 
    <property>
      <name>yarn.scheduler.capacity.maximum-am-resource-percentname>
      <value>100value>
    property>
configuration> 
1
2
3
4
5
6
7
8
9
10
11
12
13
14

---

配置 hadoop-env

$HADOOOP_HOME/etc/hadoop/hadoop-env.sh 中配置 JAVA_HOME

---

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_331.jdk/Contents/Home
1

---

---

启动/停止 Hadoop 服务

初始化 HDFS NameNode

hdfs namenode -format
1

---

1、启动

cd  $HADOOP_HOME
./start-dfs.sh 
1
2

2、在浏览器中输入url：
http://localhost:9870/dfshealth.html#tab-overview
http://localhost:9870
看到以下界面启动成功

如果打不开这个页面，而且又启动了，需要在hadoop下/etc/hadoop/hadoop-env.sh文件下第52行后面添加下方配置:

cd /usr/local/Cellar/hadoop/3.3.1/libexec/etc
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
1
2

---

3、停止hadoop服务

./stop-yarn.sh
1

---

4、启动yarn服务

cd /usr/local/Cellar/hadoop/3.3.1/libexec/sbin
./start-yarn.sh 
1
2

在浏览器中输入 http://localhost:8088/cluster 看到一下界面则启动成功

---

5、停止yarn服务

./stop-yarn.sh 
1

---

查看 hdfs report

hdfs dfsadmin -report
1

---

启动 yarn

cd $HADOOP_HOME
../sbin/start-yarn.sh 
1
2

---

浏览器输入 http://localhost:8088，出现下面界面则代表启动成功

---