大数据学习之一——Hadoop单机部署

1 背景

由于我本人服务器有限，因此尝试进行单机安装，主要为了学习使用。

2 前提

hadoop安装依赖JDK。
我这里尝试用下JDK18看看。
下载地址：https://jdk.java.net/18/

这里咱们选择Linux/x64的版本。

2.1 配置JDK环境变量

# vi /etc/profile
在文件最后补充：
export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export PATH=$PATH:$JAVA_HOME/bin
配置生效
# source /etc/profile
1
2
3
4
5
6

4 下载

http://archive.apache.org/dist/hadoop/core/hadoop-3.3.4/
在这个链接，你可以看到各个版本，我这里选择的是3.3.4
在服务器上下载命令：

# wget http://archive.apache.org/dist/hadoop/core/hadoop-3.3.4/hadoop-3.3.4.tar.gz
1

5 解压

我这边上传到centos上，进行解压

# tar -zxvf hadoop-3.3.4.tar.gz 
1

6 配置环境变量

同样，在/etc/profile文件中配置hadoop配置文件：

# vi /etc/profile
 在文件末尾补充：
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.3.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

 配置生效
# source /etc/profile
1
2
3
4
5
6
7

检查配置是否生效

[root@instance-1mfp2mc1 jdk-18.0.2.1]# hadoop version
Hadoop 3.3.4
Source code repository https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb
Compiled by stevel on 2022-07-29T12:32Z
Compiled with protoc 3.7.1
From source with checksum fb9dd8918a7b8a5b430d61af858f6ec
This command was run using /usr/local/hadoop/hadoop-3.3.4/share/hadoop/common/hadoop-common-3.3.4.jar
1
2
3
4
5
6
7

7 配置Hadoop文件

所有配置文件都在etc目录下：hadoop-3.3.4/etc/hadoop，所以，我们只要关心这个目录下相关文件的配置。
各个配置文件说明如下：

7.1 修改hadoop-env.sh

在文件末尾加上如下配置：

export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
1
2
3
4
5
6

7.2 修改 core-site.xml

将配置文件修改为：

    
        fs.defaultFS
        hdfs://localhost:9820
    
    
        hadoop.tmp.dir
        /usr/hadoop/hadoopdata
    

1
2
3
4
5
6
7
8
9
10

7.3 修改hdfs-site.xml

将配置文件修改为如下，表示只有一个副本，因为我这里是单机：

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>
1
2
3
4
5
6

7.4 修改mapred-site.xml

将配置文件修改为：

    
        mapreduce.framework.name
        yarn
    

1
2
3
4
5
6

7.5 修改yarn-site.xml

将配置文件修改为：

    
　　    yarn.nodemanager.aux-services
　　    mapreduce_shuffle
    
    
        yarn.nodemanager.env-whitelist
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    

1
2
3
4
5
6
7
8
9
10
11
12
13
14

8 启动

8.1 对安装目录进行授权

# chmod -R 777 hadoop-3.3.4
1

8.2 格式化hdfs文件系统

# cd /usr/local/hadoop/hadoop-3.3.4/
# bin/hdfs namenode -format
1
2

8.3 启动hadoop

[root@instance-1mfp2mc1 hadoop-3.3.4]# sbin/start-all.sh 
Starting namenodes on [localhost]
Last login: Wed Sep 14 21:45:50 CST 2022 from 218.76.52.120 on pts/3
Last failed login: Wed Sep 14 22:25:26 CST 2022 from 158.69.32.51 on ssh:notty
There were 10 failed login attempts since the last successful login.
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting datanodes
Last login: Wed Sep 14 22:28:54 CST 2022 on pts/3
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting secondary namenodes [instance-1mfp2mc1]
Last login: Wed Sep 14 22:28:54 CST 2022 on pts/3
instance-1mfp2mc1: Warning: Permanently added 'instance-1mfp2mc1,192.168.16.4' (ECDSA) to the list of known hosts.
instance-1mfp2mc1: root@instance-1mfp2mc1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting resourcemanager
Last login: Wed Sep 14 22:28:55 CST 2022 on pts/3
Starting nodemanagers
Last login: Wed Sep 14 22:28:59 CST 2022 on pts/3
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

不过，我这里失败了。

9 配置ssh并再次启动

操作步骤我就直接贴出来了：

[root@instance-1mfp2mc1 .ssh]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:5UT8ePM907UkTWhPTy5b8lExBD1Jbfs3woLFZaNwOY0 root@instance-1mfp2mc1
The key's randomart image is:
+---[RSA 3072]----+
|         .. +.*=+|
|         o.E B **|
|          *o* *+=|
|         +.++.o*=|
|        S +..ooB*|
|         . . oo=*|
|            . . =|
|                 |
|                 |
+----[SHA256]-----+
[root@instance-1mfp2mc1 .ssh]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@instance-1mfp2mc1 .ssh]# ssh localhost
Activate the web console with: systemctl enable --now cockpit.socket

Last failed login: Wed Sep 14 22:32:14 CST 2022 from 158.69.32.51 on ssh:notty
There were 4 failed login attempts since the last successful login.
Last login: Wed Sep 14 22:29:01 2022
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

再次启动下hadoop:

[root@instance-1mfp2mc1 ~]# cd /usr/local/hadoop/hadoop-3.3.4/
[root@instance-1mfp2mc1 hadoop-3.3.4]# sbin/start-all.sh 
Starting namenodes on [localhost]
Last login: Wed Sep 14 22:32:53 CST 2022 from ::1 on pts/4
Last failed login: Wed Sep 14 22:34:01 CST 2022 from 91.240.118.222 on ssh:notty
There was 1 failed login attempt since the last successful login.
Starting datanodes
Last login: Wed Sep 14 22:35:12 CST 2022 on pts/4
Starting secondary namenodes [instance-1mfp2mc1]
Last login: Wed Sep 14 22:35:15 CST 2022 on pts/4
Starting resourcemanager
Last login: Wed Sep 14 22:35:20 CST 2022 on pts/4
Starting nodemanagers
Last login: Wed Sep 14 22:35:26 CST 2022 on pts/4
[root@instance-1mfp2mc1 hadoop-3.3.4]# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

打开namenode，http://120.48.16.193:9870/，namenode的作用是存储元数据。

打开datanode，http://120.48.16.193:9864/，datanode作用是存储真正的数据。