-
CentOS 搭建 Hadoop3 高可用集群
Hadoop FullyDistributed Mode 完全分布式
spark101 spark102 spark103 192.168.171.101 192.168.171.102 192.168.171.103 namenode namenode journalnode journalnode journalnode datanode datanode datanode nodemanager nodemanager nodemanager recource manager recource manager job history job log job log job log 1. 准备
1.1 升级操作系统和软件
yum -y update- 1
升级后建议重启
1.2 安装常用软件
yum -y install gcc gcc-c++ autoconf automake cmake make rsync vim man zip unzip net-tools zlib zlib-devel openssl openssl-devel pcre-devel tcpdump lrzsz tar wget openssh-server- 1
1.3 修改主机名
hostnamectl set-hostname spark01- 1
hostnamectl set-hostname spark02- 1
hostnamectl set-hostname spark03- 1
1.4 修改IP地址
vim /etc/sysconfig/network-scripts/ifcfg-ens160- 1
网卡 配置文件示例
TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens32" DEVICE="ens32" ONBOOT="yes" IPADDR="192.168.171.101" PREFIX="24" GATEWAY="192.168.171.2" DNS1="192.168.171.2" IPV6_PRIVACY="no"- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
1.5 关闭防火墙
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/configsetenforce 0 systemctl stop firewalld systemctl disable firewalld- 1
- 2
- 3
1.6 修改hosts配置文件
vim /etc/hosts- 1
修改内容如下：
192.168.171.101 spark01 192.168.171.102 spark02 192.168.171.103 spark03- 1
- 2
- 3
1.7 上传软件配置环境变量
在所有主机节点创建软件目录
mkdir -p /opt/soft- 1
以下操作在 hadoop101 主机上完成
进入软件目录
cd /opt/soft- 1
下载 JDK
wget https://download.oracle.com/otn/java/jdk/8u391-b13/b291ca3e0c8548b5a51d5a5f50063037/jdk-8u391-linux-x64.tar.gz?AuthParam=1698206552_11c0bb831efdf87adfd187b0e4ccf970- 1
下载 zookeeper
wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.3/apache-zookeeper-3.8.3-bin.tar.gz- 1
下载 hadoop
wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.5/hadoop-3.3.5.tar.gz- 1
解压 JDK 修改名称
解压 zookeeper 修改名称
解压 hadoop 修改名称
tar -zxvf jdk-8u391-linux-x64.tar.gz -C /opt/soft/ mv jdk1.8.0_391/ jdk-8 tar -zxvf apache-zookeeper-3.8.3-bin.tar.gz mv apache-zookeeper-3.8.3-bin zookeeper-3 tar -zxvf hadoop-3.3.5.tar.gz -C /opt/soft/ mv hadoop-3.3.5/ hadoop-3- 1
- 2
- 3
- 4
- 5
- 6
配置环境变量
vim /etc/profile.d/my_env.sh- 1
编写以下内容：
export JAVA_HOME=/opt/soft/jdk-8 # export set JAVA_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED" export ZOOKEEPER_HOME=/opt/soft/zookeeper-3 export HDFS_NAMENODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_ZKFC_USER=root export HDFS_JOURNALNODE_USER=root export HADOOP_SHELL_EXECNAME=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root export HADOOP_HOME=/opt/soft/hadoop-3 export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
生成新的环境变量
注意：分发软件和配置文件后 在所有主机执行该步骤
source /etc/profile- 1
2. zookeeper
2.1 编辑配置文件
cd $ZOOKEEPER_HOME/conf- 1
vim zoo.cfg- 1
# 心跳单位，2s tickTime=2000 # zookeeper-3初始化的同步超时时间，10个心跳单位，也即20s initLimit=10 # 普通同步：发送一个请求并得到响应的超时时间，5个心跳单位也即10s syncLimit=5 # 内存快照数据的存储位置 dataDir=/home/zookeeper-3/data # 事务日志的存储位置 dataLogDir=/home/zookeeper-3/datalog # 当前zookeeper-3节点的端口 clientPort=2181 # 单个客户端到集群中单个节点的并发连接数，通过ip判断是否同一个客户端，默认60 maxClientCnxns=1000 # 保留7个内存快照文件在dataDir中，默认保留3个 autopurge.snapRetainCount=7 # 清除快照的定时任务，默认1小时，如果设置为0，标识关闭清除任务 autopurge.purgeInterval=1 #允许客户端连接设置的最小超时时间，默认2个心跳单位 minSessionTimeout=4000 #允许客户端连接设置的最大超时时间，默认是20个心跳单位，也即40s， maxSessionTimeout=300000 #zookeeper-3 3.5.5启动默认会把AdminService服务启动，这个服务默认是8080端口 admin.serverPort=9001 #集群地址配置 server.1=spark01:2888:3888 server.2=spark02:2888:3888 server.3=spark03:2888:3888- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/home/zookeeper-3/data dataLogDir=/home/zookeeper-3/datalog clientPort=2181 maxClientCnxns=1000 autopurge.snapRetainCount=7 autopurge.purgeInterval=1 minSessionTimeout=4000 maxSessionTimeout=300000 admin.serverPort=9001 server.1=spark01:2888:3888 server.2=spark02:2888:3888 server.3=spark03:2888:3888- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
2.2 保存后根据配置文件创建目录
在每台服务器上执行
mkdir -p /home/zookeeper-3/data mkdir -p /home/zookeeper-3/datalog- 1
- 2
2.3 myid
spark01
echo 1 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid- 1
- 2
spark02
echo 2 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid- 1
- 2
spark03
echo 3 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid- 1
- 2
2.4 编写zookeeper-3开机启动脚本
在/etc/systemd/system/文件夹下创建一个启动脚本zookeeper-3.service
注意：在每台服务器上编写
cd /etc/systemd/system vim zookeeper.service- 1
- 2
内容如下：
[Unit] Description=zookeeper After=syslog.target network.target [Service] Type=forking # 指定zookeeper-3 日志文件路径，也可以在zkServer.sh 中定义 Environment=ZOO_LOG_DIR=/home/zookeeper-3/datalog # 指定JDK路径，也可以在zkServer.sh 中定义 Environment=JAVA_HOME=/opt/soft/jdk-8 ExecStart=/opt/soft/zookeeper-3/bin/zkServer.sh start ExecStop=/opt/soft/zookeeper-3/bin/zkServer.sh stop Restart=always User=root Group=root [Install] WantedBy=multi-user.target- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
[Unit] Description=zookeeper After=syslog.target network.target [Service] Type=forking Environment=ZOO_LOG_DIR=/home/zookeeper-3/datalog Environment=JAVA_HOME=/opt/soft/jdk-8 ExecStart=/opt/soft/zookeeper-3/bin/zkServer.sh start ExecStop=/opt/soft/zookeeper-3/bin/zkServer.sh stop Restart=always User=root Group=root [Install] WantedBy=multi-user.target- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
systemctl daemon-reload # 等所有主机配置好后再执行以下命令 systemctl start zookeeper systemctl enable zookeeper systemctl status zookeeper- 1
- 2
- 3
- 4
- 5
3. hadoop
修改配置文件
cd $HADOOP_HOME/etc/hadoop- 1
- hadoop-env.sh
- core-site.xml
- hdfs-site.xml
- workers
- mapred-site.xml
- yarn-site.xml
hadoop-env.sh 文件末尾追加
export JAVA_HOME=/opt/soft/jdk-8 # export HADOOP_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED" export HDFS_NAMENODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_ZKFC_USER=root export HDFS_JOURNALNODE_USER=root export HADOOP_SHELL_EXECNAME=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
core-site.xml
<configuration> <property> <name>fs.defaultFSname> <value>hdfs://lihaozhevalue> property> <property> <name>hadoop.tmp.dirname> <value>/home/hadoop/datavalue> property> <property> <name>ha.zookeeper.quorumname> <value>spark01:2181,spark02:2181,spark03:2181value> property> <property> <name>hadoop.http.staticuser.username> <value>rootvalue> property> <property> <name>dfs.permissions.enabledname> <value>falsevalue> property> <property> <name>hadoop.proxyuser.root.hostsname> <value>*value> property> <property> <name>hadoop.proxyuser.root.groupsname> <value>*value> property> <property> <name>hadoop.proxyuser.root.usersname> <value>*value> property> configuration>- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
hdfs-site.xml
<configuration> <property> <name>dfs.nameservicesname> <value>lihaozhevalue> property> <property> <name>dfs.ha.namenodes.lihaozhename> <value>nn1,nn2value> property> <property> <name>dfs.namenode.rpc-address.lihaozhe.nn1name> <value>spark01:8020value> property> <property> <name>dfs.namenode.rpc-address.lihaozhe.nn2name> <value>spark02:8020value> property> <property> <name>dfs.namenode.http-address.lihaozhe.nn1name> <value>spark01:9870value> property> <property> <name>dfs.namenode.http-address.lihaozhe.nn2name> <value>spark02:9870value> property> <property> <name>dfs.namenode.shared.edits.dirname> <value>qjournal://spark01:8485;spark02:8485;spark03:8485/lihaozhevalue> property> <property> <name>dfs.client.failover.proxy.provider.lihaozhename> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvidervalue> property> <property> <name>dfs.ha.fencing.methodsname> <value>sshfencevalue> property> <property> <name>dfs.ha.fencing.ssh.private-key-filesname> <value>/root/.ssh/id_rsavalue> property> <property> <name>dfs.journalnode.edits.dirname> <value>/home/hadoop/journalnode/datavalue> property> <property> <name>dfs.ha.automatic-failover.enabledname> <value>truevalue> property> <property> <name>dfs.safemode.threshold.pctname> <value>1value> property> configuration>- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
workers
spark01 spark02 spark03- 1
- 2
- 3
mapred-site.xml
<configuration> <property> <name>mapreduce.framework.namename> <value>yarnvalue> property> <property> <name>mapreduce.application.classpathname> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*value> property> <property> <name>mapreduce.jobhistory.addressname> <value>spark01:10020value> property> <property> <name>mapreduce.jobhistory.webapp.addressname> <value>spark01:19888value> property> configuration>- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
yarn-site.xml
<configuration> <property> <name>yarn.resourcemanager.ha.enabledname> <value>truevalue> property> <property> <name>yarn.resourcemanager.cluster-idname> <value>cluster1value> property> <property> <name>yarn.resourcemanager.ha.rm-idsname> <value>rm1,rm2value> property> <property> <name>yarn.resourcemanager.hostname.rm1name> <value>spark01value> property> <property> <name>yarn.resourcemanager.hostname.rm2name> <value>spark02value> property> <property> <name>yarn.resourcemanager.webapp.address.rm1name> <value>spark01:8088value> property> <property> <name>yarn.resourcemanager.webapp.address.rm2name> <value>spark02:8088value> property> <property> <name>yarn.resourcemanager.zk-addressname> <value>spark01:2181,spark02:2181,spark03:2181value> property> <property> <name>yarn.nodemanager.aux-servicesname> <value>mapreduce_shufflevalue> property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.classname> <value>org.apache.hadoop.mapred.ShuffleHandlervalue> property> <property> <name>yarn.nodemanager.env-whitelistname> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOMEvalue> property> <property> <name>yarn.nodemanager.pmem-check-enabledname> <value>falsevalue> property> <property> <name>yarn.nodemanager.vmem-check-enabledname> <value>falsevalue> property> <property> <name>yarn.log-aggregation-enablename> <value>truevalue> property> <property> <name>yarn.log.server.urlname> <value>http://spark01:19888/jobhistory/logsvalue> property> <property> <name>yarn.log-aggregation.retain-secondsname> <value>604800value> property> configuration>- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
4. 配置ssh免密钥登录
创建本地秘钥并将公共秘钥写入认证文件
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa- 1
ssh-copy-id root@spark01- 1
ssh-copy-id root@spark02- 1
ssh-copy-id root@spark03- 1
ssh root@spark01 exit- 1
- 2
ssh root@spark02 exit- 1
- 2
ssh root@spark03 exit- 1
- 2
5. 分发软件和配置文件
scp -r /etc/profile.d root@spark02:/etc scp -r /etc/profile.d root@spark03:/etc scp -r /opt/soft/zookeeper-3 root@spark02:/opt/soft scp -r /opt/soft/zookeeper-3 root@spark03:/opt/soft scp -r /opt/soft/hadoop-3/etc/hadoop/* root@spark02:/opt/soft/hadoop-3/etc/hadoop/ scp -r /opt/soft/hadoop-3/etc/hadoop/* root@spark03:/opt/soft/hadoop-3/etc/hadoop/- 1
- 2
- 3
- 4
- 5
- 6
6. 在各服务器上使环境变量生效
source /etc/profile- 1
7. 启动zookeeper
7.1 myid
spark01
echo 1 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid- 1
- 2
spark02
echo 2 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid- 1
- 2
spark03
echo 3 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid- 1
- 2
7.2 启动服务
在各节点执行以下命令
systemctl daemon-reload systemctl start zookeeper systemctl enable zookeeper systemctl status zookeeper- 1
- 2
- 3
- 4
7.3 验证
jps- 1
zkServer.sh status- 1
8. Hadoop初始化
1. 启动三个zookeeper：zkServer.sh start 2. 启动三个JournalNode： hadoop-daemon.sh start journalnode 或者 hdfs --daemon start journalnode 3. 在其中一个namenode上格式化：hdfs namenode -format 4. 把刚刚格式化之后的元数据拷贝到另外一个namenode上 a) 启动刚刚格式化的namenode : hadoop-daemon.sh start namenode 或者 hdfs --daemon start namenode b) 在没有格式化的namenode上执行：hdfs namenode -bootstrapStandby c) 启动第二个namenode: hadoop-daemon.sh start namenode 或者 hdfs --daemon start namenode 5. 在其中一个namenode上初始化 hdfs zkfc -formatZK 6. 停止上面节点：stop-dfs.sh 7. 全面启动：start-all.sh 8. 启动resourcemanager节点 yarn-daemon.sh start resourcemanager 或者 start-yarn.sh http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.5.0.tar 不需要执行第 8 步 9. 启动历史服务 mapred --daemon start historyserver 10 11 12 不需要执行 10、安全模式 hdfs dfsadmin -safemode enter hdfs dfsadmin -safemode leave 11、查看哪些节点是namenodes并获取其状态 hdfs getconf -namenodes hdfs haadmin -getServiceState nn1 hdfs haadmin -getServiceState nn2 12、强制切换状态 hdfs haadmin -transitionToActive --forcemanual spark01- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
重点提示：
# 关机之前 依关闭服务 stop-yarn.sh stop-dfs.sh # 开机后 依次开启服务 start-dfs.sh start-yarn.sh- 1
- 2
- 3
- 4
- 5
- 6
或者
# 关机之前关闭服务 stop-all.sh # 开机后开启服务 start-all.sh- 1
- 2
- 3
- 4
#jps 检查进程正常后开启胡哦关闭在再做其它操作- 1
9. 修改windows下hosts文件
C:\Windows\System32\drivers\etc\hosts
追加以下内容：
192.168.171.101 hadoop101 192.168.171.102 hadoop102 192.168.171.103 hadoop103- 1
- 2
- 3
Windows11 注意 修改权限
-
开始搜索 cmd
找到命令头提示符 以管理身份运行
-
进入 C:\Windows\System32\drivers\etc 目录
cd drivers/etc- 1
-
去掉 hosts文件只读属性
attrib -r hosts- 1
-
打开 hosts 配置文件
start hosts- 1
-
追加以下内容后保存
192.168.171.101 spark01 192.168.171.102 spark02 192.168.171.103 spark03- 1
- 2
- 3
10. 测试
12.1 浏览器访问hadoop集群
浏览器访问: http://spark01:9870
浏览器访问:http://spark01:8088
浏览器访问:http://spark01:19888/
12.2 测试 hdfs
本地文件系统创建 测试文件 wcdata.txt
vim wcdata.txt- 1
Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase HiveHadoop Spark HBase StormHBase Hadoop Hive FlinkHBase Flink Hive StormHive Flink HadoopHBase Hive Spark HBaseHive Flink Storm Hadoop HBase SparkFlinkHBase StormHBase Hadoop Hive- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
- 179
- 180
- 181
- 182
- 183
- 184
- 185
- 186
- 187
- 188
- 189
- 190
- 191
- 192
- 193
- 194
- 195
- 196
- 197
- 198
- 199
- 200
- 201
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 209
- 210
- 211
- 212
- 213
- 214
- 215
- 216
- 217
- 218
- 219
- 220
- 221
- 222
- 223
- 224
在 HDFS 上创建目录 /wordcount/input
hdfs dfs -mkdir -p /wordcount/input- 1
查看 HDFS 目录结构
hdfs dfs -ls /- 1
hdfs dfs -ls /wordcount- 1
hdfs dfs -ls /wordcount/input- 1
上传本地测试文件 wcdata.txt 到 HDFS 上 /wordcount/input
hdfs dfs -put wcdata.txt /wordcount/input- 1
检查文件是否上传成功
hdfs dfs -ls /wordcount/input- 1
hdfs dfs -cat /wordcount/input/wcdata.txt- 1
12.2 测试 mapreduce
计算 PI 的值
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi 10 10- 1
单词统计
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar wordcount /wordcount/input/wcdata.txt /wordcount/result- 1
hdfs dfs -ls /wordcount/result- 1
hdfs dfs -cat /wordcount/result/part-r-00000- 1
11. 元数据
hadoop101
cd /home/hadoop_data/dfs/name/current- 1
ls- 1
看到如下内容:
edits_0000000000000000001-0000000000000000009 edits_inprogress_0000000000000000299 fsimage_0000000000000000298 VERSION edits_0000000000000000010-0000000000000000011 fsimage_0000000000000000011 fsimage_0000000000000000298.md5 edits_0000000000000000012-0000000000000000298 fsimage_0000000000000000011.md5 seen_txid- 1
- 2
- 3
- 4
查看fsimage
hdfs oiv -p XML -i fsimage_0000000000000000011- 1
将元数据内容按照指定格式读取后写入到新文件中
hdfs oiv -p XML -i fsimage_0000000000000000011 -o /opt/soft/fsimage.xml- 1
查看edits
将元数据内容按照指定格式读取后写入到新文件中
hdfs oev -p XML -i edits_inprogress_0000000000000000299 -o /opt/soft/edit.xml- 1
-
相关阅读:
【问题处理】GIT合并解决冲突后，导致其他人代码遗失的排查
【C++】面向对象编程（三）定义一个“抽象基类”的三大步骤
独孤九剑第四式-K近邻模型(KNN)
HTML网页设计结课作业 web课程设计网页规划与设计 网页设计成品DW静态网页 Web大学生网页成品 web网页设计期末课程大作业
代码随想录-45-102. 二叉树的层序遍历
【敬伟ps教程】制图操作
[含lw+源码等]微信小程序校园活动报名管理系统+后台管理系统[包运行成功]计算机毕业论文java毕业设计选题源代码
数据库：Hive转Presto（五）
Java语言基础第五天
内核开发-同步场景与概念
- 原文地址：https://blog.csdn.net/qq_24330181/article/details/134172545
